Removing duplicates in lists

Question

Pretty much I need to write a program to check if a list has any duplicates and if it does it removes them and returns a new list with the items that weren't duplicated/removed. This is what I have but to be honest I do not know what to do.

def remove_duplicates():
    t = ['a', 'b', 'c', 'd']
    t2 = ['a', 'c', 'd']
    for t in t2:
        t.append(t.remove())
    return t

Answer 1

Using a set is a typical way to create a one-of-a-kind collection of goods. Sets are groups of different things that are not arranged in any particular order. You can create a set from any iterable by using the built-in set() function. You can also give the set to the list() function if you require a real list later.

The example below should cover whatever you're wanting to accomplish:

>>> t = [1, 2, 3, 1, 2, 5, 6, 7, 8]
>>> t
[1, 2, 3, 1, 2, 5, 6, 7, 8]
>>> list(set(t))
[1, 2, 3, 5, 6, 7, 8]
>>> s = [1, 2, 3]
>>> list(set(t) - set(s))
[8, 5, 6, 7]

The original order is not retained, as you can see in the example result. As previously stated, sets are unsorted collections, hence the order is lost. When you convert a set back to a list, you get an arbitrary order.

Keeping the peace
If order is crucial to you, you'll need to utilize a different method. Using OrderedDict to retain the order of keys during insertion is a standard approach for this:

>>> from collections import OrderedDict
>>> list(OrderedDict.fromkeys(t))
[1, 2, 3, 5, 6, 7, 8]

Starting with Python 3.7, the built-in dictionary is guaranteed to keep the insertion order, thus if you're using Python 3.7 or later (or CPython 3.6), you can use it directly:

>>> list(dict.fromkeys(t))
[1, 2, 3, 5, 6, 7, 8]

It's worth noting that the overhead of first building a dictionary and then creating a list from it could be significant. You're frequently better off utilizing a set if you don't need to keep the order, especially because it provides you a lot more operations to deal with.

Finally, your items must be hashable for both the set and the OrderedDict/dict solutions to work. This usually implies that they must remain unchangeable. If you need to work with items that aren't hashable (like list objects), you'll have to take a more time-consuming technique, in which you'll have to compare each item to every other item in a nested loop.

Answer 2

list(set(t + t2))