Create an account


Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[Tut] How to Remove Duplicates From a Python List While Preserving Order?

#1
How to Remove Duplicates From a Python List While Preserving Order?

To remove duplicates from a Python list while preserving the order of the elements, use the code list(dict.fromkeys(list)) that goes through two phases: (1) Convert the list to a dict using the dict.fromkeys() function with the list elements as keys and None as dict values. (2) Convert the dictionary back to a list using the list() constructor. As dictionaries preserve the order of the keys, the list ordering is preserved.



Problem: How to remove duplicates from a Python list while keeping the order of the list elements preserved?

You may find this question a little awkward. What has removing duplicates to do with preserving the order of the elements? The reason is simple: a well-known and efficient way to remove duplicates from a list is to convert the list to a set—which is duplicated-free—and converting it back to a list. Here’s what you may find everywhere:

lst = [42, 42, 'Alice', 'Alice', 1]
dup_free = list(set(lst))
print(dup_free)
# ['Alice', 42, 1]

The back-and-forth conversion list(set(lst)) removes all duplicates from the list. However, it doesn’t preserve the order of the elements. In the example, the string 'Alice' now appears before the integer 42.

So, how to remove duplicates while preserving the order of the elements?

Remove Duplicates From List Preserve Order Python

The most Pythonic and blazingly fast approach is to use a dictionary:

lst = [3, 3, 22, 22, 1]
result = list(dict.fromkeys(lst))
print(result)
# [3, 22, 1]

The dict.fromkeys() method creates a new dictionary using the elements from an iterable as the keys. Python dictionary keys are unique by default so converting our list into a dictionary will remove duplicates automatically. Once this has been done with our initial list, converting the dictionary back results in the duplicate-free list.

This is the most Pythonic way to remove duplicates from a Python list while preserving the order.

Is this method fast? Like sets, dictionaries use hash tables, which means they are extremely fast.

Do you want to develop the skills of a well-rounded Python professional—while getting paid in the process? Become a Python freelancer and order your book Leaving the Rat Race with Python on Amazon (Kindle/Print)!

Leaving the Rat Race with Python Book

Do Python Dictionaries Preserve the Ordering of the Keys?


Surprisingly, the dictionary keys in Python preserve the order of the elements. So, yes, the order of the elements is preserved. (source)

Countless online resources like this argue that the order of dictionary keys is not preserved. They assume that the underlying implementation of the dictionary key iterables uses sets—and sets are well-known to be agnostic to the ordering of elements. But this assumption is wrong. The built-in Python dictionary implementation in cPython preserves the order.

Here’s another example:

lst = ['Alice', 'Bob', 'Bob', 1, 1, 1, 2, 3, 3]
dic = dict.fromkeys(lst)
print(dic)
# {'Alice': None, 'Bob': None, 1: None, 2: None, 3: None}

You see that the order of elements is preserved so when converting it back, the original ordering of the list elements is still preserved:

print(list(dic))
# ['Alice', 'Bob', 1, 2, 3]

However, you cannot rely on it because any Python implementation could, theoretically, decide not to preserve the order (notice the “COULD” here is 100% theoretical and does not apply to the default cPython implementation).

If you need to be certain that the order is preserved, you can use the ordered dictionary library. In cPython, this is just a wrapper for the default dict implementation.

Source Article: How to Remove Duplicates From a Python List?

Removing Duplicates From Ordered Lists For Older Versions


Dictionaries only became ordered in all Python implementations when Python 3.7 was released (this was also an implementation detail of CPython 3.6). 

So, if you’re using an older version of Python, you will need to import the OrderedDict class from the collections package in the standard library instead:

 from collections import OrderedDict lst = [1, 1, 9, 1, 9, 6, 9, 7] result = list(OrderedDict.fromkeys(lst))

The output is the following duplicate-free list with the order of the elements preserved:

 print(result) # [1, 9, 6, 7]

Interactive Code Shell


Let’s try this method in our interactive Python shell:


Exercise: Run the code. Does it work?

You can find more ways to remove duplicates while preserving the order in this detailed blog article:

Related tutorial: Python List: Remove Duplicates and Keep the Order

Where to Go From Here?


Enough theory, let’s get some practice!

To become successful in coding, you need to get out there and solve real problems for real people. That’s how you can become a six-figure earner easily. And that’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

Practice projects is how you sharpen your saw in coding!

Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?

Then become a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

Join my free webinar “How to Build Your High-Income Skill Python” and watch how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!

The post How to Remove Duplicates From a Python List While Preserving Order? first appeared on Finxter.



https://www.sickgaming.net/blog/2020/09/...ing-order/
Reply



Possibly Related Threads…
Thread Author Replies Views Last Post
  [Tut] Python’s NameError: name ‘xxx’ is not defined — How to Fix This Stupid Bug? xSicKxBot 0 8 10-16-2020, 10:30 PM
Last Post: xSicKxBot
  [Tut] Python IndentationError: unexpected indent (How to Fix This Stupid Bug) xSicKxBot 0 17 10-10-2020, 08:24 PM
Last Post: xSicKxBot
  [Tut] Yield Keyword in Python – A Simple Illustrated Guide xSicKxBot 0 18 10-09-2020, 06:13 PM
Last Post: xSicKxBot
  [Tut] Python Reverse List with Slicing — An Illustrated Guide xSicKxBot 0 29 10-02-2020, 04:24 AM
Last Post: xSicKxBot
  [Tut] How To Update A Key In A Dictionary In Python If The Key Doesn’t Exist? xSicKxBot 0 32 09-29-2020, 09:02 PM
Last Post: xSicKxBot
  [Tut] How to Get the Last Element of a Python List? xSicKxBot 0 34 09-28-2020, 02:36 AM
Last Post: xSicKxBot
  [Tut] List Changes After Assignment — How to Clone or Copy It? xSicKxBot 0 32 09-27-2020, 12:14 AM
Last Post: xSicKxBot
  [Tut] How To Format A String That Contains Curly Braces In Python? xSicKxBot 0 31 09-25-2020, 07:51 PM
Last Post: xSicKxBot
  [Tut] Python One Line Dictionary xSicKxBot 0 38 09-21-2020, 05:58 PM
Last Post: xSicKxBot
  [Tut] Python Import Error (ModuleNotFoundError) xSicKxBot 0 33 09-20-2020, 04:15 PM
Last Post: xSicKxBot

Forum Jump:

Become a Patron!
[-]
Upcoming Events

[-]
Latest Threads
[Tut] Top 3 Freelance Developer Contract...
Last Post: xSicKxBot
Today 04:49 AM
» Replies: 0
» Views: 0
(Indie Deal) Alien Cat Bundle, RIDE 4 re...
Last Post: xSicKxBot
Today 04:48 AM
» Replies: 0
» Views: 1
News - Super Smash Bros. Ultimate 9.0.1...
Last Post: xSicKxBot
Today 04:48 AM
» Replies: 0
» Views: 1
Unity Mega Bundle X 10th Anniversary Sal...
Last Post: xSicKxBot
Today 01:02 AM
» Replies: 0
» Views: 3
Microsoft - Fundraising and Engagement f...
Last Post: xSicKxBot
Today 01:02 AM
» Replies: 0
» Views: 2
Fedora - Incremental backup with Butterf...
Last Post: xSicKxBot
Today 01:01 AM
» Replies: 0
» Views: 2
News - Review: Horace – A Heart-Warming ...
Last Post: xSicKxBot
Yesterday 11:38 PM
» Replies: 0
» Views: 2
News - Matthew McConaughey Explains The ...
Last Post: xSicKxBot
Yesterday 10:20 PM
» Replies: 0
» Views: 3
PlanetSide 2 External Hack / Mod
Last Post: FarCry10
Yesterday 05:27 PM
» Replies: 6
» Views: 6812
(Indie Deal) FREE Defense of Roman Brita...
Last Post: xSicKxBot
Yesterday 03:53 PM
» Replies: 0
» Views: 4

[-]
Twitter

[-]
Sponsored
Get the Deal of the Week at RefurBees.com

Copyright © SickGaming.net 2012-2020