Sick Gaming
[Tut] How to Remove Duplicates From a Python List While Preserving Order? - Printable Version

+- Sick Gaming (https://www.sickgaming.net)
+-- Forum: Programming (https://www.sickgaming.net/forum-76.html)
+--- Forum: Python (https://www.sickgaming.net/forum-83.html)
+--- Thread: [Tut] How to Remove Duplicates From a Python List While Preserving Order? (/thread-97523.html)



[Tut] How to Remove Duplicates From a Python List While Preserving Order? - xSicKxBot - 10-01-2020

How to Remove Duplicates From a Python List While Preserving Order?

<div><p class="has-pale-cyan-blue-background-color has-background">To remove duplicates from a Python list while preserving the order of the elements, use the code <code>list(dict.fromkeys(list))</code> that goes through two phases: (1) Convert the list to a dict using the <code>dict.fromkeys()</code> function with the list elements as keys and <code>None</code> as dict values. (2) Convert the dictionary back to a list using the <code>list()</code> constructor. As dictionaries preserve the order of the keys, the list ordering is preserved.</p>
<figure class="wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio">
<div class="wp-block-embed__wrapper">
<div class="ast-oembed-container"><iframe title="How to Remove Duplicates From a Python List?" width="1400" height="788" src="https://www.youtube.com/embed/GXL23jfNk1Y?feature=oembed" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>
</div>
</figure>
<p><strong>Problem</strong>: How to remove duplicates from a <a href="https://blog.finxter.com/python-lists/" title="The Ultimate Guide to Python Lists" target="_blank" rel="noreferrer noopener">Python list</a> while keeping the order of the list elements preserved?</p>
<p>You may find this question a little awkward. What has <a href="https://blog.finxter.com/python-list-remove-duplicates-and-keep-the-order/" target="_blank" rel="noreferrer noopener" title="Python List: Remove Duplicates and Keep the Order">removing duplicates </a>to do with preserving the order of the elements? The reason is simple: a well-known and efficient way to remove duplicates from a list is to <a href="https://blog.finxter.com/python-list-to-set/" title="Python List to Set Conversion [Interactive Guide]" target="_blank" rel="noreferrer noopener">convert the list to a set</a>—which is duplicated-free—and <a href="https://blog.finxter.com/python-set-to-list/" title="Python Convert Set to List [Interactive Guide]" target="_blank" rel="noreferrer noopener">converting it back to a list</a>. Here’s what you may find everywhere:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">lst = [42, 42, 'Alice', 'Alice', 1]
dup_free = list(set(lst))
print(dup_free)
# ['Alice', 42, 1]
</pre>
<p>The back-and-forth conversion <code>list(set(lst))</code> removes all duplicates from the list. However, it doesn’t preserve the order of the elements. In the example, the string <code>'Alice'</code> now appears before the integer <code>42</code>. </p>
<p><strong><em>So, how to remove duplicates while preserving the order of the elements?</em></strong></p>
<div class="wp-block-image">
<figure class="aligncenter is-resized"><img loading="lazy" src="https://blog.finxter.com/wp-content/uploads/2020/04/removeDupsPython-1024x576.jpg" alt="Remove Duplicates From List Preserve Order Python" width="768" height="432"/></figure>
</div>
<p>The most Pythonic and blazingly fast approach is to use a <a href="https://blog.finxter.com/python-dictionary/" target="_blank" rel="noreferrer noopener">dictionary</a>:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">lst = [3, 3, 22, 22, 1]
result = list(dict.fromkeys(lst))
print(result)
# [3, 22, 1]</pre>
<p>The <code>dict.fromkeys()</code> method creates a new dictionary using the elements from an iterable as the keys. Python dictionary keys are unique by default so <a href="https://blog.finxter.com/list-to-dict-convert-a-list-into-a-dictionary-in-python/" title="List to Dict — Convert a List Into a Dictionary in Python" target="_blank" rel="noreferrer noopener">converting our list into a dictionary</a> will remove duplicates automatically. Once this has been done with our initial list, converting the dictionary back results in the duplicate-free list.</p>
<p>This is the most Pythonic way to remove duplicates from a Python list while preserving the order. </p>
<p><strong>Is this method fast? </strong>Like sets, dictionaries use <a href="https://blog.finxter.com/python-dictionary/#What_is_Hashing_in_Python" target="_blank" rel="noreferrer noopener">hash tables</a>, which means they are extremely fast.</p>
<p class="wp-block-block">Do you want to develop the skills of a <strong>well-rounded Python professional</strong>—while getting paid in the process? Become a Python freelancer and order your book <a href="https://amzn.to/2Re2JqO" target="_blank" rel="noreferrer noopener"><strong>Leaving the Rat Race with Python</strong></a> on Amazon (<em>Kindle/Print</em>)!</p>
<div class="wp-block-image">
<figure class="aligncenter size-medium is-resized"><a href="https://amzn.to/2Re2JqO" target="_blank" rel="noopener noreferrer"><img loading="lazy" src="https://blog.finxter.com/wp-content/uploads/2020/08/final_cover-200x300.jpg" alt="Leaving the Rat Race with Python Book" class="wp-image-11850" width="200" height="300" srcset="https://blog.finxter.com/wp-content/uploads/2020/08/final_cover-200x300.jpg 200w, https://blog.finxter.com/wp-content/uploads/2020/08/final_cover-scaled.jpg 683w, https://blog.finxter.com/wp-content/uploads/2020/08/final_cover-768x1152.jpg 768w, https://blog.finxter.com/wp-content/uploads/2020/08/final_cover-1024x1536.jpg 1024w, https://blog.finxter.com/wp-content/uploads/2020/08/final_cover-1365x2048.jpg 1365w, https://blog.finxter.com/wp-content/uploads/2020/08/final_cover-150x225.jpg 150w" sizes="(max-width: 200px) 100vw, 200px" /></a></figure>
</div>
<h2>Do Python Dictionaries Preserve the Ordering of the Keys?</h2>
<p><strong>Surprisingly, <em>the dictionary keys in Python preserve the order of the elements</em>. So, yes, the order of the elements is preserved. <a rel="noreferrer noopener" href="https://morepypy.blogspot.com/2015/01/faster-more-memory-efficient-and-more.html" target="_blank">(source)</a></strong></p>
<p>Countless online resources like <a rel="noreferrer noopener" href="https://stackoverflow.com/questions/5629023/the-order-of-keys-in-dictionaries" target="_blank">this</a> argue that the order of dictionary keys is not preserved. They assume that the underlying implementation of the dictionary key iterables uses <a href="https://blog.finxter.com/sets-in-python/" target="_blank" rel="noreferrer noopener" title="The Ultimate Guide to Python Sets – with Harry Potter Examples">sets</a>—and sets are well-known to be agnostic to the ordering of elements. But this assumption is wrong. The built-in Python dictionary implementation in cPython preserves the order.</p>
<p>Here’s another example:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">lst = ['Alice', 'Bob', 'Bob', 1, 1, 1, 2, 3, 3]
dic = dict.fromkeys(lst)
print(dic)
# {'Alice': None, 'Bob': None, 1: None, 2: None, 3: None}</pre>
<p>You see that the order of elements is preserved so when converting it back, the original ordering of the list elements is still preserved: </p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">print(list(dic))
# ['Alice', 'Bob', 1, 2, 3]</pre>
<p>However, you cannot rely on it because any Python implementation could, theoretically, decide not to preserve the order (notice the “COULD” here is 100% theoretical and does not apply to the default cPython implementation).</p>
<p>If you need to be certain that the order is preserved, you can use the <a rel="noreferrer noopener" href="https://docs.python.org/3/library/collections.html#collections.OrderedDict" target="_blank">ordered dictionary library</a>. In cPython, this is just a wrapper for the default dict implementation.</p>
<p><strong>Source Article:</strong> <a href="https://blog.finxter.com/how-to-remove-duplicates-from-a-python-list/" target="_blank" rel="noreferrer noopener" title="https://blog.finxter.com/how-to-remove-duplicates-from-a-python-list/">How to Remove Duplicates From a Python List?</a></p>
<h2>Removing Duplicates From Ordered Lists For Older Versions</h2>
<p>Dictionaries only became ordered in all Python implementations when <a href="https://blog.finxter.com/how-to-check-your-python-version/" target="_blank" rel="noreferrer noopener" title="How to Check Your Python Version? A Helpful Guide">Python 3.7</a> was released (this was also an implementation detail of CPython 3.6). </p>
<p>So, if you’re using an older version of Python, you will need to import the <code>OrderedDict</code> class from the collections package in the standard library instead:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""> from collections import OrderedDict lst = [1, 1, 9, 1, 9, 6, 9, 7] result = list(OrderedDict.fromkeys(lst))</pre>
<p>The output is the following duplicate-free list with the order of the elements preserved:</p>
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""> print(result) # [1, 9, 6, 7]</pre>
<h2>Interactive Code Shell</h2>
<p>Let’s try this method in our interactive Python shell:</p>
<figure><iframe src="https://repl.it/repls/GrandioseRemarkableBackups?lite=true" allowfullscreen="true" width="100%" height="400px"></iframe></figure>
<p><strong><em>Exercise: </em></strong><em>Run the code. Does it work?</em></p>
<p>You can find more ways to remove duplicates while preserving the order in this detailed blog article:</p>
<p><strong>Related tutorial:</strong> <a href="https://blog.finxter.com/python-list-remove-duplicates-and-keep-the-order/" target="_blank" rel="noreferrer noopener" title="Python List: Remove Duplicates and Keep the Order">Python List: Remove Duplicates and Keep the Order</a></p>
<h2 class="wp-block-block">Where to Go From Here?</h2>
<p>Enough theory, let’s get some practice!</p>
<p>To become successful in coding, you need to get out there and solve real problems for real people. That’s how you can become a six-figure earner easily. And that’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?</p>
<p><strong>Practice projects is how you sharpen your saw in coding!</strong></p>
<p>Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?</p>
<p>Then become a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.</p>
<p>Join my free webinar <a rel="noreferrer noopener" href="https://blog.finxter.com/webinar-freelancer/" target="_blank">“How to Build Your High-Income Skill Python”</a> and watch how I grew my coding business online and how you can, too—from the comfort of your own home.</p>
<p><a href="https://blog.finxter.com/webinar-freelancer/" target="_blank" rel="noreferrer noopener">Join the free webinar now!</a></p>
<p>The post <a href="https://blog.finxter.com/how-to-remove-duplicates-from-a-python-list-while-preserving-order/" target="_blank" rel="noopener noreferrer">How to Remove Duplicates From a Python List While Preserving Order?</a> first appeared on <a href="https://blog.finxter.com/" target="_blank" rel="noopener noreferrer">Finxter</a>.</p>
</div>


https://www.sickgaming.net/blog/2020/09/30/how-to-remove-duplicates-from-a-python-list-while-preserving-order/