Create an account


Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
[Tut] Python Unicode Encode Error

#1
Python Unicode Encode Error

Summary: The UnicodeEncodeError generally occurs while encoding a Unicode string into a certain coding. Only a limited number of Unicode characters are mapped to strings. Thus, any character that is not-represented / mapped will cause the encoding to fail and raise UnicodeEncodeError. To avoid this error use the encode(utf-8) and decode(utf-8) functions accordingly in your code.

You might be using handling an application code that needs to deal with multilingual data or web content that has plenty of emojis and special symbols. In such situations, you will possibly come across numerous problems relating to Unicode data. But python has well-defined options to deal with Unicode characters and we shall be discussing them in this article.

What is Unicode?


Unicode is a standard that facilitates character encoding using variable bit encoding. I am sure, you must have heard of ASCII if you are into the world of computer programming. ASCII represents 128 characters while Unicode defines 221 characters. Thus, Unicode can be regarded as a superset of ASCII. If you are interested in having an in-depth look at Unicode, please follow this link. 
Click on Unicode:- U+1F40D to find out what it represents! (Try it!!!?)

What is a UnicodeEncodeError?


The best way to grasp any concept is to visualize it with an example. So let us have a look at an example of the UnicodeEncodeError.

u = 'é'
print("Integer value for é: ", ord(u))
print("Converting the encoded value of é to Integer Equivalent: ", chr(233))
print("UNICODE Representation of é: ", u.encode('utf-8'))
print("ASCII Representation of é: ", u.encode('ascii'))

Output

Integer value for é: 233
Converting the encoded value of é to Integer Equivalent: é
UNICODE Representation of é: b'\xc3\xa9'
Traceback (most recent call last): File "main.py", line 5, in <module> print("ASCII Representation of é: ",u.encode('ascii'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)

In the above code, when we tried to encode the character é to its Unicode value we got an output but while trying to convert it to the ASCII equivalent we encountered an error. The error occurred because ASCII only allows 7-bit encoding and it cannot represent characters outside the range of [0..128]. 

You now have an essence of what the UnicodeEncodeError looks like. Before discussing how we can avoid such errors, I feel that there is a dire need to discuss the following concepts:

Encoding and Decoding


The process of converting human-readable data into a specified format, for the secured transmission of data is known as encoding. Decoding is the opposite of encoding that is to convert the encoded information to normal text (human-readable form).

In Python,

  • encode() is an inbuilt method used for encoding. Incase no encoding is specified, UTF-8 is used as default. 
  • decode() is an inbuilt method used for decoding. 

Example:

u = 'Πύθωνος'
print("UNICODE Representation of é: ", u.encode('utf-8'))

Output:

UNICODE Representation of é: b'\xce\xa0\xcf\x8d\xce\xb8\xcf\x89\xce\xbd\xce\xbf\xcf\x82'

The following diagram should make things a little easier:

https://blog.finxter.com/wp-content/uplo...00x156.png 300w, https://blog.finxter.com/wp-content/uplo...150x78.png 150w" sizes="(max-width: 665px) 100vw, 665px" />

Codepoint


Unicode maps the codepoint to their respective characters. So, what do we mean by a codepoint? 

  • Codepoints are numerical values or integers used to represent a character. 
  • The Unicode code point for é is U+00E9 which is integer 233. When you encode a character and print it, you will generally get its hexadecimal representation as an output instead of its binary equivalent (as seen in the examples above).
  • The byte sequence of a code point is different in different encoding schemes. For eg: the byte sequence for é in UTF-8 is \xc3\xa9 while in UTF-16 is \xff\xfe\xe9\x00.

Please have a look at the following program to get a better grip on this concept:

u = 'é'
print("INTEGER value for é: ", ord(u))
print("ENCODED Representation of é in UTF-8: ", u.encode('utf-8'))
print("ENCODED Representation of é in UTF-16: ", u.encode('utf-16'))

Output

INTEGER value for é: 233
ENCODED Representation of é in UTF-8: b'\xc3\xa9'
ENCODED Representation of é in UTF-16: b'\xff\xfe\xe9\x00'

Now that we have an overview of Unicode and UnicodeEncodeError, let us discuss how we can deal with the error and avoid it in our program.

Problem: Given a string/text to be written in a text File; how to avoid the UnicodeEncodeError and write given text in the text file.

Example:

f = open('demo.txt', 'w')
f.write('να έχεις μια όμορφη μέρα')
f.close()

Output

Traceback (most recent call last): File "uniError.py", line 2, in <module> f.write('να έχεις μια όμορφη μέρα') File "C:\Users\Shubham-PC\AppData\Local\Programs\Python\Python38-32\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-1: character maps to <undefined>

Desired Output

Solution 1: Encode String Before Writing To File And Decode While Reading


You cannot write Unicode to a file directly. This will raise an UnicodeEncodeError. To avoid this you must encode the Unicode string using the encode() function and then write it to the file as shown in the program below:

text = u'να έχεις μια όμορφη μέρα'
# write in binary mode to avoid TypeError
f = open('demo.txt', 'wb')
f.write(text.encode('utf8'))
f.close()
f = open('demo.txt', 'rb')
print(f.read().decode('utf8'))

Output

Solution 2: Open File In utf-8


If you are using Python 3 or higher, all you need to do is open the file in utf-8, as Unicode string handling is already standardized in Python 3.

text = 'να έχεις μια όμορφη μέρα'
f = open('demo2.txt', 'w', encoding="utf-8")
f.write(text)
f.close()

Output

Solution 3: Using The Codecs Module


Another approach to deal with the UnicodeEncodeError is using the codecs module. 

Let us have a look at the following code to understand how we can use the codecs module:

import codecs f = codecs.open("demo3.txt", "w", encoding='utf-8')
f.write("να έχεις μια όμορφη μέρα")
f.close()

Output

Solution 4: Using Python’s unicodecsv Module


If you are dealing with Unicode data and using a csv file for managing your data, then the unicodecsv module can be really helpful. It is an extended version of Python 2’s csv module and helps the user to handle Unicode data without any hassle.

Since the unicodecsv module is not a part of Python’s standard library, you have to install it before using it. Use the following command to install this module:

$ pip install unicodecsv

Let us have a look at the following example to get a better grip on the unicodecsv module:

import unicodecsv as csv with open('example.csv', 'wb') as f: writer = csv.writer(f, encoding='utf-8') writer.writerow(('English', 'Japanese')) writer.writerow((u'Hello', u'こんにちは'))

Output

Conclusion


In this article, we discussed some of the important concepts regarding Unicode character and then went on to learn about the UnicodeEncodeError and finally discussed the methods that we can use to avoid it. I hope by the end of this article you can handle Unicode characters in your python code with ease.

Please subscribe and stay tuned for more interesting articles!

Where to Go From Here?


Enough theory, let’s get some practice!

To become successful in coding, you need to get out there and solve real problems for real people. That’s how you can become a six-figure earner easily. And that’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

Practice projects is how you sharpen your saw in coding!

Do you want to become a code master by focusing on practical code projects that actually earn you money and solve problems for people?

Then become a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

Join my free webinar “How to Build Your High-Income Skill Python” and watch how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!



https://www.sickgaming.net/blog/2020/09/...ode-error/
Reply



Possibly Related Threads…
Thread Author Replies Views Last Post
  [Tut] Python’s NameError: name ‘xxx’ is not defined — How to Fix This Stupid Bug? xSicKxBot 0 7 10-16-2020, 10:30 PM
Last Post: xSicKxBot
  [Tut] Python IndentationError: unexpected indent (How to Fix This Stupid Bug) xSicKxBot 0 15 10-10-2020, 08:24 PM
Last Post: xSicKxBot
  [Tut] Yield Keyword in Python – A Simple Illustrated Guide xSicKxBot 0 16 10-09-2020, 06:13 PM
Last Post: xSicKxBot
  [Tut] Python Reverse List with Slicing — An Illustrated Guide xSicKxBot 0 27 10-02-2020, 04:24 AM
Last Post: xSicKxBot
  [Tut] How to Remove Duplicates From a Python List While Preserving Order? xSicKxBot 0 29 10-01-2020, 02:21 AM
Last Post: xSicKxBot
  [Tut] How To Update A Key In A Dictionary In Python If The Key Doesn’t Exist? xSicKxBot 0 30 09-29-2020, 09:02 PM
Last Post: xSicKxBot
  [Tut] How to Get the Last Element of a Python List? xSicKxBot 0 32 09-28-2020, 02:36 AM
Last Post: xSicKxBot
  [Tut] How To Format A String That Contains Curly Braces In Python? xSicKxBot 0 29 09-25-2020, 07:51 PM
Last Post: xSicKxBot
  [Tut] Python One Line Dictionary xSicKxBot 0 35 09-21-2020, 05:58 PM
Last Post: xSicKxBot
  [Tut] Python Import Error (ModuleNotFoundError) xSicKxBot 0 31 09-20-2020, 04:15 PM
Last Post: xSicKxBot

Forum Jump:

Become a Patron!
[-]
Upcoming Events

[-]
Latest Threads
[1.06] BO4 Zombie Trainer (Call of duty ...
Last Post: poochygoo
Today 06:14 AM
» Replies: 10
» Views: 376
(Indie Deal) Ultra Panic Bundle & 2K Wee...
Last Post: xSicKxBot
Today 03:05 AM
» Replies: 0
» Views: 4
News - P.T. Can't Be Downloaded On PS5, ...
Last Post: xSicKxBot
Today 03:05 AM
» Replies: 0
» Views: 4
Roblox Going Public
Last Post: xSicKxBot
Yesterday 08:41 PM
» Replies: 0
» Views: 4
Fedora - Web of Trust, Part 2: Tutorial
Last Post: xSicKxBot
Yesterday 08:41 PM
» Replies: 0
» Views: 3
News - DC’s Static Shock Movie On The Wa...
Last Post: xSicKxBot
Yesterday 08:41 PM
» Replies: 0
» Views: 3
(Indie Deal) PlugInDigital & Kalypso Str...
Last Post: xSicKxBot
Yesterday 01:57 PM
» Replies: 0
» Views: 12
News - Kingdom Hearts Director Possibly ...
Last Post: xSicKxBot
Yesterday 01:57 PM
» Replies: 0
» Views: 5
News - Double Fine Brings Classic LucasA...
Last Post: xSicKxBot
Yesterday 01:57 PM
» Replies: 0
» Views: 6
News - Video: Why your game community is...
Last Post: xSicKxBot
Yesterday 01:12 PM
» Replies: 0
» Views: 6

[-]
Twitter

[-]
Sponsored
Get the Deal of the Week at RefurBees.com

Copyright © SickGaming.net 2012-2020