Error UnicodeDecodeError when reading CSV file in Pandas with Python

0 votes

I'm running a program which is processing 30,000 similar files. A random number of them are stopping and producing this error...

   File "C:\Importer\src\dfman\importer.py", line 26, in import_chr
     data = pd.read_csv(filepath, names=fields)
   File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 400, in parser_f
     return _read(filepath_or_buffer, kwds)
   File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 205, in _read
     return parser.read()
   File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 608, in read
     ret = self._engine.read(nrows)
   File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 1028, in read
     data = self._reader.read(nrows)
   File "parser.pyx", line 706, in pandas.parser.TextReader.read (pandas\parser.c:6745)
   File "parser.pyx", line 728, in pandas.parser.TextReader._read_low_memory (pandas\parser.c:6964)
   File "parser.pyx", line 804, in pandas.parser.TextReader._read_rows (pandas\parser.c:7780)
   File "parser.pyx", line 890, in pandas.parser.TextReader._convert_column_data (pandas\parser.c:8793)
   File "parser.pyx", line 950, in pandas.parser.TextReader._convert_tokens (pandas\parser.c:9484)
   File "parser.pyx", line 1026, in pandas.parser.TextReader._convert_with_dtype (pandas\parser.c:10642)
   File "parser.pyx", line 1046, in pandas.parser.TextReader._string_convert (pandas\parser.c:10853)
   File "parser.pyx", line 1278, in pandas.parser._string_box_utf8 (pandas\parser.c:15657)
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 6: invalid    continuation byte

The source/creation of these files all come from the same place. What's the best way to correct this to proceed with the import?

Jun 15, 2020 in Python by kartik
• 37,530 points
4,172 views

1 answer to this question.

0 votes

Hello @kartik,

Try this:

import pandas as pd
df = pd.read_csv('file_name.csv', engine='python')

Alternate Solution:

  • Open the csv file in Sublime text editor.
  • Save the file in utf-8 format.
In sublime, Click File -> Save with encoding -> UTF-8

Then, you can read your file as usual:

import pandas as pd
data = pd.read_csv('file_name.csv', encoding='utf-8')

and the other different encoding types are:

encoding = "cp1252"
encoding = "ISO-8859-1"

Hope this is helpful!

answered Jun 15, 2020 by Niroj
• 82,880 points

Related Questions In Python

0 votes
1 answer

Error reading csv file in python

You are missing the extension. Try this: import ...READ MORE

answered Mar 28, 2019 in Python by Shri
1,764 views
+1 vote
1 answer

Reading a large file, line by line in Python

The correct, fully Pythonic way to read ...READ MORE

answered Aug 21, 2018 in Python by Priyaj
• 58,140 points
280 views
0 votes
1 answer

How to replace values with None in Pandas data frame in Python?

Actually in later versions of pandas this ...READ MORE

answered Aug 30, 2018 in Python by Priyaj
• 58,140 points
8,935 views
0 votes
1 answer

When to use file vs open in Python?

File() has been removed since Python 3.0 ...READ MORE

answered Oct 30, 2018 in Python by SDeb
• 13,300 points
323 views
0 votes
2 answers
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 7, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 2,479 views
0 votes
1 answer
+5 votes
6 answers

Lowercase in Python

You can simply the built-in function in ...READ MORE

answered Apr 11, 2018 in Python by hemant
• 5,810 points
1,859 views
0 votes
1 answer

How to find the value of a row in a csv file in python and print column and row of that value ?

Hello @Khanhh , Use panda to find the value of ...READ MORE

answered Oct 15, 2020 in Python by Niroj
• 82,880 points
3,122 views
0 votes
2 answers

Python Pandas error: AttributeError: 'DataFrame' object has no attribute 'rows'

Try this: data=pd.read_csv('/your file name', delim_whitespace=Tru ...READ MORE

answered Dec 10, 2020 in Python by anonymous
• 82,880 points
83,636 views