Python program to read the excel file which contains more than 65K records

0 votes

I am using this code to convert excel to csv file:

import glob 
path_to_excel_files = glob.glob('path/to/excel/files/*.xlsx')
for excel in path_to_excel_files:
 out = excel.split('.')[0]+'.csv'
 df = pd.read_excel(excel)
 df.to_csv(out)

But the problem is that it is converting only 65,000 records. How to get more than 65,000 records?

Feb 8 in Python by Rashmi
55 views

1 answer to this question.

0 votes

Hi @Rashmi!

I read your code and there is no such logic to read only 65,000 records. And the modules you are using also does not restrict the records. There is no problem in the code. So, I did a little research about this and found that 1 sheet in excel can hold only 65,000 records. Possibly, you have records in different sheets and those are not getting converted. To read records from different sheets, you will have to parse through the sheets. You can refer to the following code to do it:

import pandas as pd

df = pd.DataFrame()
xlfname = 'Productivity Report.xlsx'
xl = pd.ExcelFile(xlfname)

for sheet in xl.sheet_names:
df_tmp = xl.parse(sheet)
df = df.append(df_tmp, ignore_index=True,sort=False)

print(len(df))

csvfile = 'sample.csv'
df.to_csv(csvfile, index=False)
answered Feb 8 by Omkar
• 67,660 points

Related Questions In Python

0 votes
1 answer

How to read Excel File in Python

With pandas it is possible to get ...READ MORE

answered Oct 22, 2018 in Python by Priyaj
• 56,940 points
194 views
0 votes
1 answer

How to use read a WSDL file from the file system using Python suds?

Hi, good question. It is a very simple ...READ MORE

answered Jan 21 in Python by Nymeria
• 3,520 points
687 views
0 votes
1 answer

Need help to open excel file and read in Python

You can use pandas module to do ...READ MORE

answered Jul 22 in Python by Tina
35 views
+1 vote
3 answers

How to run the Python program forever?

you can also do the old fashioned ...READ MORE

answered Jun 10 in Python by brianno
782 views
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6 in Python by Neha
• 330 points

edited Jul 8 by Kalgi 349 views
+4 votes
6 answers
0 votes
1 answer
+1 vote
1 answer

How to read hdfs file using python?

subprocess.Popen(["hadoop", "fs", "-cat", "/path/to/myfile"], stdout ...READ MORE

answered Dec 6, 2018 in Python by Omkar
• 67,660 points
1,269 views