How to read multiple data files in python

+3 votes

I have two text Files (not in CSV) Now how to gather the these data files into one single file

Data1

Month    Spend    Sales
1    1000    9914
2    4000    40487
3    5000    54324
4    4500    50044

Data2

Month    Spend    Sales
5    3000    34719
6    4000    42551
7    9000    94871
8    11000    118914
9    15000    158484
10    12000    131348
11    7000    78504
12    3000    36284

 

Apr 6, 2018 in Python by CodingByHeart77
• 3,680 points
485 views

5 answers to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
+2 votes
Best answer

Firstly we will import pandas to read the text files

import pandas as pd

Now we will read the text file

de=pd.read_table("data1.txt")
de1=pd.read_table("data2.txt")

Convert the files into data frame

dp=pd.DataFrame(data=de)
dp1=pd.DataFrame(data=de1)

concatenate the both data frames into one

dpe=pd.concat([de,de1])
answered Apr 6, 2018 by DeepCoder786
• 1,700 points

selected Oct 12, 2018 by Omkar
+1 vote
import sys

# argv is your commandline arguments, argv[0] is your program name, so skip it
for n in sys.argv[1:]:
    print(n) #print out the filename we are currently processing
    input = open(n, "r")
    output = open(n + ".out", "w")
    # do some processing
    input.close()
    output.close()

Then call it like:

./foo.py bar.txt baz.txt
answered Oct 12, 2018 by findingbugs
• 4,730 points
+1 vote

IF all data is of the same shape then just append to a list.

all_data = [] 

and in your loop:

all_data.append(data)

finally you have

asarray(all_data)

which is an array of shape (10,50,2) (transpose if you want). If the shapes don't match, then this does not work though, numpy cannot handle rows of different shapes. Then you might need another loop which creates arrays of the largest shape, and copy your data over.

answered Oct 12, 2018 by abc
+1 vote

crude but quick

listFiles=["1.txt","2.txt", ... ,"xxx.txt"]
allData=[]
for file in listFiles:
    lines = open(file,'r').readlines()

    filedata = {}
    filedata['name'] = file
    filedata['rawLines'] = lines
    col1Vals = []
    col2Vals = []
    mapValues = {}

    for line in lines:           
       values = line.split(',')
       col1Vals.append(values[0])
       col2Vals.append(values[1])
       mapValues[values[0]] = values[1]
    filedata['col1'] = col1Vals
    filedata['col2'] = col2Vals
    filedata['map'] = mapValues
    allData.append(filedata)

answered Oct 12, 2018 by rani
+1 vote

Are you looking for an array that is [txt column1, txt column2, filename]?

file_list = glob.glob(source_dir + '/*.TXT') #Get folder path containing text files


for num,file_path in enumerate(file_list):
  data = np.genfromtxt(file_path, delimiter=',', skip_header=3, skip_footer=18)
  data = np.vstack((data.T,np.ones(data.shape[0])*num)).T
  if num==0: Output=data
  else: Output=np.vstack((Output,data))

An alternative if you dont want to transpose twice.

  data = np.vstack((data,(np.ones(data.shape[0])*num).reshape(-1,1)))
answered Oct 12, 2018 by kalpesh

Related Questions In Python

0 votes
0 answers

How to mix read() and write() on Python files in Windows

It appears that a write() immediately following a read() on a ...READ MORE

Oct 23, 2018 in Python by Aryya
• 500 points
23 views
0 votes
0 answers

How to read lines from compressed text files in Python

Is it easy to read a line ...READ MORE

Dec 27, 2018 in Python by Anirudh
• 2,050 points
53 views
0 votes
1 answer

How to replace values with None in Pandas data frame in Python?

Actually in later versions of pandas this ...READ MORE

answered Aug 30, 2018 in Python by Priyaj
• 56,100 points
273 views
0 votes
1 answer

How to read Excel File in Python

With pandas it is possible to get ...READ MORE

answered Oct 22, 2018 in Python by Priyaj
• 56,100 points
54 views
0 votes
1 answer

How can I rename multiple files in a certain directory using Python?

Use os.rename(src, dst) to rename or move a file ...READ MORE

answered Nov 23, 2018 in Python by ariaholic
• 7,320 points
57 views
0 votes
1 answer

How can I change the data type to string in Python?

You can use str(variablename) This is called conversion ...READ MORE

answered Dec 17, 2018 in Python by Shuvodip
36 views
0 votes
1 answer

how can i count the items in a list?

suppose you have a list a = [0,1,2,3,4,5,6,7,8,9,10] now ...READ MORE

answered May 2 in Python by Mohammad
• 920 points
19 views
+4 votes
6 answers

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.