How to read multiple data files in python

+3 votes

I have two text Files (not in CSV) Now how to gather the these data files into one single file

Data1

Month    Spend    Sales
1    1000    9914
2    4000    40487
3    5000    54324
4    4500    50044

Data2

Month    Spend    Sales
5    3000    34719
6    4000    42551
7    9000    94871
8    11000    118914
9    15000    158484
10    12000    131348
11    7000    78504
12    3000    36284

 

Apr 6, 2018 in Python by CodingByHeart77
• 3,690 points
2,643 views

5 answers to this question.

+2 votes
Best answer

Firstly we will import pandas to read the text files

import pandas as pd

Now we will read the text file

de=pd.read_table("data1.txt")
de1=pd.read_table("data2.txt")

Convert the files into data frame

dp=pd.DataFrame(data=de)
dp1=pd.DataFrame(data=de1)

concatenate the both data frames into one

dpe=pd.concat([de,de1])
answered Apr 6, 2018 by DeepCoder786
• 1,720 points

selected Oct 12, 2018 by Omkar
+1 vote
import sys

# argv is your commandline arguments, argv[0] is your program name, so skip it
for n in sys.argv[1:]:
    print(n) #print out the filename we are currently processing
    input = open(n, "r")
    output = open(n + ".out", "w")
    # do some processing
    input.close()
    output.close()

Then call it like:

./foo.py bar.txt baz.txt
answered Oct 12, 2018 by findingbugs
• 4,750 points
+1 vote

IF all data is of the same shape then just append to a list.

all_data = [] 

and in your loop:

all_data.append(data)

finally you have

asarray(all_data)

which is an array of shape (10,50,2) (transpose if you want). If the shapes don't match, then this does not work though, numpy cannot handle rows of different shapes. Then you might need another loop which creates arrays of the largest shape, and copy your data over.

answered Oct 12, 2018 by abc
+1 vote

crude but quick

listFiles=["1.txt","2.txt", ... ,"xxx.txt"]
allData=[]
for file in listFiles:
    lines = open(file,'r').readlines()

    filedata = {}
    filedata['name'] = file
    filedata['rawLines'] = lines
    col1Vals = []
    col2Vals = []
    mapValues = {}

    for line in lines:           
       values = line.split(',')
       col1Vals.append(values[0])
       col2Vals.append(values[1])
       mapValues[values[0]] = values[1]
    filedata['col1'] = col1Vals
    filedata['col2'] = col2Vals
    filedata['map'] = mapValues
    allData.append(filedata)

answered Oct 12, 2018 by rani
+1 vote

Are you looking for an array that is [txt column1, txt column2, filename]?

file_list = glob.glob(source_dir + '/*.TXT') #Get folder path containing text files


for num,file_path in enumerate(file_list):
  data = np.genfromtxt(file_path, delimiter=',', skip_header=3, skip_footer=18)
  data = np.vstack((data.T,np.ones(data.shape[0])*num)).T
  if num==0: Output=data
  else: Output=np.vstack((Output,data))

An alternative if you dont want to transpose twice.

  data = np.vstack((data,(np.ones(data.shape[0])*num).reshape(-1,1)))
answered Oct 12, 2018 by kalpesh

Related Questions In Python

0 votes
0 answers

How to mix read() and write() on Python files in Windows

It appears that a write() immediately following a read() on a ...READ MORE

Oct 23, 2018 in Python by Aryya
• 500 points
57 views
0 votes
0 answers

How to read lines from compressed text files in Python

Is it easy to read a line ...READ MORE

Dec 27, 2018 in Python by Anirudh
• 2,070 points
93 views
0 votes
1 answer
0 votes
0 answers

How to read data in a bubble sort algorithm in python?

I am not able to read the ...READ MORE

Jul 31 in Python by Waseem
• 4,470 points
48 views
0 votes
1 answer

How to replace values with None in Pandas data frame in Python?

Actually in later versions of pandas this ...READ MORE

answered Aug 30, 2018 in Python by Priyaj
• 56,940 points
2,088 views
0 votes
1 answer

How to read Excel File in Python

With pandas it is possible to get ...READ MORE

answered Oct 22, 2018 in Python by Priyaj
• 56,940 points
191 views
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6 in Python by Neha
• 330 points

edited Jul 8 by Kalgi 346 views
+4 votes
6 answers