How to read multiple data files in python

Question

I have two text Files (not in CSV) Now how to gather the these data files into one single file

Data1

Month    Spend    Sales
1    1000    9914
2    4000    40487
3    5000    54324
4    4500    50044

Data2

Month    Spend    Sales
5    3000    34719
6    4000    42551
7    9000    94871
8    11000    118914
9    15000    158484
10    12000    131348
11    7000    78504
12    3000    36284

 

Answer 1

Firstly we will import pandas to read the text files

import pandas as pd

Now we will read the text file

de=pd.read_table("data1.txt")
de1=pd.read_table("data2.txt")

Convert the files into data frame

dp=pd.DataFrame(data=de)
dp1=pd.DataFrame(data=de1)

concatenate the both data frames into one

dpe=pd.concat([de,de1])

Answer 2

import sys

# argv is your commandline arguments, argv[0] is your program name, so skip it
for n in sys.argv[1:]:
    print(n) #print out the filename we are currently processing
    input = open(n, "r")
    output = open(n + ".out", "w")
    # do some processing
    input.close()
    output.close()

Then call it like:

./foo.py bar.txt baz.txt

Hope this helps!!

If you need to learn more about Python, It's recommended to join Python Training today.

Thanks!

Answer 3

IF all data is of the same shape then just append to a list.

all_data = [] 

and in your loop:

all_data.append(data)

finally you have

asarray(all_data)

which is an array of shape (10,50,2) (transpose if you want). If the shapes don't match, then this does not work though, numpy cannot handle rows of different shapes. Then you might need another loop which creates arrays of the largest shape, and copy your data over.

Answer 4

crude but quick

listFiles=["1.txt","2.txt", ... ,"xxx.txt"]
allData=[]
for file in listFiles:
    lines = open(file,'r').readlines()

    filedata = {}
    filedata['name'] = file
    filedata['rawLines'] = lines
    col1Vals = []
    col2Vals = []
    mapValues = {}

    for line in lines:           
       values = line.split(',')
       col1Vals.append(values[0])
       col2Vals.append(values[1])
       mapValues[values[0]] = values[1]
    filedata['col1'] = col1Vals
    filedata['col2'] = col2Vals
    filedata['map'] = mapValues
    allData.append(filedata)

Answer 5

Are you looking for an array that is [txt column1, txt column2, filename]?

file_list = glob.glob(source_dir + '/*.TXT') #Get folder path containing text files


for num,file_path in enumerate(file_list):
  data = np.genfromtxt(file_path, delimiter=',', skip_header=3, skip_footer=18)
  data = np.vstack((data.T,np.ones(data.shape[0])*num)).T
  if num==0: Output=data
  else: Output=np.vstack((Output,data))

An alternative if you dont want to transpose twice.

  data = np.vstack((data,(np.ones(data.shape[0])*num).reshape(-1,1)))