Can t open a to big csv file in python

0 votes

I tried to open a big .csv file in python to seperate each row and append the last x lines in a new list.

btcDatear = []
btcPricear = []
btcVolumear = []
howfarback = 20000
try:
    sourceCode = open('.btceUSD.csv', 'r')
    splitSource = sourceCode.split('\n')

        for eachline in splitSource[-howfarback:]:
            splitLine = eachline.split(',')
            btcDate = splitLine[0]
            btcPrice = splitLine[1]
            btcVolume = splitLine[2]

            btcDatear.append(float(btcDate))
            btcPricear.append(float(btcPrice))
            btcVolumear.append(float(btcVolume))


except Exception, e:
    print "failed raw data", str(e)

It works with file as small as 20MB but when i try it with a file around 500MB, it does not work.

Sep 3, 2018 in Blockchain by digger
• 26,740 points
990 views

1 answer to this question.

0 votes

You can't "split a file", but you can read it line by line no matter how big. E.g:

import collections

btcDatear = []
btcPricear = []
btcVolumear = []
howfarback = 20000
try:
    with open('.btceUSD.csv', 'r') as sourceCode:
        lastNlines = collections.deque(sourceCode, howfarback)
    for eachline in lastNlines:
        splitLine = eachline.split(',')
        btcDate = splitLine[0]
        btcPrice = splitLine[1]
        btcVolume = splitLine[2]

        btcDatear.append(float(btcDate))
        btcPricear.append(float(btcPrice))
        btcVolumear.append(float(btcVolume))
except Exception as e:
    print "failed raw data", str(e)
answered Sep 3, 2018 by slayer
• 29,370 points

Related Questions In Blockchain

0 votes
1 answer

How to read file from subprocess.open in python?

you call: out, err = p.communicate() READ MORE

answered Aug 24, 2018 in Blockchain by digger
• 26,740 points
1,388 views
0 votes
1 answer

What could be the best term to use for the collection of contracts in a .sol file?

module - don't think so. Because module ...READ MORE

answered Jun 2, 2018 in Blockchain by Shashank
• 10,400 points
663 views
0 votes
1 answer

Hyperledger Fabric : How to initialise a 2D array in .cto file?

You could try something like this, with ...READ MORE

answered Oct 30, 2018 in Blockchain by Omkar
• 69,230 points
678 views
+1 vote
2 answers
0 votes
1 answer

Python request module for bitcoin json rpc

This should work: #!/usr/bin/env python import getpass import json import requests ...READ MORE

answered Aug 28, 2018 in Blockchain by digger
• 26,740 points
2,321 views
0 votes
1 answer

Truffle tests not running after truffle init

This was a bug. They've fixed it. ...READ MORE

answered Sep 11, 2018 in Blockchain by Christine
• 15,790 points
1,885 views
+1 vote
3 answers

Removing double quotes from a string from JSON response in PHP

Just remove the json_encode call, and it should work: $resp ...READ MORE

answered Sep 12, 2018 in Blockchain by digger
• 26,740 points
45,057 views
+2 votes
1 answer
0 votes
1 answer

Cant locate english.txt file while intalling Python Bitcoin module

This is related to a pybitcointools bug where the ...READ MORE

answered Aug 30, 2018 in Blockchain by slayer
• 29,370 points
725 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP