Can't open a to big .csv file in python

0 votes

I tried to open a big .csv file in python to seperate each row and append the last x lines in a new list.

btcDatear = []
btcPricear = []
btcVolumear = []
howfarback = 20000
try:
    sourceCode = open('.btceUSD.csv', 'r')
    splitSource = sourceCode.split('\n')

        for eachline in splitSource[-howfarback:]:
            splitLine = eachline.split(',')
            btcDate = splitLine[0]
            btcPrice = splitLine[1]
            btcVolume = splitLine[2]

            btcDatear.append(float(btcDate))
            btcPricear.append(float(btcPrice))
            btcVolumear.append(float(btcVolume))


except Exception, e:
    print "failed raw data", str(e)

It works with file as small as 20MB but when i try it with a file around 500MB, it does not work.

Sep 3, 2018 in Blockchain by digger
• 27,630 points
40 views

1 answer to this question.

0 votes

You can't "split a file", but you can read it line by line no matter how big. E.g:

import collections

btcDatear = []
btcPricear = []
btcVolumear = []
howfarback = 20000
try:
    with open('.btceUSD.csv', 'r') as sourceCode:
        lastNlines = collections.deque(sourceCode, howfarback)
    for eachline in lastNlines:
        splitLine = eachline.split(',')
        btcDate = splitLine[0]
        btcPrice = splitLine[1]
        btcVolume = splitLine[2]

        btcDatear.append(float(btcDate))
        btcPricear.append(float(btcPrice))
        btcVolumear.append(float(btcVolume))
except Exception as e:
    print "failed raw data", str(e)
answered Sep 3, 2018 by slayer
• 29,050 points

Related Questions In Blockchain

0 votes
1 answer

How to read file from subprocess.open in python?

you call: out, err = p.communicate() READ MORE

answered Aug 24, 2018 in Blockchain by digger
• 27,630 points
46 views
0 votes
1 answer

What could be the best term to use for the collection of contracts in a .sol file?

module - don't think so. Because module ...READ MORE

answered Jun 1, 2018 in Blockchain by Shashank
• 10,330 points
31 views
0 votes
1 answer

Hyperledger Fabric : How to initialise a 2D array in .cto file?

You could try something like this, with ...READ MORE

answered Oct 30, 2018 in Blockchain by Omkar
• 67,290 points
86 views
0 votes
1 answer

How to make sure transactions take no fee in a private Ethereum blockchain?

In a private ethereum network you have ...READ MORE

answered Mar 26, 2018 in Blockchain by Christine
• 15,790 points

edited Mar 26, 2018 by Christine 84 views
0 votes
1 answer

Python request module for bitcoin json rpc

This should work: #!/usr/bin/env python import getpass import json import requests ...READ MORE

answered Aug 28, 2018 in Blockchain by digger
• 27,630 points
209 views
0 votes
1 answer

Truffle tests not running after truffle init

This was a bug. They've fixed it. ...READ MORE

answered Sep 11, 2018 in Blockchain by Christine
• 15,790 points
125 views
+1 vote
3 answers

Removing double quotes from a string from JSON response in PHP

Just remove the json_encode call, and it should work: $resp ...READ MORE

answered Sep 12, 2018 in Blockchain by digger
• 27,630 points
2,495 views
+2 votes
1 answer
0 votes
1 answer

Cant locate english.txt file while intalling Python Bitcoin module

This is related to a pybitcointools bug where the ...READ MORE

answered Aug 30, 2018 in Blockchain by slayer
• 29,050 points
33 views