How to read a large file, line by line, in Python?

0 votes

I am trying to iterate over each line of an entire file. One way to do this is by reading the entire file, saving it to a list, then going over the line of interest. This method uses a lot of memory, so I am looking for an alternative.

My code is as follows:

for each_line in fileinput.input(input_file):
    do_something(each_line)

    for each_line_again in fileinput.input(input_file):
        do_something(each_line_again)

Executing this code gives an error message: device active.

The purpose is to calculate pair-wise string similarity, meaning for each line in file, I want to calculate the Levenshtein distance with every other line. Can anyone help me with this?

Jul 1 in Python by ana1504.k
• 7,470 points
10 views

1 answer to this question.

0 votes

The correct, fully Pythonic way to read a file is the following:

with open(...) as f:
    for line in f:
        # Do something with 'line'

The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered I/O and memory management so you don't have to worry about large files.

answered Jul 1 by SDeb
• 12,360 points

Related Questions In Python

+1 vote
1 answer

Reading a large file, line by line in Python

The correct, fully Pythonic way to read ...READ MORE

answered Aug 21, 2018 in Python by Priyaj
• 56,160 points
44 views
0 votes
1 answer

How to create and read from a temporary file in Python?

Hi, there is a very simple solution ...READ MORE

answered Jan 29 in Python by Nymeria
• 3,500 points
37 views
0 votes
1 answer

How to add a new line in Python?

You can use '\n' for a next ...READ MORE

answered May 2, 2018 in Python by aayushi
• 750 points
56 views
0 votes
1 answer

How to read Excel File in Python

With pandas it is possible to get ...READ MORE

answered Oct 22, 2018 in Python by Priyaj
• 56,160 points
95 views
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6 in Python by Neha
• 330 points

edited Jul 8 by Kalgi 130 views
0 votes
1 answer

In Python, how do I read a file line-by-line into a list?

with open(fname) as f:     content = f.readlines() # you ...READ MORE

answered Oct 9, 2018 in Python by SDeb
• 12,360 points
99 views
0 votes
1 answer

How to read numbers from file in Python?

Assuming you don't have extraneous whitespace: with open('file') ...READ MORE

answered Apr 16 in Python by SDeb
• 12,360 points
70 views