Reading a large file, line by line in Python

+1 vote

I want to iterate over each line of an entire file. One way to do this is by reading the entire file, saving it to a list, then going over the line of interest. This method uses a lot of memory, so I am looking for an alternative.

My code so far:

for each_line in fileinput.input(input_file):
    do_something(each_line)
    for each_line_again in fileinput.input(input_file):
        do_something(each_line_again)

Executing this code gives an error message: device active.

Any suggestions?

The purpose is to calculate pair-wise string similarity, meaning for each line in file, I want to calculate the Levenshtein distance with every other line.

Aug 21, 2018 in Python by bug_seeker
• 15,310 points
44 views

1 answer to this question.

0 votes

The correct, fully Pythonic way to read a file is the following:

with open(...) as f:
    for line in f:
         # Do something with 'line'

The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered I/O and memory management so you don't have to worry about large files.

answered Aug 21, 2018 by Priyaj
• 56,160 points

Related Questions In Python

0 votes
1 answer

How to read a large file, line by line, in Python?

The correct, fully Pythonic way to read ...READ MORE

answered Jul 1 in Python by SDeb
• 12,360 points
10 views
0 votes
1 answer

In Python, how do I read a file line-by-line into a list?

with open(fname) as f:     content = f.readlines() # you ...READ MORE

answered Oct 9, 2018 in Python by SDeb
• 12,360 points
99 views
0 votes
1 answer

Enter a 1-10, each on a new line in a file - python

Try something like this: i=1 for i in range(10): ...READ MORE

answered Jun 20 in Python by Greg
11 views
+3 votes
7 answers

How can I rename a file in Python?

yes, you can use "os.rename" for that. ...READ MORE

answered Mar 31, 2018 in Python by DareDev
• 6,810 points
89 views
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6 in Python by Neha
• 330 points

edited Jul 8 by Kalgi 129 views
+4 votes
6 answers
0 votes
1 answer

How do I copy a file in python?

Use the shutil module. copyfile(src, dst) Copy the contents ...READ MORE

answered Jul 31, 2018 in Python by Priyaj
• 56,160 points
29 views
+2 votes
3 answers

How can I play an audio file in the background using Python?

down voteacceptedFor windows: you could use  winsound.SND_ASYNC to play them ...READ MORE

answered Apr 3, 2018 in Python by charlie_brown
• 7,710 points
1,323 views