Reading a large file line by line in Python

+1 vote

I want to iterate over each line of an entire file. One way to do this is by reading the entire file, saving it to a list, then going over the line of interest. This method uses a lot of memory, so I am looking for an alternative.

My code so far:

for each_line in fileinput.input(input_file):
    do_something(each_line)
    for each_line_again in fileinput.input(input_file):
        do_something(each_line_again)

Executing this code gives an error message: device active.

Any suggestions?

The purpose is to calculate pair-wise string similarity, meaning for each line in file, I want to calculate the Levenshtein distance with every other line.

Aug 21, 2018 in Python by bug_seeker
• 15,550 points
264 views

1 answer to this question.

0 votes

The correct, fully Pythonic way to read a file is the following:

with open(...) as f:
    for line in f:
         # Do something with 'line'

The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered I/O and memory management so you don't have to worry about large files.

answered Aug 21, 2018 by Priyaj
• 58,140 points

Related Questions In Python

0 votes
1 answer

How to read a large file, line by line, in Python?

The correct, fully Pythonic way to read ...READ MORE

answered Jul 1, 2019 in Python by SDeb
• 13,290 points
246 views
0 votes
2 answers

In Python, how do I read a file line-by-line into a list?

readline function help to  read line in ...READ MORE

answered Jun 21, 2020 in Python by sahil
• 560 points
580 views
0 votes
1 answer

Enter a 1-10, each on a new line in a file - python

Try something like this: i=1 for i in range(10): ...READ MORE

answered Jun 20, 2019 in Python by Greg
164 views
0 votes
1 answer

I want to download a file from the website by web scraping. Can anyone explain how to do this in jupyter lab (python) with an example?

Hey, Web scraping is a technique to automatically ...READ MORE

answered Apr 7, 2020 in Python by Gitika
• 65,950 points
826 views
0 votes
2 answers
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 2,362 views
0 votes
0 answers
+5 votes
6 answers

Lowercase in Python

You can simply the built-in function in ...READ MORE

answered Apr 11, 2018 in Python by hemant
• 5,810 points
1,769 views
0 votes
1 answer

How do I copy a file in python?

Use the shutil module. copyfile(src, dst) Copy the contents ...READ MORE

answered Jul 31, 2018 in Python by Priyaj
• 58,140 points
216 views
+2 votes
3 answers

How can I play an audio file in the background using Python?

down voteacceptedFor windows: you could use  winsound.SND_ASYNC to play them ...READ MORE

answered Apr 3, 2018 in Python by charlie_brown
• 7,780 points
9,457 views