How to read a large file line by line in Python

0 votes

I am trying to iterate over each line of an entire file. One way to do this is by reading the entire file, saving it to a list, then going over the line of interest. This method uses a lot of memory, so I am looking for an alternative.

My code is as follows:

for each_line in fileinput.input(input_file):
    do_something(each_line)

    for each_line_again in fileinput.input(input_file):
        do_something(each_line_again)

Executing this code gives an error message: device active.

The purpose is to calculate pair-wise string similarity, meaning for each line in file, I want to calculate the Levenshtein distance with every other line. Can anyone help me with this?

Jul 1, 2019 in Python by ana1504.k
• 7,910 points
882 views

1 answer to this question.

0 votes

The correct, fully Pythonic way to read a file is the following:

with open(...) as f:
    for line in f:
        # Do something with 'line'

The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered I/O and memory management so you don't have to worry about large files.

answered Jul 1, 2019 by SDeb
• 13,300 points

Related Questions In Python

+1 vote
1 answer

Reading a large file, line by line in Python

The correct, fully Pythonic way to read ...READ MORE

answered Aug 21, 2018 in Python by Priyaj
• 58,020 points
1,062 views
0 votes
1 answer

How to create and read from a temporary file in Python?

Hi, there is a very simple solution ...READ MORE

answered Jan 29, 2019 in Python by Nymeria
• 3,560 points
2,080 views
–1 vote
2 answers
0 votes
1 answer

I want to download a file from the website by web scraping. Can anyone explain how to do this in jupyter lab (python) with an example?

Hey, Web scraping is a technique to automatically ...READ MORE

answered Apr 7, 2020 in Python by Gitika
• 65,770 points
2,409 views
0 votes
2 answers
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 7, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 4,463 views
0 votes
1 answer
0 votes
1 answer

In Python, how do I read a file line-by-line into a list?

with open(fname) as f:     content = f.readlines() # you ...READ MORE

answered Oct 9, 2018 in Python by SDeb
• 13,300 points
1,814 views
0 votes
1 answer

How to read numbers from file in Python?

Assuming you don't have extraneous whitespace: with open('file') ...READ MORE

answered Apr 16, 2019 in Python by SDeb
• 13,300 points
7,508 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP