Reading a large file, line by line in Python

+1 vote

I want to iterate over each line of an entire file. One way to do this is by reading the entire file, saving it to a list, then going over the line of interest. This method uses a lot of memory, so I am looking for an alternative.

My code so far:

for each_line in fileinput.input(input_file):
    do_something(each_line)
    for each_line_again in fileinput.input(input_file):
        do_something(each_line_again)

Executing this code gives an error message: device active.

Any suggestions?

The purpose is to calculate pair-wise string similarity, meaning for each line in file, I want to calculate the Levenshtein distance with every other line.

Aug 21, 2018 in Python by bug_seeker
• 14,960 points
33 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

The correct, fully Pythonic way to read a file is the following:

with open(...) as f:
    for line in f:
         # Do something with 'line'

The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered I/O and memory management so you don't have to worry about large files.

answered Aug 21, 2018 by Priyaj
• 56,100 points

Related Questions In Python

0 votes
1 answer

In Python, how do I read a file line-by-line into a list?

with open(fname) as f:     content = f.readlines() # you ...READ MORE

answered Oct 9, 2018 in Python by SDeb
• 9,380 points
61 views
+3 votes
7 answers

How can I rename a file in Python?

yes, you can use "os.rename" for that. ...READ MORE

answered Mar 31, 2018 in Python by DareDev
• 6,520 points
54 views
+2 votes
2 answers

How can I create a new file in Python?

You can try the below code which ...READ MORE

answered Mar 31, 2018 in Python by anto.trigg4
• 3,420 points
42 views
0 votes
2 answers

What do you mean by python scripting? What is a script and a module in python?

A scripting language is a programming language ...READ MORE

answered Mar 14 in Python by rajesh kumar
1,510 views
0 votes
1 answer

how can i count the items in a list?

suppose you have a list a = [0,1,2,3,4,5,6,7,8,9,10] now ...READ MORE

answered May 2 in Python by Mohammad
• 1,400 points
19 views
+4 votes
6 answers
0 votes
1 answer

How do I copy a file in python?

Use the shutil module. copyfile(src, dst) Copy the contents ...READ MORE

answered Jul 31, 2018 in Python by Priyaj
• 56,100 points
23 views
+2 votes
3 answers

How can I play an audio file in the background using Python?

down voteacceptedFor windows: you could use  winsound.SND_ASYNC to play them ...READ MORE

answered Apr 3, 2018 in Python by charlie_brown
• 7,710 points
937 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.