Need some help with Python memory leaks

Question

I have a long-running script which, if let to run long enough, will consume all the memory on my system.

I have 2 questions that needs answers:

1. Are there any best practices to follow, which will help prevent leaks from occurring?
2. What techniques are there to debug memory leaks in Python?

Thank you!

Answer 1

As far as best practices, keep an eye for recursive functions. In my case I ran into issues with recursion (where there didn't need to be). A simplified example of what I was doing:

def my_function():
    # lots of memory intensive operations
    # like operating on images or huge dictionaries and lists
    .....
    my_flag = True
    if my_flag:  # restart the function if a certain flag is true
        my_function()

def main():
    my_function()

operating in this recursive manner won't trigger the garbage collection and clear out the remains of the function, so every time through memory usage is growing and growing.

My solution was to pull the recursive call out of my_function() and have main() handle when to call it again. this way the function ends naturally and cleans up after itself.

def my_function():
    # lots of memory intensive operations
    # like operating on images or huge dictionaries and lists
    .....
    my_flag = True
    .....
    return my_flag

def main():
    result = my_function()
    if result:
        my_function()