Avoid killing children when parent process is killed

Question

I use the library multiprocessing in a flask-based web application to start long-running processes. The function that does it is the following:

def execute(self, process_id):
    self.__process_id = process_id
    process_dir = self.__dependencies["process_dir"]
    self.fit_dependencies()
    process = Process(target=self.function_wrapper, name=process_id, args=(self.__parameters, self.__config, process_dir,))
    process.start()

When I want to deploy some code on this web application, I restart a service that restarts gunicorn, served by nginx. My problem is that this restart kills all children processes started by this application as if a SIGINT signal were sent to all children. How could I avoid that ?

EDIT: After reading this post, it appears that this behavior is normal. The answer suggests to use the subprocess library instead. So I reformulate my question: how should I proceed if I want to start long-running tasks (which are python functions) in a python script and make sure they would survive the parent process OR make sure the parent process (which is a gunicorn instance) would survive a deployement ?

Answer 1

I would recommend against your design as it's quite error prone. Better solutions would de-couple the workers from the server using some sort of queuing system (RabbitMQ, Celery, Redis, ...).

Nevertheless, here's a couple of "hacks" you could try out.

1. Turn your child processes into UNIX daemons. The python daemon module could be a starting point.

2. Instruct your child processes to ignore the SIGINT signal. The service orchestrator might work around that by issuing a SIGTERM or SIGKILL signal if child processes refuse to die. You might need to disable such feature.

   To do so, just add the following line at the beginning of the function_wrapper function:

   signal.signal(signal.SIGINT, signal.SIG_IGN)