How to use multiprocessing pool map with multiple arguments

0 votes

In the Python multiprocessing library, is there a variant of pool.map which supports multiple arguments?

text = "test"
def harvester(text, case):
    X = case[0]
    text+ str(X)

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=6)
    case = RAW_DATASET
    pool.map(harvester(text,case),case, 1)
    pool.close()
    pool.join()
Nov 18, 2020 in Python by Rajiv
• 8,880 points
3,776 views

1 answer to this question.

0 votes

The answer to this is version- and situation-dependent. 

It then automatically unpacks the arguments from each tuple and passes them to the given function:

import multiprocessing
from itertools import product

def merge_names(a, b):
    return '{} & {}'.format(a, b)

if __name__ == '__main__':
    names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie']
    with multiprocessing.Pool(processes=3) as pool:
        results = pool.starmap(merge_names, product(names, repeat=2))
    print(results)

# Output: ['Brown & Brown', 'Brown & Wilson', 'Brown & Bartlett', ...

For earlier versions of Python, you'll need to write a helper function to unpack the arguments explicitly. If you want to use with, you'll also need to write a wrapper to turn Pool into a context manager.

import multiprocessing
from itertools import product
from contextlib import contextmanager

def merge_names(a, b):
    return '{} & {}'.format(a, b)

def merge_names_unpack(args):
    return merge_names(*args)

@contextmanager
def poolcontext(*args, **kwargs):
    pool = multiprocessing.Pool(*args, **kwargs)
    yield pool
    pool.terminate()

if __name__ == '__main__':
    names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie']
    with poolcontext(processes=3) as pool:
        results = pool.map(merge_names_unpack, product(names, repeat=2))
    print(results)

# Output: ['Brown & Brown', 'Brown & Wilson', 'Brown & Bartlett', ...

In simpler cases, with a fixed second argument, you can also use partial, but only in Python 2.7+.

import multiprocessing
from functools import partial
from contextlib import contextmanager

@contextmanager
def poolcontext(*args, **kwargs):
    pool = multiprocessing.Pool(*args, **kwargs)
    yield pool
    pool.terminate()

def merge_names(a, b):
    return '{} & {}'.format(a, b)

if __name__ == '__main__':
    names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie']
    with poolcontext(processes=3) as pool:
        results = pool.map(partial(merge_names, b='Sons'), names)
    print(results)

# Output: ['Brown & Sons', 'Wilson & Sons', 'Bartlett & Sons', ...
answered Nov 18, 2020 by Gitika
• 65,870 points

Related Questions In Python

+1 vote
1 answer

How to use GUI that comes with Python to test your code?

Hey @alex0809, When your testing a website ...READ MORE

answered Sep 24, 2018 in Python by Vardhan
• 13,200 points
259 views
0 votes
1 answer

How to use multiprocessing queue in Python?

This is a simple example of a ...READ MORE

answered Oct 25, 2018 in Python by SDeb
• 13,270 points
3,686 views
0 votes
1 answer

How to use MySQLdb with Python and Django?

Hello @kartik, I had the same error and pip ...READ MORE

answered Jun 24, 2020 in Python by Niroj
• 82,720 points
124 views
0 votes
1 answer

How to use Python 3 and Django with Apache?

Hello @kartik, 1) The first thing we should ...READ MORE

answered Jul 2, 2020 in Python by Niroj
• 82,720 points
406 views
0 votes
2 answers
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 6, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 2,033 views
0 votes
0 answers
+1 vote
3 answers

How can I use python to execute a curl command?

For sake of simplicity, maybe you should ...READ MORE

answered Oct 11, 2018 in Python by charlie_brown
• 7,780 points
67,350 views
+1 vote
10 answers

How to fix this? ValueError: invalid literal for int() with base 10 error in Python

Just for the record: >>> int('55063.000000') Traceback (most recent ...READ MORE

answered Aug 17, 2020 in Python by pakainfo
• 200 points
262,713 views