How to use multiprocessing queue in Python?

0 votes
I'm having much trouble trying to understand just how the multiprocessing queue works on python and how to implement it. Lets say I have two python modules that access data from a shared file, let's call these two modules a writer and a reader. My plan is to have both the reader and writer put requests into two separate multiprocessing queues, and then have a third process pop these requests in a loop and execute as such.

My main problem is that I really don't know how to implement multiprocessing.queue correctly, you cannot really instantiate the object for each process since they will be separate queues, how do you make sure that all processes relate to a shared queue.
Oct 25, 2018 in Python by ana1504.k
• 6,110 points
86 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes
This is a simple example of a reader and writer sharing a single queue... The writer sends a bunch of integers to the reader; when the writer runs out of numbers, it sends 'DONE', which lets the reader know to break out of the read loop.

from multiprocessing import Process, Queue
import time
import sys

def reader_proc(queue):
    ## Read from the queue; this will be spawned as a separate Process
    while True:
        msg = queue.get()         # Read from the queue and do nothing
        if (msg == 'DONE'):
            break

def writer(count, queue):
    ## Write to the queue
    for ii in range(0, count):
        queue.put(ii)             # Write 'count' numbers into the queue
    queue.put('DONE')

if __name__=='__main__':
    pqueue = Queue() # writer() writes to pqueue from _this_ process
    for count in [10**4, 10**5, 10**6]:             
        ### reader_proc() reads from pqueue as a separate process
        reader_p = Process(target=reader_proc, args=((pqueue),))
        reader_p.daemon = True
        reader_p.start()        # Launch reader_proc() as a separate python process

        _start = time.time()
        writer(count, pqueue)    # Send a lot of stuff to reader()
        reader_p.join()         # Wait for the reader to finish
        print("Sending {0} numbers to Queue() took {1} seconds".format(count,
            (time.time() - _start)))
answered Oct 25, 2018 by SDeb
• 9,660 points

Related Questions In Python

0 votes
2 answers

How to use threading in Python?

 Thread is the smallest unit of processing that ...READ MORE

answered Apr 6 in Python by anonymous
44 views
0 votes
1 answer

How to use “raise” keyword in Python

You can use it to raise errors ...READ MORE

answered Jul 30, 2018 in Python by Priyaj
• 56,120 points
16 views
0 votes
1 answer

How to use string.replace() in python 3.x

replace() is a method of <class 'str'> ...READ MORE

answered Aug 3, 2018 in Python by Priyaj
• 56,120 points
23 views
+1 vote
2 answers

How to use the pass statement in Python

In Python programming, pass is a null statement. The ...READ MORE

answered Apr 5 in Python by anonymous
26 views
0 votes
1 answer

Multiprocessing queue in Python?

Here you can find a brief example ...READ MORE

answered Oct 29, 2018 in Python by Priyaj
• 56,120 points
101 views
0 votes
1 answer

how can i count the items in a list?

suppose you have a list a = [0,1,2,3,4,5,6,7,8,9,10] now ...READ MORE

answered May 2 in Python by Mohammad
• 1,400 points
22 views
0 votes
1 answer

How to use a global variable in function?

The global variable can be used in ...READ MORE

answered Sep 27, 2018 in Python by SDeb
• 9,660 points
15 views
0 votes
1 answer

How to Reverse a list in Python

You can use the reversed function in ...READ MORE

answered Sep 28, 2018 in Python by SDeb
• 9,660 points
44 views

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.