How to use multiprocessing queue in Python

0 votes
I'm having much trouble trying to understand just how the multiprocessing queue works on python and how to implement it. Lets say I have two python modules that access data from a shared file, let's call these two modules a writer and a reader. My plan is to have both the reader and writer put requests into two separate multiprocessing queues, and then have a third process pop these requests in a loop and execute as such.

My main problem is that I really don't know how to implement multiprocessing.queue correctly, you cannot really instantiate the object for each process since they will be separate queues, how do you make sure that all processes relate to a shared queue.
Oct 25, 2018 in Python by ana1504.k
• 7,910 points
5,688 views

1 answer to this question.

0 votes
This is a simple example of a reader and writer sharing a single queue... The writer sends a bunch of integers to the reader; when the writer runs out of numbers, it sends 'DONE', which lets the reader know to break out of the read loop.

from multiprocessing import Process, Queue
import time
import sys

def reader_proc(queue):
    ## Read from the queue; this will be spawned as a separate Process
    while True:
        msg = queue.get()         # Read from the queue and do nothing
        if (msg == 'DONE'):
            break

def writer(count, queue):
    ## Write to the queue
    for ii in range(0, count):
        queue.put(ii)             # Write 'count' numbers into the queue
    queue.put('DONE')

if __name__=='__main__':
    pqueue = Queue() # writer() writes to pqueue from _this_ process
    for count in [10**4, 10**5, 10**6]:             
        ### reader_proc() reads from pqueue as a separate process
        reader_p = Process(target=reader_proc, args=((pqueue),))
        reader_p.daemon = True
        reader_p.start()        # Launch reader_proc() as a separate python process

        _start = time.time()
        writer(count, pqueue)    # Send a lot of stuff to reader()
        reader_p.join()         # Wait for the reader to finish
        print("Sending {0} numbers to Queue() took {1} seconds".format(count,
            (time.time() - _start)))
answered Oct 25, 2018 by SDeb
• 13,300 points

Related Questions In Python

0 votes
2 answers

How to use threading in Python?

 Thread is the smallest unit of processing that ...READ MORE

answered Apr 6, 2019 in Python by anonymous
1,027 views
0 votes
1 answer

How to use “raise” keyword in Python

You can use it to raise errors ...READ MORE

answered Jul 30, 2018 in Python by Priyaj
• 58,090 points
489 views
0 votes
1 answer

How to use string.replace() in python 3.x

replace() is a method of <class 'str'> ...READ MORE

answered Aug 3, 2018 in Python by Priyaj
• 58,090 points
850 views
+1 vote
2 answers

How to use the pass statement in Python

In Python programming, pass is a null statement. The ...READ MORE

answered Apr 5, 2019 in Python by anonymous
749 views
0 votes
1 answer

Multiprocessing queue in Python?

Here you can find a brief example ...READ MORE

answered Oct 29, 2018 in Python by Priyaj
• 58,090 points
2,640 views
0 votes
2 answers
+1 vote
2 answers

how can i count the items in a list?

Syntax :            list. count(value) Code: colors = ['red', 'green', ...READ MORE

answered Jul 7, 2019 in Python by Neha
• 330 points

edited Jul 8, 2019 by Kalgi 4,007 views
0 votes
1 answer

How to use a global variable in function?

The global variable can be used in ...READ MORE

answered Sep 27, 2018 in Python by SDeb
• 13,300 points
694 views
0 votes
1 answer

How to Reverse a list in Python

You can use the reversed function in ...READ MORE

answered Sep 29, 2018 in Python by SDeb
• 13,300 points
631 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP