Q:

Avoiding deadlocks due to queue overflow with multiprocessing.JoinableQueue

Suppose we have a multiprocessing.Pool where worker threads share a multiprocessing.JoinableQueue, dequeuing work items and potentially enqueuing more work:
def worker_main(queue):
    while True:
        work = queue.get()
        for new_work in process(work):
            queue.put(new_work)
        queue.task_done()

When the queue fills up, queue.put() will block. As long as there is at least one process reading from the queue with queue.get(), it will free up space in the queue to unblock the writers. But all of the processes could potentially block at queue.put() at the same time.
Is there a way to avoid getting jammed up like this?

A:

Depending on how often process(work) creates more items, there may be no solution beside a queue of an infinite maximum size.
In short, your queue must be large enough to accomodate the entire backlog of work items that you can have at any time.

Since queue is implemented with semaphores, there may indeed be a hard size limit of SEM_VALUE_MAX which in MacOS is 32767. So you'll need to subclass that implementation or use put(block=False) and handle queue.Full (e.g. put excess items somewhere else) if that's not enough.
Alternatively, look at one of the 3rd-party implementations of distributed work item queue for Python.

