PythonMultiprocessing-在正在运行的进程上选择like，查看哪个进程已经完成

我想运行15个命令，但一次只运行3个

testme.py

import multiprocessing
import time
import random
import subprocess

def popen_wrapper(i):
    p = subprocess.Popen( ['echo', 'hi'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = p.communicate()
    print stdout
    time.sleep(randomint(5,20)) #pretend it's doing some work
    return p.returncode

num_to_run = 15
max_parallel = 3

running = []
for i in range(num_to_run):
    p = multiprocessing.Process(target=popen_wrapper, args=(i,))
    running.append(p)
    p.start()

    if len(running) >= max_parallel:
        # blocking wait - join on whoever finishes first then continue
    else:
        # nonblocking wait- see if any processes is finished. If so, join the finished processes

我不确定如何实施以下评论：

if len(running) >= max_parallel:
    # blocking wait - join on whoever finishes first then continue
else:
    # nonblocking wait- see if any processes is finished. If so, join the finished processes

我将无法执行以下操作：

for p in running:
   p.join()

因为运行中的第二个进程可能已经完成，但我仍然阻止了第一个进程

问题：如何检查

运行中的进程在阻塞和非阻塞中是否都已完成（找到第一个已完成）

寻找与waitpid类似的东西，也许
也许最简单的安排方法是使用：

将设置一个包含3个工作进程的池。然后，您可以向池发送15个任务：

for i in range(num_to_run):
    pool.apply_async(popen_wrapper, args=(i,), callback=log_result)

协调3名工人和15项任务所需的所有机械都是
由
mp.Pool
负责


使用mp.Pool：

import multiprocessing as mp
import time
import random
import subprocess
import logging
logger = mp.log_to_stderr(logging.WARN)

def popen_wrapper(i):
    logger.warn('echo "hi"')
    return i

def log_result(retval):
    results.append(retval)

if __name__ == '__main__':

    num_to_run = 15
    max_parallel = 3
    results = []

    pool =  mp.Pool(max_parallel)
    for i in range(num_to_run):
        pool.apply_async(popen_wrapper, args=(i,), callback=log_result)
    pool.close()
    pool.join()

    logger.warn(results)

import multiprocessing as mp
import time
import random
import subprocess
import logging
logger = mp.log_to_stderr(logging.WARN)

SENTINEL = None
def popen_wrapper(inqueue, outqueue):
    for i in iter(inqueue.get, SENTINEL):
        logger.warn('echo "hi"')
        outqueue.put(i)

if __name__ == '__main__':

    num_to_run = 15
    max_parallel = 3

    inqueue = mp.Queue()
    outqueue = mp.Queue()
    procs = [mp.Process(target=popen_wrapper, args=(inqueue, outqueue)) 
             for i in range(max_parallel)]

    for p in procs:
        p.start()
    for i in range(num_to_run):
        inqueue.put(i)
    for i in range(max_parallel):
        # Put sentinels in the queue to tell `popen_wrapper` to quit
        inqueue.put(SENTINEL)

    for p in procs:
        p.join()

    results = [outqueue.get() for i in range(num_to_run)]
    logger.warn(results)

屈服

[WARNING/PoolWorker-1] echo "hi"
[WARNING/PoolWorker-3] echo "hi"
[WARNING/PoolWorker-1] echo "hi"
[WARNING/PoolWorker-1] echo "hi"
[WARNING/PoolWorker-3] echo "hi"
[WARNING/PoolWorker-1] echo "hi"
[WARNING/PoolWorker-3] echo "hi"
[WARNING/PoolWorker-1] echo "hi"
[WARNING/PoolWorker-3] echo "hi"
[WARNING/PoolWorker-1] echo "hi"
[WARNING/PoolWorker-3] echo "hi"
[WARNING/PoolWorker-1] echo "hi"
[WARNING/PoolWorker-1] echo "hi"
[WARNING/PoolWorker-3] echo "hi"
[WARNING/PoolWorker-2] echo "hi"
[WARNING/MainProcess] [0, 2, 3, 5, 4, 6, 7, 8, 9, 10, 11, 12, 14, 13, 1]

日志语句显示哪个PoolWorker处理每个任务，最后一个日志语句显示MainProcess已从对
popen\u wrapper
的15个调用中接收到返回值


如果您希望在没有池的情况下执行此操作，可以为任务设置
mp.Queue
，为返回值设置
mp.Queue
：

使用
mp.Process
和
mp.Queue
s：

import multiprocessing as mp
import time
import random
import subprocess
import logging
logger = mp.log_to_stderr(logging.WARN)

def popen_wrapper(i):
    logger.warn('echo "hi"')
    return i

def log_result(retval):
    results.append(retval)

if __name__ == '__main__':

    num_to_run = 15
    max_parallel = 3
    results = []

    pool =  mp.Pool(max_parallel)
    for i in range(num_to_run):
        pool.apply_async(popen_wrapper, args=(i,), callback=log_result)
    pool.close()
    pool.join()

    logger.warn(results)

import multiprocessing as mp
import time
import random
import subprocess
import logging
logger = mp.log_to_stderr(logging.WARN)

SENTINEL = None
def popen_wrapper(inqueue, outqueue):
    for i in iter(inqueue.get, SENTINEL):
        logger.warn('echo "hi"')
        outqueue.put(i)

if __name__ == '__main__':

    num_to_run = 15
    max_parallel = 3

    inqueue = mp.Queue()
    outqueue = mp.Queue()
    procs = [mp.Process(target=popen_wrapper, args=(inqueue, outqueue)) 
             for i in range(max_parallel)]

    for p in procs:
        p.start()
    for i in range(num_to_run):
        inqueue.put(i)
    for i in range(max_parallel):
        # Put sentinels in the queue to tell `popen_wrapper` to quit
        inqueue.put(SENTINEL)

    for p in procs:
        p.join()

    results = [outqueue.get() for i in range(num_to_run)]
    logger.warn(results)


注意，如果您使用

procs = [mp.Process(target=popen_wrapper, args=(inqueue, outqueue)) 
         for i in range(max_parallel)]

然后您强制执行正是
max_parallel
（例如3个）工作进程。然后将所有15个任务发送到一个队列：

for i in range(num_to_run):
    inqueue.put(i)

并让工作进程将任务从队列中拉出：

def popen_wrapper(inqueue, outqueue):
    for i in iter(inqueue.get, SENTINEL):
        logger.warn('echo "hi"')
        outqueue.put(i)


你也可能会感兴趣。在许多有指导意义的示例中，您会发现有一个示例显示了如何生成10个进程，并限制它们（使用
mp.Semaphore
），以便在任何给定时间只有3个进程处于活动状态。虽然这可能很有启发性，但在您的情况下，这可能不是最好的解决方案，因为您似乎没有理由希望生成3个以上的进程。
您是否可以创建一个成功完成的进程，通过队列向您的监视进程发送消息？