在Python中管理固定数量的工作人员

我需要实现一个带有主进程的系统，该主进程管理执行其他任务的从进程。我有两种不同的从机类型，每个从机需要6个实例。我已经写了一些有用的东西，但它会终止每个进程，并在任务完成时启动一个新进程。这是不可取的，因为生成新流程的成本很高。我更愿意让每个从机作为一个进程运行，并在完成时得到通知，然后使用新的输入再次运行它

下面是我当前的伪ish代码。它并不完美；我之所以这么做是因为我身上没有真正的代码

# SlaveTypeB is pretty much the same.
class SlaveTypeA(multiprocessing.Process):
    def __init__(self, val):
        self.value = val
        self.result = multiprocessing.Queue(1)
        self.start()
    def run(self):
        # In real life, run does something that takes a few seconds.
        sleep(2)
        # For SlaveTypeB, assume it writes self.val to a file instead of incrementing
        self.result.put(self.val + 1)
    def getResult(self):
        return self.result.get()[0]


if __name__ == "__main__":
    MAX_PROCESSES = 6
    # In real life, the input will grow as the while loop is being processed
    input = [1, 4, 5, 6, 9, 6, 3, 3]
    aProcessed = []
    aSlaves = []
    bSlaves = []

    while len(input) > 0 or len(aProcessed) > 0:
        if len(aSlaves) < MAX_PROCESSES and len(input) > 0:
            aSlaves.append(SlaveTypeA(input.pop(0))
        if len(bSlaves) < MAX_PROCESSES and len(aProcessed) > 0 :
            bSlaves.append(SlaveTypeB(aProcesssed.pop(0))
        for aSlave in aSlaves:
            if not aSlave.isAlive():
                aProcessed = aSlave.getResult()
                aSlaves.remove(aSlave)
        for bSlave in bSlaves:
            if not bSlave.isAlive():
                bSlaves.remove(bSlave)

编辑2 明白了，请参见下面的答案。

创建两个队列？ 比如

worktodoA

和

worktodoB

让你的员工在等待某个东西被放入队列时处于空闲状态，如果该项目被放入队列，让他们说“退出”，他们会退出吗

---

否则您应该试一试

看起来使用SyncManager是这种情况下的最佳选择

class Master(SyncManager):
    pass

input = [1, 4, 5, 6, 9, 6, 6, 3, 3]
def getNextInput():
    # Check that input isn't empty first
    return input.pop()

if __name__ == "__main__":
    MAX_PROCESSES = 6
    Master.register("getNextInput", getNextInput)
    m = Master(('localhost', 5000))
    m.start()
    for i in range(MAX_PROCESSES):
        Slave()
    while True:
        pass

class Slave(Process):
    def __init__(self):
        multiprocessing.Process.__init__(self)
        self.start()
    def run(self):
        Master.register("getNextInput", getNextInput)
        m = Master(('localhost', 5000))
        m.connect()
        while True:
            input = m.getNextInput()
            # Check for None first
            self.process(input)
    def process(self):
        print "Processed " + str(input)

---

你看过多处理池了吗？我认为不可能阻止池进程退出并重用它们，是吗？这就是我现在正在做的。问题是我需要知道工人什么时候完工。

class Master(SyncManager):
    pass

input = [1, 4, 5, 6, 9, 6, 6, 3, 3]
def getNextInput():
    # Check that input isn't empty first
    return input.pop()

if __name__ == "__main__":
    MAX_PROCESSES = 6
    Master.register("getNextInput", getNextInput)
    m = Master(('localhost', 5000))
    m.start()
    for i in range(MAX_PROCESSES):
        Slave()
    while True:
        pass

class Slave(Process):
    def __init__(self):
        multiprocessing.Process.__init__(self)
        self.start()
    def run(self):
        Master.register("getNextInput", getNextInput)
        m = Master(('localhost', 5000))
        m.connect()
        while True:
            input = m.getNextInput()
            # Check for None first
            self.process(input)
    def process(self):
        print "Processed " + str(input)