Python Multiprocessing.Pool工作者在使用Pool.map时挂起

所以我有一个脚本，它连接到大约700个设备，执行一系列命令，然后退出。我开始使用Multiprocessing.Pool和Pool.map来减少脚本的运行时间，并允许我同时登录多个设备

现在，我遇到了一些奇怪的问题，池中的工人将无限期地被绞死，我不明白他们为什么这样做。另外，我对Python还是有点陌生，所以对于我的方法的任何反馈都非常感谢。下面是我的代码：

def expect(ssh_channel_reference, pattern, command, wait, debug):
    timeout_counter = 0
    full_buffer = ''
    temp_buffer = ''
    while timeout_counter < wait:
        temp_buffer = ssh_channel_reference.recv(8192)
        # print temp_buffer
        if pattern in temp_buffer:
            ssh_channel_reference.send('%s\r' % command)
            timeout_counter = wait + 10
            full_buffer += temp_buffer
        else:
            time.sleep(0.01)
            timeout_counter += 0.01
            full_buffer += temp_buffer
        if debug:
            return full_buffer
    if timeout_counter == wait:
        raise Exception()


def backup_device(device, user, user_pass, failures):
    attempt_counter = 0
    if re.search('(^\d+\.\d+\.\d+\.\d+$)', device):
        host_name = socket.gethostbyaddr(device)
    else:
        host_name = device
    logging.info("Connecting to: %s" % host_name)
    command = 'copy config to backup'
    while attempt_counter <= 5:
        try:
            output_buffer = ''
            ssh_handle = paramiko.SSHClient()
            ssh_handle.set_missing_host_key_policy(paramiko.AutoAddPolicy())
            ssh_handle.connect(hostname=host_name, username=user, password=user_pass, timeout=20, look_for_keys=False)
            ssh_channel = ssh_handle.invoke_shell()
            output_buffer += expect(channel=ssh_channel, pattern='#', command='%s\r' % command, wait=10, debug=True)
            output_buffer += expect(channel=ssh_channel, pattern=']?', command='\n', wait=5, debug=True)
            output_buffer += expect(channel=ssh_channel, pattern=']?', command='\n', wait=5, debug=True)
            output_buffer += expect(channel=ssh_channel, pattern='#', command='exit', wait=5, debug=True)
            print output_buffer
            ssh_handle.close()
        except Exception as inst:
            logging.debug(inst.message)
            attempt_counter += 1
        else:
            logging.info('%s - backed up successfully.' % host_name)
    if attempt_counter == 5:
        logging.critical("Unable to log into device - %s" % host_name)
        failures.append(host_name)
    return

if __name__ == "__main__":
    logging.basicConfig(filename='herpderp.log', filemode='w', level=logging.DEBUG,
                        format='%(asctime)s - %(levelname)s:%(message)s')
    partial_method = partial(backup_device, user=username, user_pass=password, failures=failed_devices)
    pool = multiprocessing.Pool(processes=4, maxtasksperchild=1)
    pool_map = pool.map(func=partial_method, iterable=devices, chunksize=4)
    pool.close()
    pool.join()

截断的日志输出：

2015-05-13 16:04:11,748 - INFO:Connecting to: HOSTNAME
2015-05-13 16:04:11,760 - DEBUG:starting thread (client mode): 0x1ebaf350L
2015-05-13 16:04:11,769 - INFO:Connected (version 1.99, client Cisco-1.25)
2015-05-13 16:04:11,770 - DEBUG:kex algos:[u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa'] client encrypt:[u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] server encrypt:[u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] client mac:[u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96'] server mac:[u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96'] client compress:[u'none'] server compress:[u'none'] client lang:[u''] server lang:[u''] kex follows?False
2015-05-13 16:04:11,770 - DEBUG:Ciphers agreed: local=aes128-cbc, remote=aes128-cbc
2015-05-13 16:04:11,770 - DEBUG:using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-cbc, remote aes128-cbc; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
2015-05-13 16:04:12,038 - DEBUG:Switch to new keys ...
2015-05-13 16:04:12,064 - DEBUG:Adding ssh-rsa host key for HOSTNAME: KEY
2015-05-13 16:04:12,257 - DEBUG:userauth is OK
2015-05-13 16:04:12,258 - INFO:Auth banner: <INSERT BANNER HERE>
2015-05-13 16:04:12,314 - INFO:Authentication (password) successful!
2015-05-13 16:04:12,330 - DEBUG:[chan 0] Max packet in: 32768 bytes
2015-05-13 16:04:12,341 - DEBUG:[chan 0] Max packet out: 4096 bytes
2015-05-13 16:04:12,341 - DEBUG:Secsh channel 0 opened.
2015-05-13 16:04:12,356 - DEBUG:[chan 0] Sesch channel 0 request ok
2015-05-13 16:04:12,365 - DEBUG:[chan 0] Sesch channel 0 request ok

2015-05-13 16:04:11748-信息：连接到：主机名
2015-05-13 16:04:11760-调试：启动线程（客户端模式）：0x1ebaf350L
2015-05-13 16:04:11769-信息：已连接（版本1.99，客户端Cisco-1.25）
2015-05-13 16:04:11770-调试：kex算法：[u'diffie-hellman-group1-sha1']服务器密钥：[u'ssh-rsa']客户端加密：[u'aes128-cbc'，u'3des-cbc'，u'aes192-cbc'，u'aes256-cbc']服务器加密：[u'aes128-cbc'，u'3des-cbc'，u'aes192-cbc'，u'aes256-cbc']客户端mac:[u'hmac-sha1'，u'hmac-sha1-96'，MDU'hmac-5'，u'hmac-96']mac服务器：[u'hmac-sha1'，u'hmac-sha1-96'，u'hmac-md5'，u'hmac-md5-96']客户端压缩：[u'none']服务器压缩：[u'none']客户端语言：[u']服务器语言：[u']kex如下？False
2015-05-13 16:04:11770-调试：同意密码：本地=aes128 cbc，远程=aes128 cbc
2015-05-13 16:04:11770-调试：使用kex diffie-hellman-group1-sha1；服务器密钥类型ssh rsa；密码：本地aes128 cbc，远程aes128 cbc；mac：本地hmac-sha1，远程hmac-sha1；压缩：本地无，远程无
2015-05-13 16:04:12038-调试：切换到新键。。。
2015-05-13 16:04:12064-调试：为主机名添加ssh rsa主机密钥：密钥
2015-05-13 16:04:12257-调试：userauth正常
2015-05-13 16:04:12258-信息：授权横幅：
2015-05-13 16:04:12314-信息：身份验证（密码）成功！
2015-05-13 16:04:12330-调试：[chan 0]最大数据包长度：32768字节
2015-05-13 16:04:12341-调试：[chan 0]最大数据包输出：4096字节
2015-05-13 16:04:12341-调试：Secsh通道0打开。
2015-05-13 16:04:12356-调试：[chan 0]Sesch通道0请求正常
2015-05-13 16:04:12365-调试：[chan 0]Sesch通道0请求正常

---

上面的日志记录是我在池挂起之前看到的。

因此问题是Channel.recv（n字节）是一个阻塞调用，导致线程挂起。我添加了Channel.settimeout（float）这纠正了这个问题，唯一需要注意的是，我需要捕获由超时引起的socket.timeout异常。

我也可以添加日志输出，但这将需要一段时间，因为我需要在发布之前对其进行清理。您可能希望引发该异常，而不是返回它（在expect函数末尾）更改后，池仍然挂起。你能告诉我它挂起的代码在哪里吗？是

ssh\u channel\u reference.recv（8192）

永远阻止吗？我想你搞定了。我和一个同事认为是channel.recv（）默认情况下没有阻塞，显然您必须手动设置，或者指定一个超时。我现在正在等待脚本完成。我将很快更新。

2015-05-13 16:04:11,748 - INFO:Connecting to: HOSTNAME
2015-05-13 16:04:11,760 - DEBUG:starting thread (client mode): 0x1ebaf350L
2015-05-13 16:04:11,769 - INFO:Connected (version 1.99, client Cisco-1.25)
2015-05-13 16:04:11,770 - DEBUG:kex algos:[u'diffie-hellman-group1-sha1'] server key:[u'ssh-rsa'] client encrypt:[u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] server encrypt:[u'aes128-cbc', u'3des-cbc', u'aes192-cbc', u'aes256-cbc'] client mac:[u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96'] server mac:[u'hmac-sha1', u'hmac-sha1-96', u'hmac-md5', u'hmac-md5-96'] client compress:[u'none'] server compress:[u'none'] client lang:[u''] server lang:[u''] kex follows?False
2015-05-13 16:04:11,770 - DEBUG:Ciphers agreed: local=aes128-cbc, remote=aes128-cbc
2015-05-13 16:04:11,770 - DEBUG:using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-cbc, remote aes128-cbc; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
2015-05-13 16:04:12,038 - DEBUG:Switch to new keys ...
2015-05-13 16:04:12,064 - DEBUG:Adding ssh-rsa host key for HOSTNAME: KEY
2015-05-13 16:04:12,257 - DEBUG:userauth is OK
2015-05-13 16:04:12,258 - INFO:Auth banner: <INSERT BANNER HERE>
2015-05-13 16:04:12,314 - INFO:Authentication (password) successful!
2015-05-13 16:04:12,330 - DEBUG:[chan 0] Max packet in: 32768 bytes
2015-05-13 16:04:12,341 - DEBUG:[chan 0] Max packet out: 4096 bytes
2015-05-13 16:04:12,341 - DEBUG:Secsh channel 0 opened.
2015-05-13 16:04:12,356 - DEBUG:[chan 0] Sesch channel 0 request ok
2015-05-13 16:04:12,365 - DEBUG:[chan 0] Sesch channel 0 request ok