带有大型对象的Python多处理管道将挂起
下面我有一个简单的代码片段来演示这个问题带有大型对象的Python多处理管道将挂起,python,dictionary,multiprocessing,pipe,Python,Dictionary,Multiprocessing,Pipe,下面我有一个简单的代码片段来演示这个问题 from multiprocessing import Pipe import time recv_end, send_end = Pipe(duplex=False) d = {'word'+str(elem): elem for elem in range(3000)} start_time = time.time() send_end.send(d) print('--- %s seconds ---' % (time.time()-start
from multiprocessing import Pipe
import time
recv_end, send_end = Pipe(duplex=False)
d = {'word'+str(elem): elem for elem in range(3000)}
start_time = time.time()
send_end.send(d)
print('--- %s seconds ---' % (time.time()-start_time))
以上操作很好,速度足够快,没有问题。但如果我将大小设置为5000,它就会无限期地挂起:
from multiprocessing import Pipe
import time
recv_end, send_end = Pipe(duplex=False)
d = {'word'+str(elem): elem for elem in range(5000)} # changed to 5000
start_time = time.time()
send_end.send(d)
print('--- %s seconds ---' % (time.time()-start_time))
管道是否有尺寸限制,或者这是一个不可复制的问题?如果你把尺寸再大一点怎么样?如果有大小限制,那么避免这个问题并通过管道发送大字典的最佳方法是什么?提前谢谢 出现此问题的原因是
Pipe.send()
是一个阻塞调用,它等待接收。阅读更多。
要使其工作,您可以创建如下代码所示的流程:
#!/usr/bin/env python
from multiprocessing import Pipe, Process
import time
import sys
def foo(conn):
d = {'word'+str(elem): elem for elem in range(5000)} # changed to 5000
conn.send(d)
conn.close()
recv_end, send_end = Pipe(duplex=False)
p = Process(target=foo, args=(send_end, ))
p.start()
start_time = time.time()
recv_end.recv() # Try to comment and you will see that it waits for being received
p.join()
print('--- %s seconds ---' % (time.time()-start_time))
出现此问题的原因是
Pipe.send()
是一个阻塞调用,它等待接收。阅读更多。
要使其工作,您可以创建如下代码所示的流程:
#!/usr/bin/env python
from multiprocessing import Pipe, Process
import time
import sys
def foo(conn):
d = {'word'+str(elem): elem for elem in range(5000)} # changed to 5000
conn.send(d)
conn.close()
recv_end, send_end = Pipe(duplex=False)
p = Process(target=foo, args=(send_end, ))
p.start()
start_time = time.time()
recv_end.recv() # Try to comment and you will see that it waits for being received
p.join()
print('--- %s seconds ---' % (time.time()-start_time))
谢谢你的回答!你能详细解释一下为什么创建这样的流程会让它工作吗?谢谢你的回答!你能详细说明一下为什么创建这样的流程会让它工作吗?