Python 创建对象的副本,而不是在新的多处理过程中重新初始化
这段代码显示了我尝试执行的操作的结构Python 创建对象的副本,而不是在新的多处理过程中重新初始化,python,copy,multiprocessing,deep-copy,Python,Copy,Multiprocessing,Deep Copy,这段代码显示了我尝试执行的操作的结构 import multiprocessing from foo import really_expensive_to_compute_object ## Create a really complicated object that is *hard* to initialise. T = really_expensive_to_compute_object(10) def f(x): return T.cheap_calculation(x)
import multiprocessing
from foo import really_expensive_to_compute_object
## Create a really complicated object that is *hard* to initialise.
T = really_expensive_to_compute_object(10)
def f(x):
return T.cheap_calculation(x)
P = multiprocessing.Pool(processes=64)
results = P.map(f, range(1000000))
print results
问题是,每个进程开始时都要花费大量时间重新计算T,而不是使用原来计算过一次的T。有没有办法防止这种情况?T有一个快速(深度)复制方法,所以我可以让Python使用它而不是重新计算吗?为什么不让
f
使用T
参数而不是引用全局,然后自己进行复制
import multiprocessing, copy
from foo import really_expensive_to_compute_object
## Create a really complicated object that is *hard* to initialise.
T = really_expensive_to_compute_object(10)
def f(t, x):
return t.cheap_calculation(x)
P = multiprocessing.Pool(processes=64)
results = P.map(f, (copy.deepcopy(T) for _ in range(1000000)), range(1000000))
print results
文件
显式地将资源传递给子进程
因此,您的代码可以重写为以下内容:
import multiprocessing
import time
import functools
class really_expensive_to_compute_object(object):
def __init__(self, arg):
print 'expensive creation'
time.sleep(3)
def cheap_calculation(self, x):
return x * 2
def f(T, x):
return T.cheap_calculation(x)
if __name__ == '__main__':
## Create a really complicated object that is *hard* to initialise.
T = really_expensive_to_compute_object(10)
## helper, to pass expensive object to function
f_helper = functools.partial(f, T)
# i've reduced count for tests
P = multiprocessing.Pool(processes=4)
results = P.map(f_helper, range(100))
print results