Python 3.x multiprocessing.Pool.map函数不使用';在全局命名空间中看不到变量
此玩具脚本失败:Python 3.x multiprocessing.Pool.map函数不使用';在全局命名空间中看不到变量,python-3.x,multiprocessing,global,Python 3.x,Multiprocessing,Global,此玩具脚本失败: import numpy as np import multiprocessing as mp def myFunc1(x): return x+c if __name__ == '__main__': c = np.array([[1,1],[1,1]]) b = np.array([[[1,2],[3,4]],[[5,6],[7,8]]]) pool = mp.Pool(processes=1) result = pool.map
import numpy as np
import multiprocessing as mp
def myFunc1(x):
return x+c
if __name__ == '__main__':
c = np.array([[1,1],[1,1]])
b = np.array([[[1,2],[3,4]],[[5,6],[7,8]]])
pool = mp.Pool(processes=1)
result = pool.map(myFunc1,b)
for i in range(len(result)):
print(result[i])
错误地
RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\aldo.dottavio\anaconda3\lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:\Users\aldo.dottavio\anaconda3\lib\multiprocessing\pool.py", line 44, in mapstar
return list(map(*args))
File "C:\Users\aldo.dottavio\Projects\THEIAA\2020-02 parallel processing\untitled0.py", line 12, in myFunc1
return x+c
NameError: name 'c' is not defined
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\aldo.dottavio\Projects\THEIAA\2020-02 parallel processing\untitled0.py", line 19, in <module>
result = pool.map(myFunc1,b)
File "C:\Users\aldo.dottavio\anaconda3\lib\multiprocessing\pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "C:\Users\aldo.dottavio\anaconda3\lib\multiprocessing\pool.py", line 644, in get
raise self._value
NameError: name 'c' is not defined
结果是,在myFunc1中没有定义c。我认为这是由map造成的,并且通过使用map将myFunc1应用于数据,myFunc1将失去对全局名称空间的访问
类似的玩具脚本成功:
import numpy as np
def myFunc1(x):
return x+c
if __name__ == '__main__':
c = np.array([[1,1],[1,1]])
b = np.array([[[1,2],[3,4]],[[5,6],[7,8]]])
result = myFunc1(b)
for i in range(len(result)):
print(result[i])
并输出结果
[[2 3]
[4 5]]
[[6 7]
[8 9]]
正如所料
以上只是说明问题;在实际脚本中,函数更复杂,必须在数千个矩阵上迭代
我需要全局名称空间中的变量是函数使用的参数;它们不会被任何进程更改,并且对于所有函数调用都是相同的-它们在脚本中此时是只读的
如何从map中的函数访问全局变量?我不想复制参数-参数只有~1MB的数据,但我可以在iTable中包含15000个左右的元素。我使用starmap和itertools.product()解决了这个问题
在c周围使用itertools.product和[]可以将整个数组c作为一个元素iterable传递,而不是迭代NumPy数组的第一个索引
[[2 3]
[4 5]]
[[6 7]
[8 9]]
import numpy as np
import multiprocessing as mp
from itertools import product
def myFunc1(x,c):
return x+c
if __name__ == '__main__':
c = np.array([[1,1],[1,1]])
b = np.array([[[1,2],[3,4]],[[5,6],[7,8]]])
pool = mp.Pool(processes=1)
result = pool.starmap(myFunc1,product(b,[c]))
# result = myFunc1(b)
for i in range(len(result)):
print(result[i])