Fatal编程技术网
  • python/
  • python多处理scipy.stats.lognorm.fit

python多处理scipy.stats.lognorm.fit

python

python多处理scipy.stats.lognorm.fit,python,scipy,multiprocessing,pathos,Python,Scipy,Multiprocessing,Pathos,我想用scipy来适应许多发行版，并希望为此使用某种多处理。大概是这样的： import scipy.stats as ss from pathos.multiprocessing import ProcessingPool from multiprocessing import Pool mp = Pool() pp = ProcessingPool() l = [0,1,2,3,4,6,7,8,9] print map(ss.lognorm.fit,l) #method 0 print

我想用scipy来适应许多发行版，并希望为此使用某种多处理。大概是这样的：

import scipy.stats as ss
from pathos.multiprocessing import ProcessingPool
from multiprocessing import Pool

mp = Pool()
pp = ProcessingPool()

l = [0,1,2,3,4,6,7,8,9]
print map(ss.lognorm.fit,l) #method 0
print mp.map(ss.lognorm.fit,l) #method 1
print pp.map(ss.lognorm.fit,l) #method 2
方法0当然不是多处理，但可以工作。 方法1和2都返回长回溯。有人对此有解决办法吗

方法1错误：

Process PoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
Process PoolWorker-2:
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
Process PoolWorker-4:
    return recv()
Traceback (most recent call last):
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-5:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
Process PoolWorker-9:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 102, in worker
    task = get()
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 376, in get
    return recv()
AttributeError: ("'lognorm_gen' object has no attribute '_parse_args'", <built-in function getattr>, (<scipy.stats._continuous_distns.lognorm_gen object at 0x7fb15349ddd0>, '_parse_args'))
方法1不起作用，因为无法使用
pickle
pickle绑定实例方法。方法2不起作用，因为
scipy.stats
正在做一些“棘手”的事情，
dill
和
pathos
作者（我）在没有事先调查的情况下不太清楚它是什么

您可以看到，问题不在于
scipy.stats
使用绑定方法（对于
dill
或
pathos
），而是它在执行一些重命名魔术……这就是为什么当您从
pathos
调用中查看回溯时，您会看到
\u locate\u函数失败的原因（它失败并发现
None
）…这就是方法2不起作用的原因


>>> import scipy.stats as ss
>>>        
>>> ss.lognorm
<scipy.stats._continuous_distns.lognorm_gen object at 0x10932d6d0>



你确定你的问题不是来自-使用绑定实例方法，而绑定实例方法是无法pickle的吗？我不这么认为，或者至少，对于方法#2来说不是。至少花一个小时来尝试安装pathos.multiprocessing的全部目的是因为它不使用pickle来序列化，而且据推测，没有类似的问题。酷。抱歉进行粗略的安装--我正在努力使其不那么粗糙。

>>> import scipy.stats as ss
>>>        
>>> ss.lognorm
<scipy.stats._continuous_distns.lognorm_gen object at 0x10932d6d0>



>>> import pathos.multiprocessing as mp
>>> p = mp.ProcessingPool()
>>>        
>>> def doit(x):
...   return ss.lognorm.fit(x)
... 
>>> p.map(doit, range(5))
[(1.0, 0.0, 1.0), (1.0, 1.0, 1.0), (1.0, 2.0, 1.0), (1.0, 3.0, 1.0), (1.0, 4.0, 1.0)]