Python 使类中的方法对并发调用具有鲁棒性
我有以下实用程序类:Python 使类中的方法对并发调用具有鲁棒性,python,python-3.x,concurrency,thread-safety,Python,Python 3.x,Concurrency,Thread Safety,我有以下实用程序类: class RunningStatisticsVar: def __init__(self, ddof=0): self.mean = 0 self.var = 0 self.std = 0 self._n = 0 self._s = 0 self._ddof = ddof def update(self, value): self._n +=
class RunningStatisticsVar:
def __init__(self, ddof=0):
self.mean = 0
self.var = 0
self.std = 0
self._n = 0
self._s = 0
self._ddof = ddof
def update(self, value):
self._n += 1
old_mean = self.mean
self.mean += (value - old_mean) / self._n
self._s += (value - old_mean) * (value - self.mean)
self.var = self._s / (self._n - self._ddof) if self._n > self._ddof else 0
self.std = np.sqrt(self.var)
这将计算并存储(长)数字流的运行平均值和标准。它工作得很好,但是,因为我将类放在我的个人库中,所以我希望使它对并发执行具有健壮性。例如,我希望能够做到以下几点:
from joblib.parallel import Parallel, delayed
def execute_and_update(var):
a = do_stuff()
var.update(a)
b, c = do_more_stuff()
var.update(b)
var.update(c)
stat = RunningStatisticsVar()
Parallel()(delayed(execute_and_update)(stat) for _ in range(1000))
并使update
调用是线程安全的
谷歌搜索这一点给了我许多并发执行代码的方法,但我还没有找到让我的类安全地并发执行的方法。在Java、IIRC中,这可以通过原子方法/类来完成,但我认为Python没有
更新
在注释之后,我已更新了代码,但在尝试从并行调用我的方法时出现错误:
from joblib.parallel import Parallel, delayed
import numpy as np
from threading import Lock
class RunningStatisticsVar:
def __init__(self, ddof=0):
self.mean = 0
self.var = 0
self.std = 0
self._n = 0
self._s = 0
self._ddof = ddof
self._lock = Lock()
def update(self, value):
with self._lock:
self._n += 1
old_mean = self.mean
self.mean += (value - old_mean) / self._n
self._s += (value - old_mean) * (value - self.mean)
self.var = self._s / (self._n - self._ddof) if self._n > self._ddof else 0
self.std = np.sqrt(self.var)
samples = np.random.uniform(0, 100, [1000])
s1 = RunningStatisticsVar()
s2 = RunningStatisticsVar()
for i in samples:
s1.update(i)
Parallel(n_jobs=-1)(delayed(lambda x: s2.update(x))(i) for i in samples) #
print(s1.mean, s1.std)
print(s2.mean, s2.std)
试图运行上述代码时,在标有
的行中出现以下错误:
TypeError:无法pickle\u thread.lock对象
在输入def更新时使用(…do
.aquire
在休假时发布是的,应该可以很好地工作,谢谢你。我试图实现解决方案,但我得到了一个类型错误
。我用新的信息和代码更新了问题。我将行从多处理导入锁更改为,并保持其余不变,但是现在我得到了错误RuntimeError:Lock对象应该只通过继承在进程之间共享Read.Read