在模块化python代码库中使用Dask LocalCluster()
我正在尝试使用Dask分布式系统,使用一台机器的所有内核并行运行代码 考虑一个示例python数据管道,其文件夹结构如下在模块化python代码库中使用Dask LocalCluster(),python,python-3.x,parallel-processing,dask,dask-distributed,Python,Python 3.x,Parallel Processing,Dask,Dask Distributed,我正在尝试使用Dask分布式系统,使用一台机器的所有内核并行运行代码 考虑一个示例python数据管道,其文件夹结构如下 sample_dask_program ├── main.py ├── parallel_process_1.py ├── parallel_process_2.py ├── process_1.py ├── process_2.py └── process_3.py main.py是入口点,它在管道顺序运行时执行 例如: parallel_process_1.py和par
sample_dask_program
├── main.py
├── parallel_process_1.py
├── parallel_process_2.py
├── process_1.py
├── process_2.py
└── process_3.py
main.py是入口点,它在管道顺序运行时执行
例如:
parallel_process_1.py和parallel_process_2.py是创建Client()并用于实现并行性的模块
with Client() as client:
# list to store futures after they are submitted
futures = []
for item in items:
future = client.submit(
...
)
futures.append(future)
results = client.gather(futures)
process_1.py、process_2.py和process_3.py是执行简单计算的模块,不需要使用所有CPU内核并行运行
回溯:
File "/sm/src/calculation/parallel.py", line 140, in convert_qty_to_float
results = client.gather(futures)
File "/home/iouser/.local/lib/python3.7/site-packages/distributed/client.py", line 1894, in gather
asynchronous=asynchronous,
File "/home/iouser/.local/lib/python3.7/site-packages/distributed/client.py", line 778, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "/home/iouser/.local/lib/python3.7/site-packages/distributed/utils.py", line 348, in sync
raise exc.with_traceback(tb)
File "/home/iouser/.local/lib/python3.7/site-packages/distributed/utils.py", line 332, in f
result[0] = yield future
File "/home/iouser/.local/lib/python3.7/site-packages/tornado/gen.py", line 735, in run
value = future.result()
concurrent.futures._base.CancelledError
这是工作人员引发的错误:
distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:33901 -> tcp://127.0.0.1:38821
Traceback (most recent call last):
File "/home/iouser/.local/lib/python3.7/site-packages/distributed/comm/tcp.py", line 248, in write
future = stream.write(frame)
File "/home/iouser/.local/lib/python3.7/site-packages/tornado/iostream.py", line 546, in write
self._check_closed()
File "/home/iouser/.local/lib/python3.7/site-packages/tornado/iostream.py", line 1035, in _check_closed
raise StreamClosedError(real_error=self.error)
tornado.iostream.StreamClosedError: Stream is closed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/iouser/.local/lib/python3.7/site-packages/distributed/worker.py", line 1248, in get_data
compressed = await comm.write(msg, serializers=serializers)
File "/home/iouser/.local/lib/python3.7/site-packages/distributed/comm/tcp.py", line 255, in write
convert_stream_closed_error(self, e)
File "/home/iouser/.local/lib/python3.7/site-packages/distributed/comm/tcp.py", line 121, in convert_stream_closed_error
raise CommClosedError("in %s: %s: %s" % (obj, exc.__class__.__name__, exc))
distributed.comm.core.CommClosedError: in <closed TCP>: BrokenPipeError: [Errno 32] Broken pipe
但是,仅使用线程创建LocalCluster会将执行速度降低2-3倍
那么,找到适合该程序的适当数量的进程/线程是问题的解决方案吗?更常见的做法是创建一个Dask客户机,然后在其上运行许多工作负载
with Client() as client:
stage_one(client)
stage_two(client)
话虽如此,你所做的应该是好的。如果你能用最少的例子重现错误,那将很有用(但没有期望)。可能会有帮助。谢谢@mdurant,这个页面帮助我得到了。我正在做的计算涉及NumPy/熊猫。当进程数相对较高时,似乎会出现错误。
with Client() as client:
stage_one(client)
stage_two(client)