Dask 分布式SLURM群集保姆超时

我试图使用

dask.distributed.SLURMCluster

将批处理作业提交给超级计算集群上的SLURM作业调度器。作业都按预期提交，但在运行1分钟后引发错误：

asyncio.exceptions.TimeoutError:Nanny在60秒内启动失败

。我如何让保姆联系上

完整跟踪：

distributed.nanny - INFO -         Start Nanny at: 'tcp://206.76.203.125:38324'
distributed.dashboard.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
distributed.worker - INFO -       Start worker at: tcp://206.76.203.125:37609
distributed.worker - INFO -          Listening to: tcp://206.76.203.125:37609
distributed.worker - INFO -          dashboard at:       206.76.203.125:35505
distributed.worker - INFO - Waiting to connect to:  tcp://129.114.63.43:35489
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          8
distributed.worker - INFO -                Memory:                    2.00 GB
distributed.worker - INFO -       Local Directory: /home1/06729/tg860286/tests/dask-rsmas-presentation/dask-worker-space/worker-pu937jui
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Waiting to connect to:  tcp://129.114.63.43:35489
distributed.worker - INFO - Waiting to connect to:  tcp://129.114.63.43:35489
distributed.worker - INFO - Waiting to connect to:  tcp://129.114.63.43:35489
distributed.worker - INFO - Waiting to connect to:  tcp://129.114.63.43:35489
distributed.nanny - INFO - Closing Nanny at 'tcp://206.76.203.125:38324'
distributed.worker - INFO - Stopping worker at tcp://206.76.203.125:37609
distributed.worker - INFO - Closed worker has not yet started: None
distributed.dask_worker - INFO - End worker
Traceback (most recent call last):
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/distributed/node.py", line 173, in wait_for
    await asyncio.wait_for(future, timeout=timeout)
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/asyncio/tasks.py", line 490, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 440, in <module>
    go()
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 436, in go
    main()
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 422, in main
    loop.run_sync(run)
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/tornado/ioloop.py", line 532, in run_sync
    return future_cell[0].result()
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/distributed/cli/dask_worker.py", line 416, in run
    await asyncio.gather(*nannies)
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/asyncio/tasks.py", line 684, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/distributed/node.py", line 176, in wait_for
    raise TimeoutError(
asyncio.exceptions.TimeoutError: Nanny failed to start in 60 seconds```

distributed.nanny-INFO-启动保姆地点：'tcp://206.76.203.125:38324'
distributed.dashboard.proxy-INFO-要路由到workers diagnostics web服务器，请安装jupyter服务器代理：python-m pip安装jupyter服务器代理
distributed.worker-信息-在以下位置启动工作程序：tcp://206.76.203.125:37609
distributed.worker-信息-侦听：tcp://206.76.203.125:37609
distributed.worker-信息-仪表板电话：206.76.203.125:35505
distributed.worker-信息-正在等待连接到：tcp://129.114.63.43:35489
distributed.worker-信息--------------------------------------------------
distributed.worker-信息-线程：8
distributed.worker-信息-内存：2.00 GB
distributed.worker-INFO-本地目录：/home1/06729/tg860286/tests/dask rsmas presentation/dask worker space/worker-pu937jui
distributed.worker-信息--------------------------------------------------
distributed.worker-信息-正在等待连接到：tcp://129.114.63.43:35489
distributed.worker-信息-正在等待连接到：tcp://129.114.63.43:35489
distributed.worker-信息-正在等待连接到：tcp://129.114.63.43:35489
distributed.worker-信息-正在等待连接到：tcp://129.114.63.43:35489
distributed.nanny-信息-正在关闭“保姆”tcp://206.76.203.125:38324'
distributed.worker-信息-正在停止工作程序tcp://206.76.203.125:37609
distributed.worker-信息-已关闭的工作程序尚未启动：无
distributed.dask_worker-信息-终端worker
回溯（最近一次呼叫最后一次）：
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/distributed/node.py”，第173行，正在等待
等待异步IO。等待（未来，超时=超时）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/asyncio/tasks.py”，第490行，等待
引发异常
asyncio.exceptions.TimeoutError
在处理上述异常期间，发生了另一个异常：
回溯（最近一次呼叫最后一次）：
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/runpy.py”，第193行，位于主运行模块中
返回运行代码（代码、主全局、无、，
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/runpy.py”，第86行，在运行代码中
exec（代码、运行\全局）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/distributed/cli/dask_worker.py”，第440行，in
go（）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/distributed/cli/dask_worker.py”，第436行，在go中
main（）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/click/core.py”，第764行，在调用中__
返回self.main（*args，**kwargs）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site-packages/click/core.py”，第717行，在main中
rv=自调用（ctx）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/click/core.py”，第956行，在invoke中
返回ctx.invoke（self.callback，**ctx.params）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/click/core.py”，第555行，在invoke中
返回回调（*args，**kwargs）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/distributed/cli/dask_worker.py”，第422行，主目录
循环。运行\u同步（运行）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/tornado/ioloop.py”，第532行，运行同步
返回未来_单元格[0]。结果（）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/distributed/cli/dask_worker.py”，第416行，正在运行
等待asyncio.gather（*保姆）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/asyncio/tasks.py”，第684行，换行等待
返回（等待的收益率。uuu等待的收益率（）
文件“/home1/06729/tg860286/miniconda3/envs/daskbase/lib/python3.8/site packages/distributed/node.py”，第176行，正在等待
上升时间误差(
asyncio.exceptions.TimeoutError:Nanny在60秒内启动失败```

看起来您的工作人员无法连接到计划程序。我猜您需要指定一个网络接口。您应该询问系统管理员您应该使用哪个网络接口，然后使用

接口=

关键字指定

您可能还想通读，它提供了一个常见问题的案例研究