Python JupyterHub单用户无法使用SystemsPawner使用tensorflow gpu支持

（这是到、和的交叉过帐）

我有一个使用

SystemdSpawner

的专用JupyterHub设置，在这里我尝试使用gpu支持运行tensorflow

我遵循tensorflow，或者在AWS EC2 g4实例上使用NDVIDIA尝试了已经准备好的AWS AMI（深度学习基础AMI（Ubuntu18.04）21.0版）

在这两种设置中，我都能够在（I）Python3.6shell中使用tensorflow和gpu支持

>>> import tensorflow as tf
>>> tf.config.list_physical_devices('GPU')
2020-02-12 10:57:13.670937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-02-12 10:57:13.698230: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-12 10:57:13.699066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:1e.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.73GiB deviceMemoryBandwidth: 298.08GiB/s
2020-02-12 10:57:13.699286: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-02-12 10:57:13.700918: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-02-12 10:57:13.702512: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-02-12 10:57:13.702814: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-02-12 10:57:13.704561: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-02-12 10:57:13.705586: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-02-12 10:57:13.709171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-02-12 10:57:13.709278: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-12 10:57:13.710120: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-12 10:57:13.710893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

（关于NUMA节点的一些警告，但找到了gpu）

同时使用

nvidia smi

和

deviceQuery

显示gpu：

$ nvidia-smi
Wed Feb 12 10:39:44 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   33C    P8     9W /  70W |      0MiB / 15079MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

及

而且

我怀疑是某种“沙盒”，缺少环境变量等，因为在单用户环境中找不到gpu驱动程序，因此tensorflow gpu支持无法工作

---

有什么想法吗？这可能是一个小的配置调整，或者是由于架构根本无法解决；）

在

jupyterhub\u config.py

中设置

c.SystemdSpawner.isolation\u devices=False

以下是一段摘录：

将此设置为true将为每个用户提供单独的私有/dev。这会阻止用户直接访问硬件设备，这可能是安全问题的潜在来源/dev/null、/dev/zero、/dev/random和ttyp伪设备将已经安装，因此大多数用户在启用此功能时不会看到任何更改

c.SystemdSpawner.isolate_devices = True

这需要systemd版本>227。如果在早期版本中启用此功能，则生成将失败

Nvidia使用设备（即

/dev

中的文件）。请参阅。其中应该有名为

/dev/nvidia*

的文件。使用SystemdSpawner隔离设备将阻止访问这些Nvidia设备

---

是否仍有办法隔离设备并启用GPU支持

我不确定。。。但我可以提供指向文档的指针。设置

c.SystemdSpawner.isolate_devices=True

在最终的

systemd运行调用（）中设置
PrivateDevices=yes
。有关
PrivateDevices
选项的更多信息，请参阅

您可以保持 IsAsdiaTys=真/<代码>，然后显式安装英伟达设备。虽然我不知道怎么做…
您是否使用SystemsPawner的默认配置？这是在黑暗中拍摄的，但您可以尝试在SystemdSpawner配置中设置
c.SystemdSpawner.isolate_devices=False
。默认情况下应该是假的…我很惊讶！这就解决了我的问题！我将它设置为
True
，因为它在我看来很聪明。我从来没有在使用CPU时遇到过问题…直到现在。现在原因已经知道了。是否仍有办法隔离设备并启用GPU支持？启用隔离似乎更安全。因为我现在的用例是一个完全不需要批评的临时培训设置，我不在乎，但它可能在将来会有意义。
$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

$ /usr/local/cuda/extras/demo_suite/deviceQuery
cuda/extras/demo_suite/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 38
-> no CUDA-capable device is detected
Result = FAIL

c.SystemdSpawner.isolate_devices = True