Tensorflow 为什么在CPU而不是GPU上运行恢复检查点?
我正在使用tensorpack对GAN进行培训,培训结束后,以下是日志文件:Tensorflow 为什么在CPU而不是GPU上运行恢复检查点?,tensorflow,Tensorflow,我正在使用tensorpack对GAN进行培训,培训结束后,以下是日志文件: 2019-04-19 10:14:19.311373: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2019-04-19 10:14:19
2019-04-19 10:14:19.311373: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-04-19 10:14:19.374343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-04-19 10:14:19.374547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 1060 3GB major: 6 minor: 1 memoryClockRate(GHz): 1.759
pciBusID: 0000:01:00.0
totalMemory: 2.94GiB freeMemory: 2.17GiB
2019-04-19 10:14:19.374562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1060 3GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
[0419 10:14:20 @base.py:211] Initializing the session ...
[0419 10:14:20 @base.py:218] Graph Finalized.
[0419 10:14:20 @concurrency.py:37] Starting EnqueueThread QueueInput/input_queue ...
[0419 10:14:20 @base.py:250] Start Epoch 1
...
[0419 10:25:45 @base.py:250] Start Epoch 5
100%|#######################################|10000/10000[02:51<00:00,58.38it/s]
[0419 10:28:36 @base.py:260] Epoch 5 (global_step 50000) finished, time:2 minutes 51 seconds.
正如您所见,培训是在GPU上运行的,只需几分钟即可完成一个历元。
但训练结束后,检查站将恢复。但我发现它是在CPU而不是GPU上恢复的
这是日志文件和nvidia smi
2019-04-19 10:28:55.169308: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-04-19 10:28:55.236758: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-04-19 10:28:55.236987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 1060 3GB major: 6 minor: 1 memoryClockRate(GHz): 1.759
pciBusID: 0000:01:00.0
totalMemory: 2.94GiB freeMemory: 2.20GiB
2019-04-19 10:28:55.237005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1060 3GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
7%|##7 |700/10000[14:36<3:14:43, 0.80it/s][0419 10:28:55 @sessinit.py:117] Restoring checkpoint from train_log/TGAN_synthesizer:ISOT-1/model-50000 ...
16%|#####9 |1574/10000[31:59<2:51:25, 0.82it/s] 16%|######1 |1606/10000[32:37<2:47:25, 0.84it/s]
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 106... Off | 00000000:01:00.0 On | N/A |
| 41% 42C P8 7W / 120W | 665MiB / 3010MiB | 1% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1371 G /usr/lib/xorg/Xorg 313MiB |
| 0 2293 G compiz 196MiB |
| 0 2823 G ...uest-channel-token=17545882067829269512 97MiB |
| 0 7632 G ...-token=7C806614AA650E661A2E8895D83D4B4E 41MiB |
| 0 7824 G /opt/teamviewer/tv_bin/TeamViewer 13MiB |
+-----------------------------------------------------------------------------+
2019-04-19 10:28:55.169308:I tensorflow/core/platform/cpu_feature_guard.cc:137]您的cpu支持未编译此tensorflow二进制文件以使用的指令:SSE4.1 SSE4.2 AVX AVX2 FMA
2019-04-19 10:28:55.236758:I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895]从SysFS读取的成功NUMA节点的值为负值(-1),但必须至少有一个NUMA节点,因此返回NUMA节点零
2019-04-19 10:28:55.236987:I tensorflow/core/common_运行时/gpu/gpu_设备。cc:1105]找到了具有以下属性的设备0:
名称:GeForce GTX 1060 3GB大调:6小调:1内存锁定速率(GHz):1.759
pciBusID:0000:01:00.0
总内存:2.94GiB自由内存:2.20GiB
2019-04-19 10:28:55.237005:I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195]创建tensorflow设备(/device:gpu:0)->(设备:0,名称:GeForce GTX 1060 3GB,pci总线id:0000:01:00.0,计算能力:6.1)
7%|#| 7 | 700/10000[14:36包括您的代码片段以进行详细分析。嗯,这是一个很好的问题,但我不知道为什么
2019-04-19 10:28:55.169308: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-04-19 10:28:55.236758: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:895] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-04-19 10:28:55.236987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 1060 3GB major: 6 minor: 1 memoryClockRate(GHz): 1.759
pciBusID: 0000:01:00.0
totalMemory: 2.94GiB freeMemory: 2.20GiB
2019-04-19 10:28:55.237005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1060 3GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
7%|##7 |700/10000[14:36<3:14:43, 0.80it/s][0419 10:28:55 @sessinit.py:117] Restoring checkpoint from train_log/TGAN_synthesizer:ISOT-1/model-50000 ...
16%|#####9 |1574/10000[31:59<2:51:25, 0.82it/s] 16%|######1 |1606/10000[32:37<2:47:25, 0.84it/s]
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 106... Off | 00000000:01:00.0 On | N/A |
| 41% 42C P8 7W / 120W | 665MiB / 3010MiB | 1% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1371 G /usr/lib/xorg/Xorg 313MiB |
| 0 2293 G compiz 196MiB |
| 0 2823 G ...uest-channel-token=17545882067829269512 97MiB |
| 0 7632 G ...-token=7C806614AA650E661A2E8895D83D4B4E 41MiB |
| 0 7824 G /opt/teamviewer/tv_bin/TeamViewer 13MiB |
+-----------------------------------------------------------------------------+