Python tensorflow对象检测API：训练无声失败

我正在使用Tensorflow的对象检测API和我的自定义数据集。 我目前正在培训“ssd_mobilenet_v1_coco”

每次我尝试时，训练都会开始，但训练会无声地、随机地停止，不会出现错误消息。 （使用下面的命令，命令提示符会在一定程度上显示步骤数。） 似乎GPU（CUDA）也停止了

我已经尝试过改变批量大小（“64”显示最佳分数）和“ssd\u mobilenet\u v2\u coco”

这个参数（比如“sample\u 1\u of\n\u eval\u examples=1”）还是GPU问题

操作系统：windows10 Tensorflow版本：1.15 Python:3.6 CPU:i9-9900K GPU:NVIDIA GeForce RTX 2080

我使用的命令

python object_detection/model_main.py --pipeline_config_path="C:\Users\MYPATH\models\model\ssd_mobilenet_v1_coco.config" --model_dir="C:\Users\MYPATH\models\model" --num_train_steps=2000 --sample_1_of_n_eval_examples=1 --alsologtostderr

信息

INFO:tensorflow:Done calling model_fn.
I0106 17:49:29.545947 15104 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I0106 17:49:29.545947 15104 basic_session_run_hooks.py:541] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
I0106 17:49:32.188141 15104 monitored_session.py:240] Graph was finalized.
2020-01-06 17:49:32.200573: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-01-06 17:49:32.205758: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-01-06 17:49:32.229166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.71
pciBusID: 0000:01:00.0
2020-01-06 17:49:32.232539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
2020-01-06 17:49:32.236216: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2020-01-06 17:49:32.239801: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_100.dll
2020-01-06 17:49:32.242368: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_100.dll
2020-01-06 17:49:32.246706: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_100.dll
2020-01-06 17:49:32.250779: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_100.dll
2020-01-06 17:49:32.258807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-01-06 17:49:32.261581: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-01-06 17:49:32.700705: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-06 17:49:32.703645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0
2020-01-06 17:49:32.705343: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N
2020-01-06 17:49:32.707345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6271 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
INFO:tensorflow:Running local_init_op.
I0106 17:49:35.342885 15104 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0106 17:49:35.702204 15104 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into C:\Users\mypath\models\model\model.ckpt.
I0106 17:49:42.856755 15104 basic_session_run_hooks.py:606] Saving checkpoints for 0 into C:\Users\mypath\models\model\model.ckpt.
2020-01-06 17:49:51.489601: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-01-06 17:49:52.410981: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Internal: Invoking ptxas not supported on Windows
Relying on driver to perform ptx compilation. This message will be only logged once.
2020-01-06 17:49:52.445252: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
INFO:tensorflow:loss = 33.134163, step = 0
I0106 17:49:55.059146 15104 basic_session_run_hooks.py:262] loss = 33.134163, step = 0
INFO:tensorflow:global_step/sec: 2.58675
I0106 17:50:33.717694 15104 basic_session_run_hooks.py:692] global_step/sec: 2.58675
INFO:tensorflow:loss = 9.563588, step = 100 (38.659 sec)
I0106 17:50:33.717694 15104 basic_session_run_hooks.py:260] loss = 9.563588, step = 100 (38.659 sec)