Python Jupyter：内核似乎已经死了。它将自动重新启动。（与Keras有关）

我正在尝试训练Resnet50，但无论我做什么都失败了，因为Jupyter笔记本的内核正在消亡，内核似乎已经消亡。它将在训练纪元1/100开始时自动重新启动。我有GeForce GTX 1060 Ti，当我在持续1秒的训练中使用nvidia smi时，虽然我只看到与过去相比分配了80MB的内存，但是内核死亡，好像它尝试了但失败了

以下是要求：

pandas==0.25.1
numpy==1.17.2
opencv-python==4.1.1.26
scikit-image==0.15.0
scikit-learn==0.21.3
tensorflow-gpu==1.14.0
Keras==2.2.5
matplotlib==3.1.1
Pillow==6.1.0
albumentations==0.3.2
tqdm==4.35.0
jupyter

我很满意。以下是我如何设置培训课程的：

config = tf.ConfigProto()
config.gpu_options.allow_growth = False
config.gpu_options.per_process_gpu_memory_fraction = 0.9
sess = tf.Session(config=config) 
keras.backend.set_session(sess)

keras.__version__
os.environ["CUDA_VISIBLE_DEVICES"] = '0' #yes, this is the ID of my GPU.

# create the FCN model
model_mobilenet = ResNet50(input_shape=(1024, 1024, 3), include_top=False) # use the Resnet
model_x8_output = Conv2D(128, (1, 1), activation='relu')(model_mobilenet.layers[-95].output)
model_x8_output = UpSampling2D(size=(8, 8))(model_x8_output)
model_x8_output = Conv2D(3, (3, 3), padding='same', activation='sigmoid')(model_x8_output)
MODEL_x8 = Model(inputs=model_mobilenet.input, outputs=model_x8_output)

MODEL_x8.compile(loss='binary_crossentropy', optimizer=Adam(lr=1e-3), metrics=[jaccard_distance])

MODEL_x8.fit_generator(train_generator, steps_per_epoch=300, epochs=100, verbose=1, validation_data=val_generator, validation_steps=10)

Epoch 1/100
  1/300 [..............................] - ETA: 1:01:59 - loss: 0.7193 - jaccard_distance: 0.1125

我试过设置：

config.gpu_options.allow_growth为True。 config.gpu_options.per_process_gpu_memory_分数设置为任何其他任意值，如0.1 注释：os.environ[CUDA\u VISIBLE\u DEVICES]=0 他们都没有工作。我赞赏建设性的回答

提前谢谢

编辑：我现在尝试将其作为脚本运行，而不是作为笔记本运行，当Tensorflow会话行出现时，终端抛出以下命令：

2020-01-28 13:44:55.756819: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/username/ros_ws/devel/lib:/opt/ros/melodic/lib
2020-01-28 13:44:55.757047: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/username/ros_ws/devel/lib:/opt/ros/melodic/lib
2020-01-28 13:44:55.757313: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/username/ros_ws/devel/lib:/opt/ros/melodic/lib
2020-01-28 13:44:55.757526: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/username/ros_ws/devel/lib:/opt/ros/melodic/lib
2020-01-28 13:44:55.757736: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/username/ros_ws/devel/lib:/opt/ros/melodic/lib
2020-01-28 13:44:55.757940: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/username/ros_ws/devel/lib:/opt/ros/melodic/lib
2020-01-28 13:44:55.808416: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-01-28 13:44:55.808444: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1663] Cannot dlopen some GPU libraries. Skipping registering GPU devices...

---

这很奇怪，因为我没有CUDA10，而不是9.0，所以这甚至不应该被问到。我的Tensorflow版本错了吗？

很可能是因为没有足够的内存来存储数据/模型。您的输入图像大小也是1024x1024。我建议您尝试使用256甚至128这样的小图像大小进行训练，看看它是否有效

还有，你的GPU被TF检测到了吗？

好的，知道了

问题是我的tensorflow=gpu版本1.14与CUDA版本9.0不兼容。我必须安装一个低于1.13的版本。但这不是唯一的陷阱。我的CuDNN版本705也有问题，我不得不将Tensorflow gpu一直降低到1.9.0

---

现在一切正常。

如果您愿意，您可以保留TF gpu 1.14，只需将CUDA版本更新为10.0即可