Python Pytorch CUDA错误：配置参数无效

我最近在损失函数中添加了一个新组件。运行新代码可以在CPU上运行，但当我在GPU上运行它时，会出现以下错误，这显然与向后传递有关：

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-56dcbddd5230> in <module>
     20 recall = Recall(N_RECALL_CAND, K)
     21 #run the model
---> 22 train_loss, val_loss = fit(triplet_train_loader, triplet_test_loader, model, loss_fn, optimizer, scheduler, N_EPOCHS, cuda, LOG_INT)
     23 #measure recall

~/thesis/trainer.py in fit(train_loader, val_loader, model, loss_fn, optimizer, scheduler, n_epochs, cuda, log_interval, metrics, start_epoch)
     24         scheduler.step()
     25         # Train stage
---> 26         train_loss, metrics, writer_train_index = train_epoch(train_loader, model, loss_fn, optimizer, cuda, log_interval, metrics, writer, writer_train_index)
     27 
     28         message = 'Epoch: {}/{}. Train set: Average loss: {:.4f}'.format(epoch + 1, n_epochs, train_loss)

~/thesis/trainer.py in train_epoch(train_loader, model, loss_fn, optimizer, cuda, log_interval, metrics, writer, writer_train_index)
     80         losses.append(loss.item())
     81         total_loss += loss.item()
---> 82         loss.backward()
     83         optimizer.step()
     84 

/opt/anaconda3/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    116                 products. Defaults to ``False``.
    117         """
--> 118         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    119 
    120     def register_hook(self, hook):

/opt/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94 
     95 

RuntimeError: CUDA error: invalid configuration argument

批次大小为1时，在第一次向后传递时发生错误。来自的答案表明这与缺乏记忆有关，但我的模型并不是特别大：

TripletNet(
  (embedding_net): EmbeddingNet(
    (anchor_net): AnchorNet(anchors torch.Size([128, 192]), biases torch.Size([128]))
    (embedding): Sequential(
      (0): AnchorNet(anchors torch.Size([128, 192]), biases torch.Size([128]))
      (1): Tanh()
    )
  )
)

我的GPU上的可用内存是8GB，比型号小得多，cdist结果的大小是128x128

我不知道如何开始调试这个。如果是因为跟踪中间状态而导致内存不足的情况，我该如何导航？感谢您的帮助

编辑：监视GPU内存使用情况表明，当GPU崩溃时，我的内存很低

---

根据pytroch论坛，升级到pytroch 1.5.0应该可以解决这个问题

你找到原因了吗？@dashesy请看下面我的答案

TripletNet(
  (embedding_net): EmbeddingNet(
    (anchor_net): AnchorNet(anchors torch.Size([128, 192]), biases torch.Size([128]))
    (embedding): Sequential(
      (0): AnchorNet(anchors torch.Size([128, 192]), biases torch.Size([128]))
      (1): Tanh()
    )
  )
)