Pytorch 找不到有效的cuDNN算法来运行卷积
我在尝试运行前馈torch.nn.Conv2d时收到此消息,得到以下stacktrace:Pytorch 找不到有效的cuDNN算法来运行卷积,pytorch,Pytorch,我在尝试运行前馈torch.nn.Conv2d时收到此消息,得到以下stacktrace: --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-26-04bd4a00565d> in <module&
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-26-04bd4a00565d> in <module>
3
4 # call training function
----> 5 losses = train(D, G, n_epochs=n_epochs)
<ipython-input-24-b539315e0aa0> in train(D, G, n_epochs, print_every)
46 real_images = real_images.cuda()
47
---> 48 D_real = D(real_images)
49 d_real_loss = real_loss(D_real, True) # smoothing label 1 => 0.9
50
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
<ipython-input-14-bf68e57c25ff> in forward(self, x)
48 """
49
---> 50 x = self.leaky_relu(self.conv1(x))
51 x = self.leaky_relu(self.conv2(x))
52 x = self.leaky_relu(self.conv3(x))
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
98 def forward(self, input):
99 for module in self:
--> 100 input = module(input)
101 return input
102
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
548 result = self._slow_forward(*input, **kwargs)
549 else:
--> 550 result = self.forward(*input, **kwargs)
551 for hook in self._forward_hooks.values():
552 hook_result = hook(self, input, result)
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
347
348 def forward(self, input):
--> 349 return self._conv_forward(input, self.weight)
350
351 class Conv3d(_ConvNd):
~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
344 _pair(0), self.dilation, self.groups)
345 return F.conv2d(input, weight, self.bias, self.stride,
--> 346 self.padding, self.dilation, self.groups)
347
348 def forward(self, input):
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution
我使用的是Python 3.7、Pytorch 1.5,GPU是Nvidia GeForce GTX 770,运行在Ubuntu 18.04.2上。我在任何地方都没有找到那个错误消息。它响了吗
提前非常感谢。问题是您正在使用torch.nn.Module进行前馈,但您正在返回功能模块
F.conv2d()
。将返回代码更改为nn.Conv2d()
这可能会帮助您更多-我在测试不同EC2节点机器的推理速度时遇到了这个错误。当我翻阅日志时,我发现:
(pid=20839) /home/ubuntu/src/skai-ml/venv/lib/python3.7/site-packages/torch/cuda/__init__.py:87: UserWarning:
(pid=20839) Found GPU0 GRID K520 which is of cuda capability 3.0.
(pid=20839) PyTorch no longer supports this GPU because it is too old.
(pid=20839) The minimum cuda capability that we support is 3.5.
经验教训:不要对PyTorch模型使用g2.XX
实例类型g3.XX
和p
系列工作正常。根据tensorflow对类似问题的回答,可能是因为达到了VRAM内存限制(从错误消息来看,这是非常不直观的)
对于我的PyTorch模型培训案例,减少批量有帮助。您可以试试这个,或者减小型号以减少VRAM的消耗。同样的事情也发生在我身上,这并不是一个非常直观的信息
(pid=20839) /home/ubuntu/src/skai-ml/venv/lib/python3.7/site-packages/torch/cuda/__init__.py:87: UserWarning:
(pid=20839) Found GPU0 GRID K520 which is of cuda capability 3.0.
(pid=20839) PyTorch no longer supports this GPU because it is too old.
(pid=20839) The minimum cuda capability that we support is 3.5.