Python Theano:初始化设备gpu失败!原因=CNMEM\u状态\u内存不足
我正在运行Python Theano:初始化设备gpu失败!原因=CNMEM\u状态\u内存不足,python,python-2.7,memory,gpu,theano,Python,Python 2.7,Memory,Gpu,Theano,我正在运行Keras的kaggle\u otto\u nn.py,后端是theano。当我设置cnmem=1时,出现以下错误: cliu@cliu-ubuntu:keras-examples$ THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32,lib.cnmem=1 python kaggle_otto_nn.py Using Theano backend. ERROR (theano.sandbox.cuda): ERROR: Not u
Keras的kaggle\u otto\u nn.py
,后端是theano
。当我设置cnmem=1
时,出现以下错误:
cliu@cliu-ubuntu:keras-examples$ THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32,lib.cnmem=1 python kaggle_otto_nn.py
Using Theano backend.
ERROR (theano.sandbox.cuda): ERROR: Not using GPU. Initialisation of device gpu failed:
initCnmem: cnmemInit call failed! Reason=CNMEM_STATUS_OUT_OF_MEMORY. numdev=1
/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/tensor/signal/downsample.py:6: UserWarning: downsample module has been moved to the theano.tensor.signal.pool module.
"downsample module has been moved to the theano.tensor.signal.pool module.")
Traceback (most recent call last):
File "kaggle_otto_nn.py", line 28, in <module>
from keras.models import Sequential
File "build/bdist.linux-x86_64/egg/keras/models.py", line 15, in <module>
File "build/bdist.linux-x86_64/egg/keras/backend/__init__.py", line 46, in <module>
File "build/bdist.linux-x86_64/egg/keras/backend/theano_backend.py", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/__init__.py", line 111, in <module>
theano.sandbox.cuda.tests.test_driver.test_nvidia_driver1()
File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/sandbox/cuda/tests/test_driver.py", line 38, in test_nvidia_driver1
if not numpy.allclose(f(), a.sum()):
File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/compile/function_module.py", line 871, in __call__
storage_map=getattr(self.fn, 'storage_map', None))
File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/gof/link.py", line 314, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/theano/compile/function_module.py", line 859, in __call__
outputs = self.fn()
RuntimeError: Cuda error: kernel_reduce_ccontig_node_97496c4d3cf9a06dc4082cc141f918d2_0: out of memory. (grid: 1 x 1; block: 256 x 1 x 1)
Apply node that caused the error: GpuCAReduce{add}{1}(<CudaNdarrayType(float32, vector)>)
Toposort index: 0
Inputs types: [CudaNdarrayType(float32, vector)]
Inputs shapes: [(10000,)]
Inputs strides: [(1,)]
Inputs values: ['not shown']
Outputs clients: [[HostFromGpu(GpuCAReduce{add}{1}.0)]]
HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
cliu@cliu-ubuntu:keras示例$THEANO\u FLAGS=mode=FAST\u RUN,device=gpu,floatX=float32,lib.cnmem=1 python kaggle\u otto\u nn.py
使用Theano后端。
错误(theano.sandbox.cuda):错误:未使用GPU。初始化设备gpu失败:
initCnmem:cnmemInit调用失败!原因=CNMEM\u状态\u内存不足。numdev=1
/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/Theano/tensor/signal/downsample.py:6:UserWarning:downsample模块已移动到Theano.tensor.signal.pool模块。
“下采样模块已移动到theano.tensor.signal.pool模块。”)
回溯(最近一次呼叫最后一次):
文件“kaggle_otto_nn.py”,第28行,在
从keras.models导入顺序
文件“build/bdist.linux-x86_64/egg/keras/models.py”,第15行,在
文件“build/bdist.linux-x86_64/egg/keras/backend/_init__.py”,第46行,在
文件“build/bdist.linux-x86_64/egg/keras/backend/theano_backend.py”,第1行,在
文件“/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/Theano/__init__;u.py”,第111行,在
theano.sandbox.cuda.tests.test\u driver.test\u nvidia\u driver1()
文件“/usr/local/lib/python2.7/dist packages/Theano-0.8.0rc1-py2.7.egg/Theano/sandbox/cuda/tests/test_driver.py”,第38行,位于test_nvidia_driver 1中
如果不是numpy.allclose(f(),a.sum()):
文件“/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/Theano/compile/function_module.py”,第871行,在调用中__
存储映射=getattr(self.fn,“存储映射”,无))
文件“/usr/local/lib/python2.7/dist-packages/Theano-0.8.0rc1-py2.7.egg/Theano/gof/link.py”,第314行,带
重新释放(exc_类型、exc_值、exc_跟踪)
文件“/usr/local/lib/python2.7/dist packages/Theano-0.8.0rc1-py2.7.egg/Theano/compile/function_module.py”,第859行,在调用中__
输出=self.fn()
运行时错误:Cuda错误:内核\u减少\u计数\u节点\u 97496c4d3cf9a06dc4082cc141f918d2\u 0:内存不足。(网格:1 x 1;块:256 x 1 x 1)
导致错误的应用节点:GpuCAReduce{add}{1}()
拓扑排序索引:0
输入类型:[cudandaraytype(float32,vector)]
输入形状:[(10000,)]
输入跨步:[(1,)]
输入值:[“未显示”]
输出客户端:[[HostFromGpu(GpuCAReduce{add}{1}.0)]]
提示:在禁用大多数Theano优化的情况下重新运行可能会返回此节点创建的时间。这可以通过设置Theano标志“optimizer=fast\u compile”来完成。如果不起作用,可以使用“optimizer=None”禁用无优化。
提示:对此应用节点的调试打印和存储映射示意图使用Theano标志“exception\u verbosity=high”。
似乎我无法将cnmem
设置为非常大的值(大约>0.9),因为这可能会导致GPU内存溢出。当我设置cnmem=0.9时,它工作正常。据报道,
表示内存池的开始大小(MB或总GPU内存的百分比)
及
这可能会导致内存碎片。因此,如果在使用cnmem时出现内存错误,请尝试在开始时分配更多内存或禁用它。如果尝试此操作,请在以下位置报告结果:reftheano dev
但是如果我有内存错误,为什么我要在开始时分配更多内存?在我的情况下,在开始时分配更多内存似乎是导致错误的原因 这是问题解决的基础
如前所述,cnmem
实际上只允许分配为浮点
0:未启用
0
因此,如果cnmem=1.0
而不是cnmem=1
,它将起作用