解决PyTorch报错“cublas runtime error : library not initialized”_runtimeerror: library cudart is not initialized

运行一段深度学习的training代码时遇到如下错误：

  File "train_mdnet.py", line 69, in train_mdnet
    neg_regions = neg_regions.cuda()
  File "/home/dodge/anaconda3/envs/py27_for_pytorch020/lib/python2.7/site-packages/torch/autograd/variable.py", line 279, in cuda
    return CudaTransfer.apply(self, device_id, async)
  File "/home/dodge/anaconda3/envs/py27_for_pytorch020/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 151, in forward
    return i.cuda(async=async)
  File "/home/dodge/anaconda3/envs/py27_for_pytorch020/lib/python2.7/site-packages/torch/_utils.py", line 66, in _cuda
    return new_type(self.size()).copy_(self, async)
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1511304568725/work/torch/lib/THC/generic/THCStorage.cu:66
1
2
3
4
5
6
7
8
9

经查阅PyTorch社区，发现该问题主要可能由两方面的原因引起：

1. GPU显存不足
2. GPU缓存没有清理干净

检查主机实际运行情况，发现还有一个CUDA程序处于debug状态，应该是显存不够了所致，停止该程序，问题解决。