RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784 torch

-- Process 6 terminated with the following error:
Traceback (most recent call last):
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/detectron2/engine/launch.py", line 108, in _distributed_worker
    raise e
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/detectron2/engine/launch.py", line 103, in _distributed_worker
    timeout=timeout,
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
    barrier()
  File "/media/home/intern/anaconda3/envs/torch17/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:784, invalid usage, NCCL version 2.7.8

1
2
3
4
5
6
7
8
9
10
11
12
13
14

代码中要求用8张GPU运行，而机器只有两张卡。
所以把代码设置为两张卡运行。