解决 pytorch 报错 RuntimeError: An attempt has been made to start a new process..._runtimeerror: broken pipe

报错信息

运行 pytorch 代码报错，BrokenPipeError: [Errno 32] Broken pipe，看起来和多线程有关。

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.    ForkingPickler(file, protocol).dump(obj)

BrokenPipeError: [Errno 32] Broken pipe
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

原因分析

pytorch 加载数据集时会使用到 DataLoader 方法，其中参数 num_workers 为加载数据集所用的线程数量。线程数越多，数据集加载越快。

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=opt.batchsize,
                                             shuffle=True, num_workers=2, pin_memory=True) # 8 workers may work faster
              for x in ['train', 'val']}
1
2
3

layumi 在 issue 中也提到 Windows 下 pytorch 对多线程的支持不够好。

由于报错信息与多线程有关，因此考虑将 num_workers 设为 0，或者去掉该参数。

解决办法

1. 去掉 num_workers 参数

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=opt.batchsize,
                                             shuffle=True, pin_memory=True) # 8 workers may work faster
              for x in ['train', 'val']}
1
2
3

2. 将 num_workers 设为 0

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=opt.batchsize,
                                             shuffle=True, num_workers=0, pin_memory=True) # 8 workers may work faster
              for x in ['train', 'val']}
1
2
3

参考链接

1. pytorch使用出现"RuntimeError: An attempt has been made to start a new process before the…" 解决方法
2. https://github.com/layumi/Person_reID_baseline_pytorch/issues/188