当前位置:   article > 正文

torch.distributed

torch.distributed

参考 torch.distributed.init_process_group() - 云+社区 - 腾讯云

目录

后端

PyTorch自带的后端

使用哪个后端?

常见的环境变量

基础

初始化

torch.distributed.is_available()[source]

torch.distributed.init_process_group(backend, init_method=None, timeout=datetime.timedelta(0, 1800), world_size=-1, rank=-1, store=None, group_name='')[source]

class torch.distributed.Backend

torch.distributed.get_backend(group=)[source]

torch.distributed.get_rank(group=)[source]

torch.distributed.get_world_size(group=)[source]

torch.distributed.is_initialized()[source]

torch.distributed.is_mpi_available()[source]

torch.distributed.is_nccl_available()[source]

TCP初始化

共享文件系统初始化

环境变量初始化

torch.distributed.new_group(ranks=None, timeout=datetime.timedelta(0, 1800), backend=None)[source]

点对点通信

torch.distributed.send(tensor, dst, group=, tag=0)[source]

torch.distributed.recv(tensor, src=None, group=, tag=0)[source]

torch.distributed.isend(tensor, dst, group=, tag=0)[source]

torch.distributed.irecv(tensor, src, group=, tag=0)[source]

同步和异步收集操作

收集函数

torch.distributed.broadcast(tensor, src, group=, async_op=False)[source]

torch.distributed.all_reduce(tensor, op=ReduceOp.SUM, group=, async_op=False)[source]

torch.distributed.reduce(tensor, dst, op=ReduceOp.SUM, group=, async_op=False)[source]

torch.distributed.all_gather(tensor_list, tensor, group=, async_op=False)[source]

torch.distributed.gather(tensor, gather_list=None, dst=0, group=, async_op=False)[source]

torch.distributed.scatter(tensor, scatter_list=None, src=0, group=, async_op=False)[source]

torch.distributed.reduce_scatter(output, input_list, op=ReduceOp.SUM, group=, async_op=False)[source]

torch.distributed.all_to_all(output_tensor_list, input_tensor_list, group=, async_op=False)[source]

torch.distributed.barrier(group=, async_op=False)[source]

class torch.distributed.ReduceOp

class torch.distributed.reduce_op[source]

多GPU收集函数

torch.distributed.broadcast_multigpu(tensor_list, src, group=, async_op=False, src_tensor=0)[source]

torch.distributed.all_reduce_multigpu(tensor_list, op=ReduceOp.SUM, group=, async_op=False)[source]

torch.distributed.reduce_multigpu(tensor_list, dst, op=ReduceOp.SUM, group=, async_op=False, dst_tensor=0)[source]

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/很楠不爱3/article/detail/420503
推荐阅读
相关标签
  

闽ICP备14008679号