devbox
IT小白
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

YOLOV ：基于YOLOX，使静态图像对象检测器在视频对象检测方面表现出色，注意力机制的魅力 Loss计算部分_oom runtimeerror is raised due to the huge memory

作者：IT小白 | 2024-04-13 19:47:16

赞

踩

oom runtimeerror is raised due to the huge memory cost during label assignme

1.上一讲注意力机制后面便是loss部分，他输入的是图像、x_shifts是一张图片被分成了多少的网格数量、expanded_strides网格的长度、fc_output注意力机制后的特征、pred_idx预测

 

  1.  if self.training:
  2.  
  3.             return self.get_losses(
  4.                 imgs,
  5.                 x_shifts,
  6.                 y_shifts,
  7.                 expanded_strides,
  8.                 labels,
  9.                 torch.cat(outputs, 1),
  10.                 origin_preds,
  11.                 dtype=xin[0].dtype,
  12.                 refined_cls=fc_output,
  13.                 idx=pred_idx,
  14.                 pred_res=pred_result,
  15.             )
  16.         else:
  17.  
  18.             class_conf, class_pred = torch.max(fc_output, -1, keepdim=False)  #看哪个类别的概率最大
  19.             result, result_ori = postprocess(copy.deepcopy(pred_result), self.num_classes, fc_output,nms_thre=nms_thresh )
  20.  
  21.             return result, result_ori  # result

2.这部分比较简单，获取每一帧图像的box、真实分类 [batch,120,class+xywh]、每张图片的预测框。

  1.   bbox_preds = outputs[:, :, :4]  # [batch, n_anchors_all, 4]
  2.         obj_preds = outputs[:, :, 4].unsqueeze(-1)  # [batch, n_anchors_all, 1]
  3.         cls_preds = outputs[:, :, 5:]  # [batch, n_anchors_all, n_cls]
  4.  
  5.         # calculate targets  判断标签的类别是否是5个
  6.         mixup = labels.shape[2] > 5
  7.         if mixup:
  8.             label_cut = labels[..., :5]
  9.         else:
  10.             label_cut = labels
  11.         nlabel = (label_cut.sum(dim=2) > 0).sum(dim=1)  # number of objects    去除120中没有的标签 获取标签的真实个数
  12.  
  13.         total_num_anchors = outputs.shape[1]  # n_anchors_all
  14.         x_shifts = torch.cat(x_shifts, 1)  # [1, n_anchors_all]
  15.         y_shifts = torch.cat(y_shifts, 1)  # [1, n_anchors_all]
  16.         expanded_strides = torch.cat(expanded_strides, 1)
  17.         if self.use_l1:      # 80轮之后的L1损失
  18.             origin_preds = torch.cat(origin_preds, 1)
  19.  
  20.         cls_targets = []
  21.         reg_targets = []
  22.         l1_targets = []
  23.         obj_targets = []
  24.         fg_masks = []
  25.         ref_targets = []
  26.         num_fg = 0.0
  27.         num_gts = 0.0
  28.         ref_masks = []
  29.         for batch_idx in range(outputs.shape[0]): # batch的大小
  30.             num_gt = int(nlabel[batch_idx])
  31.             num_gts += num_gt
  32.             if num_gt == 0:
  33.                 cls_target = outputs.new_zeros((0, self.num_classes))
  34.                 reg_target = outputs.new_zeros((0, 4))
  35.                 l1_target = outputs.new_zeros((0, 4))
  36.                 obj_target = outputs.new_zeros((total_num_anchors, 1))
  37.                 fg_mask = outputs.new_zeros(total_num_anchors).bool()
  38.                 ref_target = outputs.new_zeros((idx[batch_idx].shape[0], self.num_classes + 1))
  39.                 ref_target[:, -1] = 1
  40.  
  41.             else:
  42.                 gt_bboxes_per_image = labels[batch_idx, :num_gt, 1:5]  # 每张图片的box
  43.                 gt_classes = labels[batch_idx, :num_gt, 0]             # 真实分类  [batch,120,class+xywh]
  44.                 bboxes_preds_per_image = bbox_preds[batch_idx]         # 每张图片的预测框

3.这一步是获取每一帧图像上正样本的类别、mask掩码、iou、数量等。

输入是第几个batch、真实框的数量、所有框的数量（5379）、真实框、真实框的类别、每一帧图像的预测框（5376x4）、三个特征图与原图的缩放比、左上角的xy坐标、类别预测（8x5376x30）、框的预测（8x5376x4）、置信度预测（8x5376x1）、类别、图像。

  1.  try:
  2.                     (
  3.                         gt_matched_classes,       # 正样本的类别
  4.                         fg_mask,                  # 5376中正样本30 的mask掩码
  5.                         pred_ious_this_matching,  # 正样本与它对应真实框的iou
  6.                         matched_gt_inds,          # 正样本与真实框对应
  7.                         num_fg_img,               # 正样本的数量
  8.                     ) = self.get_assignments(  # noqa
  9.                         batch_idx,
  10.                         num_gt,
  11.                         total_num_anchors,
  12.                         gt_bboxes_per_image,
  13.                         gt_classes,
  14.                         bboxes_preds_per_image,
  15.                         expanded_strides,
  16.                         x_shifts,
  17.                         y_shifts,
  18.                         cls_preds,
  19.                         bbox_preds,
  20.                         obj_preds,
  21.                         labels,
  22.                         imgs,
  23.                     )
  24.                 except RuntimeError:
  25.                     logger.error(
  26.                         "OOM RuntimeError is raised due to the huge memory cost during label assignment. \
  27.                            CPU mode is applied in this batch. If you want to avoid this issue, \
  28.                            try to reduce the batch size or image size."
  29.                     )
  30.                     torch.cuda.empty_cache()
  31.                     (
  32.                         gt_matched_classes,
  33.                         fg_mask,
  34.                         pred_ious_this_matching,
  35.                         matched_gt_inds,
  36.                         num_fg_img,
  37.                     ) = self.get_assignments(  # noqa
  38.                         batch_idx,
  39.                         num_gt,
  40.                         total_num_anchors,
  41.                         gt_bboxes_per_image,
  42.                         gt_classes,
  43.                         bboxes_preds_per_image,
  44.                         expanded_strides,
  45.                         x_shifts,
  46.                         y_shifts,
  47.                         cls_preds,
  48.                         bbox_preds,
  49.                         obj_preds,
  50.                         labels,
  51.                         imgs,
  52.                         "cpu",
  53.                     )

3.1这部分代码主要分成这几个模块：

1.将在预测框的中心点在真实框和自己设计的4.5x4.5大小框中的预测框选出来。

2.计算cost

  1.  def get_assignments(
  2.             self,
  3.             batch_idx,
  4.             num_gt,
  5.             total_num_anchors,
  6.             gt_bboxes_per_image,
  7.             gt_classes,
  8.             bboxes_preds_per_image,
  9.             expanded_strides,
  10.             x_shifts,
  11.             y_shifts,
  12.             cls_preds,
  13.             bbox_preds,
  14.             obj_preds,
  15.             labels,
  16.             imgs,
  17.             mode="gpu",
  18.     ):
  19.  
  20.         if mode == "cpu":
  21.             print("------------CPU Mode for This Batch-------------")
  22.             gt_bboxes_per_image = gt_bboxes_per_image.cpu().float()
  23.             bboxes_preds_per_image = bboxes_preds_per_image.cpu().float()
  24.             gt_classes = gt_classes.cpu().float()
  25.             expanded_strides = expanded_strides.cpu().float()
  26.             x_shifts = x_shifts.cpu()
  27.             y_shifts = y_shifts.cpu()
  28.         # 预测框的中心点既在真实框中也在4.5x4.5中的预测框
  29.         fg_mask, is_in_boxes_and_center = self.get_in_boxes_info(
  30.             gt_bboxes_per_image,
  31.             expanded_strides,
  32.             x_shifts,
  33.             y_shifts,
  34.             total_num_anchors,
  35.             num_gt,
  36.         )
  37.         # 根据是否在框中，删除不在框中的数据
  38.         bboxes_preds_per_image = bboxes_preds_per_image[fg_mask]
  39.         cls_preds_ = cls_preds[batch_idx][fg_mask]
  40.         obj_preds_ = obj_preds[batch_idx][fg_mask]
  41.         num_in_boxes_anchor = bboxes_preds_per_image.shape[0]
  42.  
  43.         if mode == "cpu":
  44.             gt_bboxes_per_image = gt_bboxes_per_image.cpu()
  45.             bboxes_preds_per_image = bboxes_preds_per_image.cpu()
  46.  
  47.         pair_wise_ious = bboxes_iou(gt_bboxes_per_image, bboxes_preds_per_image, False)     # 计算真实框和预测框的iou
  48.  
  49.         gt_cls_per_image = (     # 4x656x30 一张图片上有四个真实框 每个框的类别复制656 对应着656个预测框
  50.             F.one_hot(gt_classes.to(torch.int64), self.num_classes)
  51.                 .float()
  52.                 .unsqueeze(1)
  53.                 .repeat(1, num_in_boxes_anchor, 1)
  54.         )
  55.         pair_wise_ious_loss = -torch.log(pair_wise_ious + 1e-8)
  56.  
  57.         if mode == "cpu":
  58.             cls_preds_, obj_preds_ = cls_preds_.cpu(), obj_preds_.cpu()
  59.  
  60.         with torch.cuda.amp.autocast(enabled=False):
  61.             cls_preds_ = (     # 置信度*类别
  62.                     cls_preds_.float().unsqueeze(0).repeat(num_gt, 1, 1).sigmoid_()  # 4x656x30    656x30 复制4份
  63.                     * obj_preds_.unsqueeze(0).repeat(num_gt, 1, 1).sigmoid_()        # 4x656x1
  64.             )
  65.             pair_wise_cls_loss = F.binary_cross_entropy(      # 预测框的类别和真实框的类别做计算 得出他的类别
  66.                 cls_preds_.sqrt_(), gt_cls_per_image, reduction="none"
  67.             ).sum(-1)
  68.         del cls_preds_
  69.  
  70.         cost = (
  71.                 pair_wise_cls_loss   #分类的一个损失
  72.                 + 3.0 * pair_wise_ious_loss   #iou损失
  73.                 + 100000.0 * (~is_in_boxes_and_center)   # 如果不在里面 给她一个很大的值，cos就不会选到他
  74.         )
  75.  
  76.         (
  77.             num_fg,
  78.             gt_matched_classes,
  79.             pred_ious_this_matching,
  80.             matched_gt_inds,
  81.         ) = self.dynamic_k_matching(cost, pair_wise_ious, gt_classes, num_gt, fg_mask)
  82.         del pair_wise_cls_loss, cost, pair_wise_ious, pair_wise_ious_loss
  83.  
  84.         if mode == "cpu":
  85.             gt_matched_classes = gt_matched_classes.cuda()
  86.             fg_mask = fg_mask.cuda()
  87.             pred_ious_this_matching = pred_ious_this_matching.cuda()
  88.             matched_gt_inds = matched_gt_inds.cuda()
  89.  
  90.         return (
  91.             gt_matched_classes,
  92.             fg_mask,
  93.             pred_ious_this_matching,
  94.             matched_gt_inds,
  95.             num_fg,
  96.         )

3.1.1由于每一个格子有一个预测框，因此计算每一个格子的中心点，判断在真实框和自己设计的4.5x4.5大小的框中的中心点，去除不在这些框中的中心点。

  1.     def get_in_boxes_info(
  2.             self,
  3.             gt_bboxes_per_image,
  4.             expanded_strides,
  5.             x_shifts,
  6.             y_shifts,
  7.             total_num_anchors,
  8.             num_gt,
  9.     ):
  10.         expanded_strides_per_image = expanded_strides[0]
  11.         x_shifts_per_image = x_shifts[0] * expanded_strides_per_image
  12.         y_shifts_per_image = y_shifts[0] * expanded_strides_per_image  #左上角的坐标
  13.         x_centers_per_image = (                                      # 计算每一个格子的中心点的位置
  14.             (x_shifts_per_image + 0.5 * expanded_strides_per_image)
  15.                 .unsqueeze(0)
  16.                 .repeat(num_gt, 1)
  17.         )  # [n_anchor] -> [n_gt, n_anchor]
  18.         y_centers_per_image = (
  19.             (y_shifts_per_image + 0.5 * expanded_strides_per_image)   #每个格子的中心点
  20.                 .unsqueeze(0)
  21.                 .repeat(num_gt, 1)
  22.         )
  23.         #计算真实框的四边  l_x l_y r_x r_y
  24.         gt_bboxes_per_image_l = (
  25.             (gt_bboxes_per_image[:, 0] - 0.5 * gt_bboxes_per_image[:, 2])   #中心点和长宽
  26.                 .unsqueeze(1)
  27.                 .repeat(1, total_num_anchors)
  28.         )
  29.         gt_bboxes_per_image_r = (
  30.             (gt_bboxes_per_image[:, 0] + 0.5 * gt_bboxes_per_image[:, 2])
  31.                 .unsqueeze(1)
  32.                 .repeat(1, total_num_anchors)
  33.         )
  34.         gt_bboxes_per_image_t = (
  35.             (gt_bboxes_per_image[:, 1] - 0.5 * gt_bboxes_per_image[:, 3])
  36.                 .unsqueeze(1)
  37.                 .repeat(1, total_num_anchors)
  38.         )
  39.         gt_bboxes_per_image_b = (
  40.             (gt_bboxes_per_image[:, 1] + 0.5 * gt_bboxes_per_image[:, 3])
  41.                 .unsqueeze(1)
  42.                 .repeat(1, total_num_anchors)
  43.         )
  44.         # 判断5376个框那些中心点在真实框中
  45.         b_l = x_centers_per_image - gt_bboxes_per_image_l
  46.         b_r = gt_bboxes_per_image_r - x_centers_per_image
  47.         b_t = y_centers_per_image - gt_bboxes_per_image_t
  48.         b_b = gt_bboxes_per_image_b - y_centers_per_image
  49.         bbox_deltas = torch.stack([b_l, b_t, b_r, b_b], 2)
  50.         # 4x5376x4  四个真实框 5376个预测框 四个xy相减的值   -》4x5376 mask
  51.         is_in_boxes = bbox_deltas.min(dim=-1).values > 0.0
  52.         is_in_boxes_all = is_in_boxes.sum(dim=0) > 0   # 四个真实框都没有的预测框去除（这里真实框的数量为4，可能不同）
  53.         # in fixed center
  54.         #与上面一样
  55.         center_radius = 4.5    #生成一个4.5x4.5的格子
  56.  
  57.         gt_bboxes_per_image_l = (gt_bboxes_per_image[:, 0]).unsqueeze(1).repeat(
  58.             1, total_num_anchors
  59.         ) - center_radius * expanded_strides_per_image.unsqueeze(0)
  60.         gt_bboxes_per_image_r = (gt_bboxes_per_image[:, 0]).unsqueeze(1).repeat(
  61.             1, total_num_anchors
  62.         ) + center_radius * expanded_strides_per_image.unsqueeze(0)
  63.         gt_bboxes_per_image_t = (gt_bboxes_per_image[:, 1]).unsqueeze(1).repeat(
  64.             1, total_num_anchors
  65.         ) - center_radius * expanded_strides_per_image.unsqueeze(0)
  66.         gt_bboxes_per_image_b = (gt_bboxes_per_image[:, 1]).unsqueeze(1).repeat(
  67.             1, total_num_anchors
  68.         ) + center_radius * expanded_strides_per_image.unsqueeze(0)
  69.  
  70.         c_l = x_centers_per_image - gt_bboxes_per_image_l
  71.         c_r = gt_bboxes_per_image_r - x_centers_per_image
  72.         c_t = y_centers_per_image - gt_bboxes_per_image_t
  73.         c_b = gt_bboxes_per_image_b - y_centers_per_image
  74.         center_deltas = torch.stack([c_l, c_t, c_r, c_b], 2)
  75.         is_in_centers = center_deltas.min(dim=-1).values > 0.0
  76.         is_in_centers_all = is_in_centers.sum(dim=0) > 0
  77.  
  78.         # in boxes and in centers
  79.         is_in_boxes_anchor = is_in_boxes_all | is_in_centers_all   #两者并集   预测框或者在真实框或者在4.5x4.5中
  80.  
  81.         is_in_boxes_and_center = (
  82.                 is_in_boxes[:, is_in_boxes_anchor] & is_in_centers[:, is_in_boxes_anchor]  #两者交集
  83.         )
  84.         return is_in_boxes_anchor, is_in_boxes_and_center

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/IT小白/article/detail/418200
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号