devbox
Gausst松鼠会
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

SSD300网络结构(pytorch)+多尺度训练与测试

作者：Gausst松鼠会 | 2024-03-22 21:59:22

赞

踩

ssd300

一.SSD300

1.如图是预测框的相应feature map 

这里smin是0.2，表示最底层的scale是0.2；smax是0.9，表示最高层的scale是0.9,m代表产生尺度预测的feature map个数。

其中anchor的长宽关系，s就是上图中的scale,a就是上图中的anchor ratio

2.代码

主要由三部分组成

1.vgg作为基础网络

要注意的是作者对38*38*512进行L2正则化，并用一个可学习参数调节通道权重

2.增加大目标检测网络

3.输出包括预测框的偏移量输出与分类

偏移量计算,神经网络学习偏移量即可。

误检的HEM(hard negative mine)loss函数，用于分类

1.回归量与坐标的转换

  1. def cxcy_to_gcxgcy(cxcy, priors_cxcy):
  2.     
  3.     # See https://github.com/weiliu89/caffe/issues/155
  4.     return torch.cat([(cxcy[:, :2] - priors_cxcy[:, :2]) / (priors_cxcy[:, 2:] / 10),  # g_c_x, g_c_y
  5.                       torch.log(cxcy[:, 2:] / priors_cxcy[:, 2:]) * 5], 1)  # g_w, g_h
  6.  
  7.  
  8. def gcxgcy_to_cxcy(gcxgcy, priors_cxcy):
  9.     
  10.     return torch.cat([gcxgcy[:, :2] * priors_cxcy[:, 2:] / 10 + priors_cxcy[:, :2],  # c_x, c_y
  11.                       torch.exp(gcxgcy[:, 2:] / 5) * priors_cxcy[:, 2:]], 1)  # w, h

2.anchor与gt框匹配示例，保证每个gt至少有一个anchor

  1.  
  2. #两个gt框　3个anchor 的框分配示例
  3. import torch
  4. objects = 2
  5. overlap = torch.tensor([[0.4, 0.5, 0.6],
  6.                         [0.8, 0.9, 0.7]])
  7. iou_for_each_prior, index_for_each_prior = torch.max(overlap, dim=0)
  8. print(iou_for_each_prior, index_for_each_prior)
  9.  
  10. iou_for_each_box, index_for_each_box = torch.max(overlap, dim=1)
  11. print(iou_for_each_box, index_for_each_box)
  12.  
  13. index_for_each_prior[index_for_each_box] = torch.LongTensor(range(objects))
  14. print(index_for_each_prior)

3.gt框与对应anchor框做回归的示例，其中的true_classes是两个样本，每一个样本有3个box框的类别示例,0代表背景

  1.  
  2. #两个gt框　3个anchor 的框分配示例
  3. import torch
  4. objects = 2
  5. overlap = torch.tensor([[0.4, 0.5, 0.6],
  6.                         [0.8, 0.9, 0.7]])
  7. iou_for_each_prior, index_for_each_prior = torch.max(overlap, dim=0)
  8. print(iou_for_each_prior, index_for_each_prior)
  9.  
  10. iou_for_each_box, index_for_each_box = torch.max(overlap, dim=1)
  11. print(iou_for_each_box, index_for_each_box)
  12.  
  13. index_for_each_prior[index_for_each_box] = torch.LongTensor(range(objects))
  14. print(index_for_each_prior)
  15.  
  16. batch_size = 2
  17. true_classes = torch.tensor([[0, 1, 3],#每一个样本3个box框的类别示例,0代表背景
  18.                              [2, 4, 5]])
  19. positive_priors = true_classes != 0
  20. print('=positive_priors:\n', positive_priors)
  21.  
  22.  
  23. pre_locs = torch.rand((batch_size, 3, 4))
  24. print('==pre_locs[positive_priors].shape:\n', pre_locs[positive_priors].shape)
  25.  
  26. true_locs = torch.rand((batch_size, 3, 4))
  27. print('==true_locs[positive_priors].shape:\n', true_locs[positive_priors].shape)

4.总体代码:

  1. import torch
  2. import os
  3. from torch import nn
  4. import torch.nn.functional as F
  5. import torchvision
  6. from torchvision import models
  7. from utils import decimate, find_jaccard_overlap, cxcy_to_xy, xy_to_cxcy
  8. from utils import cxcy_to_gcxgcy as cx_cy_dxdy
  9. from math import sqrt
  10. # vgg16 = models.vgg16(pretrained=True)
  11. # print(vgg16)
  12. # vgg16_state_dict = vgg16.state_dict()
  13. # print(list(vgg16_state_dict.keys()))
  14. # print(vgg16_state_dict.values())
  15. # for key, value in vgg16.named_parameters():
  16. #     print('key:', key)
  17.  
  18. device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  19.  
  20. class VGGbase(nn.Module):
  21.     """vgg 主干网络"""
  22.     def __init__(self):
  23.         super(VGGbase, self).__init__()
  24.         self.conv1_1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1)
  25.         self.conv1_2 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)
  26.         self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
  27.  
  28.         self.conv2_1 = nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1)
  29.         self.conv2_2 = nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1)
  30.         self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
  31.  
  32.         self.conv3_1 = nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1)
  33.         self.conv3_2 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1)
  34.         self.conv3_3 = nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1)
  35.         self.pool3 = nn.MaxPool2d(kernel_size=2, stride=2, ceil_mode=True)
  36.  
  37.         self.conv4_1 = nn.Conv2d(256, 512, kernel_size=3, stride=1, padding=1)
  38.         self.conv4_2 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
  39.         self.conv4_3 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
  40.         self.pool4 = nn.MaxPool2d(kernel_size=2, stride=2)
  41.  
  42.         self.conv5_1 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
  43.         self.conv5_2 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
  44.         self.conv5_3 = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1)
  45.         self.pool5 = nn.MaxPool2d(kernel_size=3, stride=1, padding=1)#为了保证尺寸不在减少
  46.  
  47.         self.conv6 = nn.Conv2d(512, 1024, kernel_size=3, stride=1, padding=6, dilation=6)#空洞卷积扩大感受野
  48.         self.conv7 = nn.Conv2d(1024, 1024, kernel_size=1)
  49.  
  50.         self.load_pretrained_layers()#载入预训练权重
  51.     #(BS, 3, 300, 300)
  52.     def forward(self, image):
  53.         out = F.relu(self.conv1_1(image))
  54.         out = F.relu(self.conv1_2(out))
  55.         out = self.pool1(out)#(B,64, 150, 150)
  56.         out = F.relu(self.conv2_1(out))
  57.         out = F.relu(self.conv2_2(out))
  58.         out = self.pool2(out)  #(B, 128, 75, 75)
  59.         out = F.relu(self.conv3_1(out))
  60.         out = F.relu(self.conv3_2(out))
  61.         out = F.relu(self.conv3_3(out))
  62.         out = self.pool3(out)  # (B, 256, 38, 38)
  63.         out = F.relu(self.conv4_1(out))
  64.         out = F.relu(self.conv4_2(out))
  65.         out = F.relu(self.conv4_3(out))
  66.         conv4_3feats = out     # (B, 512, 38, 38)
  67.         out = self.pool4(out)  # (B, 512, 19, 19)
  68.         out = F.relu(self.conv5_1(out))
  69.         out = F.relu(self.conv5_2(out))
  70.         out = F.relu(self.conv5_3(out))
  71.         out = self.pool5(out)  # (B, 512, 19, 19)
  72.         out = F.relu(self.conv6(out))
  73.         conv7_feats = F.relu(self.conv7(out))# (B, 1024, 19, 19)
  74.         # print(out.shape)
  75.         return conv4_3feats, conv7_feats
  76.     def load_pretrained_layers(self):
  77.         state_dict = self.state_dict()
  78.         param_name = list(state_dict.keys())
  79.         print('param_name', param_name)
  80.  
  81.         pretrained_state_dict = models.vgg16(pretrained=True).state_dict()
  82.         pretrained_param_name = list(pretrained_state_dict.keys())
  83.         print('pretrained_param_name', pretrained_param_name)
  84.         #由于最后两层与原vgg网络相比多出来的，故权重和偏置要点到为止
  85.         for i, param in enumerate(param_name[:-4]):
  86.             # print('pretrained_state_dict[pretrained_param_name[i]].shape', pretrained_state_dict[pretrained_param_name[i]].shape)
  87.             state_dict[param] = pretrained_state_dict[pretrained_param_name[i]]
  88.  
  89.         # #最后两层的权重由分类器权重修改而来
  90.         # print("pretrained_state_dict['classifier.0.weight'].shape",pretrained_state_dict['classifier.0.weight'].shape)
  91.         conv_fc6_weight = pretrained_state_dict['classifier.0.weight'].reshape(4096, 512, 7, 7)
  92.         # print('===conv_fc6_weight.dim()==', conv_fc6_weight.dim())
  93.         state_dict['conv6.weight'] = decimate(conv_fc6_weight, m=[4, None, 3, 3])#(1024, 512, 3, 3)
  94.         conv_fc6_bias = pretrained_state_dict['classifier.0.bias']#(4096)
  95.         state_dict['conv6.bias'] = decimate(conv_fc6_bias, m=[4])#(1024)
  96.         # print(pretrained_state_dict['classifier.3.weight'].shape)
  97.         # print(pretrained_state_dict['classifier.6.weight'].shape)
  98.  
  99.         conv_fc7_weight = pretrained_state_dict['classifier.3.weight'].reshape(4096, 4096, 1, 1)
  100.         state_dict['conv7.weight'] = decimate(conv_fc7_weight, m=[4, 4, None, None])  # (1024, 1024, 1, 1)
  101.         conv_fc7_bias = pretrained_state_dict['classifier.3.bias']  # (4096)
  102.         state_dict['conv7.bias'] = decimate(conv_fc7_bias, m=[4])  # (1024)
  103.  
  104.         self.load_state_dict(state_dict)
  105.  
  106. class AuxiliaryConvolutions(nn.Module):
  107.     "继续在vgg基础上添加conv网络"
  108.     def __init__(self):
  109.         super(AuxiliaryConvolutions, self).__init__()#调用父类初始化
  110.         self.conv8_1 = nn.Conv2d(1024, 256, kernel_size=1, stride=1)
  111.         self.conv8_2 = nn.Conv2d(256, 512, kernel_size=3, stride=2, padding=1)
  112.  
  113.         self.conv8_1 = nn.Conv2d(1024, 256, kernel_size=1, stride=1)
  114.         self.conv8_2 = nn.Conv2d(256, 512, kernel_size=3, stride=2, padding=1)
  115.  
  116.         self.conv9_1 = nn.Conv2d(512, 128, kernel_size=1, stride=1)
  117.         self.conv9_2 = nn.Conv2d(128, 256, kernel_size=3, stride=2, padding=1)
  118.  
  119.         self.conv10_1 = nn.Conv2d(256, 128, kernel_size=1, stride=1)
  120.         self.conv10_2 = nn.Conv2d(128, 256, kernel_size=3, stride=1)
  121.  
  122.         self.conv11_1 = nn.Conv2d(256, 128, kernel_size=1, stride=1)
  123.         self.conv11_2 = nn.Conv2d(128, 256, kernel_size=3, stride=1)
  124.  
  125.         self.init_conv2d()
  126.  
  127.     def init_conv2d(self):
  128.         for c in self.children():
  129.             if isinstance(c, nn.Conv2d):
  130.                 nn.init.xavier_uniform_(c.weight)
  131.                 # nn.init.kaiming_normal_(c.weight)
  132.                 nn.init.constant_(c.bias, 0)
  133.  
  134.     def forward(self, input):
  135.         out = F.relu(self.conv8_1(input))#(B,1024,19,19)
  136.         out = F.relu(self.conv8_2(out))  #(B,512,19,19)
  137.         conv8_2feats = out
  138.         out = F.relu(self.conv9_1(out))  #(B,512,10,10)
  139.         out = F.relu(self.conv9_2(out))  ##(B,256,5,5)
  140.         conv9_2feats = out
  141.         out = F.relu(self.conv10_1(out))  # (B,128,5,5)
  142.         out = F.relu(self.conv10_2(out))  ##(B,256,3,3)
  143.         conv10_2feats = out
  144.         out = F.relu(self.conv11_1(out))  # (B,128,3,3)
  145.         out = F.relu(self.conv11_2(out))  ##(B,256,1,1)
  146.         conv11_2feats = out
  147.         # print(out.size())
  148.         return conv8_2feats, conv9_2feats, conv10_2feats, conv11_2feats
  149.  
  150. class PredictionConvolutions(nn.Module):
  151.     """卷积层输出框偏移量与分类"""
  152.     def __init__(self, n_classes):
  153.         super(PredictionConvolutions, self).__init__()
  154.         self.n_classes = n_classes
  155.         bboxs={
  156.             'conv4_3': 4,
  157.             'conv7': 6,
  158.             'conv8_2': 6,
  159.             'conv9_2': 6,
  160.             'conv10_2': 4,
  161.             'conv11_2': 4
  162.         }
  163.         self.loc_conv4_3 = nn.Conv2d(512, bboxs['conv4_3']*4, kernel_size=3, padding=1)
  164.         self.loc_conv7 = nn.Conv2d(1024, bboxs['conv7'] * 4, kernel_size=3, padding=1)
  165.         self.loc_conv8_2 = nn.Conv2d(512, bboxs['conv8_2'] * 4, kernel_size=3, padding=1)
  166.         self.loc_conv9_2 = nn.Conv2d(256, bboxs['conv9_2'] * 4, kernel_size=3, padding=1)
  167.         self.loc_conv10_2 = nn.Conv2d(256, bboxs['conv10_2'] * 4, kernel_size=3, padding=1)
  168.         self.loc_conv11_2 = nn.Conv2d(256, bboxs['conv11_2'] * 4, kernel_size=3, padding=1)
  169.  
  170.         self.cl_conv4_3 = nn.Conv2d(512, bboxs['conv4_3'] * n_classes, kernel_size=3, padding=1)
  171.         self.cl_conv7 = nn.Conv2d(1024, bboxs['conv7'] * n_classes, kernel_size=3, padding=1)
  172.         self.cl_conv8_2 = nn.Conv2d(512, bboxs['conv8_2'] * n_classes, kernel_size=3, padding=1)
  173.         self.cl_conv9_2 = nn.Conv2d(256, bboxs['conv9_2'] * n_classes, kernel_size=3, padding=1)
  174.         self.cl_conv10_2 = nn.Conv2d(256, bboxs['conv10_2'] * n_classes, kernel_size=3, padding=1)
  175.         self.cl_conv11_2 = nn.Conv2d(256, bboxs['conv11_2'] * n_classes, kernel_size=3, padding=1)
  176.         self.init_conv2d()
  177.  
  178.     def init_conv2d(self):
  179.         for c in self.children():
  180.             if isinstance(c, nn.Conv2d):
  181.                 nn.init.xavier_uniform_(c.weight)
  182.                 # nn.init.kaiming_normal_(c.weight)
  183.                 nn.init.constant_(c.bias, 0)
  184.     def forward(self, conv4_3feats,conv7_feats,conv8_2feats, conv9_2feats, conv10_2feats, conv11_2feats):
  185.         batch_size = conv4_3feats.size(0)
  186.         loc_conv4_3 = self.loc_conv4_3(conv4_3feats)#(N, 4*4, 38, 38)
  187.         loc_conv4_3 = loc_conv4_3.permute(0, 2, 3, 1)#(N, 38, 38, 4*4)
  188.         loc_conv4_3 = loc_conv4_3.reshape(batch_size, -1, 4)
  189.         # print(loc_conv4_3.shape)
  190.  
  191.         loc_conv7 = self.loc_conv7(conv7_feats)  # (N, 6*4, 19, 19)
  192.         loc_conv7 = loc_conv7.permute(0, 2, 3, 1)
  193.         loc_conv7 = loc_conv7.reshape(batch_size, -1, 4)
  194.  
  195.         loc_conv8_2 = self.loc_conv8_2(conv8_2feats)  # (N, 6*4, 10, 10)
  196.         loc_conv8_2 = loc_conv8_2.permute(0, 2, 3, 1)
  197.         loc_conv8_2 = loc_conv8_2.reshape(batch_size, -1, 4)
  198.  
  199.         loc_conv9_2 = self.loc_conv9_2(conv9_2feats)  # (N, 6*4, 5, 5)
  200.         loc_conv9_2 = loc_conv9_2.permute(0, 2, 3, 1)
  201.         loc_conv9_2 = loc_conv9_2.reshape(batch_size, -1, 4)
  202.  
  203.         loc_conv10_2 = self.loc_conv10_2(conv10_2feats)  # (N, 4*4, 3, 3)
  204.         loc_conv10_2 = loc_conv10_2.permute(0, 2, 3, 1)
  205.         loc_conv10_2 = loc_conv10_2.reshape(batch_size, -1, 4)
  206.  
  207.         loc_conv11_2 = self.loc_conv11_2(conv11_2feats)  # (N, 4*4, 1, 1)
  208.         loc_conv11_2 = loc_conv11_2.permute(0, 2, 3, 1)
  209.         loc_conv11_2 = loc_conv11_2.reshape(batch_size, -1, 4)
  210.  
  211.         cl_conv4_3 = self.cl_conv4_3(conv4_3feats)  # (N, 4*n_classes, 38, 38)
  212.         cl_conv4_3 = cl_conv4_3.permute(0, 2, 3, 1)
  213.         cl_conv4_3 = cl_conv4_3.reshape(batch_size, -1, self.n_classes)
  214.  
  215.         cl_conv7 = self.cl_conv7(conv7_feats)  # (N, 6*n_classes, 19, 19)
  216.         cl_conv7 = cl_conv7.permute(0, 2, 3, 1)
  217.         cl_conv7 = cl_conv7.reshape(batch_size, -1, self.n_classes)
  218.  
  219.         cl_conv8_2 = self.cl_conv8_2(conv8_2feats)  # (N, 6*n_classes, 10, 10)
  220.         cl_conv8_2 = cl_conv8_2.permute(0, 2, 3, 1)
  221.         cl_conv8_2 = cl_conv8_2.reshape(batch_size, -1, self.n_classes)
  222.  
  223.         cl_conv9_2 = self.cl_conv9_2(conv9_2feats)  # (N, 6*n_classes, 5, 5)
  224.         cl_conv9_2 = cl_conv9_2.permute(0, 2, 3, 1)
  225.         cl_conv9_2 = cl_conv9_2.reshape(batch_size, -1, self.n_classes)
  226.  
  227.         cl_conv10_2 = self.cl_conv10_2(conv10_2feats)  # (N, 4*n_classes, 3, 3)
  228.         cl_conv10_2 = cl_conv10_2.permute(0, 2, 3, 1)
  229.         cl_conv10_2 = cl_conv10_2.reshape(batch_size, -1, self.n_classes)
  230.  
  231.         cl_conv11_2 = self.cl_conv11_2(conv11_2feats)  # (N, 4*n_classes, 1, 1)
  232.         cl_conv11_2 = cl_conv11_2.permute(0, 2, 3, 1)
  233.         cl_conv11_2 = cl_conv11_2.reshape(batch_size, -1, self.n_classes)
  234.         # return loc_conv4_3, loc_conv7, loc_conv8_2, loc_conv9_2, loc_conv10_2, loc_conv11_2,\
  235.         #        cl_conv4_3, cl_conv7, cl_conv8_2, cl_conv9_2, cl_conv10_2, cl_conv11_2
  236.         locs = torch.cat((loc_conv4_3, loc_conv7, loc_conv8_2, loc_conv9_2, loc_conv10_2, loc_conv11_2),dim=1)
  237.         class_scores = torch.cat((cl_conv4_3, cl_conv7, cl_conv8_2, cl_conv9_2, cl_conv10_2, cl_conv11_2),dim=1)
  238.         return locs,class_scores#(10, 8732, 4) (10, 8732, 21)
  239.  
  240. class SSD300(nn.Module):
  241.     def __init__(self, n_classes):
  242.         super(SSD300, self).__init__()
  243.         self.n_classes = n_classes
  244.         self.base_vgg = VGGbase()
  245.         self.aux_convs = AuxiliaryConvolutions()
  246.         self.pre_convs = PredictionConvolutions(self.n_classes)
  247.  
  248.         #对conv4_3添加每个通道添加可学习参数，并进行L2正则化
  249.         self.rescale_factors = nn.Parameter(torch.FloatTensor(1, 512, 1, 1))
  250.         nn.init.constant_(self.rescale_factors, 20)
  251.  
  252.         self.create_prior_boxes()
  253.  
  254.     def forward(self, input):
  255.         conv4_3feats, conv7_feats = self.base_vgg(input)#(N,512,38,38) (N,1024,19,19)
  256.         norm = torch.pow(conv4_3feats, 2).sum(dim=1, keepdim=True).sqrt()#(B, 1, 38, 38)对所有通道的每一行求平方和L2正则 开更号
  257.         conv4_3feats = conv4_3feats/norm*self.rescale_factors
  258.  
  259.         conv8_2feats, conv9_2feats, conv10_2feats, conv11_2feats = self.aux_convs(conv7_feats)
  260.  
  261.         locs, class_scores = self.pre_convs(conv4_3feats, conv7_feats, conv8_2feats, conv9_2feats, conv10_2feats, conv11_2feats)
  262.         return locs, class_scores#(10, 8732, 4) (10, 8732, 21)
  263.  
  264.     def create_prior_boxes(self):
  265.         """创建SSD300的先验框(cx, cy, w, h)
  266.             (8372,4)个box"""
  267.         fmap_size = {'conv4_3': 38, 'conv7': 19, 'conv8_2': 10,
  268.                      'conv9_2': 5, 'conv10_2': 3, 'conv11_2': 1}
  269.         anchor_scale = {'conv4_3': 0.1, 'conv7': 0.2, 'conv8_2': 0.375,
  270.                         'conv9_2': 0.55, 'conv10_2': 0.725, 'conv11_2': 0.9}
  271.         anchor_ratio = {'conv4_3': [1, 2, 0.5], 'conv7': [1, 2, 3, 0.5, 0.33], 'conv8_2': [1, 2, 3, 0.5, 0.33],
  272.                         'conv9_2': [1, 2, 3, 0.5, 0.33], 'conv10_2': [1, 2, 0.5], 'conv11_2': [1, 2, 0.5]}
  273.         prior_boxes = []
  274.         for index, fmap in enumerate(fmap_size):
  275.             for i in range(fmap_size[fmap]):
  276.                 for j in range(fmap_size[fmap]):
  277.                     cy, cx = (i + 0.5) / fmap_size[fmap], (j + 0.5) / fmap_size[fmap]
  278.                     for ratio in anchor_ratio[fmap]:
  279.                         prior_boxes.append([cx, cy, anchor_scale[fmap] * sqrt(ratio), anchor_scale[fmap] / sqrt(ratio)])
  280.  
  281.                         if ratio == 1:  # 添加额外框
  282.                             try:
  283.                                 extra_scale = sqrt(anchor_scale[fmap] * anchor_scale[fmap_size[index + 1]])
  284.                             except:
  285.                                 extra_scale = 1.
  286.                             prior_boxes.append([cx, cy, extra_scale, extra_scale])
  287.             # print('len(prior_boxes)',len(prior_boxes))
  288.         # prior_boxes = [[1,2,3,4],
  289.         #                [3,4,5,6]]
  290.         prior_boxes = torch.FloatTensor(prior_boxes).to(device)
  291.         prior_boxes.clamp_(0, 1)  # 防止越界
  292.         print('prior_boxes.shape', prior_boxes.shape)
  293.         # print(prior_boxes)
  294.         return prior_boxes#(8732, 4)
  295.  
  296. class MultiBoxLoss(nn.Module):
  297.     """定位loss和分类loss,其中定位loss采用Hard Negative Mining."""
  298.     def __init__(self, prior_cxcy, threshold=0.5, neg_pos_ratio=3, alph=1.):
  299.         super(MultiBoxLoss, self).__init__()
  300.         self.prior_cxcy = prior_cxcy#(8732,4)
  301.         self.priors_xy = cxcy_to_xy(prior_cxcy)
  302.         self.threshold = threshold
  303.         self.neg_pos_ratio = neg_pos_ratio
  304.         self.alph = alph
  305.  
  306.         self.smooth_l1 = nn.L1Loss()
  307.         self.cross_entropy = nn.CrossEntropyLoss(reduce=False)#不计算batch的平均loss因为要用到hard mine模式
  308.  
  309.     def forward(self, prediction_locs, prediction_scores, boxes, labels):
  310.         """
  311.         prediction_locs,(N, 8732, 4)
  312.         prediction_scores,(N, 8732, n_classes)
  313.         boxes,[[],[[],[]]]
  314.         labels[[],[]]
  315.         """
  316.         batch_size = prediction_locs.shape[0]#(N,)
  317.         n_priors = self.prior_cxcy.shape[0]#(8732,)
  318.         n_classes = prediction_scores.shape[-1]#(n_classes)
  319.         # print('==batch_size', batch_size)
  320.         assert batch_size == len(boxes)
  321.         assert n_priors == prediction_locs.shape[1] == prediction_scores.shape[1]
  322.  
  323.         true_locs = torch.zeros((batch_size, n_priors, 4),dtype=torch.float)#(N, 8732, 4)
  324.         true_classes = torch.zeros((batch_size, n_priors),dtype=torch.long)#(N, 8732)
  325.         for i in range(batch_size):
  326.             # print('===boxes[i]', boxes[i])
  327.             objects = boxes[i].shape[0]   #(objects, 4)  (8732, 4)
  328.             overlap = find_jaccard_overlap(boxes[i], self.priors_xy)#(objects, 8732)
  329.             # 每个先验框与gt框的最大IOU 以及索引
  330.             iou_for_each_prior, index_for_each_prior = overlap.max(dim=0)
  331.             # 每个gt框与先验框的最大IOU 以及索引
  332.             iou_for_each_box, index_for_each_box = overlap.max(dim=1)
  333.  
  334.             #为了防止没有相应的先验框与gt相交
  335.             index_for_each_prior[index_for_each_box] = torch.LongTensor(range(objects)).to(device)
  336.             iou_for_each_prior[index_for_each_box] = 1.
  337.  
  338.             label_for_each_prior = labels[i][index_for_each_prior]#得到对应的每个先验框的标签
  339.             label_for_each_prior[iou_for_each_prior<self.threshold] = 0#将小于阈值的置为背景
  340.  
  341.             #依次存储batchsize
  342.             true_classes[i] = label_for_each_prior
  343.             true_locs[i] = cx_cy_dxdy(xy_to_cxcy(boxes[i][index_for_each_prior]), self.prior_cxcy)#得到偏移量
  344.         print('true_classes.dtype',true_classes.dtype)
  345.         positive_priors = true_classes != 0#batch_size 正样本(N,8732)
  346.         print('positive_priors.dtype',positive_priors.dtype)
  347.         print('==positive_priors.shape', positive_priors.shape)
  348.         print('==positive_priors', positive_priors)
  349.         loc_loss = self.smooth_l1(prediction_locs[positive_priors], true_locs[positive_priors])
  350.         n_postives = positive_priors.sum(dim=1)#(N,)
  351.         n_hard_negatives = self.neg_pos_ratio*n_postives#(N,)
  352.  
  353.         confidence_loss_all = self.cross_entropy(prediction_scores.reshape(-1, n_classes), true_classes.reshape(-1))
  354.         confidence_loss_all = confidence_loss_all.reshape(batch_size, n_priors)
  355.         print('==confidence_loss_all.shape', confidence_loss_all.shape)
  356.         confidence_loss_pos = confidence_loss_all[positive_priors]
  357.         #
  358.         print('==confidence_loss_pos.shape', confidence_loss_pos.shape)
  359.  
  360.         confidence_loss_neg = confidence_loss_all.clone()#(N, 8732)
  361.         confidence_loss_neg[positive_priors] = 0.#(N, 8732)#把正样本loss清零再去做HEM
  362.         confidence_loss_neg, _ = confidence_loss_neg.sort(dim=1, descending=True)#(N,8732)按行从大到小
  363.         hardness_ranks = torch.LongTensor(range(n_priors)).unsqueeze(0).expand_as(confidence_loss_neg) # (N, 8732)
  364.         hard_negatives = hardness_ranks < n_hard_negatives.unsqueeze(1)  # (N, 8732)
  365.         confidence_loss_hard = confidence_loss_all[hard_negatives]
  366.         # print('==confidence_loss_hard.shape', confidence_loss_hard.shape)
  367.         confidence_loss = (confidence_loss_pos.sum()+confidence_loss_hard.sum())/n_postives.sum().float()
  368.  
  369.         return loc_loss+self.alph*confidence_loss
  370.  
  371. def test_vgg_base():
  372.     model = VGGbase()
  373.     x = torch.rand((10, 3, 300, 300))
  374.     conv4_3feats, conv7_feats = model(x)
  375.     print('conv4_3feats.shape:', conv4_3feats.shape)
  376.     print('conv7_feats.shape:', conv7_feats.shape)
  377. def test_AUx_conv():
  378.     model = AuxiliaryConvolutions()
  379.     # (B, 1024, 19, 19)
  380.     x = torch.rand((10, 1024, 19, 19))
  381.     conv8_2feats, conv9_2feats, conv10_2feats, conv11_2feats = model(x)
  382.     print('conv8_2feats.shape:', conv8_2feats.shape)
  383.     print('conv9_2feats.shape:', conv9_2feats.shape)
  384.     print('conv10_2feats.shape:', conv10_2feats.shape)
  385.     print('conv11_2feats.shape:', conv11_2feats.shape)
  386.  
  387. def test_pre_conv():
  388.     n_classes = 21
  389.     model = PredictionConvolutions(n_classes)
  390.     conv4_3feats = torch.rand((10, 512, 38, 38))
  391.     conv7_feats = torch.rand((10, 1024, 19, 19))
  392.     conv8_2feats = torch.rand((10, 512, 10, 10))
  393.     conv9_2feats = torch.rand((10, 256, 5, 5))
  394.     conv10_2feats = torch.rand((10, 256, 3, 3))
  395.     conv11_2feats = torch.rand((10, 256, 1, 1))
  396.     locs, class_scores = model(conv4_3feats, conv7_feats, conv8_2feats, conv9_2feats, conv10_2feats, conv11_2feats)
  397.     # print(loc_conv4_3.shape, loc_conv7.shape, loc_conv8_2.shape, loc_conv9_2.shape,
  398.     #       loc_conv10_2.shape, loc_conv11_2.shape,\
  399.     #            cl_conv4_3.shape, cl_conv7.shape, cl_conv8_2.shape, cl_conv9_2.shape,
  400.     #       cl_conv10_2.shape, cl_conv11_2.shape)
  401.     print(locs.shape)
  402.     print(class_scores.shape)
  403. def test_SSD300():
  404.     os.environ["CUDA_VISIBLE_DEVICES"] = '0'
  405.  
  406.     n_classes = 21
  407.     model = SSD300(n_classes)
  408.     print('==model', model)
  409.     x = torch.rand((10, 3, 300, 300))
  410.     locs, class_scores = model(x)
  411.     print('locs.shape', locs.shape)
  412.     print('class_scores.shape', class_scores.shape)
  413.  
  414. def test_mutiboxloss():
  415.     prior_boxes = create_prior_boxes()
  416.     loss_model = MultiBoxLoss(prior_boxes)
  417.     prediction_locs = torch.rand(2, 8732, 4)
  418.     prediction_scores = torch.rand(2, 8732, 21)
  419.     boxes = [torch.tensor([[0.1040, 0.1946, 0.9400, 0.9480],
  420.              [0.3140, 0.0973, 0.5760, 0.3756]]).to(device),
  421.              torch.tensor([[0.0000, 0.6107, 0.8540, 0.7787]]).to(device)]
  422.     labels = [torch.tensor([13,  15]).to(device),
  423.               torch.tensor([4]).to(device)]
  424.     # boxes = torch.tensor([[[1, 2, 3, 4]],
  425.     #                       [[7, 8, 9, 10],
  426.     #                       [4, 5, 6, 7]]])
  427.     # labels = torch.tensor([[1],
  428.     #                     [1, 3]])
  429.     loss_sclar = loss_model(prediction_locs, prediction_scores, boxes, labels)
  430.     print('==loss_sclar',loss_sclar)
  431.  
  432. def create_prior_boxes():
  433.     """创建SSD300的先验框(cx, cy, w, h)
  434.         (8prediction_locs, prediction_scores, boxes, labels372,4)个box"""
  435.     os.environ["CUDA_VISIBLE_DEVICES"] = '0'
  436.     device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
  437.     from math import sqrt
  438.     fmap_size = {'conv4_3':38, 'conv7':19, 'conv8_2':10,
  439.                  'conv9_2':5, 'conv10_2':3, 'conv11_2':1}
  440.     anchor_scale = {'conv4_3':0.1,'conv7':0.2,'conv8_2':0.375,
  441.                  'conv9_2':0.55,'conv10_2':0.725,'conv11_2':0.9}
  442.     anchor_ratio = {'conv4_3':[1,2,0.5], 'conv7':[1,2,3,0.5,0.33], 'conv8_2':[1,2,3,0.5,0.33],
  443.                    'conv9_2':[1,2,3,0.5,0.33], 'conv10_2':[1,2,0.5], 'conv11_2':[1,2,0.5]}
  444.     prior_boxes = []
  445.     for index,fmap in enumerate(fmap_size):
  446.         for i in range(fmap_size[fmap]):
  447.             for j in range(fmap_size[fmap]):
  448.                 cy,cx = (i+0.5)/fmap_size[fmap], (j+0.5)/fmap_size[fmap]
  449.                 for ratio in anchor_ratio[fmap]:
  450.                     prior_boxes.append([cx, cy, anchor_scale[fmap]*sqrt(ratio), anchor_scale[fmap]/sqrt(ratio)])
  451.  
  452.                     if ratio==1:#添加额外框
  453.                         try:
  454.                             extra_scale = sqrt(anchor_scale[fmap]*anchor_scale[fmap_size[index+1]])
  455.                         except:
  456.                             extra_scale = 1.
  457.                         prior_boxes.append([cx, cy, extra_scale, extra_scale])
  458.         # print('len(prior_boxes)',len(prior_boxes))
  459.     # prior_boxes = [[1,2,3,4],
  460.     #                [3,4,5,6]]
  461.     prior_boxes = torch.FloatTensor(prior_boxes).to(device)
  462.     prior_boxes.clamp_(0,1)#防止越界
  463.     print('prior_boxes.shape', prior_boxes.shape)
  464.     # print(prior_boxes)
  465.     return prior_boxes
  466.  
  467. def decimate(tensor, m):
  468.     """
  469.     Decimate a tensor by a factor 'm', i.e. downsample by keeping every 'm'th value.
  471.     This is used when we convert FC layers to equivalent Convolutional layers, BUT of a smaller size.
  473.     :param tensor: tensor to be decimated
  474.     :param m: list of decimation factors for each dimension of the tensor; None if not to be decimated along a dimension
  475.     :return: decimated tensor
  476.     """
  477.     assert tensor.dim() == len(m)
  478.     for d in range(tensor.dim()):
  479.         if m[d] is not None:
  480.             tensor = tensor.index_select(dim=d,
  481.                                          index=torch.arange(start=0, end=tensor.size(d), step=m[d]).long())
  482.             # print('==tensor.shape:', tensor.shape)
  483.     return tensor
  484. def test_fc_conv():
  485.     """fc (4096,25088)-->conv (1024,512,3,3)"""
  486.     fc_weight_init = torch.rand(4096, 25088)
  487.     fc_weight = fc_weight_init.reshape(4096, 512, 7, 7)
  488.     m = [4, None, 3, 3]
  489.     conv_weight = decimate(fc_weight, m)
  490.     print('==conv_weight.shape', conv_weight.shape)
  491.  
  492. def index_select():
  493.     x = torch.linspace(1, 12, steps=12, requires_grad=True).reshape(3, 4)
  494.     print('==x', x)
  495.     print(x.dtype)
  496.     print(x.data)
  497.     print(x.data.dtype)
  498.     # indices = torch.LongTensor([0, 2])
  499.     # y = torch.index_select(x, 0, indices)  # 对行操作
  500.     # print('==y', y)
  501.     #
  502.     # z = torch.index_select(x, 1, indices)  # 对列操作
  503.     # print('==z', z)
  504.     #
  505.     # z = torch.index_select(y, 1, indices)  # 对列操作
  506.     # print('==z', z)
  507.  
  508.  
  509. if __name__ == '__main__':
  510.     os.environ["CUDA_VISIBLE_DEVICES"] = '0'
  511.     # test_vgg_base()
  512.     # test_AUx_conv()
  513.     # test_pre_conv()
  514.     # test_fc_conv()
  515.     # index_select()
  516.     # create_prior_boxes()
  517.     # test_SSD300()
  518.     test_mutiboxloss()
  519.  
  520.  
  521.  
  522.

二.多尺度训练与测试

1.多尺度训练

目的：用不同的尺度去帮助模型适应各种大小的目标，获得对尺寸的鲁棒性。一般是每个batch随机选择一个合适的尺度进行训练即可．

2.多尺度测试

2.1 one-stage 多尺度测试

对单个尺度的结果先进行NMS,在resize成同一个尺度大小在进行一次NMS.先对单个尺度结果进行NMS可以减少推理时间．

2.2 two-stage 多尺度测试

(1) 不同尺度图，通过Backbone+RPN和各自的NMS之后，会得到各自的proposals。再把尺度统一到同一张图的大小上去，然后合并到一起做阈值为0.7的NMS，得到Proposals。

(2) R-CNN阶段依然希望用多尺度，所以需要把proposals分别resize到橙色和绿色的图的尺寸上去，然后各自过R-CNN。后面的步骤与RPN和one stage是一样的，先各自做NMS，然后Resize到统一尺寸后再合并做阈值为0.5的NMS。

参考：
https://mp.weixin.qq.com/s/lBhPjOiT_05WXwxFCXj2mQ

本文内容由网友自发贡献，转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号