devbox
正经夜光杯
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

【YOLOv5/v7改进系列】替换特征融合网络为AFPN_特征融合方式 afpn

作者：正经夜光杯 | 2024-07-23 09:50:12

赞

踩

特征融合方式 afpn
一、导言

介绍了一种新的特征金字塔网络结构——渐近特征金字塔网络（Asymptotic Feature Pyramid Network, AFPN），旨在解决目标检测任务中多尺度特征提取的问题，特别是非相邻层级间特征信息的损失和退化。以下是AFPN的主要优点：

  1. 直接交互非相邻层级：AFPN通过渐进的方式使低层特征与高层特征直接融合，从一开始就融合两个不同分辨率的低层特征，并逐渐加入更多高层特征，直至融合骨干网络的顶层特征。这种设计避免了传统方法中因间接交互导致的非相邻层级间较大的语义差距。

  2. 保持细节与语义信息：在融合过程中，低层特征与高层的语义信息结合，同时高层特征也融入来自低层的详细信息。这确保了检测对象的特征既包含丰富的细节也具备充分的语义理解，有助于提高检测精度。

  3. 自适应空间融合操作：为了解决不同层级特征在同一位置可能存在的信息冲突问题，AFPN引入了自适应空间融合操作。这一操作能够筛选出多级融合中的有益信息，避免简单的元素相加可能导致的信息混乱，提高了特征融合的有效性。

  4. 性能提升与效率优化：实验结果显示，将AFPN应用于Faster R-CNN框架时，在MS COCO 2017数据集上，使用ResNet-50和ResNet-101作为主干网络分别获得了1.6%和2.6%的性能提升。此外，与最先进的特征金字塔网络相比，AFPN不仅具有更优的检测性能，还实现了最低的计算量（以FLOPs衡量）。在YOLOv5这样的单阶段检测器上，AFPN同样表现出色，提升了性能的同时减少了参数数量。

  5. 广泛适用性：研究证明AFPN在两阶段和一阶段检测框架上均能显著提升检测性能，显示了其方法的通用性和灵活性。

  6. 资源效率：虽然AFPN的参数数量比基础的FPN增加了21.0%，但它在所有比较方法中具有最低的GFLOPs，这主要归功于特征维度的降低，表明了其在资源利用上的高效性。

综上所述，AFPN通过创新的渐近融合策略和自适应空间融合技术，有效解决了多尺度特征融合中的关键问题，为对象检测任务提供了更为高效、准确的解决方案。

二、准备工作

首先在YOLOv5/v7的models文件夹下新建文件afpnv.py，导入如下代码

  1. from models.common import *
  2.  
  3. class Upsample(nn.Module):
  4.     """Applies convolution followed by upsampling."""
  5.  
  6.     def __init__(self, c1, c2, scale_factor=2):
  7.         super().__init__()
  8.         if scale_factor == 2:
  9.             self.cv1 = nn.ConvTranspose2d(c1, c2, 2, 2, 0, bias=True)  # nn.Upsample(scale_factor=2, mode='nearest')
  10.         elif scale_factor == 4:
  11.             self.cv1 = nn.ConvTranspose2d(c1, c2, 4, 4, 0, bias=True)  # nn.Upsample(scale_factor=4, mode='nearest')
  12.  
  13.     def forward(self, x):
  14.         # return self.upsample(self.cv1(x))
  15.         return self.cv1(x)
  16.  
  17.  
  18. class ASFF2(nn.Module):
  19.     """ASFF2 module for YOLO AFPN head https://arxiv.org/abs/2306.15988"""
  20.  
  21.     def __init__(self, c1, c2, level=0):
  22.         super().__init__()
  23.         c1_l, c1_h = c1[0], c1[1]
  24.         self.level = level
  25.         self.dim = c1_l, c1_h
  26.         self.inter_dim = self.dim[self.level]
  27.         compress_c = 8
  28.  
  29.         if level == 0:
  30.             self.stride_level_1 = Upsample(c1_h, self.inter_dim)
  31.         if level == 1:
  32.             self.stride_level_0 = Conv(c1_l, self.inter_dim, 2, 2, 0)  # downsample 2x
  33.  
  34.         self.weight_level_0 = Conv(self.inter_dim, compress_c, 1, 1)
  35.         self.weight_level_1 = Conv(self.inter_dim, compress_c, 1, 1)
  36.  
  37.         self.weights_levels = nn.Conv2d(compress_c * 2, 2, kernel_size=1, stride=1, padding=0)
  38.         self.conv = Conv(self.inter_dim, self.inter_dim, 3, 1)
  39.  
  40.     def forward(self, x):
  41.         x_level_0, x_level_1 = x[0], x[1]
  42.  
  43.         if self.level == 0:
  44.             level_0_resized = x_level_0
  45.             level_1_resized = self.stride_level_1(x_level_1)
  46.         elif self.level == 1:
  47.             level_0_resized = self.stride_level_0(x_level_0)
  48.             level_1_resized = x_level_1
  49.  
  50.         level_0_weight_v = self.weight_level_0(level_0_resized)
  51.         level_1_weight_v = self.weight_level_1(level_1_resized)
  52.         levels_weight_v = torch.cat((level_0_weight_v, level_1_weight_v), 1)
  53.         levels_weight = self.weights_levels(levels_weight_v)
  54.         levels_weight = F.softmax(levels_weight, dim=1)
  55.  
  56.         fused_out_reduced = level_0_resized * levels_weight[:, 0:1] + level_1_resized * levels_weight[:, 1:2]
  57.         return self.conv(fused_out_reduced)
  58.  
  59.  
  60. class ASFF3(nn.Module):
  61.     """ASFF3 module for YOLO AFPN head https://arxiv.org/abs/2306.15988"""
  62.  
  63.     def __init__(self, c1, c2, level=0):
  64.         super().__init__()
  65.         c1_l, c1_m, c1_h = c1[0], c1[1], c1[2]
  66.         self.level = level
  67.         self.dim = c1_l, c1_m, c1_h
  68.         self.inter_dim = self.dim[self.level]
  69.         compress_c = 8
  70.  
  71.         if level == 0:
  72.             self.stride_level_1 = Upsample(c1_m, self.inter_dim)
  73.             self.stride_level_2 = Upsample(c1_h, self.inter_dim, scale_factor=4)
  74.  
  75.         if level == 1:
  76.             self.stride_level_0 = Conv(c1_l, self.inter_dim, 2, 2, 0)  # downsample 2x
  77.             self.stride_level_2 = Upsample(c1_h, self.inter_dim)
  78.  
  79.         if level == 2:
  80.             self.stride_level_0 = Conv(c1_l, self.inter_dim, 4, 4, 0)  # downsample 4x
  81.             self.stride_level_1 = Conv(c1_m, self.inter_dim, 2, 2, 0)  # downsample 2x
  82.  
  83.         self.weight_level_0 = Conv(self.inter_dim, compress_c, 1, 1)
  84.         self.weight_level_1 = Conv(self.inter_dim, compress_c, 1, 1)
  85.         self.weight_level_2 = Conv(self.inter_dim, compress_c, 1, 1)
  86.  
  87.         self.weights_levels = nn.Conv2d(compress_c * 3, 3, kernel_size=1, stride=1, padding=0)
  88.         self.conv = Conv(self.inter_dim, self.inter_dim, 3, 1)
  89.  
  90.     def forward(self, x):
  91.         x_level_0, x_level_1, x_level_2 = x[0], x[1], x[2]
  92.  
  93.         if self.level == 0:
  94.             level_0_resized = x_level_0
  95.             level_1_resized = self.stride_level_1(x_level_1)
  96.             level_2_resized = self.stride_level_2(x_level_2)
  97.  
  98.         elif self.level == 1:
  99.             level_0_resized = self.stride_level_0(x_level_0)
  100.             level_1_resized = x_level_1
  101.             level_2_resized = self.stride_level_2(x_level_2)
  102.  
  103.         elif self.level == 2:
  104.             level_0_resized = self.stride_level_0(x_level_0)
  105.             level_1_resized = self.stride_level_1(x_level_1)
  106.             level_2_resized = x_level_2
  107.  
  108.         level_0_weight_v = self.weight_level_0(level_0_resized)
  109.         level_1_weight_v = self.weight_level_1(level_1_resized)
  110.         level_2_weight_v = self.weight_level_2(level_2_resized)
  111.  
  112.         levels_weight_v = torch.cat((level_0_weight_v, level_1_weight_v, level_2_weight_v), 1)
  113.         w = self.weights_levels(levels_weight_v)
  114.         w = F.softmax(w, dim=1)
  115.  
  116.         fused_out_reduced = level_0_resized * w[:, :1] + level_1_resized * w[:, 1:2] + level_2_resized * w[:, 2:]
  117.         return self.conv(fused_out_reduced)

其次在在YOLOv5/v7项目文件下的models/yolo.py中在文件首部添加代码

from models.afpn import *

并搜索def parse_model(d, ch)

定位到如下行添加以下代码

  1.         elif m is ASFF2:
  2.             c1, c2 = [ch[f[0]], ch[f[1]]], args[0]
  3.             c2 = make_divisible(c2 * gw, 8)
  4.             args = [c1, c2, *args[1:]]
  5.         elif m is ASFF3:
  6.             c1, c2 = [ch[f[0]], ch[f[1]], ch[f[2]]], args[0]
  7.             c2 = make_divisible(c2 * gw, 8)
  8.             args = [c1, c2, *args[1:]]
三、YOLOv7-tiny改进工作

完成二后，在YOLOv7项目文件下的models文件夹下创建新的文件yolov7-tiny-afpn.yaml，导入如下代码。

  1. # parameters
  2. nc: 80  # number of classes
  3. depth_multiple: 1.0  # model depth multiple
  4. width_multiple: 1.0  # layer channel multiple
  5.  
  6. # anchors
  7. anchors:
  8.   - [10,13, 16,30, 33,23]  # P3/8
  9.   - [30,61, 62,45, 59,119]  # P4/16
  10.   - [116,90, 156,198, 373,326]  # P5/32
  11.  
  12. # yolov7-tiny backbone
  13. backbone:
  14.   # [from, number, module, args] c2, k=1, s=1, p=None, g=1, act=True
  15.   [[-1, 1, Conv, [32, 3, 2, None, 1, nn.LeakyReLU(0.1)]],  # 0-P1/2
  16.  
  17.    [-1, 1, Conv, [64, 3, 2, None, 1, nn.LeakyReLU(0.1)]],  # 1-P2/4
  18.  
  19.    [-1, 1, Conv, [32, 1, 1, None, 1, nn.LeakyReLU(0.1)]],
  20.    [-2, 1, Conv, [32, 1, 1, None, 1, nn.LeakyReLU(0.1)]],
  21.    [-1, 1, Conv, [32, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  22.    [-1, 1, Conv, [32, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  23.    [[-1, -2, -3, -4], 1, Concat, [1]],
  24.    [-1, 1, Conv, [64, 1, 1, None, 1, nn.LeakyReLU(0.1)]],  # 7
  25.  
  26.    [-1, 1, MP, []],  # 8-P3/8
  27.    [-1, 1, Conv, [64, 1, 1, None, 1, nn.LeakyReLU(0.1)]],
  28.    [-2, 1, Conv, [64, 1, 1, None, 1, nn.LeakyReLU(0.1)]],
  29.    [-1, 1, Conv, [64, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  30.    [-1, 1, Conv, [64, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  31.    [[-1, -2, -3, -4], 1, Concat, [1]],
  32.    [-1, 1, Conv, [128, 1, 1, None, 1, nn.LeakyReLU(0.1)]],  # 14
  33.  
  34.    [-1, 1, MP, []],  # 15-P4/16
  35.    [-1, 1, Conv, [128, 1, 1, None, 1, nn.LeakyReLU(0.1)]],
  36.    [-2, 1, Conv, [128, 1, 1, None, 1, nn.LeakyReLU(0.1)]],
  37.    [-1, 1, Conv, [128, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  38.    [-1, 1, Conv, [128, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  39.    [[-1, -2, -3, -4], 1, Concat, [1]],
  40.    [-1, 1, Conv, [256, 1, 1, None, 1, nn.LeakyReLU(0.1)]],  # 21
  41.  
  42.    [-1, 1, MP, []],  # 22-P5/32
  43.    [-1, 1, Conv, [256, 1, 1, None, 1, nn.LeakyReLU(0.1)]],
  44.    [-2, 1, Conv, [256, 1, 1, None, 1, nn.LeakyReLU(0.1)]],
  45.    [-1, 1, Conv, [256, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  46.    [-1, 1, Conv, [256, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  47.    [[-1, -2, -3, -4], 1, Concat, [1]],
  48.    [-1, 1, Conv, [512, 1, 1, None, 1, nn.LeakyReLU(0.1)]],  # 28
  49.   ]
  50.  
  51. # yolov7-tiny head
  52. head:
  53.   [[14, 1, Conv, [64, 1, 1, None, 1, nn.LeakyReLU(0.1)]], # 29 downsample backbone P3
  54.    [21, 1, Conv, [128, 1, 1, None, 1, nn.LeakyReLU(0.1)]], # 30 downsample backbone P4
  55.  
  56.    [[29, 30], 1, ASFF2, [64, 0]], # 31
  57.    [[29, 30], 1, ASFF2, [128, 1]], # 32
  58.  
  59.    [-2, 1, Conv, [64, 3, 1, None, 1, nn.LeakyReLU(0.1)]], # 33
  60.    [-2, 1, Conv, [128, 3, 1, None, 1, nn.LeakyReLU(0.1)]], # 34
  61.  
  62.    [28, 1, Conv, [256, 1, 1, None, 1, nn.LeakyReLU(0.1)]], # 35 downsample backbone P5
  63.  
  64.    [[33, 34, 35], 1, ASFF3, [64, 0]], # 36
  65.    [[33, 34, 35], 1, ASFF3, [128, 1]], # 37
  66.    [[33, 34, 35], 1, ASFF3, [256, 2]], # 38
  67.  
  68.    [36, 1, Conv, [128, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  69.    [37, 1, Conv, [256, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  70.    [38, 1, Conv, [512, 3, 1, None, 1, nn.LeakyReLU(0.1)]],
  71.  
  72.   [[39, 40, 41], 1, IDetect, [nc, anchors]],  # Detect(P3, P4, P5)
  73.     ]
  74.  
  75.  
  1.  
  2.                  from  n    params  module                                  arguments                     
  3.   0                -1  1       928  models.common.Conv                      [3, 32, 3, 2, None, 1, LeakyReLU(negative_slope=0.1)]
  4.   1                -1  1     18560  models.common.Conv                      [32, 64, 3, 2, None, 1, LeakyReLU(negative_slope=0.1)]
  5.   2                -1  1      2112  models.common.Conv                      [64, 32, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  6.   3                -2  1      2112  models.common.Conv                      [64, 32, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  7.   4                -1  1      9280  models.common.Conv                      [32, 32, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  8.   5                -1  1      9280  models.common.Conv                      [32, 32, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  9.   6  [-1, -2, -3, -4]  1         0  models.common.Concat                    [1]                           
  10.   7                -1  1      8320  models.common.Conv                      [128, 64, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  11.   8                -1  1         0  models.common.MP                        []                            
  12.   9                -1  1      4224  models.common.Conv                      [64, 64, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  13.  10                -2  1      4224  models.common.Conv                      [64, 64, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  14.  11                -1  1     36992  models.common.Conv                      [64, 64, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  15.  12                -1  1     36992  models.common.Conv                      [64, 64, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  16.  13  [-1, -2, -3, -4]  1         0  models.common.Concat                    [1]                           
  17.  14                -1  1     33024  models.common.Conv                      [256, 128, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  18.  15                -1  1         0  models.common.MP                        []                            
  19.  16                -1  1     16640  models.common.Conv                      [128, 128, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  20.  17                -2  1     16640  models.common.Conv                      [128, 128, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  21.  18                -1  1    147712  models.common.Conv                      [128, 128, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  22.  19                -1  1    147712  models.common.Conv                      [128, 128, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  23.  20  [-1, -2, -3, -4]  1         0  models.common.Concat                    [1]                           
  24.  21                -1  1    131584  models.common.Conv                      [512, 256, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  25.  22                -1  1         0  models.common.MP                        []                            
  26.  23                -1  1     66048  models.common.Conv                      [256, 256, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  27.  24                -2  1     66048  models.common.Conv                      [256, 256, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  28.  25                -1  1    590336  models.common.Conv                      [256, 256, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  29.  26                -1  1    590336  models.common.Conv                      [256, 256, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  30.  27  [-1, -2, -3, -4]  1         0  models.common.Concat                    [1]                           
  31.  28                -1  1    525312  models.common.Conv                      [1024, 512, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  32.  29                14  1      8320  models.common.Conv                      [128, 64, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  33.  30                21  1     33024  models.common.Conv                      [256, 128, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  34.  31          [29, 30]  1     70914  models.afpn.ASFF2                       [[64, 128], 64, 0]            
  35.  32          [29, 30]  1    182850  models.afpn.ASFF2                       [[64, 128], 128, 1]           
  36.  33                -2  1     36992  models.common.Conv                      [64, 64, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  37.  34                -2  1    147712  models.common.Conv                      [128, 128, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  38.  35                28  1    131584  models.common.Conv                      [512, 256, 1, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  39.  36      [33, 34, 35]  1    333691  models.afpn.ASFF3                       [[64, 128, 256], 64, 0]       
  40.  37      [33, 34, 35]  1    315131  models.afpn.ASFF3                       [[64, 128, 256], 128, 1]      
  41.  38      [33, 34, 35]  1    990843  models.afpn.ASFF3                       [[64, 128, 256], 256, 2]      
  42.  39                36  1     73984  models.common.Conv                      [64, 128, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  43.  40                37  1    295424  models.common.Conv                      [128, 256, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  44.  41                38  1   1180672  models.common.Conv                      [256, 512, 3, 1, None, 1, LeakyReLU(negative_slope=0.1)]
  45.  42      [39, 40, 41]  1     17132  models.yolo.IDetect                     [1, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
  46.  
  47. Model Summary: 251 layers, 6282689 parameters, 6282689 gradients, 18.9 GFLOPS

运行后若打印出如上文本代表改进成功。

四、YOLOv5s改进工作

完成二后，在YOLOv5项目文件下的models文件夹下创建新的文件yolov5s-afpn.yaml，导入如下代码。

  1. # YOLOv5 
    声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/正经夜光杯/article/detail/869350
    推荐阅读
    相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。