devbox
从前慢现在也慢
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

YOLOv8改进之C2f-DySnakeConv（动态蛇形卷积Dynamic Snake Convolution）_dysnakeconv论文

作者：从前慢现在也慢 | 2024-06-12 21:55:39

赞

踩

dysnakeconv论文

一、动态蛇形卷积Dynamic Snake Convolution论文

论文地址：2307.08388.pdf (arxiv.org)

二、动态蛇形卷积的简要介绍

   动态蛇形卷积（Dynamic Snake Convolution）的设计灵感来源于蛇形的形状，用于改善对目标形状和边界的敏感性。能够帮助神经网络更好地捕捉目标的形状信息，特别是对于复杂的或不规则形状的目标。通过引入动态的、可变形的卷积核来实现这一目标。这种可变形的卷积核能够根据目标的形状和边界信息进行调整，从而更好地适应目标的特定形状。

   传统的卷积操作在处理目标形状变化较大的情况下可能存在一定的局限性，而动态蛇形卷积则能够通过自适应性地调整卷积核的形状和大小，更有效地捕获目标的特征。

    这种模块的应用通常能够增强目标检测模型对不同尺度、形状和姿态的目标的感知能力，从而提高目标检测的准确性和鲁棒性。虽然这只是目标检测中的一种模块，但它代表了在深度学习领域中不断创新和改进的努力，以提高模型对复杂场景的理解能力。

三、算法实现

 1、在官方的yolov8包中ultralytics\ultralytics\nn的目录下新创建一个文件夹other_modules，用于存放额外添加的改进模块。

在other_modules文件夹下创建dynamic_snake_conv.py文件，存放dynamic_snake_conv动态蛇形卷积代码：

  1. import torch
  2. import torch.nn as nn
  3. from ..modules.conv import Conv
  4.  
  5. __all__ = ['DySnakeConv']
  7. class DySnakeConv(nn.Module):
  8.     def __init__(self, inc, ouc, k=3) -> None:
  9.         super().__init__()
  10.         
  11.         self.conv_0 = Conv(inc, ouc, k)
  12.         self.conv_x = DSConv(inc, ouc, 0, k)
  13.         self.conv_y = DSConv(inc, ouc, 1, k)
  14.     
  15.     def forward(self, x):
  16.         return torch.cat([self.conv_0(x), self.conv_x(x), self.conv_y(x)], dim=1)
  18. class DSConv(nn.Module):
  19.     def __init__(self, in_ch, out_ch, morph, kernel_size=3, if_offset=True, extend_scope=1):
  20.         """
  21.         The Dynamic Snake Convolution
  22.         :param in_ch: input channel
  23.         :param out_ch: output channel
  24.         :param kernel_size: the size of kernel
  25.         :param extend_scope: the range to expand (default 1 for this method)
  26.         :param morph: the morphology of the convolution kernel is mainly divided into two types
  27.                         along the x-axis (0) and the y-axis (1) (see the paper for details)
  28.         :param if_offset: whether deformation is required, if it is False, it is the standard convolution kernel
  29.         """
  30.         super(DSConv, self).__init__()
  31.         # use the <offset_conv> to learn the deformable offset
  32.         self.offset_conv = nn.Conv2d(in_ch, 2 * kernel_size, 3, padding=1)
  33.         self.bn = nn.BatchNorm2d(2 * kernel_size)
  34.         self.kernel_size = kernel_size
  36.         # two types of the DSConv (along x-axis and y-axis)
  37.         self.dsc_conv_x = nn.Conv2d(
  38.             in_ch,
  39.             out_ch,
  40.             kernel_size=(kernel_size, 1),
  41.             stride=(kernel_size, 1),
  42.             padding=0,
  43.         )
  44.         self.dsc_conv_y = nn.Conv2d(
  45.             in_ch,
  46.             out_ch,
  47.             kernel_size=(1, kernel_size),
  48.             stride=(1, kernel_size),
  49.             padding=0,
  50.         )
  52.         self.gn = nn.GroupNorm(out_ch // 4, out_ch)
  53.         self.act = Conv.default_act
  55.         self.extend_scope = extend_scope
  56.         self.morph = morph
  57.         self.if_offset = if_offset
  59.     def forward(self, f):
  60.         offset = self.offset_conv(f)
  61.         offset = self.bn(offset)
  62.         # We need a range of deformation between -1 and 1 to mimic the snake's swing
  63.         offset = torch.tanh(offset)
  64.         input_shape = f.shape
  65.         dsc = DSC(input_shape, self.kernel_size, self.extend_scope, self.morph)
  66.         deformed_feature = dsc.deform_conv(f, offset, self.if_offset)
  67.         if self.morph == 0:
  68.             x = self.dsc_conv_x(deformed_feature.type(f.dtype))
  69.             x = self.gn(x)
  70.             x = self.act(x)
  71.             return x
  72.         else:
  73.             x = self.dsc_conv_y(deformed_feature.type(f.dtype))
  74.             x = self.gn(x)
  75.             x = self.act(x)
  76.             return x
  79. # Core code, for ease of understanding, we mark the dimensions of input and output next to the code
  80. class DSC(object):
  81.     def __init__(self, input_shape, kernel_size, extend_scope, morph):
  82.         self.num_points = kernel_size
  83.         self.width = input_shape[2]
  84.         self.height = input_shape[3]
  85.         self.morph = morph
  86.         self.extend_scope = extend_scope  # offset (-1 ~ 1) * extend_scope
  88.         # define feature map shape
  89.         """
  90.         B: Batch size  C: Channel  W: Width  H: Height
  91.         """
  92.         self.num_batch = input_shape[0]
  93.         self.num_channels = input_shape[1]
  95.     """
  96.     input: offset [B,2*K,W,H]  K: Kernel size (2*K: 2D image, deformation contains <x_offset> and <y_offset>)
  97.     output_x: [B,1,W,K*H]   coordinate map
  98.     output_y: [B,1,K*W,H]   coordinate map
  99.     """
  101.     def _coordinate_map_3D(self, offset, if_offset):
  102.         device = offset.device
  103.         # offset
  104.         y_offset, x_offset = torch.split(offset, self.num_points, dim=1)
  106.         y_center = torch.arange(0, self.width).repeat([self.height])
  107.         y_center = y_center.reshape(self.height, self.width)
  108.         y_center = y_center.permute(1, 0)
  109.         y_center = y_center.reshape([-1, self.width, self.height])
  110.         y_center = y_center.repeat([self.num_points, 1, 1]).float()
  111.         y_center = y_center.unsqueeze(0)
  113.         x_center = torch.arange(0, self.height).repeat([self.width])
  114.         x_center = x_center.reshape(self.width, self.height)
  115.         x_center = x_center.permute(0, 1)
  116.         x_center = x_center.reshape([-1, self.width, self.height])
  117.         x_center = x_center.repeat([self.num_points, 1, 1]).float()
  118.         x_center = x_center.unsqueeze(0)
  120.         if self.morph == 0:
  121.             """
  122.             Initialize the kernel and flatten the kernel
  123.                 y: only need 0
  124.                 x: -num_points//2 ~ num_points//2 (Determined by the kernel size)
  125.                 !!! The related PPT will be submitted later, and the PPT will contain the whole changes of each step
  126.             """
  127.             y = torch.linspace(0, 0, 1)
  128.             x = torch.linspace(
  129.                 -int(self.num_points // 2),
  130.                 int(self.num_points // 2),
  131.                 int(self.num_points),
  132.             )
  134.             y, x = torch.meshgrid(y, x)
  135.             y_spread = y.reshape(-1, 1)
  136.             x_spread = x.reshape(-1, 1)
  138.             y_grid = y_spread.repeat([1, self.width * self.height])
  139.             y_grid = y_grid.reshape([self.num_points, self.width, self.height])
  140.             y_grid = y_grid.unsqueeze(0)  # [B*K*K, W,H]
  142.             x_grid = x_spread.repeat([1, self.width * self.height])
  143.             x_grid = x_grid.reshape([self.num_points, self.width, self.height])
  144.             x_grid = x_grid.unsqueeze(0)  # [B*K*K, W,H]
  146.             y_new = y_center + y_grid
  147.             x_new = x_center + x_grid
  149.             y_new = y_new.repeat(self.num_batch, 1, 1, 1).to(device)
  150.             x_new = x_new.repeat(self.num_batch, 1, 1, 1).to(device)
  152.             y_offset_new = y_offset.detach().clone()
  154.             if if_offset:
  155.                 y_offset = y_offset.permute(1, 0, 2, 3)
  156.                 y_offset_new = y_offset_new.permute(1, 0, 2, 3)
  157.                 center = int(self.num_points // 2)
  159.                 # The center position remains unchanged and the rest of the positions begin to swing
  160.                 # This part is quite simple. The main idea is that "offset is an iterative process"
  161.                 y_offset_new[center] = 0
  162.                 for index in range(1, center):
  163.                     y_offset_new[center + index] = (y_offset_new[center + index - 1] + y_offset[center + index])
  164.                     y_offset_new[center - index] = (y_offset_new[center - index + 1] + y_offset[center - index])
  165.                 y_offset_new = y_offset_new.permute(1, 0, 2, 3).to(device)
  166.                 y_new = y_new.add(y_offset_new.mul(self.extend_scope))
  168.             y_new = y_new.reshape(
  169.                 [self.num_batch, self.num_points, 1, self.width, self.height])
  170.             y_new = y_new.permute(0, 3, 1, 4, 2)
  171.             y_new = y_new.reshape([
  172.                 self.num_batch, self.num_points * self.width, 1 * self.height
  173.             ])
  174.             x_new = x_new.reshape(
  175.                 [self.num_batch, self.num_points, 1, self.width, self.height])
  176.             x_new = x_new.permute(0, 3, 1, 4, 2)
  177.             x_new = x_new.reshape([
  178.                 self.num_batch, self.num_points * self.width, 1 * self.height
  179.             ])
  180.             return y_new, x_new
  182.         else:
  183.             """
  184.             Initialize the kernel and flatten the kernel
  185.                 y: -num_points//2 ~ num_points//2 (Determined by the kernel size)
  186.                 x: only need 0
  187.             """
  188.             y = torch.linspace(
  189.                 -int(self.num_points // 2),
  190.                 int(self.num_points // 2),
  191.                 int(self.num_points),
  192.             )
  193.             x = torch.linspace(0, 0, 1)
  195.             y, x = torch.meshgrid(y, x)
  196.             y_spread = y.reshape(-1, 1)
  197.             x_spread = x.reshape(-1, 1)
  199.             y_grid = y_spread.repeat([1, self.width * self.height])
  200.             y_grid = y_grid.reshape([self.num_points, self.width, self.height])
  201.             y_grid = y_grid.unsqueeze(0)
  203.             x_grid = x_spread.repeat([1, self.width * self.height])
  204.             x_grid = x_grid.reshape([self.num_points, self.width, self.height])
  205.             x_grid = x_grid.unsqueeze(0)
  207.             y_new = y_center + y_grid
  208.             x_new = x_center + x_grid
  210.             y_new = y_new.repeat(self.num_batch, 1, 1, 1)
  211.             x_new = x_new.repeat(self.num_batch, 1, 1, 1)
  213.             y_new = y_new.to(device)
  214.             x_new = x_new.to(device)
  215.             x_offset_new = x_offset.detach().clone()
  217.             if if_offset:
  218.                 x_offset = x_offset.permute(1, 0, 2, 3)
  219.                 x_offset_new = x_offset_new.permute(1, 0, 2, 3)
  220.                 center = int(self.num_points // 2)
  221.                 x_offset_new[center] = 0
  222.                 for index in range(1, center):
  223.                     x_offset_new[center + index] = (x_offset_new[center + index - 1] + x_offset[center + index])
  224.                     x_offset_new[center - index] = (x_offset_new[center - index + 1] + x_offset[center - index])
  225.                 x_offset_new = x_offset_new.permute(1, 0, 2, 3).to(device)
  226.                 x_new = x_new.add(x_offset_new.mul(self.extend_scope))
  228.             y_new = y_new.reshape(
  229.                 [self.num_batch, 1, self.num_points, self.width, self.height])
  230.             y_new = y_new.permute(0, 3, 1, 4, 2)
  231.             y_new = y_new.reshape([
  232.                 self.num_batch, 1 * self.width, self.num_points * self.height
  233.             ])
  234.             x_new = x_new.reshape(
  235.                 [self.num_batch, 1, self.num_points, self.width, self.height])
  236.             x_new = x_new.permute(0, 3, 1, 4, 2)
  237.             x_new = x_new.reshape([
  238.                 self.num_batch, 1 * self.width, self.num_points * self.height
  239.             ])
  240.             return y_new, x_new
  242.     """
  243.     input: input feature map [N,C,D,W,H]；coordinate map [N,K*D,K*W,K*H] 
  244.     output: [N,1,K*D,K*W,K*H]  deformed feature map
  245.     """
  246.     def _bilinear_interpolate_3D(self, input_feature, y, x):
  247.         device = input_feature.device
  248.         y = y.reshape([-1]).float()
  249.         x = x.reshape([-1]).float()
  251.         zero = torch.zeros([]).int()
  252.         max_y = self.width - 1
  253.         max_x = self.height - 1
  255.         # find 8 grid locations
  256.         y0 = torch.floor(y).int()
  257.         y1 = y0 + 1
  258.         x0 = torch.floor(x).int()
  259.         x1 = x0 + 1
  261.         # clip out coordinates exceeding feature map volume
  262.         y0 = torch.clamp(y0, zero, max_y)
  263.         y1 = torch.clamp(y1, zero, max_y)
  264.         x0 = torch.clamp(x0, zero, max_x)
  265.         x1 = torch.clamp(x1, zero, max_x)
  267.         input_feature_flat = input_feature.flatten()
  268.         input_feature_flat = input_feature_flat.reshape(
  269.             self.num_batch, self.num_channels, self.width, self.height)
  270.         input_feature_flat = input_feature_flat.permute(0, 2, 3, 1)
  271.         input_feature_flat = input_feature_flat.reshape(-1, self.num_channels)
  272.         dimension = self.height * self.width
  274.         base = torch.arange(self.num_batch) * dimension
  275.         base = base.reshape([-1, 1]).float()
  277.         repeat = torch.ones([self.num_points * self.width * self.height
  278.                              ]).unsqueeze(0)
  279.         repeat = repeat.float()
  281.         base = torch.matmul(base, repeat)
  282.         base = base.reshape([-1])
  284.         base = base.to(device)
  286.         base_y0 = base + y0 * self.height
  287.         base_y1 = base + y1 * self.height
  289.         # top rectangle of the neighbourhood volume
  290.         index_a0 = base_y0 - base + x0
  291.         index_c0 = base_y0 - base + x1
  293.         # bottom rectangle of the neighbourhood volume
  294.         index_a1 = base_y1 - base + x0
  295.         index_c1 = base_y1 - base + x1
  297.         # get 8 grid values
  298.         value_a0 = input_feature_flat[index_a0.type(torch.int64)].to(device)
  299.         value_c0 = input_feature_flat[index_c0.type(torch.int64)].to(device)
  300.         value_a1 = input_feature_flat[index_a1.type(torch.int64)].to(device)
  301.         value_c1 = input_feature_flat[index_c1.type(torch.int64)].to(device)
  303.         # find 8 grid locations
  304.         y0 = torch.floor(y).int()
  305.         y1 = y0 + 1
  306.         x0 = torch.floor(x).int()
  307.         x1 = x0 + 1
  309.         # clip out coordinates exceeding feature map volume
  310.         y0 = torch.clamp(y0, zero, max_y + 1)
  311.         y1 = torch.clamp(y1, zero, max_y + 1)
  312.         x0 = torch.clamp(x0, zero, max_x + 1)
  313.         x1 = torch.clamp(x1, zero, max_x + 1)
  315.         x0_float = x0.float()
  316.         x1_float = x1.float()
  317.         y0_float = y0.float()
  318.         y1_float = y1.float()
  320.         vol_a0 = ((y1_float - y) * (x1_float - x)).unsqueeze(-1).to(device)
  321.         vol_c0 = ((y1_float - y) * (x - x0_float)).unsqueeze(-1).to(device)
  322.         vol_a1 = ((y - y0_float) * (x1_float - x)).unsqueeze(-1).to(device)
  323.         vol_c1 = ((y - y0_float) * (x - x0_float)).unsqueeze(-1).to(device)
  325.         outputs = (value_a0 * vol_a0 + value_c0 * vol_c0 + value_a1 * vol_a1 +
  326.                    value_c1 * vol_c1)
  328.         if self.morph == 0:
  329.             outputs = outputs.reshape([
  330.                 self.num_batch,
  331.                 self.num_points * self.width,
  332.                 1 * self.height,
  333.                 self.num_channels,
  334.             ])
  335.             outputs = outputs.permute(0, 3, 1, 2)
  336.         else:
  337.             outputs = outputs.reshape([
  338.                 self.num_batch,
  339.                 1 * self.width,
  340.                 self.num_points * self.height,
  341.                 self.num_channels,
  342.             ])
  343.             outputs = outputs.permute(0, 3, 1, 2)
  344.         return outputs
  346.     def deform_conv(self, input, offset, if_offset):
  347.         y, x = self._coordinate_map_3D(offset, if_offset)
  348.         deformed_feature = self._bilinear_interpolate_3D(input, y, x)
  349.         return deformed_feature

在 ultralytics\ultralytics\nn\other_modules\block.py文件中开头声明动态蛇卷积：

  1. from ultralytics.nn.modules import Conv
  2. from .dynamic_snake_conv import DySnakeConv

并在__all__中添加：

然后还是在这个block文件中添加C2f DySnakeConv代码：

  1. #######      添加 C2f DySnakeConv       ##########
  2. class Bottleneck_DySnakeConv(Bottleneck):
  3.     """Standard bottleneck with DySnakeConv."""
  4.  
  5.     def __init__(self, c1, c2, shortcut=True, g=1, k=(3, 3), e=0.5):  # ch_in, ch_out, shortcut, groups, kernels, expand
  6.         super().__init__(c1, c2, shortcut, g, k, e)
  7.         c_ = int(c2 * e)  # hidden channels
  8.         self.cv2 = DySnakeConv(c_, c2, k[1])
  9.         self.cv3 = Conv(c2 * 3, c2, k=1)
  10.  
  11.     def forward(self, x):
  12.         """'forward()' applies the YOLOv5 FPN to input data."""
  13.         return x + self.cv3(self.cv2(self.cv1(x))) if self.add else self.cv3(self.cv2(self.cv1(x)))
  14.  
  15.  
  16. class C2f_DySnakeConv(C2f):
  17.     def __init__(self, c1, c2, n=1, shortcut=False, g=1, e=0.5):
  18.         super().__init__(c1, c2, n, shortcut, g, e)
  19.         self.m = nn.ModuleList(Bottleneck_DySnakeConv(self.c, self.c, shortcut, g, k=(3, 3), e=1.0) for _ in range(n))

2、在ultralytics\ultralytics\nn\tasks.py文件里修改

首先在该文件开头引入上一步创建的文件夹other_modules：

  1. from ultralytics.nn.other_modules.block import C2f_DySnakeConv
  2. from ultralytics.nn.other_modules import *

然后在下边的然后def parse_model模块中添加warehouse_manager：在括号里面添加一个参数warehouse_manager：

def parse_model(d, ch, verbose=True, warehouse_manager=None):

 然后在该模块的下边加入C2f_DySnakeConv的运行算法，将原来程序中的685-694行中的内容：

修改为加入C2f_DySnakeConv的运行算法：

  1.         if m in (Classify, Conv, ConvTranspose, GhostConv, Bottleneck, GhostBottleneck, SPP, SPPF, DWConv, Focus,
  2.                  BottleneckCSP, C1, C2, C2f, C3, C3TR, C3Ghost, nn.ConvTranspose2d, DWConvTranspose2d, C3x, RepC3,
  3.                  C2f_DBB,C2f_DySnakeConv):
  4.             if args[0] == 'head_channel':
  5.                 args[0] = d[args[0]]
  6.             c1, c2 = ch[f], args[0]
  7.             if c2 != nc:  # if c2 not equal to number of classes (i.e. for Classify() output)
  8.                 c2 = make_divisible(min(c2, max_channels) * width, 8)
  9.  
  10.             args = [c1, c2, *args[1:]]
  11.  
  12.  
  13.             if m in (
  14.             BottleneckCSP, C1, C2, C2f, C3, C3TR, C3Ghost, C3x, RepC3, C2f_DBB,C2f_DySnakeConv):
  15.                 args.insert(2, n)  # number of repeats
  16.                 n = 1

3、创建YOLOv8+C2f-DySnakeConv的yaml文件 ：

  1. # Ultralytics YOLO 
    声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/从前慢现在也慢/article/detail/709785
    推荐阅读
    相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。