【注意力】ESE：Effective Squeeze-and-Excitation Block

原文：https://arxiv.org/pdf/1911.06667.pdf

 ESE(Effective Squeeze and Extraction) layer是模型中的一个block，基于SE（Squeeze and Extraction）而来。与SE的区别在于，ESE block只有一个fc层，《CenterMask : Real-Time Anchor-Free Instance Segmentation》的作者注意到SE模块有一个缺点：由于维度的减少导致的通道信息损失。为了避免这种大模型的计算负担，se的2个fc层需要减少通道维度。特别的，当第一个fc层使用r减少输入特征通道，将通道数从c变为c/r的时候，第二个fc层又需要扩张减少的通道数到原始的通道c.在这个过程中，通道维度的减少导致了通道信息的损失。因而，effective SE(eSE)仅仅使用一个通道数为c的fc层代替了两个fc层，避免了通道信息DE丢失；

代码：

def get_act_fn(act=None, trt=False):
    assert act is None or isinstance(act, (
        str, dict)), 'name of activation should be str, dict or None'
    if not act:
        return identity
 
    if isinstance(act, dict):
        name = act['name']
        act.pop('name')
        kwargs = act
    else:
        name = act
        kwargs = dict()
 
    if trt and name in TRT_ACT_SPEC:
        fn = TRT_ACT_SPEC[name]
    elif name in ACT_SPEC:
        fn = ACT_SPEC[name]
    else:
        fn = getattr(F, name)
 
    return lambda x: fn(x, **kwargs)
 
class EffectiveSELayer(nn.Layer):
    """ Effective Squeeze-Excitation
    From `CenterMask : Real-Time Anchor-Free Instance Segmentation` - https://arxiv.org/abs/1911.06667
    """
 
    def __init__(self, channels, act='hardsigmoid'):
        super(EffectiveSELayer, self).__init__()
        self.fc = nn.Conv2D(channels, channels, kernel_size=1, padding=0)
        self.act = get_act_fn(act) if act is None or isinstance(act, (
            str, dict)) else act
 
    def forward(self, x):
        x_se = x.mean((2, 3), keepdim=True)
        x_se = self.fc(x_se)
        return x * self.act(x_se)

代码摘自pp-yoloe（https://github.com/PaddlePaddle/PaddleDetection）