devbox
小蓝xlanll
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

PaddleDetection代码解析之Transformer encoder源码实现分析（下）_attn_bias_layer

作者：小蓝xlanll | 2024-04-09 20:01:47

赞

踩

attn_bias_layer

2021SC@SDUSC

接我的上一篇博客：PaddleDetection代码解析之Transformer encoder源码实现分析（上）。

依旧是对代码的解释在下面代码的注释中：

MultiHeadAttentionLayer

几个维度：

self._emb_size = config['hidden_size']   # 768

d_key=self._emb_size // self._n_head,

d_value=self._emb_size // self._n_head,

d_model=self._emb_size,

d_inner_hid=self._emb_size * 4

得到q、k、v的矩阵：

  
        q = self._q_fc(queries)
        k = self._k_fc(keys)
        v = self._v_fc(values)
  1. class MultiHeadAttentionLayer(Layer):
  2.     """
  3.     MultiHeadAttentionLayer
  4.     """
  5.  
  6.     def __init__(self,
  7.                  d_key,
  8.                  d_value,
  9.                  d_model,
  10.                  n_head=1,
  11.                  dropout_rate=0.,
  12.                  cache=None,
  13.                  gather_idx=None,
  14.                  static_kv=False,
  15.                  param_initializer=None,
  16.                  name=""):
  17.         super(MultiHeadAttentionLayer, self).__init__()
  18.         self._n_head = n_head
  19.         self._d_key = d_key
  20.         self._d_value = d_value
  21.         self._d_model = d_model
  22.         self._dropout_rate = dropout_rate
  23.  
  24.         self._q_fc = Linear(
  25.             input_dim=d_model,
  26.             output_dim=d_key * n_head,
  27.             param_attr=fluid.ParamAttr(
  28.                 name=name + '_query_fc.w_0', initializer=param_initializer),
  29.             bias_attr=name + '_query_fc.b_0')
  30.  
  31.         self._k_fc = Linear(
  32.             input_dim=d_model,
  33.             output_dim=d_key * n_head,
  34.             param_attr=fluid.ParamAttr(
  35.                 name=name + '_key_fc.w_0', initializer=param_initializer),
  36.             bias_attr=name + '_key_fc.b_0')
  37.  
  38.         self._v_fc = Linear(
  39.             input_dim=d_model,
  40.             output_dim=d_value * n_head,
  41.             param_attr=fluid.ParamAttr(
  42.                 name=name + '_value_fc.w_0', initializer=param_initializer),
  43.             bias_attr=name + '_value_fc.b_0')
  44.  
  45.         self._proj_fc = Linear(
  46.             input_dim=d_value * n_head,
  47.             output_dim=d_model,
  48.             param_attr=fluid.ParamAttr(
  49.                 name=name + '_output_fc.w_0', initializer=param_initializer),
  50.             bias_attr=name + '_output_fc.b_0')
  51.  
  52.     def forward(self, queries, keys, values, attn_bias):
  53.         """
  54.         forward
  55.         :param queries:
  56.         :param keys:
  57.         :param values:
  58.         :param attn_bias:
  59.         :return:
  60.         """
  61.         # compute q ,k ,v
  62.         keys = queries if keys is None else keys
  63.         values = keys if values is None else values
  64.         # 得到q k v 矩阵
  65.         q = self._q_fc(queries)
  66.         k = self._k_fc(keys)
  67.         v = self._v_fc(values)
  68.  
  69.         # split head
  70.  
  71.         q_hidden_size = q.shape[-1]     
  72.         eshaped_q = fluid.layers.reshape(
  73.             x=q,
  74.             shape=[0, 0, self._n_head, q_hidden_size // self._n_head],
  75.             inplace=False)
  76.         transpose_q = fluid.layers.transpose(x=reshaped_q, perm=[0, 2, 1, 3])
  77.  
  78.         k_hidden_size = k.shape[-1]
  79.         reshaped_k = fluid.layers.reshape(
  80.             x=k,
  81.             shape=[0, 0, self._n_head, k_hidden_size // self._n_head],
  82.             inplace=False)
  83.         transpose_k = fluid.layers.transpose(x=reshaped_k, perm=[0, 2, 1, 3])
  84.  
  85.         v_hidden_size = v.shape[-1]
  86.         reshaped_v = fluid.layers.reshape(
  87.             x=v,
  88.             shape=[0, 0, self._n_head, v_hidden_size // self._n_head],
  89.             inplace=False)
  90.         transpose_v = fluid.layers.transpose(x=reshaped_v, perm=[0, 2, 1, 3])
  91.  
  92.         scaled_q = fluid.layers.scale(x=transpose_q, scale=self._d_key**-0.5)
  93.         # scale dot product attention
  94.         product = fluid.layers.matmul(
  95.             #x=transpose_q,
  96.             x=scaled_q,
  97.             y=transpose_k,
  98.             transpose_y=True)
  99.         #alpha=self._d_model**-0.5)
  100.         if attn_bias:
  101.             product += attn_bias
  102.         weights = fluid.layers.softmax(product)
  103.         if self._dropout_rate:
  104.             weights_droped = fluid.layers.dropout(
  105.                 weights,
  106.                 dropout_prob=self._dropout_rate,
  107.                 dropout_implementation="upscale_in_train",
  108.                 is_test=False)
  109.             out = fluid.layers.matmul(weights_droped, transpose_v)
  110.         else:       
  111.                 out = fluid.layers.matmul(weights, transpose_v)
  112.  
  113.         # combine heads
  114.         if len(out.shape) != 4:
  115.             raise ValueError("Input(x) should be a 4-D Tensor.")
  116.         trans_x = fluid.layers.transpose(out, perm=[0, 2, 1, 3])
  117.         final_out = fluid.layers.reshape(
  118.             x=trans_x,
  119.             shape=[0, 0, trans_x.shape[2] * trans_x.shape[3]],
  120.             inplace=False)
  121.  
  122.         # fc to output
  123.         proj_out = self._proj_fc(final_out)
  124.         return proj_out

EncoderSubLayer

预处理：

 
        self._preprocess_layer = PrePostProcessLayer(
            self._preprocess_cmd,
            d_model,
            prepostprocess_dropout,
            name=name + "_pre_att")

多头注意力：

        
        self._multihead_attention_layer = MultiHeadAttentionLayer(
            d_key,
            d_value,
            d_model,
            n_head,
            attention_dropout,
            None,
            None,
            False,
            param_initializer,
            name=name + "_multi_head_att")

  1. class EncoderSubLayer(Layer):
  2.     """
  3.     EncoderSubLayer
  4.     """
  5.  
  6.     def __init__(self,
  7.                  hidden_act,
  8.                  n_head,
  9.                  d_key,
  10.                  d_value,
  11.                  d_model,
  12.                  d_inner_hid,
  13.                  prepostprocess_dropout,
  14.                  attention_dropout,
  15.                  relu_dropout,
  16.                  preprocess_cmd="n",
  17.                  postprocess_cmd="da",
  18.                  param_initializer=None,
  19.                  name=""):
  20.  
  21.         super(EncoderSubLayer, self).__init__()
  22.         self.name = name
  23.         self._preprocess_cmd = preprocess_cmd
  24.         self._postprocess_cmd = postprocess_cmd
  25.         self._prepostprocess_dropout = prepostprocess_dropout
  26.         # 预处理
  27.         self._preprocess_layer = PrePostProcessLayer(
  28.             self._preprocess_cmd,
  29.             d_model,
  30.             prepostprocess_dropout,
  31.             name=name + "_pre_att")
  32.         # 多头注意力
  33.         self._multihead_attention_layer = MultiHeadAttentionLayer(
  34.             d_key,
  35.             d_value,
  36.             d_model,
  37.             n_head,
  38.             attention_dropout,
  39.             None,
  40.             None,
  41.             False,
  42.             param_initializer,
  43.             name=name + "_multi_head_att")
  44.  
  45.         self._postprocess_layer = PrePostProcessLayer(
  46.             self._postprocess_cmd,
  47.             d_model,
  48.             self._prepostprocess_dropout,
  49.             name=name + "_post_att")
  50.         self._preprocess_layer2 = PrePostProcessLayer(
  51.             self._preprocess_cmd,
  52.             d_model,
  53.             self._prepostprocess_dropout,
  54.             name=name + "_pre_ffn")
  55.  
  56.         self._positionwise_feed_forward = PositionwiseFeedForwardLayer(
  57.             hidden_act,
  58.             d_inner_hid,
  59.             d_model,
  60.             relu_dropout,
  61.             param_initializer,
  62.             name=name + "_ffn")
  63.  
  64.         self._postprocess_layer2 = PrePostProcessLayer(
  65.             self._postprocess_cmd,
  66.             d_model,
  67.             self._prepostprocess_dropout,
  68.             name=name + "_post_ffn")
  69.  
  70.     def forward(self, enc_input, attn_bias):
  71.         """
  72.         forward
  73.         :param enc_input: encoder 输入
  74.         :param attn_bias: attention 偏置
  75.         :return: 一层encoder encode输入之后的结果
  76.         """
  77.         # 在进行多头attention前，先进行预处理
  78.         pre_process_multihead = self._preprocess_layer(enc_input)
  79.         # 预处理之后的结果给到多头attention层
  80.         attn_output = self._multihead_attention_layer(pre_process_multihead,
  81.                                                       None, None, attn_bias)
  82.         # 经过attention之后进行后处理
  83.         attn_output = self._postprocess_layer(attn_output, enc_input)
  84.         # 在给到FFN层前进行预处理
  85.         pre_process2_output = self._preprocess_layer2(attn_output)
  86.         # 得到FFN层的结果
  87.         ffd_output = self._positionwise_feed_forward(pre_process2_output)
  88.         # 返回后处理后的结果
  89.         return self._postprocess_layer2(ffd_output, attn_output)

EncoderLayer

# 使用add_sublayer方法添加子层：

                self.add_sublayer(
                    'esl_%d' % i,
                    EncoderSubLayer(
                        hidden_act,
                        n_head,
                        d_key,
                        d_value,
                        d_model,
                        d_inner_hid,
                        prepostprocess_dropout,
                        attention_dropout,
                        relu_dropout,
                        preprocess_cmd,
                        postprocess_cmd,
                        param_initializer,
                        name=name + '_layer_' + str(i))))
  1. class EncoderLayer(Layer):
  2.     """
  3.     encoder
  4.     """
  5.  
  6.     def __init__(self,
  7.                  hidden_act,
  8.                  n_layer, # encoder子层数量 / encoder深度
  9.                  n_head, # 注意力机制中head数量
  10.                  d_key,
  11.                  d_value,
  12.                  d_model,
  13.                  d_inner_hid,
  14.                  prepostprocess_dropout, # 处理层的dropout概率
  15.                  attention_dropout, # attention层的dropout概率
  16.                  relu_dropout, # 激活函数层的dropout概率
  17.                  preprocess_cmd="n", # 前处理，正则化
  18.                  postprocess_cmd="da", # 后处理，dropout + 残差连接
  19.                  param_initializer=None,
  20.                  name=""):
  21.  
  22.         super(EncoderLayer, self).__init__()
  23.         self._preprocess_cmd = preprocess_cmd
  24.         self._encoder_sublayers = list()
  25.         self._prepostprocess_dropout = prepostprocess_dropout
  26.         self._n_layer = n_layer
  27.         self._hidden_act = hidden_act
  28.         # 后处理层，这里是层正则化
  29.         self._preprocess_layer = PrePostProcessLayer(
  30.             self._preprocess_cmd, 3, self._prepostprocess_dropout,
  31.             "post_encoder")
  32.         # 根据n_layer的设置（bert_base中是12）迭代定义几个encoder子层
  33.         for i in range(n_layer):
  34.             self._encoder_sublayers.append(
  35.                 # 使用add_sublayer方法添加子层
  36.                 self.add_sublayer(
  37.                     'esl_%d' % i,
  38.                     EncoderSubLayer(
  39.                         hidden_act,
  40.                         n_head,
  41.                         d_key,
  42.                         d_value,
  43.                         d_model,
  44.                         d_inner_hid,
  45.                         prepostprocess_dropout,
  46.                         attention_dropout,
  47.                         relu_dropout,
  48.                         preprocess_cmd,
  49.                         postprocess_cmd,
  50.                         param_initializer,
  51.                         name=name + '_layer_' + str(i))))
  52.  
  53.     def forward(self, enc_input, attn_bias):
  54.         """
  55.         forward
  56.         :param enc_input: 模型输入
  57.         :param attn_bias: bias项可根据具体情况选择是否保留
  58.         :return: encode之后的结果
  59.         """
  60.         # 迭代多个encoder子层，例如 bert base 的encoder子层数为12(self._n_layer)
  61.         for i in range(self._n_layer):
  62.             # 得到子层的输出，参数为 enc_input, attn_bias
  63.             enc_output = self._encoder_sublayers[i](enc_input, attn_bias)
  64.             # 该子层的输出作为下一子层的输入
  65.             enc_input = enc_output
  66.         # 返回处理过的层
  67.         return self._preprocess_layer(enc_output)
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/小蓝xlanll/article/detail/394867
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号