devbox
Cpp五条
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

深度强化学习（DRL）算法 附录 6 —— NLP 回顾之预训练模型篇

作者：Cpp五条 | 2024-04-14 08:37:19

赞

踩

深度强化学习（DRL）算法 附录 6 —— NLP 回顾之预训练模型篇

Self-Attention

模型结构

上图架构以 batch_size 为 1，两个时间步的 X 为例子，计算过程如下：

位置编码

根据 self-attention 的模型结构，改变 X 的输入顺序，不影响 attention 的结果，所以还需要引入额外的位置信息，即位置编码。

图里计算机二进制编码的低位和位置编码矩阵的前面几列对应。

除了上面捕获绝对位置信息之外，上述的位置编码还允许模型学习得到输入序列中相对位置信息。 这是因为对于任何确定的位置偏移δ，位置 i+δ 处的位置编码可以线性投影位置 i 处的位置编码来表示。

\begin{aligned} & {\left[\begin{array}{cc} \cos \left(\delta \omega_j\right) & \sin \left(\delta \omega_j\right) \\ -\sin \left(\delta \omega_j\right) & \cos \left(\delta \omega_j\right) \end{array}\right]\left[\begin{array}{c} p_{i, 2 j} \\ p_{i, 2 j+1} \end{array}\right] } \\ = & {\left[\begin{array}{c} \cos \left(\delta \omega_j\right) \sin \left(i \omega_j\right)+\sin \left(\delta \omega_j\right) \cos \left(i \omega_j\right) \\ -\sin \left(\delta \omega_j\right) \sin \left(i \omega_j\right)+\cos \left(\delta \omega_j\right) \cos \left(i \omega_j\right) \end{array}\right] } \\ = & {\left[\begin{array}{l} \sin \left((i+\delta) \omega_j\right) \\ \cos \left((i+\delta) \omega_j\right) \end{array}\right] } \\ = & {\left[\begin{array}{c} p_{i+\delta, 2 j} \\ p_{i+\delta, 2 j+1} \end{array}\right] } \end{aligned}

代码

  1. #@save
  2. class PositionalEncoding(nn.Module):
  3.     """位置编码"""
  4.     def __init__(self, num_hiddens, dropout, max_len=1000):
  5.         super(PositionalEncoding, self).__init__()
  6.         self.dropout = nn.Dropout(dropout)
  7.         # 创建一个足够长的P
  8.         self.P = torch.zeros((1, max_len, num_hiddens))
  9.         X = torch.arange(max_len, dtype=torch.float32).reshape(
  10.             -1, 1) / torch.pow(10000, torch.arange(
  11.             0, num_hiddens, 2, dtype=torch.float32) / num_hiddens)
  12.         self.P[:, :, 0::2] = torch.sin(X)
  13.         self.P[:, :, 1::2] = torch.cos(X)
  14.  
  15.     def forward(self, X):
  16.         X = X + self.P[:, :X.shape[1], :].to(X.device)
  17.         return self.dropout(X)

多头注意力

模型结构

  • 两头注意力

  • 七头注意力

添加图片注释，不超过 140 字（可选）

  • 七头注意力连接进行信息融合

  • 掩码多头注意力

和多头注意力是一样的，只不过每个头的 self-attention 变成了 masked self-attention

代码

  1. import math
  2. import torch
  3. from torch import nn
  4. from d2l import torch as d2l
  5.  
  6. #@save
  7. def transpose_qkv(X, num_heads):
  8.     """为了多注意力头的并行计算而变换形状"""
  9.     # 输入X的形状:(batch_size，查询或者“键－值”对的个数，num_hiddens)
  10.     # 输出X的形状:(batch_size，查询或者“键－值”对的个数，num_heads，
  11.     # num_hiddens/num_heads)
  12.     X = X.reshape(X.shape[0], X.shape[1], num_heads, -1)
  13.  
  14.     # 输出X的形状:(batch_size，num_heads，查询或者“键－值”对的个数,
  15.     # num_hiddens/num_heads)
  16.     X = X.permute(0, 2, 1, 3)
  17.  
  18.     # 最终输出的形状:(batch_size*num_heads,查询或者“键－值”对的个数,
  19.     # num_hiddens/num_heads)
  20.     return X.reshape(-1, X.shape[2], X.shape[3])
  21.  
  22.  
  23. #@save
  24. def transpose_output(X, num_heads):
  25.     """逆转transpose_qkv函数的操作"""
  26.     X = X.reshape(-1, num_heads, X.shape[1], X.shape[2])
  27.     X = X.permute(0, 2, 1, 3)
  28.     return X.reshape(X.shape[0], X.shape[1], -1)
  29.  
  30. #@save
  31. class DotProductAttention(nn.Module):
  32.     """Scaled dot product attention.
  33.     Defined in :numref:`subsec_additive-attention`"""
  34.     def __init__(self, dropout, **kwargs):
  35.         super(DotProductAttention, self).__init__(**kwargs)
  36.         self.dropout = nn.Dropout(dropout)
  37.  
  38.     # Shape of `queries`: (`batch_size`, no. of queries, `d`)
  39.     # Shape of `keys`: (`batch_size`, no. of key-value pairs, `d`)
  40.     # Shape of `values`: (`batch_size`, no. of key-value pairs, value
  41.     # dimension)
  42.     # Shape of `valid_lens`: (`batch_size`,) or (`batch_size`, no. of queries)
  43.     def forward(self, queries, keys, values, valid_lens=None):
  44.         d = queries.shape[-1]
  45.         # Set `transpose_b=True` to swap the last two dimensions of `keys`
  46.         scores = torch.bmm(queries, keys.transpose(1,2)) / math.sqrt(d)
  47.         self.attention_weights = masked_softmax(scores, valid_lens)
  48.         return torch.bmm(self.dropout(self.attention_weights), values)
  49.  
  50. #@save
  51. class MultiHeadAttention(nn.Module):
  52.     """多头注意力"""
  53.     def __init__(self, key_size, query_size, value_size, num_hiddens,
  54.                  num_heads, dropout, bias=False, **kwargs):
  55.         super(MultiHeadAttention, self).__init__(**kwargs)
  56.         self.num_heads = num_heads
  57.         self.attention = d2l.DotProductAttention(dropout)
  58.         self.W_q = nn.Linear(query_size, num_hiddens, bias=bias)
  59.         self.W_k = nn.Linear(key_size, num_hiddens, bias=bias)
  60.         self.W_v = nn.Linear(value_size, num_hiddens, bias=bias)
  61.         self.W_o = nn.Linear(num_hiddens, num_hiddens, bias=bias)
  62.  
  63.     def forward(self, queries, keys, values, valid_lens):
  64.         # queries，keys，values的形状:
  65.         # (batch_size，查询或者“键－值”对的个数，num_hiddens)
  66.         # valid_lens　的形状:
  67.         # (batch_size，)或(batch_size，查询的个数)
  68.         # 经过变换后，输出的queries，keys，values　的形状:
  69.         # (batch_size*num_heads，查询或者“键－值”对的个数，
  70.         # num_hiddens/num_heads)
  71.         queries = transpose_qkv(self.W_q(queries), self.num_heads)
  72.         keys = transpose_qkv(self.W_k(keys), self.num_heads)
  73.         values = transpose_qkv(self.W_v(values), self.num_heads)
  74.  
  75.         if valid_lens is not None:
  76.             # 在轴0，将第一项（标量或者矢量）复制num_heads次，
  77.             # 然后如此复制第二项，然后诸如此类。
  78.             valid_lens = torch.repeat_interleave(
  79.                 valid_lens, repeats=self.num_heads, dim=0)
  80.  
  81.         # output的形状:(batch_size*num_heads，查询的个数，
  82.         # num_hiddens/num_heads)
  83.         output = self.attention(queries, keys, values, valid_lens)
  84.  
  85.         # output_concat的形状:(batch_size，查询的个数，num_hiddens)
  86.         output_concat = transpose_output(output, self.num_heads)
  87.         return self.W_o(output_concat)

Transformer

模型结构

encoder

decoder

代码

  1. import math
  2. import pandas as pd
  3. import torch
  4. from torch import nn
  5. from d2l import torch as d2l
  6.  
  7. #@save
  8. class PositionWiseFFN(nn.Module):
  9.     """基于位置的前馈网络"""
  10.     def __init__(self, ffn_num_input, ffn_num_hiddens, ffn_num_outputs,
  11.                  **kwargs):
  12.         super(PositionWiseFFN, self).__init__(**kwargs)
  13.         self.dense1 = nn.Linear(ffn_num_input, ffn_num_hiddens)
  14.         self.relu = nn.ReLU()
  15.         self.dense2 = nn.Linear(ffn_num_hiddens, ffn_num_outputs)
  16.  
  17.     def forward(self, X):
  18.         return self.dense2(self.relu(self.dense1(X)))
  19.  
  20. #@save
  21. class AddNorm(nn.Module):
  22.     """残差连接后进行层规范化"""
  23.     def __init__(self, normalized_shape, dropout, **kwargs):
  24.         super(AddNorm, self).__init__(**kwargs)
  25.         self.dropout = nn.Dropout(dropout)
  26.         self.ln = nn.LayerNorm(normalized_shape)
  27.  
  28.     def forward(self, X, Y):
  29.         return self.ln(self.dropout(Y) + X)
  30.  
  31. #@save
  32. class EncoderBlock(nn.Module):
  33.     """Transformer编码器块"""
  34.     def __init__(self, key_size, query_size, value_size, num_hiddens,
  35.                  norm_shape, ffn_num_input, ffn_num_hiddens, num_heads,
  36.                  dropout, use_bias=False, **kwargs):
  37.         super(EncoderBlock, self).__init__(**kwargs)
  38.         self.attention = d2l.MultiHeadAttention(
  39.             key_size, query_size, value_size, num_hiddens, num_heads, dropout,
  40.             use_bias)
  41.         self.addnorm1 = AddNorm(norm_shape, dropout)
  42.         self.ffn = PositionWiseFFN(
  43.             ffn_num_input, ffn_num_hiddens, num_hiddens)
  44.         self.addnorm2 = AddNorm(norm_shape, dropout)
  45.  
  46.     def forward(self, X, valid_lens):
  47.         Y = self.addnorm1(X, self.attention(X, X, X, valid_lens))
  48.         return self.addnorm2(Y, self.ffn(Y))
  49.  
  50. #@save
  51. class TransformerEncoder(d2l.Encoder):
  52.     """Transformer编码器"""
  53.     def __init__(self, vocab_size, key_size, query_size, value_size,
  54.                  num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens,
  55.                  num_heads, num_layers, dropout, use_bias=False, **kwargs):
  56.         super(TransformerEncoder, self).__init__(**kwargs)
  57.         self.num_hiddens = num_hiddens
  58.         self.embedding = nn.Embedding(vocab_size, num_hiddens)
  59.         self.pos_encoding = d2l.PositionalEncoding(num_hiddens, dropout)
  60.         self.blks = nn.Sequential()
  61.         for i in range(num_layers):
  62.             self.blks.add_module("block"+str(i),
  63.                 EncoderBlock(key_size, query_size, value_size, num_hiddens,
  64.                              norm_shape, ffn_num_input, ffn_num_hiddens,
  65.                              num_heads, dropout, use_bias))
  66.  
  67.     def forward(self, X, valid_lens, *args):
  68.         # 因为位置编码值在-1和1之间，
  69.         # 因此嵌入值乘以嵌入维度的平方根进行缩放，
  70.         # 然后再与位置编码相加。
  71.         X = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens))
  72.         self.attention_weights = [None] * len(self.blks)
  73.         for i, blk in enumerate(self.blks):
  74.             X = blk(X, valid_lens)
  75.             self.attention_weights[
  76.                 i] = blk.attention.attention.attention_weights
  77.         return X
  78.  
  79. class DecoderBlock(nn.Module):
  80.     """解码器中第i个块"""
  81.     def __init__(self, key_size, query_size, value_size, num_hiddens,
  82.                  norm_shape, ffn_num_input, ffn_num_hiddens, num_heads,
  83.                  dropout, i, **kwargs):
  84.         super(DecoderBlock, self).__init__(**kwargs)
  85.         self.i = i
  86.         self.attention1 = d2l.MultiHeadAttention(
  87.             key_size, query_size, value_size, num_hiddens, num_heads, dropout)
  88.         self.addnorm1 = AddNorm(norm_shape, dropout)
  89.         self.attention2 = d2l.MultiHeadAttention(
  90.             key_size, query_size, value_size, num_hiddens, num_heads, dropout)
  91.         self.addnorm2 = AddNorm(norm_shape, dropout)
  92.         self.ffn = PositionWiseFFN(ffn_num_input, ffn_num_hiddens,
  93.                                    num_hiddens)
  94.         self.addnorm3 = AddNorm(norm_shape, dropout)
  95.  
  96.     def forward(self, X, state):
  97.         enc_outputs, enc_valid_lens = state[0], state[1]
  98.         # 训练阶段，输出序列的所有词元都在同一时间处理，
  99.         # 因此state[2][self.i]初始化为None。
  100.         # 预测阶段，输出序列是通过词元一个接着一个解码的，
  101.         # 因此state[2][self.i]包含着直到当前时间步第i个块解码的输出表示
  102.         if state[2][self.i] is None:
  103.             key_values = X
  104.         else:
  105.             key_values = torch.cat((state[2][self.i], X), axis=1)
  106.         state[2][self.i] = key_values
  107.         if self.training:
  108.             batch_size, num_steps, _ = X.shape
  109.             # dec_valid_lens的开头:(batch_size,num_steps),
  110.             # 其中每一行是[1,2,...,num_steps]
  111.             dec_valid_lens = torch.arange(
  112.                 1, num_steps + 1, device=X.device).repeat(batch_size, 1)
  113.         else:
  114.             dec_valid_lens = None
  115.  
  116.         # 自注意力
  117.         X2 = self.attention1(X, key_values, key_values, dec_valid_lens)
  118.         Y = self.addnorm1(X, X2)
  119.         # 编码器－解码器注意力。
  120.         # enc_outputs的开头:(batch_size,num_steps,num_hiddens)
  121.         Y2 = self.attention2(Y, enc_outputs, enc_outputs, enc_valid_lens)
  122.         Z = self.addnorm2(Y, Y2)
  123.         return self.addnorm3(Z, self.ffn(Z)), state
  124.  
  125. class TransformerDecoder(d2l.AttentionDecoder):
  126.     def __init__(self, vocab_size, key_size, query_size, value_size,
  127.                  num_hiddens, norm_shape, ffn_num_input, ffn_num_hiddens,
  128.                  num_heads, num_layers, dropout, **kwargs):
  129.         super(TransformerDecoder, self).__init__(**kwargs)
  130.         self.num_hiddens = num_hiddens
  131.         self.num_layers = num_layers
  132.         self.embedding = nn.Embedding(vocab_size, num_hiddens)
  133.         self.pos_encoding = d2l.PositionalEncoding(num_hiddens, dropout)
  134.         self.blks = nn.Sequential()
  135.         for i in range(num_layers):
  136.             self.blks.add_module("block"+str(i),
  137.                 DecoderBlock(key_size, query_size, value_size, num_hiddens,
  138.                              norm_shape, ffn_num_input, ffn_num_hiddens,
  139.                              num_heads, dropout, i))
  140.         self.dense = nn.Linear(num_hiddens, vocab_size)
  141.  
  142.     def init_state(self, enc_outputs, enc_valid_lens, *args):
  143.         return [enc_outputs, enc_valid_lens, [None] * self.num_layers]
  144.  
  145.     def forward(self, X, state):
  146.         X = self.pos_encoding(self.embedding(X) * math.sqrt(self.num_hiddens))
  147.         self._attention_weights = [[None] * len(self.blks) for _ in range (2)]
  148.         for i, blk in enumerate(self.blks):
  149.             X, state = blk(X, state)
  150.             # 解码器自注意力权重
  151.             self._attention_weights[0][
  152.                 i] = blk.attention1.attention.attention_weights
  153.             # “编码器－解码器”自注意力权重
  154.             self._attention_weights[1][
  155.                 i] = blk.attention2.attention.attention_weights
  156.         return self.dense(X), state
  157.  
  158.     @property
  159.     def attention_weights(self):
  160.         return self._attention_weights
  161.  
  162. num_hiddens, num_layers, dropout, batch_size, num_steps = 32, 2, 0.1, 64, 10
  163. lr, num_epochs, device = 0.005, 200, d2l.try_gpu()
  164. ffn_num_input, ffn_num_hiddens, num_heads = 32, 64, 4
  165. key_size, query_size, value_size = 32, 32, 32
  166. norm_shape = [32]
  167.  
  168. train_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)
  169.  
  170. encoder = TransformerEncoder(
  171.     len(src_vocab), key_size, query_size, value_size, num_hiddens,
  172.     norm_shape, ffn_num_input, ffn_num_hiddens, num_heads,
  173.     num_layers, dropout)
  174. decoder = TransformerDecoder(
  175.     len(tgt_vocab), key_size, query_size, value_size, num_hiddens,
  176.     norm_shape, ffn_num_input, ffn_num_hiddens, num_heads,
  177.     num_layers, dropout)
  178. net = d2l.EncoderDecoder(encoder, decoder)
  179. d2l.train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)

Bert

bert 开启了预训练模型的风潮，使用了带掩码的语言模型，具体就是通过大量的数据，模型获取了语言信息抽取的能力，从而可以通过 fine-tune 应用到各种 NLP 任务上。

3w 的词典，使用了 WordPiece。[cls] A [seq] B [seq]

位置嵌入换成了学习的矩阵。

模型结构

截取了 transformer 的 encoder（代码没有改动）

不同点：

  • 输入

  • 训练（类似完形填空，以及下一个句子预测）

尽管掩蔽语言建模能够编码双向上下文来表示单词，但它不能显式地建模文本对之间的逻辑关系。为了帮助理解两个文本序列之间的关系，BERT在预训练中考虑了一个二元分类任务——下一句预测。在为预训练生成句子对时，有一半的时间它们确实是标签为“真”的连续句子；在另一半的时间里，第二个句子是从语料库中随机抽取的，标记为“假”。

模型参数计算

BERT-base（H = 768，L = 12，A = 12）

Transformer encoder block 里面主要参数有：

  1. 嵌入层：H x 30000（vocab_size 约等于 30000）

2. 全连接层：H x 4H + 4H x H（一个 block 里面有两个全连接层）

3. 多头注意力机制层：H x H / head_num x 3（一个头的参数，3代表 Q，K，V 用不同矩阵做线性变换），所有头加起来 H x H x 3，再加上多头注意力机制层的线性变换 H x H，这里可以结合下图理解：

添加图片注释，不超过 140 字（可选）

1，2，3 加起来就是 BERT-base 的参数数量。

计算公式： L*12H^{2} + 30000*H \approx 110M (H=768, L=12)

BERT-large 同理可以计算出参数数量约等于 340M。

GPT-3

截取了 transformer 的 decoder（代码没有改动）

参考

51 序列模型【动手学深度学习v2】-跟李沐学AI-【完结】动手学深度学习 PyTorch版-哔哩哔哩视频

8.1. 序列模型 - 动手学深度学习 2.0.0 documentation

Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)

The Illustrated Transformer

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/Cpp五条/article/detail/421144
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号