devbox
你好赵伟
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

文本分类(二) | (4) 模型及其配置的定义_self.pad_size

作者：你好赵伟 | 2024-04-30 17:14:16

赞

踩

self.pad_size

完整项目​​​​​​​

本篇博客，主要介绍各个模型的模块定义，包括模型本身的定义以及模型对应的配置(超参数)的定义，二者在一个模块文件中。

目录

1. FastText

2. TextCNN

3. TextRNN

4. TextRCNN

5. TextRNN_Atten

6. DPCNN

7. Transformer

1. FastText

  • 配置类
  1. class Config(object):
  2.  
  3.     """FastText配置参数"""
  4.     def __init__(self, dataset, embedding):
  5.         self.model_name = 'FastText'
  6.         #训练集、验证集、测试集路径
  7.         self.train_path = dataset + '/data/train.txt'
  8.         self.dev_path = dataset + '/data/dev.txt'
  9.         self.test_path = dataset + '/data/test.txt'
  10.         #数据集的所有类别
  11.         self.class_list = [x.strip() for x in open(
  12.             dataset + '/data/class.txt').readlines()]
  13.         #构建好的词/字典路径
  14.         self.vocab_path = dataset + '/data/vocab.pkl'
  15.         #训练好的模型参数保存路径
  16.         self.save_path = dataset + '/saved_dict/' + self.model_name + '.ckpt'
  17.         #模型日志保存路径
  18.         self.log_path = dataset + '/log/' + self.model_name
  19.         #如果词/字嵌入矩阵不随机初始化 则加载初始化好的词/字嵌入矩阵 类别为float32 并转换为tensor 否则为None
  20.         self.embedding_pretrained = torch.tensor(
  21.             np.load(dataset + '/data/' + embedding)["embeddings"].astype('float32'))\
  22.             if embedding != 'random' else None
  23.         self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')   # 设备
  24.  
  25.         self.dropout = 0.5                                              # 随机失活 丢弃率
  26.         self.require_improvement = 1000                                 # 若超过1000batch效果还没提升，则提前结束训练
  27.         self.num_classes = len(self.class_list)                         # 类别数
  28.         self.n_vocab = 0                                                # 词表大小，在运行时赋值
  29.         self.num_epochs = 20                                            # epoch数
  30.         self.batch_size = 128                                           # mini-batch大小
  31.         self.pad_size = 32                                              # 每句话处理成的长度(短填长切)
  32.         self.learning_rate = 1e-3                                       # 学习率
  33.         self.embed = self.embedding_pretrained.size(1)\
  34.             if self.embedding_pretrained is not None else 300           # 字向量维度
  35.         self.hidden_size = 256                                          # 隐藏层大小
  36.         self.n_gram_vocab = 250499        #n-gram词表大小
  • 模型定义类
  1. class Model(nn.Module):
  2.     def __init__(self, config):
  3.         super(Model, self).__init__()
  4.  
  5.         if config.embedding_pretrained is not None: #加载初始化好的预训练词/字嵌入矩阵  微调funetuning
  6.             self.embedding = nn.Embedding.from_pretrained(config.embedding_pretrained, freeze=False)
  7.         else: #否则随机初始化词/字嵌入矩阵 指定填充对应的索引
  8.             self.embedding = nn.Embedding(config.n_vocab, config.embed, padding_idx=config.n_vocab - 1)
  9.         
  10.         #分别随机初始化 bi-gram tri-gram对应的词嵌入矩阵 
  11.         self.embedding_ngram2 = nn.Embedding(config.n_gram_vocab, config.embed)
  12.         self.embedding_ngram3 = nn.Embedding(config.n_gram_vocab, config.embed)
  13.         #dropout
  14.         self.dropout = nn.Dropout(config.dropout)
  15.         #隐层
  16.         self.fc1 = nn.Linear(config.embed * 3, config.hidden_size)
  17.         # self.dropout2 = nn.Dropout(config.dropout)
  18.         #输出层
  19.         self.fc2 = nn.Linear(config.hidden_size, config.num_classes)
  20.  
  21.     def forward(self, x):
  22.         #x (uni-gram,seq_len,bi-gram,tri-gram)
  23.         #基于uni-gram、bi-gram、tri-gram对应的索引 在各自的词嵌入矩阵中查询 得到词嵌入
  24.         #（batch,seq_len,embed）
  25.         out_word = self.embedding(x[0])  
  26.         out_bigram = self.embedding_ngram2(x[2])
  27.         out_trigram = self.embedding_ngram3(x[3])
  28.         #三种嵌入进行拼接 (batch,seq,embed*3)
  29.         out = torch.cat((out_word, out_bigram, out_trigram), -1)
  30.         
  31.         #沿长度维 作平均 (batch,embed*3)
  32.         out = out.mean(dim=1)
  33.         #通过fropout
  34.         out = self.dropout(out)
  35.         #通过隐层 (batch,hidden_size)
  36.         out = self.fc1(out)
  37.         out = F.relu(out)
  38.         #输出层 (batch,classes)
  39.         out = self.fc2(out)
  40.         return out

 

2. TextCNN

  • 配置类
  1. class Config(object):
  2.  
  3.     """TextCNN配置参数"""
  4.     def __init__(self, dataset, embedding):
  5.         
  6.         self.model_name = 'TextCNN'
  7.         # 训练集、验证集、测试集路径
  8.         self.train_path = dataset + '/data/train.txt'
  9.         self.dev_path = dataset + '/data/dev.txt'
  10.         self.test_path = dataset + '/data/test.txt'
  11.         # 数据集的所有类别
  12.         self.class_list = [x.strip() for x in open(
  13.             dataset + '/data/class.txt').readlines()]
  14.         # 构建好的词/字典路径
  15.         self.vocab_path = dataset + '/data/vocab.pkl'
  16.         # 训练好的模型参数保存路径
  17.         self.save_path = dataset + '/saved_dict/' + self.model_name + '.ckpt'
  18.         # 模型日志保存路径
  19.         self.log_path = dataset + '/log/' + self.model_name
  20.         # 如果词/字嵌入矩阵不随机初始化 则加载初始化好的词/字嵌入矩阵 类别为float32 并转换为tensor 否则为None
  21.         self.embedding_pretrained = torch.tensor(
  22.             np.load(dataset + '/data/' + embedding)["embeddings"].astype('float32')) \
  23.             if embedding != 'random' else None
  24.         self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  # 设备
  25.  
  26.         self.dropout = 0.5                                              # 随机失活
  27.         self.require_improvement = 1000                                 # 若超过1000batch效果还没提升，则提前结束训练
  28.         self.num_classes = len(self.class_list)                         # 类别数
  29.         self.n_vocab = 0                                                # 词表大小，在运行时赋值
  30.         self.num_epochs = 20                                            # epoch数
  31.         self.batch_size = 128                                           # mini-batch大小
  32.         self.pad_size = 32                                              # 每句话处理成的长度(短填长切)
  33.         self.learning_rate = 1e-3                                       # 学习率
  34.         self.embed = self.embedding_pretrained.size(1)\
  35.             if self.embedding_pretrained is not None else 300           # 字向量维度
  36.         self.filter_sizes = (2, 3, 4)                                   # 不同大小卷积核尺寸
  37.         self.num_filters = 256                                          # 每种卷积核数量
  • 模型定义类
  1. class Model(nn.Module):
  2.     def __init__(self, config):
  3.         super(Model, self).__init__()
  4.  
  5.         if config.embedding_pretrained is not None:  # 加载初始化好的预训练词/字嵌入矩阵  微调funetuning
  6.             self.embedding = nn.Embedding.from_pretrained(config.embedding_pretrained, freeze=False)
  7.         else:  # 否则随机初始化词/字嵌入矩阵 指定填充对应的索引
  8.             self.embedding = nn.Embedding(config.n_vocab, config.embed, padding_idx=config.n_vocab - 1)
  9.         
  10.         #不同大小卷积核对应的卷积操作
  11.         self.convs = nn.ModuleList(
  12.             [nn.Conv2d(1, config.num_filters, (k, config.embed)) for k in config.filter_sizes])
  13.         
  14.         self.dropout = nn.Dropout(config.dropout)
  15.         self.fc = nn.Linear(config.num_filters * len(config.filter_sizes), config.num_classes)
  16.  
  17.     def conv_and_pool(self, x, conv):
  18.         x = F.relu(conv(x)).squeeze(3) #（batch,num_filters,height）
  19.         x = F.max_pool1d(x, x.size(2)).squeeze(2) #(batch,num_filters) 全局最大池化
  20.         return x
  21.  
  22.     def forward(self, x):
  23.         out = self.embedding(x[0]) #（batch,seq） -> (batch,seq,embed)
  24.         out = out.unsqueeze(1) #添加通道维 (batch,1,seq,embed)
  25.         #通过不同大小的卷积核提取特征 并对池化结果进行拼接  (batch,num_filters*len(filter_size))
  26.         out = torch.cat([self.conv_and_pool(out, conv) for conv in self.convs], 1)
  27.         out = self.dropout(out)
  28.         out = self.fc(out) #(batch,classes)
  29.         return out

3. TextRNN

  • 配置类
  1. class Config(object):
  2.  
  3.     """配置参数"""
  4.     def __init__(self, dataset, embedding):
  5.         self.model_name = 'TextRNN'
  6.         # 训练集、验证集、测试集路径
  7.         self.train_path = dataset + '/data/train.txt'
  8.         self.dev_path = dataset + '/data/dev.txt'
  9.         self.test_path = dataset + '/data/test.txt'
  10.         # 数据集的所有类别
  11.         self.class_list = [x.strip() for x in open(
  12.             dataset + '/data/class.txt').readlines()]
  13.         # 构建好的词/字典路径
  14.         self.vocab_path = dataset + '/data/vocab.pkl'
  15.         # 训练好的模型参数保存路径
  16.         self.save_path = dataset + '/saved_dict/' + self.model_name + '.ckpt'
  17.         # 模型日志保存路径
  18.         self.log_path = dataset + '/log/' + self.model_name
  19.         # 如果词/字嵌入矩阵不随机初始化 则加载初始化好的词/字嵌入矩阵 类别为float32 并转换为tensor 否则为None
  20.         self.embedding_pretrained = torch.tensor(
  21.             np.load(dataset + '/data/' + embedding)["embeddings"].astype('float32')) \
  22.             if embedding != 'random' else None
  23.         self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  # 设备
  24.  
  25.         self.dropout = 0.5                                              # 随机失活
  26.         self.require_improvement = 1000                                 # 若超过1000batch效果还没提升，则提前结束训练
  27.         self.num_classes = len(self.class_list)                         # 类别数
  28.         self.n_vocab = 0                                                # 词表大小，在运行时赋值
  29.         self.num_epochs = 10                                            # epoch数
  30.         self.batch_size = 128                                           # mini-batch大小
  31.         self.pad_size = 32                                              # 每句话处理成的长度(短填长切)
  32.         self.learning_rate = 1e-3                                       # 学习率
  33.         self.embed = self.embedding_pretrained.size(1)\
  34.             if self.embedding_pretrained is not None else 300           # 字向量维度, 若使用了预训练词向量，则维度统一
  35.         self.hidden_size = 128                                          # lstm隐藏单元数
  36.         self.num_layers = 2                                             # lstm层数
  • 模型定义类
  1. class Model(nn.Module):
  2.     def __init__(self, config):
  3.         super(Model, self).__init__()
  4.         
  5.         if config.embedding_pretrained is not None:  # 加载初始化好的预训练词/字嵌入矩阵  微调funetuning
  6.             self.embedding = nn.Embedding.from_pretrained(config.embedding_pretrained, freeze=False)
  7.         else:  # 否则随机初始化词/字嵌入矩阵 指定填充对应的索引
  8.             self.embedding = nn.Embedding(config.n_vocab, config.embed, padding_idx=config.n_vocab - 1)
  9.  
  10.         #2层双向lstm batch_size为第一维度
  11.         self.lstm = nn.LSTM(config.embed, config.hidden_size, config.num_layers,
  12.                             bidirectional=True, batch_first=True, dropout=config.dropout)
  13.         #输出层
  14.         self.fc = nn.Linear(config.hidden_size * 2, config.num_classes)
  15.  
  16.     def forward(self, x):
  17.         x, _ = x     #（batch,SEQ_LEN）
  18.         out = self.embedding(x)  # [batch_size, seq_len, embeding]=[128, 32, 300]
  19.         out, _ = self.lstm(out) #(batch_size,seq_len,hidden_size*2)
  20.         out = self.fc(out[:, -1, :])  # 句子最后时刻的 hidden state (batch,hidden_size*2)->(batch,classes)
  21.         return out
  22.  
  23.     '''变长RNN，效果差不多，甚至还低了点...'''
  24.     # def forward(self, x):
  25.     #     x, seq_len = x
  26.     #     out = self.embedding(x)
  27.     #     _, idx_sort = torch.sort(seq_len, dim=0, descending=True)  # 长度从长到短排序（index）
  28.     #     _, idx_unsort = torch.sort(idx_sort)  # 排序后，原序列的 index
  29.     #     out = torch.index_select(out, 0, idx_sort)
  30.     #     seq_len = list(seq_len[idx_sort])
  31.     #     out = nn.utils.rnn.pack_padded_sequence(out, seq_len, batch_first=True)
  32.     #     # [batche_size, seq_len, num_directions * hidden_size]
  33.     #     out, (hn, _) = self.lstm(out)
  34.     #     out = torch.cat((hn[2], hn[3]), -1)
  35.     #     # out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first=True)
  36.     #     out = out.index_select(0, idx_unsort)
  37.     #     out = self.fc(out)
  38.     #     return out

4. TextRCNN

  1. class Config(object):
  2.  
  3.     """配置参数"""
  4.     def __init__(self, dataset, embedding):
  5.         self.model_name = 'TextRCNN'
  6.         # 训练集、验证集、测试集路径
  7.         self.train_path = dataset + '/data/train.txt'
  8.         self.dev_path = dataset + '/data/dev.txt'
  9.         self.test_path = dataset + '/data/test.txt'
  10.         # 数据集的所有类别
  11.         self.class_list = [x.strip() for x in open(
  12.             dataset + '/data/class.txt').readlines()]
  13.         # 构建好的词/字典路径
  14.         self.vocab_path = dataset + '/data/vocab.pkl'
  15.         # 训练好的模型参数保存路径
  16.         self.save_path = dataset + '/saved_dict/' + self.model_name + '.ckpt'
  17.         # 模型日志保存路径
  18.         self.log_path = dataset + '/log/' + self.model_name
  19.         # 如果词/字嵌入矩阵不随机初始化 则加载初始化好的词/字嵌入矩阵 类别为float32 并转换为tensor 否则为None
  20.         self.embedding_pretrained = torch.tensor(
  21.             np.load(dataset + '/data/' + embedding)["embeddings"].astype('float32')) \
  22.             if embedding != 'random' else None
  23.         self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  # 设备
  24.  
  25.         self.dropout = 1.0                                              # 随机失活
  26.         self.require_improvement = 1000                                 # 若超过1000batch效果还没提升，则提前结束训练
  27.         self.num_classes = len(self.class_list)                         # 类别数
  28.         self.n_vocab = 0                                                # 词表大小，在运行时赋值
  29.         self.num_epochs = 10                                            # epoch数
  30.         self.batch_size = 128                                           # mini-batch大小
  31.         self.pad_size = 32                                              # 每句话处理成的长度(短填长切)
  32.         self.learning_rate = 1e-3                                       # 学习率
  33.         self.embed = self.embedding_pretrained.size(1)\
  34.             if self.embedding_pretrained is not None else 300           # 字向量维度, 若使用了预训练词向量，则维度统一
  35.         self.hidden_size = 256                                          # lstm隐藏单元数
  36.         self.num_layers = 1                                             # lstm层数
  • 模型定义类
  1. class Model(nn.Module):
  2.     def __init__(self, config):
  3.         super(Model, self).__init__()
  4.         
  5.         if config.embedding_pretrained is not None:  # 加载初始化好的预训练词/字嵌入矩阵  微调funetuning
  6.             self.embedding = nn.Embedding.from_pretrained(config.embedding_pretrained, freeze=False)
  7.         else:  # 否则随机初始化词/字嵌入矩阵 指定填充对应的索引
  8.             self.embedding = nn.Embedding(config.n_vocab, config.embed, padding_idx=config.n_vocab - 1)
  9.         
  10.         #单层双向lstm batch_size为第一维度
  11.         self.lstm = nn.LSTM(config.embed, config.hidden_size, config.num_layers,
  12.                             bidirectional=True, batch_first=True, dropout=config.dropout)
  13.         
  14.         self.maxpool = nn.MaxPool1d(config.pad_size) #沿长度方向作全局最大池化
  15.         #输出层
  16.         self.fc = nn.Linear(config.hidden_size * 2 + config.embed, config.num_classes)
  17.  
  18.     def forward(self, x):
  19.         x, _ = x #（batch,seq_len）
  20.         embed = self.embedding(x)  # [batch_size, seq_len, embeding]=[64, 32, 64]
  21.         out, _ = self.lstm(embed)  #(batch_size,seq_len,hidden_size*2)
  22.         out = torch.cat((embed, out), 2) #把词嵌入和lstm输出进行拼接 (batch,seq_len.embed+hidden_size*2)
  23.         out = F.relu(out)
  24.         out = out.permute(0, 2, 1) #(batch,embed+hidden_size*2,seq_len)
  25.         out = self.maxpool(out).squeeze() #（batch,embed+hidden_size*2）
  26.         out = self.fc(out) #(batch,classes)
  27.         return out

 

5. TextRNN_Atten

  • 配置类
  1. class Config(object):
  2.  
  3.     """配置参数"""
  4.     def __init__(self, dataset, embedding):
  5.         self.model_name = 'TextRNN_Att'
  6.         # 训练集、验证集、测试集路径
  7.         self.train_path = dataset + '/data/train.txt'
  8.         self.dev_path = dataset + '/data/dev.txt'
  9.         self.test_path = dataset + '/data/test.txt'
  10.         # 数据集的所有类别
  11.         self.class_list = [x.strip() for x in open(
  12.             dataset + '/data/class.txt').readlines()]
  13.         # 构建好的词/字典路径
  14.         self.vocab_path = dataset + '/data/vocab.pkl'
  15.         # 训练好的模型参数保存路径
  16.         self.save_path = dataset + '/saved_dict/' + self.model_name + '.ckpt'
  17.         # 模型日志保存路径
  18.         self.log_path = dataset + '/log/' + self.model_name
  19.         # 如果词/字嵌入矩阵不随机初始化 则加载初始化好的词/字嵌入矩阵 类别为float32 并转换为tensor 否则为None
  20.         self.embedding_pretrained = torch.tensor(
  21.             np.load(dataset + '/data/' + embedding)["embeddings"].astype('float32')) \
  22.             if embedding != 'random' else None
  23.         self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  # 设备
  24.  
  25.         self.dropout = 0.5                                              # 随机失活
  26.         self.require_improvement = 1000                                 # 若超过1000batch效果还没提升，则提前结束训练
  27.         self.num_classes = len(self.class_list)                         # 类别数
  28.         self.n_vocab = 0                                                # 词表大小，在运行时赋值
  29.         self.num_epochs = 10                                            # epoch数
  30.         self.batch_size = 128                                           # mini-batch大小
  31.         self.pad_size = 32                                              # 每句话处理成的长度(短填长切)
  32.         self.learning_rate = 1e-3                                       # 学习率
  33.         self.embed = self.embedding_pretrained.size(1)\
  34.             if self.embedding_pretrained is not None else 300           # 字向量维度, 若使用了预训练词向量，则维度统一
  35.         self.hidden_size = 128                                          # lstm隐藏单元数
  36.         self.num_layers = 2                                             # lstm层数
  37.         self.hidden_size2 = 64                                          #全连接层隐藏单元数
  • 模型定义类
  1. class Model(nn.Module):
  2.     def __init__(self, config):
  3.         super(Model, self).__init__()
  4.         
  5.         if config.embedding_pretrained is not None:  # 加载初始化好的预训练词/字嵌入矩阵  微调funetuning
  6.             self.embedding = nn.Embedding.from_pretrained(config.embedding_pretrained, freeze=False)
  7.         else:  # 否则随机初始化词/字嵌入矩阵 指定填充对应的索引
  8.             self.embedding = nn.Embedding(config.n_vocab, config.embed, padding_idx=config.n_vocab - 1)
  9.         
  10.         #2层双向 LSTM batch_size为第一维度
  11.         self.lstm = nn.LSTM(config.embed, config.hidden_size, config.num_layers,
  12.                             bidirectional=True, batch_first=True, dropout=config.dropout)
  13.         self.tanh1 = nn.Tanh()
  14.         # self.u = nn.Parameter(torch.Tensor(config.hidden_size * 2, config.hidden_size * 2))
  15.         #定义一个参数向量 作为Query
  16.         self.w = nn.Parameter(torch.Tensor(config.hidden_size * 2))
  17.         self.tanh2 = nn.Tanh()
  18.         #隐层
  19.         self.fc1 = nn.Linear(config.hidden_size * 2, config.hidden_size2)
  20.         #输出层
  21.         self.fc = nn.Linear(config.hidden_size2, config.num_classes)
  22.  
  23.     def forward(self, x):
  24.         x, _ = x #（batch,seq_len)
  25.         emb = self.embedding(x)  # [batch_size, seq_len, embeding]=[128, 32, 300]
  26.         H, _ = self.lstm(emb)  # [batch_size, seq_len, hidden_size * num_direction]=[128, 32, 256] 各时刻隐状态 作为value
  27.  
  28.         M = self.tanh1(H)  # [128, 32, 256] 各时刻隐状态通过tanh激活函数 作为Key
  29.         # M = torch.tanh(torch.matmul(H, self.u))
  30.         #Key和Query作运算 在通过softmax 得到每个时刻对应的权重
  31.         alpha = F.softmax(torch.matmul(M, self.w), dim=1).unsqueeze(-1)  # [128, 32, 1]
  32.         #各时刻的权重和各时刻的隐状态Value对应相乘
  33.         out = H * alpha  # [128, 32, 256]
  34.         #再相加
  35.         out = torch.sum(out, 1)  # [128, 256] （batch,hidden*2）
  36.         out = F.relu(out)
  37.         out = self.fc1(out) #(batch,hidden2）
  38.         out = self.fc(out)  # (batch,classes）
  39.         return out

 

6. DPCNN

  • 配置类
  1. class Config(object):
  2.  
  3.     """配置参数"""
  4.     def __init__(self, dataset, embedding):
  5.         self.model_name = 'DPCNN'
  6.         # 训练集、验证集、测试集路径
  7.         self.train_path = dataset + '/data/train.txt'
  8.         self.dev_path = dataset + '/data/dev.txt'
  9.         self.test_path = dataset + '/data/test.txt'
  10.         # 数据集的所有类别
  11.         self.class_list = [x.strip() for x in open(
  12.             dataset + '/data/class.txt').readlines()]
  13.         # 构建好的词/字典路径
  14.         self.vocab_path = dataset + '/data/vocab.pkl'
  15.         # 训练好的模型参数保存路径
  16.         self.save_path = dataset + '/saved_dict/' + self.model_name + '.ckpt'
  17.         # 模型日志保存路径
  18.         self.log_path = dataset + '/log/' + self.model_name
  19.         # 如果词/字嵌入矩阵不随机初始化 则加载初始化好的词/字嵌入矩阵 类别为float32 并转换为tensor 否则为None
  20.         self.embedding_pretrained = torch.tensor(
  21.             np.load(dataset + '/data/' + embedding)["embeddings"].astype('float32')) \
  22.             if embedding != 'random' else None
  23.         self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  # 设备
  24.  
  25.         self.dropout = 0.5                                              # 随机失活
  26.         self.require_improvement = 1000                                 # 若超过1000batch效果还没提升，则提前结束训练
  27.         self.num_classes = len(self.class_list)                         # 类别数
  28.         self.n_vocab = 0                                                # 词表大小，在运行时赋值
  29.         self.num_epochs = 20                                            # epoch数
  30.         self.batch_size = 128                                           # mini-batch大小
  31.         self.pad_size = 32                                              # 每句话处理成的长度(短填长切)
  32.         self.learning_rate = 1e-3                                       # 学习率
  33.         self.embed = self.embedding_pretrained.size(1)\
  34.             if self.embedding_pretrained is not None else 300           # 字/词向量维度
  35.         self.num_filters = 250                                          # 卷积核数量(channels数)
  • 模型定义类
  1.  
  2. class Model(nn.Module):
  3.     def __init__(self, config):
  4.         super(Model, self).__init__()
  5.  
  6.         if config.embedding_pretrained is not None:  # 加载初始化好的预训练词/字嵌入矩阵  微调funetuning
  7.             self.embedding = nn.Embedding.from_pretrained(config.embedding_pretrained, freeze=False)
  8.         else:  # 否则随机初始化词/字嵌入矩阵 指定填充对应的索引
  9.             self.embedding = nn.Embedding(config.n_vocab, config.embed, padding_idx=config.n_vocab - 1)
  10.  
  11.         #region embedding 类似于TextCNN中的卷积操作
  12.         self.conv_region = nn.Conv2d(1, config.num_filters, (3, config.embed), stride=1)
  13.         
  14.         self.conv = nn.Conv2d(config.num_filters, config.num_filters, (3, 1), stride=1)
  15.         self.max_pool = nn.MaxPool2d(kernel_size=(3, 1), stride=2)
  16.         self.padding1 = nn.ZeroPad2d((0, 0, 1, 1))  # top bottom 上下各添加1个0
  17.         self.padding2 = nn.ZeroPad2d((0, 0, 0, 1))  # bottom 下添加一个0
  18.         self.relu = nn.ReLU()
  19.         self.fc = nn.Linear(config.num_filters, config.num_classes)
  20.  
  21.     def forward(self, x):
  22.         x = x[0]    #（batch,seq_len）
  23.         x = self.embedding(x) #(batch,seq_len,embed)
  24.         x = x.unsqueeze(1)  # 添加通道维 进行2d卷积 (batch,1,seq_len,embed)
  25.         x = self.conv_region(x)  # (batch,num_filters,seq_len-3+1,1)
  26.         #先卷积 再填充 等价于等长卷积  序列长度不变
  27.         x = self.padding1(x)  # [batch_size, num_filters, seq_len, 1]
  28.         x = self.relu(x)
  29.         x = self.conv(x)  # [batch_size, num_filters, seq_len-3+1, 1]
  30.         x = self.padding1(x)  # [batch_size, num_filters, seq_len, 1]
  31.         x = self.relu(x)
  32.         x = self.conv(x)  # [batch_size, num_filters, seq_len-3+1, 1]
  33.         while x.size()[2] > 2: 
  34.             x = self._block(x)
  35.         x = x.squeeze()  # [batch_size, num_filters]
  36.         x = self.fc(x)  #（batch,classes）
  37.         return x
  38.  
  39.     def _block(self, x):
  40.         x = self.padding2(x) #[batch_size, num_filters, seq_len-1, 1]
  41.         #长度减半
  42.         px = self.max_pool(x) #[batch_size, num_filters, （seq_len-1）/2, 1]
  43.  
  44.         #等长卷积 长度不变
  45.         x = self.padding1(px)
  46.         x = F.relu(x)
  47.         x = self.conv(x)
  48.  
  49.         # 等长卷积 长度不变
  50.         x = self.padding1(x)
  51.         x = F.relu(x)
  52.         x = self.conv(x)
  53.  
  54.         # Short Cut
  55.         x = x + px
  56.         return x

7. Transformer

  • 配置类
  1. class Config(object):
  2.  
  3.     """配置参数"""
  4.     def __init__(self, dataset, embedding):
  5.         self.model_name = 'Transformer'
  6.         # 训练集、验证集、测试集路径
  7.         self.train_path = dataset + '/data/train.txt'
  8.         self.dev_path = dataset + '/data/dev.txt'
  9.         self.test_path = dataset + '/data/test.txt'
  10.         # 数据集的所有类别
  11.         self.class_list = [x.strip() for x in open(
  12.             dataset + '/data/class.txt').readlines()]
  13.         # 构建好的词/字典路径
  14.         self.vocab_path = dataset + '/data/vocab.pkl'
  15.         # 训练好的模型参数保存路径
  16.         self.save_path = dataset + '/saved_dict/' + self.model_name + '.ckpt'
  17.         # 模型日志保存路径
  18.         self.log_path = dataset + '/log/' + self.model_name
  19.         # 如果词/字嵌入矩阵不随机初始化 则加载初始化好的词/字嵌入矩阵 类别为float32 并转换为tensor 否则为None
  20.         self.embedding_pretrained = torch.tensor(
  21.             np.load(dataset + '/data/' + embedding)["embeddings"].astype('float32')) \
  22.             if embedding != 'random' else None
  23.         self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  # 设备
  24.  
  25.         self.dropout = 0.5                                              # 随机失活
  26.         self.require_improvement = 2000                                 # 若超过2000batch效果还没提升，则提前结束训练
  27.         self.num_classes = len(self.class_list)                         # 类别数
  28.         self.n_vocab = 0                                                # 词表大小，在运行时赋值
  29.         self.num_epochs = 20                                            # epoch数
  30.         self.batch_size = 128                                           # mini-batch大小
  31.         self.pad_size = 32                                              # 每句话处理成的长度(短填长切)
  32.         self.learning_rate = 5e-4                                       # 学习率
  33.         self.embed = self.embedding_pretrained.size(1)\
  34.             if self.embedding_pretrained is not None else 300           # 字向量维度
  35.         self.dim_model = 300
  36.         self.hidden = 1024
  37.         self.last_hidden = 512
  38.         self.num_head = 5               #5头注意力机制
  39.         self.num_encoder = 2           #两个transformer encoder block
  • 模型定义类
  1.  
  2. class Model(nn.Module):
  3.     def __init__(self, config):
  4.         super(Model, self).__init__()
  5.         
  6.         #词/字嵌入
  7.         if config.embedding_pretrained is not None:  # 加载初始化好的预训练词/字嵌入矩阵  微调funetuning
  8.             self.embedding = nn.Embedding.from_pretrained(config.embedding_pretrained, freeze=False)
  9.         else:  # 否则随机初始化词/字嵌入矩阵 指定填充对应的索引
  10.             self.embedding = nn.Embedding(config.n_vocab, config.embed, padding_idx=config.n_vocab - 1)
  11.         
  12.         #位置编码
  13.         self.postion_embedding = Positional_Encoding(config.embed, config.pad_size, config.dropout, config.device)
  14.         
  15.         #transformer encoder block
  16.         self.encoder = Encoder(config.dim_model, config.num_head, config.hidden, config.dropout)
  17.         
  18.         #多个transformer encoder block
  19.         self.encoders = nn.ModuleList([
  20.             copy.deepcopy(self.encoder)
  21.             # Encoder(config.dim_model, config.num_head, config.hidden, config.dropout)
  22.             for _ in range(config.num_encoder)])
  23.         
  24.         #输出层
  25.         self.fc1 = nn.Linear(config.pad_size * config.dim_model, config.num_classes)
  26.         # self.fc2 = nn.Linear(config.last_hidden, config.num_classes)
  27.         # self.fc1 = nn.Linear(config.dim_model, config.num_classes)
  28.  
  29.     def forward(self, x):
  30.         out = self.embedding(x[0]) #(batch,seq_len) -> (batch,seq_len,embed)
  31.         out = self.postion_embedding(out) # 添加位置编码 (batch,seq_len,embed)
  32.         for encoder in self.encoders: #通过多个ender block
  33.             out = encoder(out)  #(batch,seq_len,dim_model)
  34.         out = out.view(out.size(0), -1) #（batch,seq_len*dim_model）
  35.         # out = torch.mean(out, 1)
  36.         out = self.fc1(out) #(batch,classes)
  37.         return out
  38.  
  39.  
  40. class Encoder(nn.Module):
  41.     def __init__(self, dim_model, num_head, hidden, dropout):
  42.         super(Encoder, self).__init__()
  43.         #多头注意力机制
  44.         self.attention = Multi_Head_Attention(dim_model, num_head, dropout)
  45.         #两个全连接层
  46.         self.feed_forward = Position_wise_Feed_Forward(dim_model, hidden, dropout)
  47.  
  48.     def forward(self, x): #x (batch,seq_len,embed_size)  embed_size = dim_model
  49.         out = self.attention(x) #计算多头注意力结果 (batch,seq_len,dim_model)
  50.         out = self.feed_forward(out) #通过两个全连接层增加 非线性转换能力 (batch,seq_len,dim_model)
  51.         return out
  52.  
  53.  
  54. class Positional_Encoding(nn.Module):
  55.     #位置编码
  56.     def __init__(self, embed, pad_size, dropout, device):
  57.         super(Positional_Encoding, self).__init__()
  58.         self.device = device
  59.         #利用sin cos生成绝对位置编码
  60.         self.pe = torch.tensor([[pos / (10000.0 ** (i // 2 * 2.0 / embed)) for i in range(embed)] for pos in range(pad_size)])
  61.         self.pe[:, 0::2] = np.sin(self.pe[:, 0::2])
  62.         self.pe[:, 1::2] = np.cos(self.pe[:, 1::2])
  63.         self.dropout = nn.Dropout(dropout)
  64.  
  65.     def forward(self, x): 
  66.         #token embedding + 绝对位置编码
  67.         out = x + nn.Parameter(self.pe, requires_grad=False).to(self.device)
  68.         #再通过dropout
  69.         out = self.dropout(out)
  70.         return out
  71.  
  72.  
  73. class Scaled_Dot_Product_Attention(nn.Module):
  74.     '''Scaled Dot-Product Attention '''
  75.     def __init__(self):
  76.         super(Scaled_Dot_Product_Attention, self).__init__()
  77.  
  78.     def forward(self, Q, K, V, scale=None):
  79.         '''
  80.         Args:
  81.             Q: [batch_size, len_Q, dim_Q]
  82.             K: [batch_size, len_K, dim_K]
  83.             V: [batch_size, len_V, dim_V]
  84.             scale: 缩放因子 论文为根号dim_K
  85.         Return:
  86.             self-attention后的张量，以及attention张量
  87.         '''
  88.         #Q与K的第2、3维转置计算内积  (batch*num_head,seq_len,seq_len)
  89.         attention = torch.matmul(Q, K.permute(0, 2, 1))
  90.         if scale: #作缩放 减小结果的方差 
  91.             attention = attention * scale
  92.         # if mask:  # TODO change this 
  93.         #     attention = attention.masked_fill_(mask == 0, -1e9)
  94.         attention = F.softmax(attention, dim=-1) #转换为权重
  95.         context = torch.matmul(attention, V) #再与V运算 得到结果 (batch*num_head,seq_len,dim_head)
  96.         return context
  97.  
  98.  
  99. class Multi_Head_Attention(nn.Module):
  100.     #多头注意力机制 encoder block的第一部分
  101.     def __init__(self, dim_model, num_head, dropout=0.0):
  102.         super(Multi_Head_Attention, self).__init__()
  103.         self.num_head = num_head  #头数 
  104.         assert dim_model % num_head == 0 #必须整除
  105.         self.dim_head = dim_model // self.num_head 
  106.         #分别通过三个Dense层 生成Q、K、V
  107.         self.fc_Q = nn.Linear(dim_model, num_head * self.dim_head)
  108.         self.fc_K = nn.Linear(dim_model, num_head * self.dim_head)
  109.         self.fc_V = nn.Linear(dim_model, num_head * self.dim_head)
  110.         #Attention计算
  111.         self.attention = Scaled_Dot_Product_Attention()
  112.         self.fc = nn.Linear(num_head * self.dim_head, dim_model)
  113.         self.dropout = nn.Dropout(dropout)
  114.         #层归一化
  115.         self.layer_norm = nn.LayerNorm(dim_model)
  116.  
  117.     def forward(self, x): #（batch,seq_len,dim_model）
  118.         batch_size = x.size(0) 
  119.         # Q,K,V维度 (batch,seq_len,dim_head*num_head)
  120.         Q = self.fc_Q(x)  
  121.         K = self.fc_K(x)
  122.         V = self.fc_V(x)
  123.         #沿第三个维度进行切分 切分为num_head份 再沿第一个维度拼接 多个注意力头并行计算
  124.         #Q,K,V维度 (batch*num_head,seq_len,dim_head)
  125.         Q = Q.view(batch_size * self.num_head, -1, self.dim_head) 
  126.         K = K.view(batch_size * self.num_head, -1, self.dim_head)
  127.         V = V.view(batch_size * self.num_head, -1, self.dim_head)
  128.         # if mask:  # TODO
  129.         #     mask = mask.repeat(self.num_head, 1, 1)  # TODO change this
  130.         scale = K.size(-1) ** -0.5  # 缩放因子 dim_head的开放取倒数 对内积结果进行缩放 减小结果的方差 有利于训练
  131.         #attention计算 多个注意力头并行计算（矩阵运算）
  132.         context = self.attention(Q, K, V, scale)
  133.         
  134.         #多头注意力计算结果 沿第一个维度进行切分 再沿第三个维度拼接 转为原来的维度(batch,seq_len,dim_head*num_head)
  135.         context = context.view(batch_size, -1, self.dim_head * self.num_head)
  136.         
  137.         out = self.fc(context)#(batch,seq_len,dim_model)
  138.         out = self.dropout(out)
  139.         out = out + x  # 残差连接
  140.         out = self.layer_norm(out)
  141.         return out
  142.  
  143.  
  144. class Position_wise_Feed_Forward(nn.Module):
  145.     #encoder block的第二部分
  146.     def __init__(self, dim_model, hidden, dropout=0.0):
  147.         #定义两个全连接层 多头注意力的计算结果 通过两个全连接层 增加非线性 
  148.         super(Position_wise_Feed_Forward, self).__init__()
  149.         self.fc1 = nn.Linear(dim_model, hidden)
  150.         self.fc2 = nn.Linear(hidden, dim_model)
  151.         self.dropout = nn.Dropout(dropout)
  152.         self.layer_norm = nn.LayerNorm(dim_model)
  153.  
  154.     def forward(self, x): #（batch,seq_len,dim_model）
  155.         out = self.fc1(x)  #（batch,seq_len,hidden）
  156.         out = F.relu(out)
  157.         out = self.fc2(out) #（batch,seq_len,dim_model）
  158.         out = self.dropout(out)
  159.         out = out + x  # 残差连接
  160.         out = self.layer_norm(out) #层归一化
  161.         return out

 

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/你好赵伟/article/detail/514365
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号