devbox
知新_RL
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

lstm古诗生成-pytorch_lstm生成古诗

作者：知新_RL | 2024-08-14 05:54:12

赞

踩

lstm生成古诗

本文为RNN做古诗生成的一个小demo，只要是为了完成课上的作业（由于训练比较慢，所以周期仅设置为3，大一点性能可能会更好），如有需要可以在这基础之上进行加工，数据集没办法上传，如有需要，可以私信我。

LSTM：

如上图所示LSTM神经元存在两个状态向量:h(t)和c(t)（可将h(t)视为短期状态，c(t)视为长期状态） 首先，将当前输入向量x(t)和先前的短期状态h(t-1)馈入四个不同的全连接层(FC)。它们都有不同的目的:

主要层是输出g(t)的层：它通常的作用是分析当前输入x(t)和先前(短期)状态 h(t-1)，得到本时间步的信息。

遗忘门(由f(t)控制)：控制长期状态的哪些部分应当被删除。

输入门(由i(t)控制)：控制应将g(t)的哪些部分添加到长期状态。

输出门(由o(t)控制)：控制应在此时间步长读取长期状态的哪些部分并输出 到h(t)和y(t)。

如图1，LSTM神经元运用了三个sigmoid激活函数和一个tanh激活函数，

Tanh 作用在于帮助调节流经网络的值，使得数值始终限制在 -1 和 1 之间。
Sigmoid 激活函数与 tanh 函数类似，不同之处在于 sigmoid 是把值压缩到0~1 这样的设置有助于更新或忘记信息，可将其理解为比例（任何数乘以 0 都得 0，这部分信息就会剔除掉；同样的，任何数乘以 1 都得到它本身，这部分信息就会完美地保存下来）因记忆能力有限，记住重要的，忘记不重要的。
例子：以输入门为例，首先输入x(t)和先前(短期)状态 h(t-1)，得到本时间步的信息向量g(t) = (g1(t),g2(t),g3(t)……gn(t))(其中n个神经元的个数，g1(t)取值范围为(-1,1))，然后与向量i(t)=(i1(t),i2(t),i3(t)……in(t))(ii(t)取值范围为(0,1))对应元素相乘，得到向量(g1(t)*i1(t), g2(t)*i2(t)……gn(t)*in(t))，即本时间步有用信息，然后把他加上长期记忆c(t-1)中进行保存。

LSM关键的思想是网络可以学习长期状态下存储的内容、丢弃的内容以及从中读取的内容。当长期状态c(t-1)从左到右遍历网络时，可以看到它首先经过一个遗 忘门，丢掉了一些记忆，然后通过加法操作添加了一些新的记忆(由输入门选择的记忆)。结果c(t)直接送出来，无须任何进一步的转换。因此，在每个时间步长中，都会 丢掉一些记忆，并添加一些记忆。此外，在加法运算之后，长期状态被复制并通过tanh函数传输，然后结果被输出门滤波。这将产生短期状态h(t)(等于该时间步长的单元输出 y(t))。

 原理：

本文使用LSTM生成古诗，那么RNN是怎么用作我们的文本生成呢?话不多说,其实用RNN来生成的思想很简单, 就是将前一个字进行词嵌入,后一个字作为标签,将这个组合输入到RNN的网络里面等待训练拟合之后,再用一个引导词,训练出它的预测结果,再用其预测结果,来训练下一个词,循环往复,从而实现RNN生成文本的效果.
 

main.py

  1. import numpy as np
  2. import collections
  3. import torch
  4. from torch.autograd import Variable
  5. import torch.optim as optim
  6.  
  7. import rnn
  8.  
  9. start_token = 'G'
  10. end_token = 'E'
  11. batch_size = 64
  12.  
  13. def process_poems1(file_name):
  14.     """
  16.     :param file_name:
  17.     :return: poems_vector  have two dimmention ,first is the poem, the second is the word_index
  18.     e.g. [[1,2,3,4,5,6,7,8,9,10],[9,6,3,8,5,2,7,4,1]]
  20.     """
  21.     poems = []
  22.     i = 1
  23.     with open(file_name, "r", encoding='utf-8', ) as f:
  24.         for line in f.readlines():
  25.             try:
  26.                 i = i+1
  27.  
  28.                 title, content = line.strip().split(':')
  29.                 # content = content.replace(' ', '').replace('，','').replace('。','')
  30.                 content = content.replace(' ', '')
  31.                 if '_' in content or '(' in content or '（' in content or '《' in content or '[' in content or \
  32.                                 start_token in content or end_token in content:
  33.                     continue
  34.                 if len(content) < 5 or len(content) > 80:
  35.                     continue
  36.                 content = start_token + content + end_token
  37.                 poems.append(content)
  38.             except ValueError as e:
  39.                 print(line)
  40.                 print(i)
  41.                 print("error")
  42.                 pass
  43.     # 按诗的字数排序
  44.     poems = sorted(poems, key=lambda line: len(line))
  45.     # print(poems)
  46.     # 统计每个字出现次数
  47.     all_words = []
  48.     j = 0
  49.     for poem in poems:
  50.         all_words += [word for word in poem]  # 数据连接
  51.     counter = collections.Counter(all_words)  # 统计词和词频。
  52.  
  53.     count_pairs = sorted(counter.items(), key=lambda x: -x[1])  # d.items() 以列表的形式返回可遍历的元组数组 逆序排序
  54.     words, _ = zip(*count_pairs) # zip(*) 可理解为解压，返回二维矩阵式
  55.  
  56.     words = words[:len(words)] + (' ',) #（‘ ’，） 为一个元素的元祖
  57.  
  58.     word_int_map = dict(zip(words, range(len(words))))
  59.     poems_vector = [list(map(word_int_map.get, poem)) for poem in poems] # 第一位为一个函数，后一位为一个迭代器
  60.     return poems_vector, word_int_map, words # 诗句的向量表示，单词映射表，单词表 
  61.  
  62. def process_poems2(file_name):
  63.     """
  64.     :param file_name:
  65.     :return: poems_vector  have tow dimmention ,first is the poem, the second is the word_index
  66.     e.g. [[1,2,3,4,5,6,7,8,9,10],[9,6,3,8,5,2,7,4,1]]
  68.     """
  69.     poems = []
  70.     with open(file_name, "r", encoding='utf-8', ) as f:
  71.         # content = ''
  72.         for line in f.readlines():
  73.             try:
  74.                 line = line.strip()
  75.                 if line:
  76.                     content = line.replace(' '' ', '').replace('，','').replace('。','')
  77.                     if '_' in content or '(' in content or '（' in content or '《' in content or '[' in content or \
  78.                                     start_token in content or end_token in content:
  79.                         continue
  80.                     if len(content) < 5 or len(content) > 80:
  81.                         continue
  82.                     # print(content)
  83.                     content = start_token + content + end_token
  84.                     poems.append(content)
  85.                     # content = ''
  86.             except ValueError as e:
  87.                 # print("error")
  88.                 pass
  89.     # 按诗的字数排序
  90.     poems = sorted(poems, key=lambda line: len(line))
  91.     # print(poems)
  92.     # 统计每个字出现次数
  93.     all_words = []
  94.     for poem in poems:
  95.         all_words += [word for word in poem]
  96.     counter = collections.Counter(all_words)  # 统计词和词频。
  97.     count_pairs = sorted(counter.items(), key=lambda x: -x[1])  # 排序
  98.     words, _ = zip(*count_pairs)
  99.     words = words[:len(words)] + (' ',)
  100.     word_int_map = dict(zip(words, range(len(words))))
  101.     poems_vector = [list(map(word_int_map.get, poem)) for poem in poems]
  102.     return poems_vector, word_int_map, words
  103.  
  104. def generate_batch(batch_size, poems_vec, word_to_int):
  105.     #生成训练数据
  106.  
  107.     n_chunk = len(poems_vec) // batch_size  #34813/100 = 348 古诗的向量表示
  108.     x_batches = []
  109.     y_batches = []
  110.     for i in range(n_chunk):
  111.         start_index = i * batch_size
  112.         end_index = start_index + batch_size
  113.         x_data = poems_vec[start_index:end_index]
  114.         y_data = []
  115.         for row in x_data:
  116.             y  = row[1:]
  117.             y.append(row[-1])
  118.             y_data.append(y)
  119.         """
  120.         x_data             y_data
  121.         [6,2,4,6,9]       [2,4,6,9,9]  文本生成，所以用后面一位数据做label
  122.         [1,4,2,8,5]       [4,2,8,5,5]
  123.         """
  124.         # print(x_data[0])
  125.         # print(y_data[0])
  126.         # exit(0)
  127.         x_batches.append(x_data)
  128.         y_batches.append(y_data)
  129.     return x_batches, y_batches
  130.  
  131.  
  132. def run_training():
  133.     # 处理数据集
  134.     # poems_vector, word_to_int, vocabularies = process_poems2('./tangshi.txt')
  135.     poems_vector, word_to_int, vocabularies = process_poems1('./poems.txt') 
  136.     # 生成batch
  137.     print("finish  loadding data")
  138.     BATCH_SIZE = 100
  139.  
  140.     torch.manual_seed(5)
  141.     word_embedding = rnn.word_embedding( vocab_length= len(word_to_int) + 1 , embedding_dim= 100) #6123 x 100
  142.     #print(word_embedding.shape)
  143.  
  144.     rnn_model = rnn.RNN_model(batch_sz = BATCH_SIZE,vocab_len = len(word_to_int) + 1 ,word_embedding = word_embedding ,embedding_dim= 100, lstm_hidden_dim=128)
  145.     # optimizer = optim.Adam(rnn_model.parameters(), lr= 0.001)
  146.     optimizer=optim.RMSprop(rnn_model.parameters(), lr=0.01)
  147.  
  148.     loss_fun = torch.nn.NLLLoss()
  149.     # rnn_model.load_state_dict(torch.load('./poem_generator_rnn'))  # if you have already trained your model you can load it by this line.
  150.  
  151.     for epoch in range(3):
  152.         batches_inputs, batches_outputs = generate_batch(BATCH_SIZE, poems_vector, word_to_int) #生成训练数据 由batch组成的数组 348
  153.         n_chunk = len(batches_inputs)
  154.         for batch in range(n_chunk):
  155.             batch_x = batches_inputs[batch]
  156.             batch_y = batches_outputs[batch] # (batch , time_step)
  157.  
  158.             loss = 0
  159.             for index in range(BATCH_SIZE):  #batch_size = 100
  160.                 x = np.array(batch_x[index], dtype = np.int64)
  161.                 y = np.array(batch_y[index], dtype = np.int64)
  162.         
  163.                 x = Variable(torch.from_numpy(np.expand_dims(x,axis=1))) #将数组转换成张量 np.expand_dims扩展数据的形状 x.sahpe = 7x1, 
  164.                 y = Variable(torch.from_numpy(y ))
  165.                 pre = rnn_model(x) # 7 x 6125
  166.                 loss += loss_fun(pre , y)
  167.                 if index == 0:
  168.                     _, pre = torch.max(pre, dim=1)# pre为张量，tolist转换成列表　
  169.                     print('prediction', pre.data.tolist()) # the following  three line can print the output and the prediction
  170.                     print('b_y       ', y.data.tolist())   # And you need to take a screenshot and then past is to your homework paper.
  171.                     print('*' * 30)
  172.             loss  = loss  / BATCH_SIZE
  173.             print("epoch  ",epoch,'batch number',batch,"loss is: ", loss.data.tolist())
  174.             optimizer.zero_grad()
  175.             loss.backward()
  176.             torch.nn.utils.clip_grad_norm(rnn_model.parameters(), 1)    # 梯度裁剪 可以预防梯度爆炸,参数的平方和
  177.             optimizer.step()  #训练参数
  178.  
  179.             if batch % 20 ==0:
  180.                 torch.save(rnn_model.state_dict(), './poem_generator_rnn')
  181.                 print("finish  save model")
  182.  
  183.  
  184.  
  185. def to_word(predict, vocabs):  # 预测的结果转化成汉字
  186.     sample = np.argmax(predict)
  187.  
  188.     if sample >= len(vocabs):
  189.         sample = len(vocabs) - 1
  190.     return vocabs[sample]
  191.  
  192.  
  193. def pretty_print_poem(poem):  # 令打印的结果更工整
  194.     shige=[]
  195.     for w in poem:
  196.         if w == start_token or w == end_token:
  197.             break
  198.         shige.append(w)
  199.     poem_sentences = poem.split('。')
  200.     for s in poem_sentences:
  201.         if s != '' and len(s) > 2:
  202.         #     print(s + '。')
  203.             print(s + '。')
  204.  
  205.  
  206. def gen_poem(begin_word):
  207.     # poems_vector, word_int_map, vocabularies = process_poems2('./tangshi.txt')  #  use the other dataset to train the network
  208.     poems_vector, word_int_map, vocabularies = process_poems1('./poems.txt')
  209.     word_embedding = rnn.word_embedding(vocab_length=len(word_int_map) + 1, embedding_dim=100)
  210.     rnn_model = rnn.RNN_model(batch_sz=64, vocab_len=len(word_int_map) + 1, word_embedding=word_embedding,
  211.                                    embedding_dim=100, lstm_hidden_dim=128)
  212.  
  213.     rnn_model.load_state_dict(torch.load('./poem_generator_rnn'))
  214.     # 指定开始的字
  215.  
  216.     poem = begin_word
  217.     word = begin_word
  218.     while word != end_token:
  219.         input = np.array([word_int_map[w] for w in poem],dtype= np.int64)
  220.         input = Variable(torch.from_numpy(input))
  221.         output = rnn_model(input, is_test=True)
  222.         word = to_word(output.data.tolist(), vocabularies)
  223.         poem += word
  224.         if len(poem) > 30:
  225.             break
  226.     return poem
  227.  
  228.  
  229.  
  230. #run_training()  # 如果不是训练阶段 ，请注销这一行 。 网络训练时间很长。
  231.  
  232.  
  233. pretty_print_poem(gen_poem("日"))
  234. pretty_print_poem(gen_poem("红"))
  235. pretty_print_poem(gen_poem("山"))
  236. pretty_print_poem(gen_poem("夜"))
  237. pretty_print_poem(gen_poem("湖"))
  238. pretty_print_poem(gen_poem("湖"))
  239. pretty_print_poem(gen_poem("湖"))
  240. pretty_print_poem(gen_poem("君"))

rnn.py

  1. import torch.nn as nn
  2. import torch
  3. from torch.autograd import Variable
  4. import torch.nn.functional as F
  5.  
  6. import numpy as np
  7.  
  8. def weights_init(m):
  9.     classname = m.__class__.__name__  #   obtain the class name
  10.     if classname.find('Linear') != -1:
  11.         weight_shape = list(m.weight.data.size()) #6123 x 128
  12.         fan_in = weight_shape[1]
  13.         fan_out = weight_shape[0]
  14.         w_bound = np.sqrt(6. / (fan_in + fan_out))
  15.         m.weight.data.uniform_(-w_bound, w_bound)
  16.         m.bias.data.fill_(0)
  17.         print("inital  linear weight ")
  18.  
  19.  
  20. class word_embedding(nn.Module):
  21.     def __init__(self,vocab_length , embedding_dim):
  22.         super(word_embedding, self).__init__()
  23.         w_embeding_random_intial = np.random.uniform(-1,1,size=(vocab_length ,embedding_dim)) #生成服从均匀分布的随机数
  24.         self.word_embedding = nn.Embedding(vocab_length,embedding_dim) #创建一个embedding层
  25.         self.word_embedding.weight.data.copy_(torch.from_numpy(w_embeding_random_intial))
  26.     def forward(self,input_sentence):
  27.         """
  28.         :param input_sentence:  a tensor ,contain several word index.
  29.         :return: a tensor ,contain word embedding tensor
  30.         """
  31.         sen_embed = self.word_embedding(input_sentence)
  32.         return sen_embed
  33.  
  34.  
  35. class RNN_model(nn.Module):
  36.     def __init__(self, batch_sz ,vocab_len ,word_embedding,embedding_dim, lstm_hidden_dim):
  37.         super(RNN_model,self).__init__()
  38.  
  39.         self.word_embedding_lookup = word_embedding
  40.         self.batch_size = batch_sz
  41.         self.vocab_length = vocab_len
  42.         self.word_embedding_dim = embedding_dim
  43.         self.lstm_dim = lstm_hidden_dim
  44.         #########################################
  45.         # here you need to define the "self.rnn_lstm"  the input size is "embedding_dim" and the output size is "lstm_hidden_dim"
  46.         # the lstm should have two layers, and the  input and output tensors are provided as (batch, seq, feature)
  47.         # ???
  48.  
  49.         self.rnn_lstm = nn.LSTM(input_size=embedding_dim,hidden_size=lstm_hidden_dim, num_layers=2,batch_first=True)
  50.  
  51.         ##########################################
  52.         self.fc = nn.Linear(lstm_hidden_dim, vocab_len )
  53.         self.apply(weights_init) # call the weights initial function.
  54.         self.softmax = nn.LogSoftmax() # the activation function.
  55.         # self.tanh = nn.Tanh()
  56.     def forward(self,sentence,is_test = False):
  57.         batch_input = self.word_embedding_lookup(sentence).view(1,-1,self.word_embedding_dim)  # sentence=[7,1] [7x1x100] batch_input=[1,7,100])
  58.         # print(batch_input.size()) # print the size of the input
  59.         ################################################
  60.         # here you need to put the "batch_input"  input the self.lstm which is defined before.
  61.         # the hidden output should be named as output, the initial hidden state and cell state set to zero.
  62.         # ???
  63.         #print(batch_input.shape)
  64.         output,_ = self.rnn_lstm(batch_input) # 1x7x128
  65.         ################################################
  66.         out = output.contiguous().view(-1,self.lstm_dim)  #1x128
  67.         #print(out.shape)
  68.         out =  F.relu(self.fc(out))
  69.         out = self.softmax(out)
  70.  
  71.         if is_test:
  72.             prediction = out[ -1, : ].view(1,-1) #[1,6125]
  73.             #prediction = torch.max(out,0)
  74.             output = prediction
  75.         else:
  76.            output = out
  77.  
  78.         # print(out)
  79.         return output
  80.

本文内容由网友自发贡献，转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号