devbox
你好赵伟
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

基于术语词典干预的机器翻译挑战赛baseline及改进（Datawhale AI 夏令营）_机器翻译比赛

作者：你好赵伟 | 2024-08-07 08:56:29

赞

踩

机器翻译比赛

比赛背景

        目前，神经机器翻译（NMT）技术在翻译质量和速度方面已经取得了显著进展。然而，在特定领域或行业中，NMT仍面临一些挑战，尤其是在术语一致性方面。对于术语名词、人名地名等特定词汇，机器翻译经常会出现不准确的结果，这会导致翻译的混淆或歧义。通过引入术语词典，可以纠正这些错误，从而提高翻译质量。

赛事任务

        本次比赛的任务是基于术语词典干预的英文到中文的机器翻译。大赛提供了以下数据：

  • 训练集：中英双语数据，共计14万余对句子。
  • 开发集：英中双语数据，共计1000对句子。
  • 测试集：英中双语数据，共计1000对句子。
  • 术语词典：包含英中对照的2226条术语。

        参赛队伍需要使用提供的训练数据，构建并训练多语言机器翻译模型，并基于测试集和术语词典，提供最终的翻译结果。

数据说明

        所有文件均为UTF-8编码，训练集、开发集、测试集和术语词典的格式如下：

  • 训练集：每行为一个句对样本，格式如图1所示。

    示例：

图1 训练集格式

术语词典格式如图2所示。

图2 术语词典格式

评估指标

对于参赛队伍提交的测试集翻译结果文件，采用自动评价指标BLUE-4进行评价，具体工具使用sacrebleu开源版本。

Baseline

  • 加载和预处理训练数据和术语词典。
  • 定义序列到序列的神经网络模型，包括编码器和解码器。
  • 使用训练数据训练模型，并保存训练好的模型参数。
  • 在测试集上进行推理，并利用术语词典干预翻译结果，确保术语的一致性。
  1. # 安装torchtext
  2. !pip install torchtext
  3.  
  4. import torch
  5. import torch.nn as nn
  6. import torch.optim as optim
  7. from torch.utils.data import Dataset, DataLoader
  8. from torchtext.data.utils import get_tokenizer
  9. from collections import Counter
  10. import random
  11. from torch.utils.data import Subset, DataLoader
  12. import time
  13.  
  14. # 定义数据集类，处理术语词典
  15. class TranslationDataset(Dataset):
  16.     def __init__(self, filename, terminology):
  17.         self.data = []
  18.         with open(filename, 'r', encoding='utf-8') as f:
  19.             for line in f:
  20.                 en, zh = line.strip().split('\t')
  21.                 self.data.append((en, zh))
  22.         
  23.         self.terminology = terminology
  24.         
  25.         # 创建词汇表，确保术语词典中的词也被包含在词汇表中
  26.         self.en_tokenizer = get_tokenizer('basic_english')
  27.         self.zh_tokenizer = list  # 使用字符级分词
  28.         
  29.         en_vocab = Counter(self.terminology.keys())  # 确保术语在词汇表中
  30.         zh_vocab = Counter()
  31.         
  32.         for en, zh in self.data:
  33.             en_vocab.update(self.en_tokenizer(en))
  34.             zh_vocab.update(self.zh_tokenizer(zh))
  35.         
  36.         self.en_vocab = ['<pad>', '<sos>', '<eos>'] + list(self.terminology.keys()) + [word for word, _ in en_vocab.most_common(10000)]
  37.         self.zh_vocab = ['<pad>', '<sos>', '<eos>'] + [word for word, _ in zh_vocab.most_common(10000)]
  38.         
  39.         self.en_word2idx = {word: idx for idx, word in enumerate(self.en_vocab)}
  40.         self.zh_word2idx = {word: idx for idx, word in enumerate(self.zh_vocab)}
  41.  
  42.     def __len__(self):
  43.         return len(self.data)
  44.  
  45.     def __getitem__(self, idx):
  46.         en, zh = self.data[idx]
  47.         en_tensor = torch.tensor([self.en_word2idx.get(word, self.en_word2idx['<sos>']) for word in self.en_tokenizer(en)] + [self.en_word2idx['<eos>']])
  48.         zh_tensor = torch.tensor([self.zh_word2idx.get(word, self.zh_word2idx['<sos>']) for word in self.zh_tokenizer(zh)] + [self.zh_word2idx['<eos>']])
  49.         return en_tensor, zh_tensor
  50.  
  51. def collate_fn(batch):
  52.     en_batch, zh_batch = [], []
  53.     for en_item, zh_item in batch:
  54.         en_batch.append(en_item)
  55.         zh_batch.append(zh_item)
  56.     
  57.     en_batch = nn.utils.rnn.pad_sequence(en_batch, padding_value=0, batch_first=True)
  58.     zh_batch = nn.utils.rnn.pad_sequence(zh_batch, padding_value=0, batch_first=True)
  59.     
  60.     return en_batch, zh_batch
  61.  
  62. class Encoder(nn.Module):
  63.     def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
  64.         super().__init__()
  65.         self.embedding = nn.Embedding(input_dim, emb_dim)
  66.         self.rnn = nn.GRU(emb_dim, hid_dim, n_layers, dropout=dropout, batch_first=True)
  67.         self.dropout = nn.Dropout(dropout)
  68.  
  69.     def forward(self, src):
  70.         embedded = self.dropout(self.embedding(src))
  71.         outputs, hidden = self.rnn(embedded)
  72.         return outputs, hidden
  73.  
  74. class Decoder(nn.Module):
  75.     def __init__(self, output_dim, emb_dim, hid_dim, n_layers, dropout):
  76.         super().__init__()
  77.         self.output_dim = output_dim
  78.         self.embedding = nn.Embedding(output_dim, emb_dim)
  79.         self.rnn = nn.GRU(emb_dim, hid_dim, n_layers, dropout=dropout, batch_first=True)
  80.         self.fc_out = nn.Linear(hid_dim, output_dim)
  81.         self.dropout = nn.Dropout(dropout)
  82.  
  83.     def forward(self, input, hidden):
  84.         embedded = self.dropout(self.embedding(input))
  85.         output, hidden = self.rnn(embedded, hidden)
  86.         prediction = self.fc_out(output.squeeze(1))
  87.         return prediction, hidden
  88.  
  89. class Seq2Seq(nn.Module):
  90.     def __init__(self, encoder, decoder, device):
  91.         super().__init__()
  92.         self.encoder = encoder
  93.         self.decoder = decoder
  94.         self.device = device
  95.  
  96.     def forward(self, src, trg, teacher_forcing_ratio=0.5):
  97.         batch_size = src.shape[0]
  98.         trg_len = trg.shape[1]
  99.         trg_vocab_size = self.decoder.output_dim
  100.  
  101.         outputs = torch.zeros(batch_size, trg_len, trg_vocab_size).to(self.device)
  102.         
  103.         _, hidden = self.encoder(src)
  104.         
  105.         input = trg[:, 0].unsqueeze(1)
  106.  
  107.         for t in range(1, trg_len):
  108.             output, hidden = self.decoder(input, hidden)
  109.             outputs[:, t, :] = output
  110.             teacher_force = random.random() < teacher_forcing_ratio
  111.             top1 = output.argmax(1)
  112.             input = trg[:, t].unsqueeze(1) if teacher_force else top1.unsqueeze(1)
  113.  
  114.         return outputs
  115.  
  116. def load_terminology_dictionary(dict_file):
  117.     terminology = {}
  118.     with open(dict_file, 'r', encoding='utf-8') as f:
  119.         for line in f:
  120.             en_term, ch_term = line.strip().split('\t')
  121.             terminology[en_term] = ch_term
  122.     return terminology
  123.  
  124. def train(model, iterator, optimizer, criterion, clip):
  125.     model.train()
  126.     epoch_loss = 0
  127.     for i, (src, trg) in enumerate(iterator):
  128.         src, trg = src.to(device), trg.to(device)
  129.         optimizer.zero_grad()
  130.         output = model(src, trg)
  131.         output_dim = output.shape[-1]
  132.         output = output[:, 1:].contiguous().view(-1, output_dim)
  133.         trg = trg[:, 1:].contiguous().view(-1)
  134.         loss = criterion(output, trg)
  135.         loss.backward()
  136.         torch.nn.utils.clip_grad_norm_(model.parameters(), clip)
  137.         optimizer.step()
  138.         epoch_loss += loss.item()
  139.     return epoch_loss / len(iterator)
  140.  
  141. def evaluate_bleu(model, dataset, src_file, ref_file, terminology, device):
  142.     model.eval()
  143.     src_sentences = load_sentences(src_file)
  144.     ref_sentences = load_sentences(ref_file)
  145.     
  146.     translated_sentences = []
  147.     for src in src_sentences:
  148.         translated = translate_sentence(src, model, dataset, terminology, device)
  149.         translated_sentences.append(translated)
  150.     
  151.     bleu = BLEU()
  152.     score = bleu.corpus_score(translated_sentences, [ref_sentences])
  153.     
  154.     return score
  155.  
  156. def translate_sentence(sentence, model, dataset, terminology, device, max_length=50):
  157.     model.eval()
  158.     tokens = dataset.en_tokenizer(sentence)
  159.     tensor = torch.LongTensor([dataset.en_word2idx.get(token, dataset.en_word2idx['<sos>']) for token in tokens]).unsqueeze(0).to(device)
  160.     
  161.     with torch.no_grad():
  162.         _, hidden = model.encoder(tensor)
  163.  
  164.     translated_tokens = []
  165.     input_token = torch.LongTensor([[dataset.zh_word2idx['<sos>']]]).to(device)
  166.  
  167.     for _ in range(max_length):
  168.         output, hidden = model.decoder(input_token, hidden)
  169.         top_token = output.argmax(1)
  170.         translated_token = dataset.zh_vocab[top_token.item()]
  171.         
  172.         if translated_token == '<eos>':
  173.             break
  174.         
  175.         if translated_token in terminology.values():
  176.             for en_term, ch_term in terminology.items():
  177.                 if translated_token == ch_term:
  178.                     translated_token = en_term
  179.                     break
  180.         
  181.         translated_tokens.append(translated_token)
  182.         input_token = top_token.unsqueeze(1)
  183.  
  184.     return ''.join(translated_tokens)
  185.  
  186. def inference(model, dataset, src_file, save_dir, terminology, device):
  187.     model.eval()
  188.     src_sentences = load_sentences(src_file)
  189.     
  190.     translated_sentences = []
  191.     for src in src_sentences:
  192.         translated = translate_sentence(src, model, dataset, terminology, device)
  193.         translated_sentences.append(translated)
  194.     
  195.     text = '\n'.join(translated_sentences)
  196.     with open(save_dir, 'w', encoding='utf-8') as f:
  197.         f.write(text)
  198.  
  199. def load_sentences(file_path):
  200.     with open(file_path, 'r', encoding='utf-8') as f:
  201.         return [line.strip() for line in f]
  202.  
  203. # 主函数
  204. if __name__ == '__main__':
  205.     start_time = time.time()
  206.  
  207.     device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
  208.  
  209.     # 加载术语词典
  210.     terminology = load_terminology_dictionary('../dataset/en-zh.dic')
  211.  
  212.     # 加载数据
  213.     dataset = TranslationDataset('../dataset/train.txt', terminology)
  214.     N = 1000
  215.     subset_indices = list(range(N))
  216.     subset_dataset = Subset(dataset, subset_indices)
  217.     train_loader = DataLoader(subset_dataset, batch_size=32, shuffle=True, collate_fn=collate_fn)
  218.  
  219.     INPUT_DIM = len(dataset.en_vocab)
  220.     OUTPUT_DIM = len(dataset.zh_vocab)
  221.     ENC_EMB_DIM = 256
  222.     DEC_EMB_DIM = 256
  223.     HID_DIM = 512
  224.     N_LAYERS = 2
  225.     ENC_DROPOUT = 0.5
  226.     DEC_DROPOUT = 0.5
  227.  
  228.     enc = Encoder(INPUT_DIM, ENC_EMB_DIM, HID_DIM, N_LAYERS, ENC_DROPOUT)
  229.     dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, HID_DIM, N_LAYERS, DEC_DROPOUT)
  230.     model = Seq2Seq(enc, dec, device).to(device)
  231.  
  232.     optimizer = optim.Adam(model.parameters())
  233.     criterion = nn.CrossEntropyLoss(ignore_index=dataset.zh_word2idx['<pad>'])
  234.  
  235.     N_EPOCHS = 10
  236.     CLIP = 1
  237.  
  238.     for epoch in range(N_EPOCHS):
  239.         train_loss = train(model, train_loader, optimizer, criterion, CLIP)
  240.         print(f'Epoch: {epoch+1:02} | Train Loss: {train_loss:.3f}')
  241.         
  242.     torch.save(model.state_dict(), './translation_model_GRU.pth')
  243.     
  244.     end_time = time.time()
  245.     elapsed_time_minute = (end_time - start_time)/60
  246.     print(f"Total running time: {elapsed_time_minute:.2f} minutes")
  247.  
  248.     bleu_score = evaluate_bleu(model, dataset, '../dataset/dev_en.txt', '../dataset/dev_zh.txt', terminology, device)
  249.     print(f'BLEU-4 score: {bleu_score.score:.2f}')
  250.  
  251.     save_dir = '../dataset/submit.txt'
  252.     inference(model, dataset, "../dataset/test_en.txt", save_dir, terminology, device)
  253.     print(f"翻译完成！文件已保存到{save_dir}")

baseline改进

  • 增加训练样本。
  • 增加训练轮数。
  1. # 选择数据集的前N个样本进行训练
  2.     N = int(len(dataset) * 0.8)
  3. # 训练模型
  4.     N_EPOCHS = 100

如果你觉得这篇博文对你有帮助，请点赞、收藏、关注我，并且可以打赏支持我！

欢迎关注我的后续博文，我将分享更多关于人工智能、自然语言处理和计算机视觉的精彩内容。

谢谢大家的支持！

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/你好赵伟/article/detail/941807
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号