devbox
从前慢现在也慢
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

自然语言处理(NLP)——前馈网络

作者：从前慢现在也慢 | 2024-06-27 04:08:28

赞

踩

自然语言处理(NLP)——前馈网络

一、前馈网络概要

1、定义

前馈神经网络(FNN)是一种最简单的神经网络结构，其中信息在一个方向上流动，从输入层到输出层，不会有反馈环路。该网络由多个神经元组成，这些神经元通常被组织成几个层，包括输入层、隐藏层和输出层。

2、特点

简单直观：前馈神经网络的结构比较简单明了，信息只朝一个方向传播，便于理解和实现。

高效性能：尽管简单，前馈神经网络在许多任务中表现出色，尤其在处理大规模数据和复杂模式时有很高的性能表现。

可扩展性：前馈神经网络可以很容易地扩展到更深、更宽的网络结构，以适应不同的任务和数据。

并行计算：由于前馈神经网络的结构，其各个神经元之间的计算是独立的，可以实现高效的并行计算加速。

适用性广泛：前馈神经网络广泛应用于图像识别、语音识别、文本分类、回归分析等多个领域，具有很好的适用性和灵活性。

易于训练：前馈神经网络通常使用反向传播算法进行训练，这是一种有效的优化算法，可以通过调整网络的权重和偏差来最小化损失函数，提高网络的性能。

二、自然语言处理与前馈神经网络

前馈神经网络的结构简单，易于实现和训练并且能够处理高维数据，而在自然语言处理中文本数据通常是高维的，故前馈神经网络十分适用于完成各种自然语言处理的任务。现在前馈神经网络被广泛应用于各种自然语言处理任务，包括但不限于以下几个方面：

  1. 文本分类：前馈神经网络可以用于文本分类任务，例如情感分析、垃圾邮件过滤、新闻分类等。输入文本数据经过处理后，通过前馈神经网络可以实现对文本进行自动分类。

  2. 语言模型：前馈神经网络可以用于构建语言模型，从而实现自然语言生成和语言理解任务。通过训练前馈神经网络，可以学习文本序列之间的模式和关系，从而生成自然流畅的文本或对文本进行理解。

  3. 序列标注：在词性标注、命名实体识别、文本分类等序列标注任务中，前馈神经网络也被广泛应用。通过将输入序列数据经过前馈神经网络的处理，可以实现对序列数据进行标注或分类。

  4. 机器翻译：前馈神经网络在机器翻译任务中也有应用。通过构建编码器-解码器结构的前馈神经网络，可以实现将一种语言的文本翻译成另一种语言的功能。

  5. 文本生成：前馈神经网络还可用于文本生成任务，如对话生成、摘要生成等。通过学习文本序列的模式和关系，前馈神经网络可以生成自然语言文本。

三、基于pytorch的一个简单实现

任务描述：我们将多层感知机应用于将姓氏分类到其原籍国的任务。

代码实现思路：

1、数据预处理

2、构建多层感知机模型

3、训练模型

4、预测结果

以下是具体代码部分：

1、数据预处理

数据集名为surname.csv，它从互联网上不同的姓名来源收集了了来自18个不同国家的10,000个姓氏。数据预处理的目的一是为了平衡数据集中18个国家的姓氏在数据集中的比例，均匀的比例分布有利于训练有效的模型。另外要将数据集分为三个部分:70%到训练数据集，15%到验证数据集，最后15%到测试数据集，以便跨这些部分的类标签分布具有可比性。

  1. from argparse import Namespace
  2. from collections import Counter
  3. import json
  4. import os
  5. import string
  6.  
  7. import numpy as np
  8. import pandas as pd
  9.  
  10. import torch
  11. import torch.nn as nn
  12. import torch.nn.functional as F
  13. import torch.optim as optim
  14. from torch.utils.data import Dataset, DataLoader
  15. from tqdm import tqdm_notebook
  16.  
  17. class Vocabulary(object):
  18.     """Class to process text and extract vocabulary for mapping"""
  19.  
  20.     def __init__(self, token_to_idx=None, add_unk=True, unk_token="<UNK>"):
  21.         """
  22.         Args:
  23.             token_to_idx (dict): a pre-existing map of tokens to indices
  24.             add_unk (bool): a flag that indicates whether to add the UNK token
  25.             unk_token (str): the UNK token to add into the Vocabulary
  26.         """
  27.  
  28.         if token_to_idx is None:
  29.             token_to_idx = {}
  30.         self._token_to_idx = token_to_idx
  31.  
  32.         self._idx_to_token = {idx: token 
  33.                               for token, idx in self._token_to_idx.items()}
  34.         
  35.         self._add_unk = add_unk
  36.         self._unk_token = unk_token
  37.         
  38.         self.unk_index = -1
  39.         if add_unk:
  40.             self.unk_index = self.add_token(unk_token) 
  41.         
  42.         
  43.     def to_serializable(self):
  44.         """ returns a dictionary that can be serialized """
  45.         return {'token_to_idx': self._token_to_idx, 
  46.                 'add_unk': self._add_unk, 
  47.                 'unk_token': self._unk_token}
  48.  
  49.     @classmethod
  50.     def from_serializable(cls, contents):
  51.         """ instantiates the Vocabulary from a serialized dictionary """
  52.         return cls(**contents)
  53.  
  54.     def add_token(self, token):
  55.         """Update mapping dicts based on the token.
  57.         Args:
  58.             token (str): the item to add into the Vocabulary
  59.         Returns:
  60.             index (int): the integer corresponding to the token
  61.         """
  62.         try:
  63.             index = self._token_to_idx[token]
  64.         except KeyError:
  65.             index = len(self._token_to_idx)
  66.             self._token_to_idx[token] = index
  67.             self._idx_to_token[index] = token
  68.         return index
  69.     
  70.     def add_many(self, tokens):
  71.         """Add a list of tokens into the Vocabulary
  72.         
  73.         Args:
  74.             tokens (list): a list of string tokens
  75.         Returns:
  76.             indices (list): a list of indices corresponding to the tokens
  77.         """
  78.         return [self.add_token(token) for token in tokens]
  79.  
  80.     def lookup_token(self, token):
  81.         """Retrieve the index associated with the token 
  82.           or the UNK index if token isn't present.
  83.         
  84.         Args:
  85.             token (str): the token to look up 
  86.         Returns:
  87.             index (int): the index corresponding to the token
  88.         Notes:
  89.             `unk_index` needs to be >=0 (having been added into the Vocabulary) 
  90.               for the UNK functionality 
  91.         """
  92.         if self.unk_index >= 0:
  93.             return self._token_to_idx.get(token, self.unk_index)
  94.         else:
  95.             return self._token_to_idx[token]
  96.  
  97.     def lookup_index(self, index):
  98.         """Return the token associated with the index
  99.         
  100.         Args: 
  101.             index (int): the index to look up
  102.         Returns:
  103.             token (str): the token corresponding to the index
  104.         Raises:
  105.             KeyError: if the index is not in the Vocabulary
  106.         """
  107.         if index not in self._idx_to_token:
  108.             raise KeyError("the index (%d) is not in the Vocabulary" % index)
  109.         return self._idx_to_token[index]
  110.  
  111.     def __str__(self):
  112.         return "<Vocabulary(size=%d)>" % len(self)
  113.  
  114.     def __len__(self):
  115.         return len(self._token_to_idx)
  116.  
  117. class SurnameVectorizer(object):
  118.     """ The Vectorizer which coordinates the Vocabularies and puts them to use"""
  119.     def __init__(self, surname_vocab, nationality_vocab):
  120.         """
  121.         Args:
  122.             surname_vocab (Vocabulary): maps characters to integers
  123.             nationality_vocab (Vocabulary): maps nationalities to integers
  124.         """
  125.         self.surname_vocab = surname_vocab
  126.         self.nationality_vocab = nationality_vocab
  127.  
  128.     def vectorize(self, surname):
  129.         """
  130.         Args:
  131.             surname (str): the surname
  133.         Returns:
  134.             one_hot (np.ndarray): a collapsed one-hot encoding 
  135.         """
  136.         vocab = self.surname_vocab
  137.         one_hot = np.zeros(len(vocab), dtype=np.float32)
  138.         for token in surname:
  139.             one_hot[vocab.lookup_token(token)] = 1
  140.  
  141.         return one_hot
  142.  
  143.     @classmethod
  144.     def from_dataframe(cls, surname_df):
  145.         """Instantiate the vectorizer from the dataset dataframe
  146.         
  147.         Args:
  148.             surname_df (pandas.DataFrame): the surnames dataset
  149.         Returns:
  150.             an instance of the SurnameVectorizer
  151.         """
  152.         surname_vocab = Vocabulary(unk_token="@")
  153.         nationality_vocab = Vocabulary(add_unk=False)
  154.  
  155.         for index, row in surname_df.iterrows():
  156.             for letter in row.surname:
  157.                 surname_vocab.add_token(letter)
  158.             nationality_vocab.add_token(row.nationality)
  159.  
  160.         return cls(surname_vocab, nationality_vocab)
  161.  
  162.     @classmethod
  163.     def from_serializable(cls, contents):
  164.         surname_vocab = Vocabulary.from_serializable(contents['surname_vocab'])
  165.         nationality_vocab =  Vocabulary.from_serializable(contents['nationality_vocab'])
  166.         return cls(surname_vocab=surname_vocab, nationality_vocab=nationality_vocab)
  167.  
  168.     def to_serializable(self):
  169.         return {'surname_vocab': self.surname_vocab.to_serializable(),
  170.                 'nationality_vocab': self.nationality_vocab.to_serializable()}
  171.  
  172. class SurnameDataset(Dataset):
  173.     def __init__(self, surname_df, vectorizer):
  174.         """
  175.         Args:
  176.             surname_df (pandas.DataFrame): the dataset
  177.             vectorizer (SurnameVectorizer): vectorizer instatiated from dataset
  178.         """
  179.         self.surname_df = surname_df
  180.         self._vectorizer = vectorizer
  181.  
  182.         self.train_df = self.surname_df[self.surname_df.split=='train']
  183.         self.train_size = len(self.train_df)
  184.  
  185.         self.val_df = self.surname_df[self.surname_df.split=='val']
  186.         self.validation_size = len(self.val_df)
  187.  
  188.         self.test_df = self.surname_df[self.surname_df.split=='test']
  189.         self.test_size = len(self.test_df)
  190.  
  191.         self._lookup_dict = {'train': (self.train_df, self.train_size),
  192.                              'val': (self.val_df, self.validation_size),
  193.                              'test': (self.test_df, self.test_size)}
  194.  
  195.         self.set_split('train')
  196.         
  197.         # Class weights
  198.         class_counts = surname_df.nationality.value_counts().to_dict()
  199.         def sort_key(item):
  200.             return self._vectorizer.nationality_vocab.lookup_token(item[0])
  201.         sorted_counts = sorted(class_counts.items(), key=sort_key)
  202.         frequencies = [count for _, count in sorted_counts]
  203.         self.class_weights = 1.0 / torch.tensor(frequencies, dtype=torch.float32)
  204.  
  205.     @classmethod
  206.     def load_dataset_and_make_vectorizer(cls, surname_csv):
  207.         """Load dataset and make a new vectorizer from scratch
  208.         
  209.         Args:
  210.             surname_csv (str): location of the dataset
  211.         Returns:
  212.             an instance of SurnameDataset
  213.         """
  214.         surname_df = pd.read_csv(surname_csv)
  215.         train_surname_df = surname_df[surname_df.split=='train']
  216.         return cls(surname_df, SurnameVectorizer.from_dataframe(train_surname_df))
  217.  
  218.     @classmethod
  219.     def load_dataset_and_load_vectorizer(cls, surname_csv, vectorizer_filepath):
  220.         """Load dataset and the corresponding vectorizer. 
  221.         Used in the case in the vectorizer has been cached for re-use
  222.         
  223.         Args:
  224.             surname_csv (str): location of the dataset
  225.             vectorizer_filepath (str): location of the saved vectorizer
  226.         Returns:
  227.             an instance of SurnameDataset
  228.         """
  229.         surname_df = pd.read_csv(surname_csv)
  230.         vectorizer = cls.load_vectorizer_only(vectorizer_filepath)
  231.         return cls(surname_df, vectorizer)
  232.  
  233.     @staticmethod
  234.     def load_vectorizer_only(vectorizer_filepath):
  235.         """a static method for loading the vectorizer from file
  236.         
  237.         Args:
  238.             vectorizer_filepath (str): the location of the serialized vectorizer
  239.         Returns:
  240.             an instance of SurnameVectorizer
  241.         """
  242.         with open(vectorizer_filepath) as fp:
  243.             return SurnameVectorizer.from_serializable(json.load(fp))
  244.  
  245.     def save_vectorizer(self, vectorizer_filepath):
  246.         """saves the vectorizer to disk using json
  247.         
  248.         Args:
  249.             vectorizer_filepath (str): the location to save the vectorizer
  250.         """
  251.         with open(vectorizer_filepath, "w") as fp:
  252.             json.dump(self._vectorizer.to_serializable(), fp)
  253.  
  254.     def get_vectorizer(self):
  255.         """ returns the vectorizer """
  256.         return self._vectorizer
  257.  
  258.     def set_split(self, split="train"):
  259.         """ selects the splits in the dataset using a column in the dataframe """
  260.         self._target_split = split
  261.         self._target_df, self._target_size = self._lookup_dict[split]
  262.  
  263.     def __len__(self):
  264.         return self._target_size
  265.  
  266.     def __getitem__(self, index):
  267.         """the primary entry point method for PyTorch datasets
  268.         
  269.         Args:
  270.             index (int): the index to the data point 
  271.         Returns:
  272.             a dictionary holding the data point's:
  273.                 features (x_surname)
  274.                 label (y_nationality)
  275.         """
  276.         row = self._target_df.iloc[index]
  277.  
  278.         surname_vector = \
  279.             self._vectorizer.vectorize(row.surname)
  280.  
  281.         nationality_index = \
  282.             self._vectorizer.nationality_vocab.lookup_token(row.nationality)
  283.  
  284.         return {'x_surname': surname_vector,
  285.                 'y_nationality': nationality_index}
  286.  
  287.     def get_num_batches(self, batch_size):
  288.         """Given a batch size, return the number of batches in the dataset
  289.         
  290.         Args:
  291.             batch_size (int)
  292.         Returns:
  293.             number of batches in the dataset
  294.         """
  295.         return len(self) // batch_size
  296.  
  297.     
  298. def generate_batches(dataset, batch_size, shuffle=True,
  299.                      drop_last=True, device="cpu"): 
  300.     """
  301.     A generator function which wraps the PyTorch DataLoader. It will 
  302.       ensure each tensor is on the write device location.
  303.     """
  304.     dataloader = DataLoader(dataset=dataset, batch_size=batch_size,
  305.                             shuffle=shuffle, drop_last=drop_last)
  306.  
  307.     for data_dict in dataloader:
  308.         out_data_dict = {}
  309.         for name, tensor in data_dict.items():
  310.             out_data_dict[name] = data_dict[name].to(device)
  311.         yield out_data_dict

2、构建多层感知机模型

  1. class SurnameClassifier(nn.Module):
  2.     """ A 2-layer Multilayer Perceptron for classifying surnames """
  3.     def __init__(self, input_dim, hidden_dim, output_dim):
  4.         """
  5.         Args:
  6.             input_dim (int): the size of the input vectors
  7.             hidden_dim (int): the output size of the first Linear layer
  8.             output_dim (int): the output size of the second Linear layer
  9.         """
  10.         super(SurnameClassifier, self).__init__()
  11.         self.fc1 = nn.Linear(input_dim, hidden_dim)
  12.         self.fc2 = nn.Linear(hidden_dim, output_dim)
  13.  
  14.     def forward(self, x_in, apply_softmax=False):
  15.         """The forward pass of the classifier
  16.         
  17.         Args:
  18.             x_in (torch.Tensor): an input data tensor. 
  19.                 x_in.shape should be (batch, input_dim)
  20.             apply_softmax (bool): a flag for the softmax activation
  21.                 should be false if used with the Cross Entropy losses
  22.         Returns:
  23.             the resulting tensor. tensor.shape should be (batch, output_dim)
  24.         """
  25.         intermediate_vector = F.relu(self.fc1(x_in))
  26.         prediction_vector = self.fc2(intermediate_vector)
  27.  
  28.         if apply_softmax:
  29.             prediction_vector = F.softmax(prediction_vector, dim=1)
  30.  
  31.         return prediction_vector
  32.  
  33. def make_train_state(args):
  34.     return {'stop_early': False,
  35.             'early_stopping_step': 0,
  36.             'early_stopping_best_val': 1e8,
  37.             'learning_rate': args.learning_rate,
  38.             'epoch_index': 0,
  39.             'train_loss': [],
  40.             'train_acc': [],
  41.             'val_loss': [],
  42.             'val_acc': [],
  43.             'test_loss': -1,
  44.             'test_acc': -1,
  45.             'model_filename': args.model_state_file}
  46.  
  47. def update_train_state(args, model, train_state):
  48.     """Handle the training state updates.
  50.     Components:
  51.      - Early Stopping: Prevent overfitting.
  52.      - Model Checkpoint: Model is saved if the model is better
  54.     :param args: main arguments
  55.     :param model: model to train
  56.     :param train_state: a dictionary representing the training state values
  57.     :returns:
  58.         a new train_state
  59.     """
  60.  
  61.     # Save one model at least
  62.     if train_state['epoch_index'] == 0:
  63.         torch.save(model.state_dict(), train_state['model_filename'])
  64.         train_state['stop_early'] = False
  65.  
  66.     # Save model if performance improved
  67.     elif train_state['epoch_index'] >= 1:
  68.         loss_tm1, loss_t = train_state['val_loss'][-2:]
  69.  
  70.         # If loss worsened
  71.         if loss_t >= train_state['early_stopping_best_val']:
  72.             # Update step
  73.             train_state['early_stopping_step'] += 1
  74.         # Loss decreased
  75.         else:
  76.             # Save the best model
  77.             if loss_t < train_state['early_stopping_best_val']:
  78.                 torch.save(model.state_dict(), train_state['model_filename'])
  79.  
  80.             # Reset early stopping step
  81.             train_state['early_stopping_step'] = 0
  82.  
  83.         # Stop early ?
  84.         train_state['stop_early'] = \
  85.             train_state['early_stopping_step'] >= args.early_stopping_criteria
  86.  
  87.     return train_state
  88.  
  89. def compute_accuracy(y_pred, y_target):
  90.     _, y_pred_indices = y_pred.max(dim=1)
  91.     n_correct = torch.eq(y_pred_indices, y_target).sum().item()
  92.     return n_correct / len(y_pred_indices) * 100
  93.  
  94. def set_seed_everywhere(seed, cuda):
  95.     np.random.seed(seed)
  96.     torch.manual_seed(seed)
  97.     if cuda:
  98.         torch.cuda.manual_seed_all(seed)
  99.  
  100. def handle_dirs(dirpath):
  101.     if not os.path.exists(dirpath):
  102.         os.makedirs(dirpath)

3、训练模型

  1. args = Namespace(
  2.     # Data and path information
  3.     surname_csv="surnames_with_splits.csv",
  4.     vectorizer_file="vectorizer.json",
  5.     model_state_file="model.pth",
  6.     save_dir="model_storage/ch4/surname_mlp",
  7.     # Model hyper parameters
  8.     hidden_dim=300,
  9.     # Training  hyper parameters
  10.     seed=1337,
  11.     num_epochs=100,
  12.     early_stopping_criteria=5,
  13.     learning_rate=0.001,
  14.     batch_size=64,
  15.     # Runtime options
  16.     cuda=False,
  17.     reload_from_files=False,
  18.     expand_filepaths_to_save_dir=True,
  19. )
  20.  
  21. if args.expand_filepaths_to_save_dir:
  22.     args.vectorizer_file = os.path.join(args.save_dir,
  23.                                         args.vectorizer_file)
  24.  
  25.     args.model_state_file = os.path.join(args.save_dir,
  26.                                          args.model_state_file)
  27.     
  28.     print("Expanded filepaths: ")
  29.     print("\t{}".format(args.vectorizer_file))
  30.     print("\t{}".format(args.model_state_file))
  31.     
  32. # Check CUDA
  33. if not torch.cuda.is_available():
  34.     args.cuda = False
  35.  
  36. args.device = torch.device("cuda" if args.cuda else "cpu")
  37.     
  38. print("Using CUDA: {}".format(args.cuda))
  39.  
  40.  
  41. # Set seed for reproducibility
  42. set_seed_everywhere(args.seed, args.cuda)
  43.  
  44. # handle dirs
  45. handle_dirs(args.save_dir)
  46.  
  47. if args.reload_from_files:
  48.     # training from a checkpoint
  49.     print("Reloading!")
  50.     dataset = SurnameDataset.load_dataset_and_load_vectorizer(args.surname_csv,
  51.                                                               args.vectorizer_file)
  52. else:
  53.     # create dataset and vectorizer
  54.     print("Creating fresh!")
  55.     dataset = SurnameDataset.load_dataset_and_make_vectorizer(args.surname_csv)
  56.     dataset.save_vectorizer(args.vectorizer_file)
  57.     
  58. vectorizer = dataset.get_vectorizer()
  59. classifier = SurnameClassifier(input_dim=len(vectorizer.surname_vocab), 
  60.                                hidden_dim=args.hidden_dim, 
  61.                                output_dim=len(vectorizer.nationality_vocab))
  62.  
  63. classifier = classifier.to(args.device)
  64. dataset.class_weights = dataset.class_weights.to(args.device)
  65.  
  66.     
  67. loss_func = nn.CrossEntropyLoss(dataset.class_weights)
  68. optimizer = optim.Adam(classifier.parameters(), lr=args.learning_rate)
  69. scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimizer,
  70.                                                  mode='min', factor=0.5,
  71.                                                  patience=1)
  72.  
  73. train_state = make_train_state(args)
  74.  
  75. epoch_bar = tqdm_notebook(desc='training routine', 
  76.                           total=args.num_epochs,
  77.                           position=0)
  78.  
  79. dataset.set_split('train')
  80. train_bar = tqdm_notebook(desc='split=train',
  81.                           total=dataset.get_num_batches(args.batch_size), 
  82.                           position=1, 
  83.                           leave=True)
  84. dataset.set_split('val')
  85. val_bar = tqdm_notebook(desc='split=val',
  86.                         total=dataset.get_num_batches(args.batch_size), 
  87.                         position=1, 
  88.                         leave=True)
  89.  
  90. try:
  91.     for epoch_index in range(args.num_epochs):
  92.         train_state['epoch_index'] = epoch_index
  93.  
  94.         # Iterate over training dataset
  95.  
  96.         # setup: batch generator, set loss and acc to 0, set train mode on
  97.  
  98.         dataset.set_split('train')
  99.         batch_generator = generate_batches(dataset, 
  100.                                            batch_size=args.batch_size, 
  101.                                            device=args.device)
  102.         running_loss = 0.0
  103.         running_acc = 0.0
  104.         classifier.train()
  105.  
  106.         for batch_index, batch_dict in enumerate(batch_generator):
  107.             # the training routine is these 5 steps:
  108.  
  109.             # --------------------------------------
  110.             # step 1. zero the gradients
  111.             optimizer.zero_grad()
  112.  
  113.             # step 2. compute the output
  114.             y_pred = classifier(batch_dict['x_surname'])
  115.  
  116.             # step 3. compute the loss
  117.             loss = loss_func(y_pred, batch_dict['y_nationality'])
  118.             loss_t = loss.item()
  119.             running_loss += (loss_t - running_loss) / (batch_index + 1)
  120.  
  121.             # step 4. use loss to produce gradients
  122.             loss.backward()
  123.  
  124.             # step 5. use optimizer to take gradient step
  125.             optimizer.step()
  126.             # -----------------------------------------
  127.             # compute the accuracy
  128.             acc_t = compute_accuracy(y_pred, batch_dict['y_nationality'])
  129.             running_acc += (acc_t - running_acc) / (batch_index + 1)
  130.  
  131.             # update bar
  132.             train_bar.set_postfix(loss=running_loss, acc=running_acc, 
  133.                             epoch=epoch_index)
  134.             train_bar.update()
  135.  
  136.         train_state['train_loss'].append(running_loss)
  137.         train_state['train_acc'].append(running_acc)
  138.  
  139.         # Iterate over val dataset
  140.  
  141.         # setup: batch generator, set loss and acc to 0; set eval mode on
  142.         dataset.set_split('val')
  143.         batch_generator = generate_batches(dataset, 
  144.                                            batch_size=args.batch_size, 
  145.                                            device=args.device)
  146.         running_loss = 0.
  147.         running_acc = 0.
  148.         classifier.eval()
  149.  
  150.         for batch_index, batch_dict in enumerate(batch_generator):
  151.  
  152.             # compute the output
  153.             y_pred =  classifier(batch_dict['x_surname'])
  154.  
  155.             # step 3. compute the loss
  156.             loss = loss_func(y_pred, batch_dict['y_nationality'])
  157.             loss_t = loss.to("cpu").item()
  158.             running_loss += (loss_t - running_loss) / (batch_index + 1)
  159.  
  160.             # compute the accuracy
  161.             acc_t = compute_accuracy(y_pred, batch_dict['y_nationality'])
  162.             running_acc += (acc_t - running_acc) / (batch_index + 1)
  163.             val_bar.set_postfix(loss=running_loss, acc=running_acc, 
  164.                             epoch=epoch_index)
  165.             val_bar.update()
  166.  
  167.         train_state['val_loss'].append(running_loss)
  168.         train_state['val_acc'].append(running_acc)
  169.  
  170.         train_state = update_train_state(args=args, model=classifier,
  171.                                          train_state=train_state)
  172.  
  173.         scheduler.step(train_state['val_loss'][-1])
  174.  
  175.         if train_state['stop_early']:
  176.             break
  177.  
  178.         train_bar.n = 0
  179.         val_bar.n = 0
  180.         epoch_bar.update()
  181. except KeyboardInterrupt:
  182.     print("Exiting loop")

4、预测结果

  1. # compute the loss & accuracy on the test set using the best available model
  2.  
  3. classifier.load_state_dict(torch.load(train_state['model_filename']))
  4.  
  5. classifier = classifier.to(args.device)
  6. dataset.class_weights = dataset.class_weights.to(args.device)
  7. loss_func = nn.CrossEntropyLoss(dataset.class_weights)
  8.  
  9. dataset.set_split('test')
  10. batch_generator = generate_batches(dataset, 
  11.                                    batch_size=args.batch_size, 
  12.                                    device=args.device)
  13. running_loss = 0.
  14. running_acc = 0.
  15. classifier.eval()
  16.  
  17. for batch_index, batch_dict in enumerate(batch_generator):
  18.     # compute the output
  19.     y_pred =  classifier(batch_dict['x_surname'])
  20.     
  21.     # compute the loss
  22.     loss = loss_func(y_pred, batch_dict['y_nationality'])
  23.     loss_t = loss.item()
  24.     running_loss += (loss_t - running_loss) / (batch_index + 1)
  25.  
  26.     # compute the accuracy
  27.     acc_t = compute_accuracy(y_pred, batch_dict['y_nationality'])
  28.     running_acc += (acc_t - running_acc) / (batch_index + 1)
  29.  
  30. train_state['test_loss'] = running_loss
  31. train_state['test_acc'] = running_acc
  32.  
  33. print("Test loss: {};".format(train_state['test_loss']))
  34. print("Test Accuracy: {}".format(train_state['test_acc']))

  1. def predict_nationality(surname, classifier, vectorizer):
  2.     """Predict the nationality from a new surname
  3.     
  4.     Args:
  5.         surname (str): the surname to classifier
  6.         classifier (SurnameClassifer): an instance of the classifier
  7.         vectorizer (SurnameVectorizer): the corresponding vectorizer
  8.     Returns:
  9.         a dictionary with the most likely nationality and its probability
  10.     """
  11.     vectorized_surname = vectorizer.vectorize(surname)
  12.     vectorized_surname = torch.tensor(vectorized_surname).view(1, -1)
  13.     result = classifier(vectorized_surname, apply_softmax=True)
  14.  
  15.     probability_values, indices = result.max(dim=1)
  16.     index = indices.item()
  17.  
  18.     predicted_nationality = vectorizer.nationality_vocab.lookup_index(index)
  19.     probability_value = probability_values.item()
  20.  
  21.     return {'nationality': predicted_nationality, 'probability': probability_value}
  22.  
  23. new_surname = input("Enter a surname to classify: ")
  24. classifier = classifier.to("cpu")
  25. prediction = predict_nationality(new_surname, classifier, vectorizer)
  26. print("{} -> {} (p={:0.2f})".format(new_surname,
  27.                                     prediction['nationality'],
  28.                                     prediction['probability']))

  1. vectorizer.nationality_vocab.lookup_index(8)
  2.  
  3. def predict_topk_nationality(name, classifier, vectorizer, k=5):
  4.     vectorized_name = vectorizer.vectorize(name)
  5.     vectorized_name = torch.tensor(vectorized_name).view(1, -1)
  6.     prediction_vector = classifier(vectorized_name, apply_softmax=True)
  7.     probability_values, indices = torch.topk(prediction_vector, k=k)
  8.     
  9.     # returned size is 1,k
  10.     probability_values = probability_values.detach().numpy()[0]
  11.     indices = indices.detach().numpy()[0]
  12.     
  13.     results = []
  14.     for prob_value, index in zip(probability_values, indices):
  15.         nationality = vectorizer.nationality_vocab.lookup_index(index)
  16.         results.append({'nationality': nationality, 
  17.                         'probability': prob_value})
  18.     
  19.     return results
  20.  
  21.  
  22. new_surname = input("Enter a surname to classify: ")
  23. classifier = classifier.to("cpu")
  24.  
  25. k = int(input("How many of the top predictions to see? "))
  26. if k > len(vectorizer.nationality_vocab):
  27.     print("Sorry! That's more than the # of nationalities we have.. defaulting you to max size :)")
  28.     k = len(vectorizer.nationality_vocab)
  29.     
  30. predictions = predict_topk_nationality(new_surname, classifier, vectorizer, k=k)
  31.  
  32. print("Top {} predictions:".format(k))
  33. print("===================")
  34. for prediction in predictions:
  35.     print("{} -> {} (p={:0.2f})".format(new_surname,
  36.                                         prediction['nationality'],
  37.                                         prediction['probability']))

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/从前慢现在也慢/article/detail/761289
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号