devbox
我家小花儿
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

昇思25天学习打卡营第17天|LLM-基于MindSpore的GPT2文本摘要

作者：我家小花儿 | 2024-07-22 11:49:16

赞

踩

昇思25天学习打卡营第17天|LLM-基于MindSpore的GPT2文本摘要

打卡

目录

打卡

环境准备

准备阶段

数据加载与预处理

BertTokenizer

部分输出

模型构建

gpt2模型结构输出

训练流程

部分输出

部分输出2（减少训练数据）

推理流程

环境准备

  1. pip install -i https://pypi.mirrors.ustc.edu.cn/simple mindspore==2.2.14
  2.  
  3. pip install tokenizers==0.15.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
  4. # 该案例在 mindnlp 0.3.1 版本完成适配，如果发现案例跑不通，可以指定mindnlp版本，执行`!pip install mindnlp==0.3.1`
  5.  
  6. pip install mindnlp

准备阶段

nlpcc2017摘要数据，内容为新闻正文及其摘要，总计50000个样本。

来源：nlpcc2017摘要数据

数据加载与预处理

  • 原始数据格式：
  1. article: [CLS] article_context [SEP]
  2. summary: [CLS] summary_context [SEP]
  • 预处理后的数据格式：
[CLS] article_context [SEP] summary_context [SEP]

BertTokenizer

因GPT2无中文的tokenizer，使用BertTokenizer替代。代码如下：

  1. from mindspore.dataset import TextFileDataset
  2. import json
  3. import numpy as np
  4. from mindnlp.transformers import BertTokenizer
  5.  
  6.  
  7. # preprocess dataset
  8. def process_dataset(dataset, tokenizer, batch_size=6, max_seq_len=1024, shuffle=False):
  9.     def read_map(text):
  10.         data = json.loads(text.tobytes())
  11.         return np.array(data['article']), np.array(data['summarization'])
  12.  
  13.     def merge_and_pad(article, summary):
  14.         # tokenization
  15.         # pad to max_seq_length, only truncate the article
  16.         tokenized = tokenizer(text=article, text_pair=summary,
  17.                               padding='max_length', truncation='only_first', max_length=max_seq_len)
  18.         return tokenized['input_ids'], tokenized['input_ids']
  19.     
  20.     dataset = dataset.map(read_map, 'text', ['article', 'summary'])
  21.     # change column names to input_ids and labels for the following training
  22.     dataset = dataset.map(merge_and_pad, ['article', 'summary'], ['input_ids', 'labels'])
  23.  
  24.     dataset = dataset.batch(batch_size)
  25.     if shuffle:
  26.         dataset = dataset.shuffle(batch_size)
  27.  
  28.     return dataset
  29.  
  30.  
  31. # load dataset
  32. dataset = TextFileDataset(str(path), shuffle=False)
  33. print(dataset.get_dataset_size())   ### 50000
  34.  
  35. # split into training and testing dataset
  36. train_dataset, test_dataset = dataset.split([0.9, 0.1], randomize=False)
  37. print(len(train_dataset))  ### 45000
  38.  
  39. # We use BertTokenizer for tokenizing chinese context.
  40. tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
  41. len(tokenizer)
  42.  
  43. train_dataset = process_dataset(train_dataset, tokenizer, batch_size=4)
  44. ## next(train_dataset.create_tuple_iterator())
  45.  
  46.

部分输出

模型构建

如下，通过两个类实现：

  1. 构建GPT2ForSummarization模型，注意shift right的操作。
  2. 动态学习率
  1. from mindspore import ops
  2. from mindnlp.transformers import GPT2LMHeadModel 
  3. from mindspore.nn.learning_rate_schedule import LearningRateSchedule
  4.  
  5. from mindspore import nn
  6. from mindnlp.transformers import GPT2Config, GPT2LMHeadModel
  7. from mindnlp._legacy.engine import Trainer
  8. from mindnlp._legacy.engine.callbacks import CheckpointCallback
  9.  
  10.  
  11.  
  12. class GPT2ForSummarization(GPT2LMHeadModel):
  13.     def construct(
  14.         self,
  15.         input_ids = None,
  16.         attention_mask = None,
  17.         labels = None,
  18.     ):
  19.         outputs = super().construct(input_ids=input_ids, attention_mask=attention_mask)
  20.         shift_logits = outputs.logits[..., :-1, :]
  21.         shift_labels = labels[..., 1:]
  22.         # Flatten the tokens
  23.         loss = ops.cross_entropy(shift_logits.view(-1, shift_logits.shape[-1]), shift_labels.view(-1), ignore_index=tokenizer.pad_token_id)
  24.         return loss
  25.  
  26.  
  27. class LinearWithWarmUp(LearningRateSchedule):
  28.     """
  29.     Warmup-decay learning rate.
  30.     """
  31.     def __init__(self, learning_rate, num_warmup_steps, num_training_steps):
  32.         super().__init__()
  33.         self.learning_rate = learning_rate
  34.         self.num_warmup_steps = num_warmup_steps
  35.         self.num_training_steps = num_training_steps
  36.  
  37.     def construct(self, global_step):
  38.         if global_step < self.num_warmup_steps:
  39.             return global_step / float(max(1, self.num_warmup_steps)) * self.learning_rate
  40.         return ops.maximum(
  41.             0.0, (self.num_training_steps - global_step) / (max(1, self.num_training_steps - self.num_warmup_steps))
  42.         ) * self.learning_rate
  43.  
  44.  
  45. ## 训练参数设置
  46. num_epochs = 1
  47. warmup_steps = 2000
  48. learning_rate = 1.5e-4
  49.  
  50. num_training_steps = num_epochs * train_dataset.get_dataset_size()
  51.  
  52.  
  53. config = GPT2Config(vocab_size=len(tokenizer))
  54. model = GPT2ForSummarization(config)
  55.  
  56. lr_scheduler = LinearWithWarmUp(
  57.         learning_rate=learning_rate, 
  58.         num_warmup_steps=warmup_steps, 
  59.         num_training_steps=num_training_steps)
  60. optimizer = nn.AdamWeightDecay(model.trainable_params(), 
  61.                                learning_rate=lr_scheduler)
  62.  
  63. # 记录模型参数数量
  64. print('number of model parameters: {}'.format(model.num_parameters()))

gpt2模型结构输出

1. 1级主类：GPT2ForSummarization

2. 2级类：GPT2Model 层，是transformer 结构，是模型的核心部分。

3. 2级类：lm_head 结构的 Dense 全连接层 , dim[in, out]=[768,  21128]。

4. GPT2Model 结构下的3级类组件分三层：

        >> wte 嵌入层：dim[in, out]=[21128, 768] ，即使用了 21128 个词汇，每个词汇映射到一个768 维的向量。

        >> wpe 嵌入层：dim[in, out]=[1024, 768] 

        >> drop 层。

        >> layers h 隐网络结构层：Transformer模型的主体，包含 12 个 GPT2Block。  

        >> ln_f LayerNorm 最后的层归一化。        

5. GPT2Block 的结构：

        》》ln_1 LayerNorm层，层归一化，用于在注意力机制之前对输入进行归一化。

        》》attn GPT2Attention层，自注意力机制，用于计算输入序列中不同位置的注意力权重。共包括3层：Conv1D、Conv1D、CustomDropout、CustomDropout。

        》》ln_2 LayerNorm层，用于自注意力之后的归一化。

        》》mlp  GPT2MLP层，多层感知机，用于对自注意力层的输出进行进一步的非线性变换。这里使用的操作包括：Conv1D、Conv1D、GELU、CustomDropout。
 

  1. $ print(model)
  2.  
  3. GPT2ForSummarization<
  4.   (transformer): GPT2Model<
  5.     (wte): Embedding<vocab_size=21128, embedding_size=768, use_one_hot=False, weight=Parameter (Tensor(shape=[21128, 768], dtype=Float32, value=[...], name=transformer.wte.weight), requires_grad=True), dtype=Float32, padding_idx=None>
  6.     (wpe): Embedding<vocab_size=1024, embedding_size=768, use_one_hot=False, weight=Parameter (Tensor(shape=[1024, 768], dtype=Float32, value=[...], name=transformer.wpe.weight), requires_grad=True), dtype=Float32, padding_idx=None>
  7.     (drop): CustomDropout<>
  8.     (h): CellList<
  9.       (0): GPT2Block<
  10.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.0.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.0.ln_1.bias), requires_grad=True)>
  11.         (attn): GPT2Attention<
  12.           (c_attn): Conv1D<
  13.             (matmul): Matmul<>
  14.             >
  15.           (c_proj): Conv1D<
  16.             (matmul): Matmul<>
  17.             >
  18.           (attn_dropout): CustomDropout<>
  19.           (resid_dropout): CustomDropout<>
  20.           >
  21.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.0.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.0.ln_2.bias), requires_grad=True)>
  22.         (mlp): GPT2MLP<
  23.           (c_fc): Conv1D<
  24.             (matmul): Matmul<>
  25.             >
  26.           (c_proj): Conv1D<
  27.             (matmul): Matmul<>
  28.             >
  29.           (act): GELU<>
  30.           (dropout): CustomDropout<>
  31.           >
  32.         >
  33.       (1): GPT2Block<
  34.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.1.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.1.ln_1.bias), requires_grad=True)>
  35.         (attn): GPT2Attention<
  36.           (c_attn): Conv1D<
  37.             (matmul): Matmul<>
  38.             >
  39.           (c_proj): Conv1D<
  40.             (matmul): Matmul<>
  41.             >
  42.           (attn_dropout): CustomDropout<>
  43.           (resid_dropout): CustomDropout<>
  44.           >
  45.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.1.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.1.ln_2.bias), requires_grad=True)>
  46.         (mlp): GPT2MLP<
  47.           (c_fc): Conv1D<
  48.             (matmul): Matmul<>
  49.             >
  50.           (c_proj): Conv1D<
  51.             (matmul): Matmul<>
  52.             >
  53.           (act): GELU<>
  54.           (dropout): CustomDropout<>
  55.           >
  56.         >
  57.       (2): GPT2Block<
  58.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.2.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.2.ln_1.bias), requires_grad=True)>
  59.         (attn): GPT2Attention<
  60.           (c_attn): Conv1D<
  61.             (matmul): Matmul<>
  62.             >
  63.           (c_proj): Conv1D<
  64.             (matmul): Matmul<>
  65.             >
  66.           (attn_dropout): CustomDropout<>
  67.           (resid_dropout): CustomDropout<>
  68.           >
  69.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.2.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.2.ln_2.bias), requires_grad=True)>
  70.         (mlp): GPT2MLP<
  71.           (c_fc): Conv1D<
  72.             (matmul): Matmul<>
  73.             >
  74.           (c_proj): Conv1D<
  75.             (matmul): Matmul<>
  76.             >
  77.           (act): GELU<>
  78.           (dropout): CustomDropout<>
  79.           >
  80.         >
  81.       (3): GPT2Block<
  82.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.3.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.3.ln_1.bias), requires_grad=True)>
  83.         (attn): GPT2Attention<
  84.           (c_attn): Conv1D<
  85.             (matmul): Matmul<>
  86.             >
  87.           (c_proj): Conv1D<
  88.             (matmul): Matmul<>
  89.             >
  90.           (attn_dropout): CustomDropout<>
  91.           (resid_dropout): CustomDropout<>
  92.           >
  93.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.3.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.3.ln_2.bias), requires_grad=True)>
  94.         (mlp): GPT2MLP<
  95.           (c_fc): Conv1D<
  96.             (matmul): Matmul<>
  97.             >
  98.           (c_proj): Conv1D<
  99.             (matmul): Matmul<>
  100.             >
  101.           (act): GELU<>
  102.           (dropout): CustomDropout<>
  103.           >
  104.         >
  105.       (4): GPT2Block<
  106.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.4.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.4.ln_1.bias), requires_grad=True)>
  107.         (attn): GPT2Attention<
  108.           (c_attn): Conv1D<
  109.             (matmul): Matmul<>
  110.             >
  111.           (c_proj): Conv1D<
  112.             (matmul): Matmul<>
  113.             >
  114.           (attn_dropout): CustomDropout<>
  115.           (resid_dropout): CustomDropout<>
  116.           >
  117.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.4.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.4.ln_2.bias), requires_grad=True)>
  118.         (mlp): GPT2MLP<
  119.           (c_fc): Conv1D<
  120.             (matmul): Matmul<>
  121.             >
  122.           (c_proj): Conv1D<
  123.             (matmul): Matmul<>
  124.             >
  125.           (act): GELU<>
  126.           (dropout): CustomDropout<>
  127.           >
  128.         >
  129.       (5): GPT2Block<
  130.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.5.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.5.ln_1.bias), requires_grad=True)>
  131.         (attn): GPT2Attention<
  132.           (c_attn): Conv1D<
  133.             (matmul): Matmul<>
  134.             >
  135.           (c_proj): Conv1D<
  136.             (matmul): Matmul<>
  137.             >
  138.           (attn_dropout): CustomDropout<>
  139.           (resid_dropout): CustomDropout<>
  140.           >
  141.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.5.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.5.ln_2.bias), requires_grad=True)>
  142.         (mlp): GPT2MLP<
  143.           (c_fc): Conv1D<
  144.             (matmul): Matmul<>
  145.             >
  146.           (c_proj): Conv1D<
  147.             (matmul): Matmul<>
  148.             >
  149.           (act): GELU<>
  150.           (dropout): CustomDropout<>
  151.           >
  152.         >
  153.       (6): GPT2Block<
  154.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.6.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.6.ln_1.bias), requires_grad=True)>
  155.         (attn): GPT2Attention<
  156.           (c_attn): Conv1D<
  157.             (matmul): Matmul<>
  158.             >
  159.           (c_proj): Conv1D<
  160.             (matmul): Matmul<>
  161.             >
  162.           (attn_dropout): CustomDropout<>
  163.           (resid_dropout): CustomDropout<>
  164.           >
  165.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.6.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.6.ln_2.bias), requires_grad=True)>
  166.         (mlp): GPT2MLP<
  167.           (c_fc): Conv1D<
  168.             (matmul): Matmul<>
  169.             >
  170.           (c_proj): Conv1D<
  171.             (matmul): Matmul<>
  172.             >
  173.           (act): GELU<>
  174.           (dropout): CustomDropout<>
  175.           >
  176.         >
  177.       (7): GPT2Block<
  178.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.7.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.7.ln_1.bias), requires_grad=True)>
  179.         (attn): GPT2Attention<
  180.           (c_attn): Conv1D<
  181.             (matmul): Matmul<>
  182.             >
  183.           (c_proj): Conv1D<
  184.             (matmul): Matmul<>
  185.             >
  186.           (attn_dropout): CustomDropout<>
  187.           (resid_dropout): CustomDropout<>
  188.           >
  189.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.7.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.7.ln_2.bias), requires_grad=True)>
  190.         (mlp): GPT2MLP<
  191.           (c_fc): Conv1D<
  192.             (matmul): Matmul<>
  193.             >
  194.           (c_proj): Conv1D<
  195.             (matmul): Matmul<>
  196.             >
  197.           (act): GELU<>
  198.           (dropout): CustomDropout<>
  199.           >
  200.         >
  201.       (8): GPT2Block<
  202.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.8.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.8.ln_1.bias), requires_grad=True)>
  203.         (attn): GPT2Attention<
  204.           (c_attn): Conv1D<
  205.             (matmul): Matmul<>
  206.             >
  207.           (c_proj): Conv1D<
  208.             (matmul): Matmul<>
  209.             >
  210.           (attn_dropout): CustomDropout<>
  211.           (resid_dropout): CustomDropout<>
  212.           >
  213.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.8.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.8.ln_2.bias), requires_grad=True)>
  214.         (mlp): GPT2MLP<
  215.           (c_fc): Conv1D<
  216.             (matmul): Matmul<>
  217.             >
  218.           (c_proj): Conv1D<
  219.             (matmul): Matmul<>
  220.             >
  221.           (act): GELU<>
  222.           (dropout): CustomDropout<>
  223.           >
  224.         >
  225.       (9): GPT2Block<
  226.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.9.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.9.ln_1.bias), requires_grad=True)>
  227.         (attn): GPT2Attention<
  228.           (c_attn): Conv1D<
  229.             (matmul): Matmul<>
  230.             >
  231.           (c_proj): Conv1D<
  232.             (matmul): Matmul<>
  233.             >
  234.           (attn_dropout): CustomDropout<>
  235.           (resid_dropout): CustomDropout<>
  236.           >
  237.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.9.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.9.ln_2.bias), requires_grad=True)>
  238.         (mlp): GPT2MLP<
  239.           (c_fc): Conv1D<
  240.             (matmul): Matmul<>
  241.             >
  242.           (c_proj): Conv1D<
  243.             (matmul): Matmul<>
  244.             >
  245.           (act): GELU<>
  246.           (dropout): CustomDropout<>
  247.           >
  248.         >
  249.       (10): GPT2Block<
  250.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.10.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.10.ln_1.bias), requires_grad=True)>
  251.         (attn): GPT2Attention<
  252.           (c_attn): Conv1D<
  253.             (matmul): Matmul<>
  254.             >
  255.           (c_proj): Conv1D<
  256.             (matmul): Matmul<>
  257.             >
  258.           (attn_dropout): CustomDropout<>
  259.           (resid_dropout): CustomDropout<>
  260.           >
  261.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.10.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.10.ln_2.bias), requires_grad=True)>
  262.         (mlp): GPT2MLP<
  263.           (c_fc): Conv1D<
  264.             (matmul): Matmul<>
  265.             >
  266.           (c_proj): Conv1D<
  267.             (matmul): Matmul<>
  268.             >
  269.           (act): GELU<>
  270.           (dropout): CustomDropout<>
  271.           >
  272.         >
  273.       (11): GPT2Block<
  274.         (ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.11.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.11.ln_1.bias), requires_grad=True)>
  275.         (attn): GPT2Attention<
  276.           (c_attn): Conv1D<
  277.             (matmul): Matmul<>
  278.             >
  279.           (c_proj): Conv1D<
  280.             (matmul): Matmul<>
  281.             >
  282.           (attn_dropout): CustomDropout<>
  283.           (resid_dropout): CustomDropout<>
  284.           >
  285.         (ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.11.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.11.ln_2.bias), requires_grad=True)>
  286.         (mlp): GPT2MLP<
  287.           (c_fc): Conv1D<
  288.             (matmul): Matmul<>
  289.             >
  290.           (c_proj): Conv1D<
  291.             (matmul): Matmul<>
  292.             >
  293.           (act): GELU<>
  294.           (dropout): CustomDropout<>
  295.           >
  296.         >
  297.       >
  298.     (ln_f): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.ln_f.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.ln_f.bias), requires_grad=True)>
  299.     >
  300.   (lm_head): Dense<input_channels=768, output_channels=21128>
  301.   >

训练流程

  1. from mindspore import nn
  2. from mindnlp.transformers import GPT2Config, GPT2LMHeadModel
  3. from mindnlp._legacy.engine import Trainer
  4. from mindnlp._legacy.engine.callbacks import CheckpointCallback
  5.  
  6.  
  7. # 记录模型参数数量
  8. print('number of model parameters: {}'.format(model.num_parameters()))
  9.  
  10. ckpoint_cb = CheckpointCallback(save_path='checkpoint', ckpt_name='gpt2_summarization',
  11.                                 epochs=1, keep_checkpoint_max=2)
  12.  
  13. trainer = Trainer(network=model, 
  14.                   train_dataset=train_dataset,
  15.                   epochs=1, 
  16.                   optimizer=optimizer, 
  17.                   callbacks=ckpoint_cb)
  18. trainer.set_amp(level='O1')  # 开启混合精度
  19.  
  20. trainer.run(tgt_columns="labels")

部分输出

注：建议使用较高规格的算力，训练时间较长

部分输出2（减少训练数据）

此次活动的 notebook 只可以连续运行8小时，此次目的也不是性能优化，故此，我将训练数据减少到了1/10，此时的部分输出如下。

推理流程

  1. ## 向量数据转为中文数据
  2. def process_test_dataset(dataset, tokenizer, batch_size=1, max_seq_len=1024, max_summary_len=100):
  3.     def read_map(text):
  4.         data = json.loads(text.tobytes())
  5.         return np.array(data['article']), np.array(data['summarization'])
  6.  
  7.     def pad(article):
  8.         tokenized = tokenizer(text=article, truncation=True, max_length=max_seq_len-max_summary_len)
  9.         return tokenized['input_ids']
  10.  
  11.     dataset = dataset.map(read_map, 'text', ['article', 'summary'])
  12.     dataset = dataset.map(pad, 'article', ['input_ids'])
  13.     
  14.     dataset = dataset.batch(batch_size)
  15.  
  16.     return dataset
  17.  
  18.  
  19. test_dataset = process_test_dataset(test_dataset, tokenizer, batch_size=1)
  20. print(next(test_dataset.create_tuple_iterator(output_numpy=True)))
  21.  
  22. model = GPT2LMHeadModel.from_pretrained('./checkpoint/gpt2_summarization_epoch_0.ckpt', config=config)
  23.  
  24. model.set_train(False)
  25. model.config.eos_token_id = model.config.sep_token_id
  26. i = 0
  27. for (input_ids, raw_summary) in test_dataset.create_tuple_iterator():
  28.     output_ids = model.generate(input_ids, max_new_tokens=50, num_beams=5, no_repeat_ngram_size=2)
  29.     output_text = tokenizer.decode(output_ids[0].tolist())
  30.     print(output_text)
  31.     i += 1
  32.     if i == 1:
  33.         break

减少训练数据后的模型推理结果展示。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/我家小花儿/article/detail/865033
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号