知识图谱：【知识图谱问答KBQA（七）】——P-tuning V2训练代码核心网络层解析_token classification是什么任务

在P-Tuning V2代码中，包括四类NLP任务：

• token_classification:对应序列标注任务
• sequence_classification:对应文本分类任务
• question_answering:对应问答任务
• multiple_choice:对应阅读理解任务

class TaskType(Enum):
    TOKEN_CLASSIFICATION = 1,
    SEQUENCE_CLASSIFICATION = 2,
    QUESTION_ANSWERING = 3,
    MULTIPLE_CHOICE = 4
1
2
3
4
5

其次，P-Tuning V2中训练方法又分为三大类，每一类又可以通过不同的预训练模型实现：

• prefix_models:对应P-Tuning V2方法

  • bert
  • roberta
  • deberta
  • deberta-v2

• prompt_models:对应P-Tuning方法

  • bert
  • roberta

• auto_models:对应fine-tuning方法

下面以sequence_classification任务中P-Tuning V2方法的roberta模型实现代码RobertaPrefixForSequenceClassification为例，介绍下P-Tuning V2的网络结构：

一.网络初始化函数__init__（）

    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels
        self.config = config
        self.roberta = RobertaModel(config)  ## roberta基础模型层
        self.dropout = torch.nn.Dropout(config.hidden_dropout_prob) 
        self.classifier = torch.nn.Linear(config.hidden_size, config.num_labels)  ## 全连接层
        self.init_weights()

        for param in self.roberta.parameters():
            param.requires_grad = False
        
        self.pre_seq_len = config.pre_seq_len  ## The length of prompt
        self.n_layer = config.num_hidden_layers
        self.n_head = config.num_attention_heads
        self.n_embd = config.hidden_size // config.num_attention_heads

        self.prefix_tokens = torch.arange(self.pre_seq_len).long()  ## 初始prefix_tokens
        self.prefix_encoder = PrefixEncoder(config)  ## Prefix编码层

        bert_param = 0
        for name, param in self.roberta.named_parameters():
            bert_param += param.numel()
        all_param = 0
        for name, param in self.named_parameters():
            all_param += param.numel()
        total_param = all_param - bert_param
        print('total param is {}'.format(total_param)) 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

二.get_prompt()函数

    def get_prompt(self, batch_size):
        prefix_tokens = self.prefix_tokens.unsqueeze(0).expand(batch_size, -1).to(self.roberta.device) ## 根据batch_size将初始化的prefix_tokens扩展成（batch_size,pre_seq_len）的张量
        ## 将prefix_tokens输入prefix_encoder层得到past_key_values
        past_key_values = self.prefix_encoder(prefix_tokens)
        ## 对past_key_values进行reshape
        past_key_values = past_key_values.view(
            batch_size,
            self.pre_seq_len,
            self.n_layer * 2, 
            self.n_head,
            self.n_embd
        )
        ## 对past_key_values进行dropout
        past_key_values = self.dropout(past_key_values)
        ## 对past_key_values进行维度调整并分成两两一组
        past_key_values = past_key_values.permute([2, 0, 3, 1, 4]).split(2)
        return past_key_values
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

三.前向传播函数forward()

    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        output_attentions=None,
        output_hidden_states=None,
        return_dict=None,
    ):
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        batch_size = input_ids.shape[0]
        ## get_prompt
        past_key_values = self.get_prompt(batch_size=batch_size)
        prefix_attention_mask = torch.ones(batch_size, self.pre_seq_len).to(self.roberta.device)
        ## prefix_attention_mash与attention_mask进行拼接
        attention_mask = torch.cat((prefix_attention_mask, attention_mask), dim=1)
        ## roberta前向传播
        outputs = self.roberta(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            past_key_values=past_key_values,
        )

        pooled_output = outputs[1]

        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)

        loss = None
        ## 根据相关配置进行loss选择与计算
        if labels is not None:
            if self.config.problem_type is None:
                if self.num_labels == 1:
                    self.config.problem_type = "regression"
                elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):
                    self.config.problem_type = "single_label_classification"
                else:
                    self.config.problem_type = "multi_label_classification"

            if self.config.problem_type == "regression":
                loss_fct = MSELoss()
                if self.num_labels == 1:
                    loss = loss_fct(logits.squeeze(), labels.squeeze())
                else:
                    loss = loss_fct(logits, labels)
            elif self.config.problem_type == "single_label_classification":
                loss_fct = CrossEntropyLoss()
                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
            elif self.config.problem_type == "multi_label_classification":
                loss_fct = BCEWithLogitsLoss()
                loss = loss_fct(logits, labels)
        
        ## 根据return_dict返回前向传播的结果
        if not return_dict:
            output = (logits,) + outputs[2:]
            return ((loss,) + output) if loss is not None else output

        return SequenceClassifierOutput(
            loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions,
        )
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75