devbox
你好赵伟
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

深度学习（8）：基于BERT算法的文本内容情感分析_基于bert的文本情感分析

作者：你好赵伟 | 2024-05-25 23:26:10

赞

踩

基于bert的文本情感分析

目标：基于BERT网络实现对文本的情感进行分析，将网络上的商品评论内容经过预处理后输入BERT模型训练和推理，最后将判断结果进行输出。

一、原理

1.了解BERT算法的基本原理

BERT介绍

谷歌AI团队发布的BERT模型在11种不同的自然语言处理任务中创出佳成绩，为自然语言处理带来里程碑式的改变，也是自然语言处理领域近期重要的进展。

BERT是一种对语言表征进行预训练的方法, 即是经过大型文本语料库（如维基百科）训练后获得的通用“语言理解”模型,该模型可用于自然语言处理下游任务（如自动问答）。BERT之所以表现得比过往的方法要好, 是因为它是首个用于自然语言处理预训练的无监督、深度双向系统。BERT的优势是能够轻松适用多种类型的自然语言处理任务。

Bert最关键有两点，第一点是特征抽取器采用Transformer，第二点是预训练的时候采用双向语言模型。
 

参考博客：Bert文本分类实战（附代码讲解）_Dr.sky_的博客-CSDN博客_bert实战

2.熟悉文本分类的常规方法

文本分类流程：1.输入文本预处理，2.文本表示及特征提取，3.构造分类器模型，4.文本分类。

文本分类技术参考博客：一文读懂文本分类技术路线_Yunlord的博客-CSDN博客_文本分类技术

二、过程

1.准备数据

  1. #准备数据，从OSS中获取数据并解压到当前目录：
  2.  
  3. import os
  4. import oss2
  5. access_key_id = os.getenv('OSS_TEST_ACCESS_KEY_ID', 'LTAI4G1MuHTUeNrKdQEPnbph')
  6. access_key_secret = os.getenv('OSS_TEST_ACCESS_KEY_SECRET', 'm1ILSoVqcPUxFFDqer4tKDxDkoP1ji')
  7. bucket_name = os.getenv('OSS_TEST_BUCKET', 'mldemo')
  8. endpoint = os.getenv('OSS_TEST_ENDPOINT', 'https://oss-cn-shanghai.aliyuncs.com')
  9. # 创建Bucket对象，所有Object相关的接口都可以通过Bucket对象来进行
  10. bucket = oss2.Bucket(oss2.Auth(access_key_id, access_key_secret), endpoint, bucket_name)
  11. # 下载到本地文件
  12. bucket.get_object_to_file('data/c12/bert_data.zip', 'bert_data.zip')

  1. #解压数据
  2. !unzip  -o -q bert_data.zip
  3. !rm -rf __MACOSX

!ls bert_input_data -ilht

 2.导入库

  1. import collections
  2. import csv
  3. import errno
  4. import tensorflow as tf
  5. import logging
  6. import logging as log
  7. import sys, os
  8. import traceback
  9. from sklearn.utils import shuffle
  10. import pandas as pd
  11. import numpy as np
  12. import modeling
  13. import optimization
  14. import tokenization
  15. %matplotlib inline

 

 3.读取数据

  1. #读取数据
  2. df = pd.read_csv("./bert_input_data/train_data.tsv", header=0,sep='\t').sample(n=100,random_state=1)
  1. df.columns=['baseid','xtext','category']
  2. df.head(5)

df.info()

  1. # number of different class
  2. print('\nnumber of different class: ', len(list(set(df.category))))
  3. print(list(set(df.category)))

 

查看不同类别数据对比情况：

df.category.value_counts().plot(kind='bar')

 

4.构建训练集

构建训练数据集，首先随机化数据：

  1. from sklearn.utils import shuffle
  2. df = shuffle(df,random_state=0)
  1. #查看随机化之后的数据情况
  2. df.head()

 

 将数据按照8:1:1分为训练集、验证信和测试集三部分、

  1. msk = np.random.rand(len(df)) < 0.8
  2. train = df[msk]
  3. dev_test = df[~msk]
  4. msk = np.random.rand(len(dev_test)) < 0.5
  5.  
  6. dev = dev_test[msk]
  7. test = dev_test[~msk]

 将数据集存为tsv格式，作为BERT模型的输入

  1. export_csv_train = train.to_csv ('./bert_input_data/level1_train.tsv', sep='\t', index = None, header=None)
  2. export_csv_dev = dev.to_csv ('./bert_input_data/level1_dev.tsv', sep='\t',index = None, header=None)
  3. export_csv_test = test.to_csv ('./bert_input_data/level1_test.tsv', sep='\t',index = None, header=None)

5.模型训练

定义模型超参

  1. MODEL_OUTPUT_DIR = "./bert_output/"
  2. init_checkpoint = "./chinese_wwm_ext_L-12_H-768_A-12/bert_model.ckpt"
  3. bert_config_file = "./chinese_wwm_ext_L-12_H-768_A-12/bert_config.json"
  4. vocab_file = "./chinese_wwm_ext_L-12_H-768_A-12/vocab.txt"
  5. save_checkpoints_steps = 200
  6. iterations_per_loop = 100
  7. num_tpu_cores = 4
  8. warmup_proportion =100
  9. train_batch_size = 1
  10. learning_rate=5e-5
  11. eval_batch_size =1
  12. predict_batch_size=2
  13. max_seq_length =16
  14. data_dir = "./bert_input_data/"
  1. #清理模型输出目录，减少磁盘占用
  2. !rm bert_output -rf

BERT算法数据准备及训练代码：

  1. class InputExample(object):
  2.     """A single training/test example for simple sequence classification."""
  3.  
  4.     def __init__(self, guid, text_a, text_b=None, label=None):
  5.         self.guid = guid
  6.         self.text_a = text_a
  7.         self.text_b = text_b
  8.         self.label = label
  9.  
  10.  
  11. class PaddingInputExample(object):
  12.     pass
  13.  
  14.  
  15. class InputFeatures(object):
  16.     """A single set of features of data."""
  17.  
  18.     def __init__(self,
  19.                              input_ids,
  20.                              input_mask,
  21.                              segment_ids,
  22.                              label_id,
  23.                              is_real_example=True):
  24.         self.input_ids = input_ids
  25.         self.input_mask = input_mask
  26.         self.segment_ids = segment_ids
  27.         self.label_id = label_id
  28.         self.is_real_example = is_real_example
  29.  
  30.  
  31. class DataProcessor(object):
  32.     """Base class for data converters for sequence classification data sets."""
  33.  
  34.     def get_train_examples(self, data_dir):
  35.         """Gets a collection of `InputExample`s for the train set."""
  36.         raise NotImplementedError()
  37.  
  38.     def get_dev_examples(self, data_dir):
  39.         """Gets a collection of `InputExample`s for the dev set."""
  40.         raise NotImplementedError()
  41.  
  42.     def get_test_examples(self, data_dir):
  43.         """Gets a collection of `InputExample`s for prediction."""
  44.         raise NotImplementedError()
  45.  
  46.     def get_labels(self):
  47.         """Gets the list of labels for this data set."""
  48.         raise NotImplementedError()
  49.  
  50.     @classmethod
  51.     def _read_tsv(cls, input_file, quotechar=None):
  52.         """Reads a tab separated value file."""
  53.         with tf.gfile.Open(input_file, "r") as f:
  54.             reader = csv.reader(f, delimiter="\t", quotechar=quotechar)
  55.             lines = []
  56.             for line in reader:
  57.                 lines.append(line)
  58.             return lines
  59.  
  60. class SHLibProcessor(DataProcessor):
  61.     """Processor for the SHlib data set ."""
  62.     def __init__(self, label_list):
  63.         self.static_label_list = load_labels(label_list)
  64.  
  65.     def get_train_examples(self, train_lines):
  66.         """See base class."""
  67.         return self._create_examples(train_lines, "train")
  68.  
  69.     def get_dev_examples(self, eval_lines):
  70.         """See base class."""
  71.         return self._create_examples(eval_lines, "dev")
  72.  
  73.     def get_test_examples(self, predict_lines):
  74.         """See base class."""
  75.         return self._create_examples(predict_lines, "test")
  76.  
  77.     def get_labels(self):
  78.         """See base class."""
  79.         return self.static_label_list
  80.  
  81.  
  82.     def _create_examples(self, lines, set_type):
  83.         """Creates examples for the training and dev sets."""
  84.         examples = []
  85.         for (i, line) in enumerate(lines):
  86.  
  87.             guid = line[0]# "%s-%s" % (set_type, i)
  88.             if set_type == "test":
  89.                 text_a = tokenization.convert_to_unicode(line[1])
  90.                 label = tokenization.convert_to_unicode(line[2])
  91.             else:
  92.                 text_a = tokenization.convert_to_unicode(line[1])
  93.                 label = tokenization.convert_to_unicode(line[2])
  94.             examples.append(
  95.                 InputExample(guid=guid, text_a=text_a, text_b=None, label=label))
  96.         return examples
  97.  
  98.  
  99.     def load_labels(self, label_file_path):
  100.         with open(label_file_path,'r') as label_file:
  101.                 static_label_list = list(label_file.read().splitlines())
  102.                 print(static_label_list)
  103.         return static_label_list
  104.  
  105. def convert_single_example(ex_index, example, label_list, max_seq_length,
  106.                                                          tokenizer):
  107.     """Converts a single `InputExample` into a single `InputFeatures`."""
  108.  
  109.     if isinstance(example, PaddingInputExample):
  110.         return InputFeatures(
  111.                 input_ids=[0] * max_seq_length,
  112.                 input_mask=[0] * max_seq_length,
  113.                 segment_ids=[0] * max_seq_length,
  114.                 label_id=0,
  115.                 is_real_example=False)
  116.  
  117.     label_map = {}
  118.     for (i, label) in enumerate(label_list):
  119.         label_map[label] = i
  120.  
  121.     tokens_a = tokenizer.tokenize(example.text_a)
  122.     tokens_b = None
  123.     if example.text_b:
  124.         tokens_b = tokenizer.tokenize(example.text_b)
  125.  
  126.     if tokens_b:
  127.         _truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
  128.     else:
  129.     # Account for [CLS] and [SEP] with "- 2"
  130.         if len(tokens_a) > max_seq_length - 2:
  131.             tokens_a = tokens_a[0:(max_seq_length - 2)]
  132.  
  133.     tokens = []
  134.     segment_ids = []
  135.     tokens.append("[CLS]")
  136.     segment_ids.append(0)
  137.     for token in tokens_a:
  138.         tokens.append(token)
  139.         segment_ids.append(0)
  140.     tokens.append("[SEP]")
  141.     segment_ids.append(0)
  142.  
  143.     if tokens_b:
  144.         for token in tokens_b:
  145.             tokens.append(token)
  146.             segment_ids.append(1)
  147.         tokens.append("[SEP]")
  148.         segment_ids.append(1)
  149.  
  150.     input_ids = tokenizer.convert_tokens_to_ids(tokens)
  151.  
  152.     # The mask has 1 for real tokens and 0 for padding tokens. Only real
  153.     # tokens are attended to.
  154.     input_mask = [1] * len(input_ids)
  155.  
  156.     # Zero-pad up to the sequence length.
  157.     while len(input_ids) < max_seq_length:
  158.         input_ids.append(0)
  159.         input_mask.append(0)
  160.         segment_ids.append(0)
  161.  
  162.     assert len(input_ids) == max_seq_length
  163.     assert len(input_mask) == max_seq_length
  164.     assert len(segment_ids) == max_seq_length
  165.  
  166.     label_id = label_map[example.label]
  167.     if ex_index < 3:
  168.         print("*** Example ***")
  169.         print("guid: %s" % (example.guid))
  170.         print("tokens: %s" % " ".join(
  171.             [tokenization.printable_text(x) for x in tokens]))
  172.         print("input_ids: %s" % " ".join([str(x) for x in input_ids]))
  173.         print("input_mask: %s" % " ".join([str(x) for x in input_mask]))
  174.         print("segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
  175.         print("label: %s (id = %d)" % (example.label, label_id))
  176.  
  177.     feature = InputFeatures(
  178.             input_ids=input_ids,
  179.             input_mask=input_mask,
  180.             segment_ids=segment_ids,
  181.             label_id=label_id,
  182.             is_real_example=True)
  183.     return feature
  184.  
  185.  
  186. def file_based_convert_examples_to_features(examples, label_list, max_seq_length, tokenizer, output_file):
  187.     """Convert a set of `InputExample`s to a TFRecord file."""
  188.  
  189.     writer = tf.python_io.TFRecordWriter(output_file)
  190.  
  191.     for (ex_index, example) in enumerate(examples):
  192.         if ex_index % 10000 == 0:
  193.             tf.logging.info("Writing example %d of %d" % (ex_index, len(examples)))
  194.  
  195.         feature = convert_single_example(ex_index, example, label_list,
  196.                                                                          max_seq_length, tokenizer)
  197.  
  198.         def create_int_feature(values):
  199.             f = tf.train.Feature(int64_list=tf.train.Int64List(value=list(values)))
  200.             return f
  201.  
  202.         features = collections.OrderedDict()
  203.         features["input_ids"] = create_int_feature(feature.input_ids)
  204.         features["input_mask"] = create_int_feature(feature.input_mask)
  205.         features["segment_ids"] = create_int_feature(feature.segment_ids)
  206.         features["label_ids"] = create_int_feature([feature.label_id])
  207.         features["is_real_example"] = create_int_feature(
  208.             [int(feature.is_real_example)])
  209.  
  210.         tf_example = tf.train.Example(features=tf.train.Features(feature=features))
  211.         writer.write(tf_example.SerializeToString())
  212.     writer.close()
  213.  
  214.  
  215. def file_based_input_fn_builder(input_file, seq_length, is_training,
  216.                                                                     drop_remainder):
  217.     """Creates an `input_fn` closure to be passed to TPUEstimator."""
  218.  
  219.     name_to_features = {
  220.         "input_ids": tf.FixedLenFeature([seq_length], tf.int64),
  221.         "input_mask": tf.FixedLenFeature([seq_length], tf.int64),
  222.         "segment_ids": tf.FixedLenFeature([seq_length], tf.int64),
  223.         "label_ids": tf.FixedLenFeature([], tf.int64),
  224.         "is_real_example": tf.FixedLenFeature([], tf.int64),
  225.     }
  226.  
  227.     def _decode_record(record, name_to_features):
  228.         """Decodes a record to a TensorFlow example."""
  229.         example = tf.parse_single_example(record, name_to_features)
  230.  
  231.     # tf.Example only supports tf.int64, but the TPU only supports tf.int32.
  232.     # So cast all int64 to int32.
  233.         for name in list(example.keys()):
  234.             t = example[name]
  235.             if t.dtype == tf.int64:
  236.                 t = tf.to_int32(t)
  237.             example[name] = t
  238.  
  239.         return example
  240.  
  241.     def input_fn(params):
  242.         """The actual input function."""
  243.         batch_size = params["batch_size"]
  244.  
  245.     # For training, we want a lot of parallel reading and shuffling.
  246.     # For eval, we want no shuffling and parallel reading doesn't matter.
  247.         d = tf.data.TFRecordDataset(input_file)
  248.         if is_training:
  249.             d = d.repeat()
  250.             d = d.shuffle(buffer_size=100)
  251.  
  252.         d = d.apply(
  253.                 tf.contrib.data.map_and_batch(
  254.                         lambda record: _decode_record(record, name_to_features),
  255.                         batch_size=batch_size,
  256.                         drop_remainder=drop_remainder))
  257.  
  258.         return d
  259.  
  260.     return input_fn
  261.  
  262.  
  263. def _truncate_seq_pair(tokens_a, tokens_b, max_length):
  264.     """Truncates a sequence pair in place to the maximum length."""
  265.  
  266.     # This is a simple heuristic which will always truncate the longer sequence
  267.     # one token at a time. This makes more sense than truncating an equal percent
  268.     # of tokens from each, since if one sequence is very short then each token
  269.     # that's truncated likely contains more information than a longer sequence.
  270.     while True:
  271.         total_length = len(tokens_a) + len(tokens_b)
  272.         if total_length <= max_length:
  273.             break
  274.         if len(tokens_a) > len(tokens_b):
  275.             tokens_a.pop()
  276.         else:
  277.             tokens_b.pop()
  278.  
  279.  
  280. def create_model(bert_config, is_training, input_ids, input_mask, segment_ids,
  281.                                      labels, num_labels, use_one_hot_embeddings):
  282.     """Creates a classification model."""
  283.     model = modeling.BertModel(
  284.             config=bert_config,
  285.             is_training=is_training,
  286.             input_ids=input_ids,
  287.             input_mask=input_mask,
  288.             token_type_ids=segment_ids,
  289.             use_one_hot_embeddings=use_one_hot_embeddings)
  290.  
  291.     # In the demo, we are doing a simple classification task on the entire
  292.     # segment.
  293.     #
  294.     # If you want to use the token-level output, use model.get_sequence_output()
  295.     # instead.
  296.     output_layer = model.get_pooled_output()
  297.  
  298.     hidden_size = output_layer.shape[-1].value
  299.  
  300.     output_weights = tf.get_variable(
  301.         "output_weights", [num_labels, hidden_size],
  302.             initializer=tf.truncated_normal_initializer(stddev=0.02))
  303.  
  304.     output_bias = tf.get_variable(
  305.         "output_bias", [num_labels], initializer=tf.zeros_initializer())
  306.  
  307.     with tf.variable_scope("loss"):
  308.         if is_training:
  309.         # I.e., 0.1 dropout
  310.             output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)
  311.  
  312.         logits = tf.matmul(output_layer, output_weights, transpose_b=True)
  313.         logits = tf.nn.bias_add(logits, output_bias)
  314.         probabilities = tf.nn.softmax(logits, axis=-1)
  315.         log_probs = tf.nn.log_softmax(logits, axis=-1)
  316.  
  317.         one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)
  318.  
  319.         per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
  320.         loss = tf.reduce_mean(per_example_loss)
  321.  
  322.         return (loss, per_example_loss, logits, probabilities)
  323.  
  324.  
  325. def model_fn_builder(bert_config, num_labels, init_checkpoint, learning_rate,
  326.                                              num_train_steps, num_warmup_steps, use_tpu,
  327.                                              use_one_hot_embeddings):
  328.     """Returns `model_fn` closure for TPUEstimator."""
  329.  
  330.     def model_fn(features, labels, mode, params):    # pylint: disable=unused-argument
  331.         """The `model_fn` for TPUEstimator."""
  332.  
  333.         tf.logging.info("*** Features ***")
  334.         for name in sorted(features.keys()):
  335.             print("        name = %s, shape = %s" % (name, features[name].shape))
  336.  
  337.         input_ids = features["input_ids"]
  338.         input_mask = features["input_mask"]
  339.         segment_ids = features["segment_ids"]
  340.         label_ids = features["label_ids"]
  341.         is_real_example = None
  342.         if "is_real_example" in features:
  343.             is_real_example = tf.cast(features["is_real_example"], dtype=tf.float32)
  344.         else:
  345.             is_real_example = tf.ones(tf.shape(label_ids), dtype=tf.float32)
  346.  
  347.         is_training = (mode == tf.estimator.ModeKeys.TRAIN)
  348.  
  349.         (total_loss, per_example_loss, logits, probabilities) = create_model(
  350.                 bert_config, is_training, input_ids, input_mask, segment_ids, label_ids,
  351.                 num_labels, use_one_hot_embeddings)
  352.  
  353.         tvars = tf.trainable_variables()
  354.         initialized_variable_names = {}
  355.         scaffold_fn = None
  356.         if init_checkpoint:
  357.             (assignment_map, initialized_variable_names) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint)
  358.             if use_tpu:
  359.  
  360.                 def tpu_scaffold():
  361.                     tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
  362.                     return tf.train.Scaffold()
  363.  
  364.                 scaffold_fn = tpu_scaffold
  365.             else:
  366.                 tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
  367.  
  368.         tf.logging.info("**** Trainable Variables ****")
  369.         for var in tvars:
  370.             init_string = ""
  371.             if var.name in initialized_variable_names:
  372.                 init_string = ", *INIT_FROM_CKPT*"
  373.             tf.logging.info("        name = %s, shape = %s%s", var.name, var.shape,
  374.                                             init_string)
  375.  
  376.         output_spec = None
  377.         if mode == tf.estimator.ModeKeys.TRAIN:
  378.  
  379.             train_op = optimization.create_optimizer(
  380.                     total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)
  381.  
  382.             output_spec = tf.contrib.tpu.TPUEstimatorSpec(
  383.                     mode=mode,
  384.                     loss=total_loss,
  385.                     train_op=train_op,
  386.                     scaffold_fn=scaffold_fn)
  387.         elif mode == tf.estimator.ModeKeys.EVAL:
  388.  
  389.             def metric_fn(per_example_loss, label_ids, logits, is_real_example):
  390.                 predictions = tf.argmax(logits, axis=-1, output_type=tf.int32)
  391.                 accuracy = tf.metrics.accuracy(
  392.                         labels=label_ids, predictions=predictions, weights=is_real_example)
  393.                 loss = tf.metrics.mean(values=per_example_loss, weights=is_real_example)
  394.                 return {
  395.                     "eval_accuracy": accuracy,
  396.                     "eval_loss": loss,
  397.             }
  398.  
  399.             eval_metrics = (metric_fn,
  400.                                         [per_example_loss, label_ids, logits, is_real_example])
  401.             output_spec = tf.contrib.tpu.TPUEstimatorSpec(
  402.                     mode=mode,
  403.                     loss=total_loss,
  404.                     eval_metrics=eval_metrics,
  405.                     scaffold_fn=scaffold_fn)
  406.         else:
  407.         # predictions = tf.argmax(logits, axis=-1, output_type=tf.int32)
  408.         # is_predicting = True
  409.         # (predicted_labels, log_probs) = create_model(
  410.         #                 is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)
  411.  
  412.             log_probs = tf.nn.log_softmax(logits, axis=-1)
  413.             probabilities = tf.nn.softmax(logits, axis=-1)
  414.  
  415.             predicted_labels = tf.squeeze(tf.argmax(log_probs, axis=-1, output_type=tf.int32))
  416.             predictions = {
  417.                     'probabilities': log_probs,
  418.                     'labels': predicted_labels
  419.                     
  420.         }
  421.             output_spec = tf.contrib.tpu.TPUEstimatorSpec(
  422.                     mode=mode,
  423.                     predictions= predictions,#{"probabilities": probabilities},
  424.                     scaffold_fn=scaffold_fn)
  425.         return output_spec
  426.  
  427.     return model_fn
  428.  
  429.  
  430. # This function is not used by this file but is still used by the Colab and
  431. # people who depend on it.
  432. def input_fn_builder(features, seq_length, is_training, drop_remainder):
  433.     """Creates an `input_fn` closure to be passed to TPUEstimator."""
  434.  
  435.     all_input_ids = []
  436.     all_input_mask = []
  437.     all_segment_ids = []
  438.     all_label_ids = []
  439.  
  440.     for feature in features:
  441.         all_input_ids.append(feature.input_ids)
  442.         all_input_mask.append(feature.input_mask)
  443.         all_segment_ids.append(feature.segment_ids)
  444.         all_label_ids.append(feature.label_id)
  445.  
  446.     def input_fn(params):
  447.         """The actual input function."""
  448.         batch_size = params["batch_size"]
  449.  
  450.         num_examples = len(features)
  451.  
  452.     # This is for demo purposes and does NOT scale to large data sets. We do
  453.     # not use Dataset.from_generator() because that uses tf.py_func which is
  454.     # not TPU compatible. The right way to load data is with TFRecordReader.
  455.         d = tf.data.Dataset.from_tensor_slices({
  456.             "input_ids":
  457.                         tf.constant(
  458.                                 all_input_ids, shape=[num_examples, seq_length],
  459.                                 dtype=tf.int32),
  460.             "input_mask":
  461.                         tf.constant(
  462.                                 all_input_mask,
  463.                                 shape=[num_examples, seq_length],
  464.                                 dtype=tf.int32),
  465.             "segment_ids":
  466.                         tf.constant(
  467.                                 all_segment_ids,
  468.                                 shape=[num_examples, seq_length],
  469.                                 dtype=tf.int32),
  470.             "label_ids":
  471.                         tf.constant(all_label_ids, shape=[num_examples], dtype=tf.int32),
  472.     })
  473.  
  474.         if is_training:
  475.             d = d.repeat()
  476.             d = d.shuffle(buffer_size=100)
  477.  
  478.         d = d.batch(batch_size=batch_size, drop_remainder=drop_remainder)
  479.         return d
  480.  
  481.     return input_fn
  482.  
  483.  
  484. # This function is not used by this file but is still used by the Colab and
  485. # people who depend on it.
  486. def convert_examples_to_features(examples, label_list, max_seq_length,
  487.                                                                      tokenizer):
  488.     """Convert a set of `InputExample`s to a list of `InputFeatures`."""
  489.  
  490.     features = []
  491.     for (ex_index, example) in enumerate(examples):
  492.         if ex_index % 1000 == 0:
  493.             tf.logging.info("Writing example %d of %d" % (ex_index, len(examples)))
  494.  
  495.         feature = convert_single_example(ex_index, example, label_list,
  496.                                                                          max_seq_length, tokenizer)
  497.  
  498.         features.append(feature)
  499.     return features
  1. class BTrainer(object):
  2.     def __init__(self, train_list, predict_list, lable_list, output_dir, num_train_epochs):
  3.         self.output_dir = output_dir
  4.         tokenization.validate_case_matches_checkpoint(True, init_checkpoint)
  5.  
  6.         self.bert_config = modeling.BertConfig.from_json_file(bert_config_file)
  7.         tf.gfile.MakeDirs(self.output_dir)
  8.  
  9.         self.processor = SHLibProcessor(lable_list)
  10.         self.label_list = self.processor.get_labels()
  11.  
  12.         self.tokenizer = tokenization.FullTokenizer(
  13.           vocab_file=vocab_file, do_lower_case=True)
  14.  
  15.         tpu_cluster_resolver = None
  16.  
  17.         is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2
  18.         self.run_config = tf.contrib.tpu.RunConfig(
  19.           cluster=tpu_cluster_resolver,
  20.           keep_checkpoint_max=1,
  21.           master=None,
  22.           model_dir=self.output_dir,
  23.           save_checkpoints_steps=save_checkpoints_steps,
  24.           tpu_config=tf.contrib.tpu.TPUConfig(
  25.               iterations_per_loop=iterations_per_loop,
  26.               num_shards=num_tpu_cores,
  27.               per_host_input_for_training=is_per_host))
  28.  
  29.         num_train_steps = None
  30.         num_warmup_steps = None
  31.  
  32.         self.train_examples = self.processor.get_train_examples(train_list)
  33.         self.num_train_steps = int(len(self.train_examples) / train_batch_size * num_train_epochs)
  34.         num_warmup_steps = int(self.num_train_steps * warmup_proportion)
  35.  
  36.         self.predict_examples = self.processor.get_test_examples(predict_list)
  37.  
  38.         model_fn = model_fn_builder(
  39.           bert_config=self.bert_config,
  40.           num_labels=len(self.label_list),
  41.           init_checkpoint=init_checkpoint,
  42.           learning_rate=learning_rate,
  43.           num_train_steps=self.num_train_steps,
  44.           num_warmup_steps=num_warmup_steps,
  45.           use_tpu=False,
  46.           use_one_hot_embeddings=False)
  47.  
  48.         # If TPU is not available, this will fall back to normal Estimator on CPU
  49.         # or GPU.
  50.         self.estimator = tf.contrib.tpu.TPUEstimator(
  51.           use_tpu=False,
  52.           model_fn=model_fn,
  53.           config=self.run_config,
  54.           train_batch_size=train_batch_size,
  55.           eval_batch_size=eval_batch_size,
  56.           predict_batch_size=predict_batch_size)
  57.  
  58.     def do_train(self):
  59.         try:
  60.             train_file = os.path.join(self.output_dir, "train.tf_record")
  61.             file_based_convert_examples_to_features(
  62.                 self.train_examples, self.label_list, max_seq_length, self.tokenizer, train_file)
  63.             print("***** Running training *****")
  64.  
  65.             train_input_fn = file_based_input_fn_builder(
  66.                 input_file=train_file,
  67.                 seq_length=max_seq_length,
  68.                 is_training=True,
  69.                 drop_remainder=True)
  70.             self.estimator.train(input_fn=train_input_fn, max_steps=self.num_train_steps)
  71.             print("train complete")
  72.         except Exception:
  73.             traceback.print_exc()
  74.             return -4
  75.  
  76.         return 1
  77.  
  78.     def do_predict(self):
  79.         num_actual_predict_examples = len(self.predict_examples)
  80.  
  81.         predict_file = os.path.join(self.output_dir, "predict.tf_record")
  82.         file_based_convert_examples_to_features(self.predict_examples, self.label_list,
  83.                                                 max_seq_length, self.tokenizer,
  84.                                                 predict_file)
  85.  
  86.         predict_drop_remainder = True
  87.         predict_input_fn = file_based_input_fn_builder(
  88.             input_file=predict_file,
  89.             seq_length=max_seq_length,
  90.             is_training=False,
  91.             drop_remainder=predict_drop_remainder)
  92.  
  93.         result = self.estimator.predict(input_fn=predict_input_fn)
  94.         acc = 0
  95.         output_predict_file = os.path.join(self.output_dir, "test_results.tsv")
  96.         with tf.gfile.GFile(output_predict_file, "w") as writer:
  97.             num_written_lines = 0
  98.             print("***** Predict results *****")
  99.             correct_count = 0
  100.             for (i, prediction) in enumerate(result):
  101.                 if i >= num_actual_predict_examples:
  102.                     break
  103.                 if self.predict_examples[i].label == self.label_list[prediction['labels']]:
  104.                     correct_count += 1
  105.                 num_written_lines += 1
  106.                 writer.write(str(self.predict_examples[i].guid) + "\t" + str(self.predict_examples[i].text_a) 
  107.                              + "\t" + str(self.predict_examples[i].label) 
  108.                              + "\t" + str(self.label_list[prediction['labels']]) + "\n")
  109.  
  110.             acc = correct_count/num_written_lines
  111.             print("total count:", num_written_lines, " correct:",correct_count," accuracy:",acc)
  112.         return acc

#定义读取tsv和加载标签的方法

  1. def _read_tsv(input_file, quotechar=None):
  2.     """Reads a tab separated value file."""
  3.     with tf.gfile.Open(input_file, "r") as f:
  4.         reader = csv.reader(f, delimiter="\t", quotechar=quotechar)
  5.         lines = []
  6.         for line in reader:
  7.             lines.append(line)
  8.         return lines
  9.  
  10. def load_labels(label_file_path):
  11.     with open(label_file_path,'r') as label_file:
  12.         static_label_list = list(label_file.read().splitlines())
  13.     return static_label_list

准备训练数据

  1. train_file_path = "./bert_input_data/level1_train.tsv"
  2. dev_file_path = "./bert_input_data/level1_dev.tsv"
  3. test_file_path = "./bert_input_data/level1_test.tsv"
  4.  
  5. label_file_path = "./bert_input_data/label.txt"
  6. train_epoch = 1

定义模型训练方法

  1. ef f_train_model(train_file_path,dev_file_path, lable_file_path,train_epoch):
  2.     if not os.path.exists(train_file_path) or not os.path.exists(train_file_path):
  3.         ret_value = 3
  4.         return ret_value
  5.  
  6.     label_list = load_labels(lable_file_path)
  7.     if len(label_list) <= 1:
  8.         ret_value = 4
  9.         return ret_value
  10.  
  11.     train = _read_tsv(train_file_path)
  12.     if len(train) <= 20:
  13.         ret_value = 5
  14.         return ret_value
  15.  
  16.  
  17.     dev = _read_tsv(dev_file_path)
  18.     
  19.     print("train length:",len(train))
  20.     print("val length:",len(dev))
  21.     
  22.     trainer = BTrainer(train, dev, lable_file_path, MODEL_OUTPUT_DIR, train_epoch)
  23.     return trainer.do_train()
  1. #训练模型
  2. f_train_model(train_file_path, dev_file_path, label_file_path, train_epoch)

 不知道怎么解决啊！

6.模型应用

定义模型验证方法

  1. def f_test_model(train_file_path,test_file_path, lable_file_path):
  2.     if not os.path.exists(train_file_path) or not os.path.exists(train_file_path):
  3.         ret_value = 3
  4.         return ret_value
  5.  
  6.     label_list = load_labels(lable_file_path)
  7.     if len(label_list) <= 1:
  8.         ret_value = 4
  9.         return ret_value
  10.  
  11.     train = _read_tsv(train_file_path)
  12.     if len(train) <= 10:
  13.         ret_value = 5
  14.         return ret_value
  15.  
  16.     test = _read_tsv(test_file_path)
  17.     
  18.     print("test length:",len(test))
  19.     
  20.     trainer = BTrainer(train, test, lable_file_path, MODEL_OUTPUT_DIR, 0)
  21.     return trainer.do_predict()
  1. #模训测试
  2. f_test_model(train_file_path, test_file_path, label_file_path)

 

好难搞啊，先放一放。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/你好赵伟/article/detail/624146
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号