devbox
小蓝xlanll
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

工程师&程序员的自我修养 Episode.4 基于百度飞桨PaddlePaddle框架的女朋友情绪分析&防被打消息推荐深度学习系统_baidu paddle uie情感分析

作者：小蓝xlanll | 2024-03-25 17:33:51

赞

踩

baidu paddle uie情感分析

具体为什么想到这个题目呢。。。大概是我也想不出别的什么有趣的话题或者项目的工作了吧。

有一天，柏拉图问老师苏格拉底什么是爱情？老师就让他到理论麦田里去，摘一棵全麦田里最大最金黄的麦穗来，期间只能摘一次，并且只可向前走，不能回头。柏拉图于是按照老师说的去做了。结果他两手空空走出了田地，老师问他为什么摘不到。他说：“因为期间只能摘一次，又不能走回头路，期间即使见到最大最金黄的，因为不知前面是否有更好的，所以没有摘；走到前面时，又发觉总不及之前见到的好，原来最大最金黄的麦穗早错过了；于是我什么也没有摘。”老师说：这就是“爱情”。

但是现实很骨感。当前在读信息、计科及机械等硬核工科男大多还处在“啊女朋友啊，今天一句话都没来得及说。”的状态中，而且经常代码写不来，实验做不好，debug完不成的卷容易造成说话用语的玄学。因而简单实现一个简单的利用微信qq消息推测女朋友当前隐含情绪以及推荐防被打消息的工具是大有裨益的。

友情提示：本深度学习模型的实战测试可能会有生命危险，请谨慎操作！

文章整体结构分为：

一、百度UNIT架构实现的自然多轮对话机器人功能

二、对话情绪识别理论

三、PaddleHub实现对话情绪识别

四、PaddleNLP实现对话情绪识别升级版

五、PaddlePaddle开发的情绪识别详细代码

六、PaddlePaddle结合Python爬虫的女票微博情绪监控

一、百度UNIT架构实现的自然多轮对话机器人功能：

以下感谢 @没入门的研究生 相关文章及百度暂开放使用的UNIT对话机器人平台服务，深受启发。

1、准备工作：

百度UNIT，即百度智能对话定制与服务平台（Understanding and Interaction Technology），是百度积累多年的自然语言处理技术的集大成者，通过简单的四个步骤：创建技能、配置意图及词槽、配置训练数据、训练模型即可从无到有得到一个对话系统。

UNIT是一个商业服务平台，对于高级的对话功能提供了付费服务。不过，对于注册的开发者，UNIT提供了五个免费的技能对话的研发环境。想试一试的朋友可以利用这个免费技能实现一些非常有意思的功能。通过UNIT实现多轮对话的前提，是我们要注册成为该平台的开发者，并在百度控制台申请到UNIT功能的ID和KEY。以下详谈。

进入UNIT官网后，点击进入即可看到注册界面，按照要求注册完成后，便可以进入到UNIT的技能库，如下图所示：

此时我们的技能库还是空的。左上角有“我的机器人”和“我的闲聊”两个板块，实现多轮对话便会用到这两块。

之后点击“我的闲聊”，然后点击新建我的闲聊技能，这里有普通，专业，增强版可选。其中增强版即百度Plato对话模型的中文对话模型。

点击“我的机器人”，然后点击“+”号，创建一个新的机器人：

创建完成后，点击刚刚创建的机器人，进入机器人页面，点击“添加技能”，添加刚刚创建的闲聊技能。

最后登录百度控制台,需要注意的是这个也需要注册，没有注册的朋友请自行注册完成。进入主页后，在产品服务下找到“智能对话定制与服务平台UNIT”，点击进入；点击“创建应用”，输入应用名称，应用描述后后点击“立即创建”即可创建应用。创建后，可以在页面看到创建应用的信息，纪录APPID和APIKEY信息。

至此，多轮对话的准备工作便已经完成了。

2、本机代码调用实现：

准备工作完成后，我们可以通过请求调用API的形式，接收返回结果进行多轮对话。更加详细的调用方法可以在UNIT官网技术文档获取，这里我直接提供出现成的调用对话函数，如下。其中APPID，SECRETKEY为控制台中申请应用得到的APPID和APIKEY，SERVICEID为机器人的ID，SKILLID为闲聊技能ID，而函数的参数解释如下：

text: 当前对话内容（人发出的对话）

user_id: 对话人的编号（对不同人进行不同的编号，来区分不同对象）

session_id, history: 见后边解释

log_id: 非必要，纪录日志的id号

  1. # encoding:utf-8
  2. import requests
  3.  
  4. APPID = "*************"
  5. SECRETKEY = "****************"
  6. SERVICEID = '****************'
  7. SKILLID = '***************'
  8. # client_id 为官网获取的AK， client_secret 为官网获取的SK
  9. host = 'https://aip.baidubce.com/oauth/2.0/token?grant_type=client_credentials&client_id=%s&client_secret=%s' \
  10.        % (APPKEY, SECRETKEY)
  11. response = requests.get(host)
  12.  
  13. access_token = response.json()["access_token"]
  14. url = 'https://aip.baidubce.com/rpc/2.0/unit/service/chat?access_token=' + access_token
  15.  
  16.  
  17. def dialog(text, user_id, session_id='', history='', log_id='LOG_FOR_TEST'):
  18.     post_data = "{\"log_id\":\"%s\",\"version\":\"2.0\",\"service_id\":\"%s\",\"session_id\":\"%s\"," \
  19.                 "\"request\":{\"query\":\"%s\",\"user_id\":\"%s\"}," \
  20.                 "\"dialog_state\":{\"contexts\":{\"SYS_REMEMBERED_SKILLS\":[\"%s\"]}}}"\
  21.                 %(log_id, SERVICEID, session_id, text, user_id, SKILLID)
  22.     if len(history) > 0:
  23.         post_data = "{\"log_id\":\"%s\",\"version\":\"2.0\",\"service_id\":\"%s\",\"session_id\":\"\"," \
  24.                 "\"request\":{\"query\":\"%s\",\"user_id\":\"%s\"}," \
  25.                 "\"dialog_state\":{\"contexts\":{\"SYS_REMEMBERED_SKILLS\":[\"%s\"], " \
  26.                     "\"SYS_CHAT_HIST\":[%s]}}}" \
  27.                 %(log_id, SERVICEID, text, user_id, SKILLID, history)
  28.     post_data = post_data.encode('utf-8')
  29.     headers = {'content-type': 'application/x-www-form-urlencoded'}
  30.     response = requests.post(url, data=post_data, headers=headers)
  31.     resp = response.json()
  32.     ans = resp["result"]["response_list"][0]["action_list"][0]['say']
  33.     session_id = resp['result']['session_id']
  34.     return ans, session_id

需要说明的是，多轮对话的历史信息，我们可以通过记录请求响应结果中的session_id实现，也可以手动记录对话纪录history，并写入请求的dialog_state中的SYS_CHAT_HIST。为此，我们定义一个用户类，来存储每个用户的对话历史，或相应的session_id。当输入了history时，session_id无效；当没有history时，务必输入session_id以确保对话过程中可以考虑历史对话信息。如下：

  1. class User:
  2.     def __init__(self, user_id):
  3.         self.user_id = user_id
  4.         self.session_id = ''
  5.         self._history = []
  6.         self.history = ''
  7.         self.MAX_TURN = 7
  8.  
  9.     def get_service_id(self, session_id):
  10.         self.session_id = session_id
  11.  
  12.     def update_history(self, text):
  13.         self._history.append(text)
  14.         self._history = self._history[-self.MAX_TURN*2-1:]
  15.         self.history = ','.join(["\""+sent+"\"" for sent in self._history])
  16.  
  17.     def start_new_dialog(self):
  18.         self.session_id = ''
  19.         self._history = []
  20.         self.history = ''
  21.  
  22.     def change_max_turn(self, max_turn):
  23.         self.MAX_TURN = max_turn

 接下来，便可以进行对话了：

  1. from dialog import dialog
  2. from user import User
  3.  
  4.  
  5. user_id = 'test_user'
  6. user = User(user_id)
  7.  
  8. while True:
  9.     human_ans = input()
  10.     if len(human_ans) > 0:
  11.         user.update_history(human_ans)
  12.         robot_resp, session_id = dialog(human_ans, user.user_id, user.session_id, user.history)
  13.         user.session_id = session_id
  14.         user.update_history(robot_resp)
  15.         print("Robot: %s" % robot_resp)
  16.     else:
  17.         break
  18. user.start_new_dialog()

 实际演示对话的话可以利用如下代码段实现：

  1. from dialog import dialog
  2. from user import User
  3.  
  4.  
  5. user_id = 'test_user'
  6. user = User(user_id)
  7.  
  8. while True:
  9.     human_ans = input()
  10.     if len(human_ans) > 0:
  11.         user.update_history(human_ans)
  12.         robot_resp, session_id = dialog(human_ans, user.user_id, user.session_id, user.history)
  13.         user.session_id = session_id
  14.         user.update_history(robot_resp)
  15.         print("You: %s" % human_ans)
  16.         print("Robot: %s" % robot_resp)
  17.     else:
  18.         break
  19. user.start_new_dialog()

这里You后紧接我们自己想说的话，Robot后为UNIT自动生成的回复，部分问题也可能是AI问我们，挑选出其中非问题的对话内容可以作为之后情绪识别的验证。 

二、对话情绪识别理论：

此处感谢飞桨官方文章 七夕礼物没送对？飞桨PaddlePaddle帮你读懂女朋友的小心思 深有启发。

对话情绪识别适用于聊天、客服等多个场景，能够帮助企业更好地把握对话质量、改善产品的用户交互体验，也能分析客服服务质量、降低人工质检成本。对话情绪识别（Emotion Detection，简称EmoTect），专注于识别智能对话场景中用户的情绪，针对智能对话场景中的用户文本，自动判断该文本的情绪类别并给出相应的置信度，情绪类型分为积极、消极、中性。

基于百度自建测试集（包含闲聊、客服）和nlpcc2014微博情绪数据集评测效果如下表所示，此外，PaddleNLP还开源了百度基于海量数据训练好的模型，该模型在聊天对话语料上fine-tune之后，可以得到更好的效果。

  • BOW：Bag Of Words，是一个非序列模型，使用基本的全连接结构。

  • CNN：浅层CNN模型，能够处理变长的序列输入，提取一个局部区域之内的特征。

  • TextCNN：多卷积核CNN模型，能够更好地捕捉句子局部相关性。

  • LSTM：单层LSTM模型，能够较好地解决序列文本中长距离依赖的问题。

  • BI-LSTM：双向单层LSTM模型，采用双向LSTM结构，更好地捕获句子中的语义特征。

  • ERNIE：百度自研基于海量数据和先验知识训练的通用文本语义表示模型，并基于此在对话情绪分类数据集上进行fine-tune获得。

在百度PaddlePaddle飞桨下属的各个高层API中，PaddleNLP是专门用于自然语言处理的。PaddleHub则是深度学习现成预训练仓库，也可以调用其中训练好的模型进行使用。这两种方法相对简单顶层，但是直观性比较差。此外以下我们会大致给出一个从PaddlePaddle底层搭建的情绪分析代码。

技术上，基于百度飞桨PaddleNLP的“对话情绪识别”模型则特别针对中文表达中口语化、语气词多、词汇乱序等常见情况与难题，优化除去口语化、同义词转换等预处理方式，保证识别数据的干净有效，从而让机器“更懂”用户所表达的“中心思想”。因此，该模型可以广泛的适用于电商、教育、地图导航等场景，帮助“机器”更好地理解“人心”。

三、PaddleHub实现对话情绪识别：

1、准备工作：

包括查看当前的数据集和工作区文件，并且将PaddlePaddle和PaddleHub库统一升级到2.0版本。

  1. # 查看当前挂载的数据集目录, 该目录下的变更重启环境后会自动还原
  2. # View dataset directory. This directory will be recovered automatically after resetting environment. 
  3. !ls /home/aistudio/data
  4.  
  5. # 查看工作区文件, 该目录下的变更将会持久保存. 请及时清理不必要的文件, 避免加载过慢.
  6. # View personal work directory. All changes under this directory will be kept even after reset. Please clean unnecessary files in time to speed up environment loading.
  7. !ls /home/aistudio/work
  8.  
  9.  
  10. #需要将PaddleHub和PaddlePaddle统一升级到2.0版本
  11. !pip install paddlehub==2.0.0 -i https://pypi.tuna.tsinghua.edu.cn/simple 
  12. !pip install paddlepaddle==2.0.0 -i https://pypi.tuna.tsinghua.edu.cn/simple

2、引入库文件和hub中关于情感分析的模型：

  1. import paddlehub as hub
  2.  
  3. senta = hub.Module(name="senta_bilstm")

 3、调入我们需要检测的文本内容，放置在text变量中，每一句消息作为一个字符串存放在一个一维列表：

  1. test_text = [
  2.     "天啦，千万别多说，扰乱军心，哈哈", 
  3.     "该做什么的时候就得好好做，别多想了",
  4.     "你的老师和伙伴都你需要专心，哈哈" ,
  5.     "其实你说你想我我肯定很开心，说明你在乎呀" ,
  6.     "成大事者需要专注力哟~",
  7.     "晚安啦~"
  8. ]

4、模型参数读取及调用预测：

  1. input_dict = {"text": test_text}
  2. results = senta.sentiment_classify(data=input_dict)
  3.  
  4. for result in results:
  5.     print(result['text'])
  6.     print(result['sentiment_label'])
  7.     print(result['sentiment_key'])
  8.     print(result['positive_probs'])
  9.     print(result['negative_probs'])

5、预测结果：

其中数字2表示积极，1表示中性，0表示消极。后两排的概率表示PaddleHub_Senta_bilstm模型认为的积极和消极程度（以可能性作为评判标准）的深浅。

四、基于PaddleNLP的对话情绪分析升级版：

1、准备工作：

包括升级PaddleNLP到2.0版本和导入必要的库文件。

  1. #首先，需要安装paddlenlp2.0。
  2. !pip install paddlenlp
  1. #导入相关的模块
  2. import paddle
  3. import paddlenlp as ppnlp
  4. from paddlenlp.data import Stack, Pad, Tuple
  5. import paddle.nn.functional as F
  6. import numpy as np
  7. from functools import partial #partial()函数可以用来固定某些参数值，并返回一个新的callable对象

2、数据集准备：

数据集为公开中文情感分析数据集ChnSenticorp。使用PaddleNLP的.datasets.ChnSentiCorp.get_datasets方法即可以加载该数据集。

  1. #采用paddlenlp内置的ChnSentiCorp语料，该语料主要可以用来做情感分类。训练集用来训练模型，验证集用来选择模型，测试集用来评估模型泛化性能。
  2. train_ds, dev_ds, test_ds = ppnlp.datasets.ChnSentiCorp.get_datasets(['train','dev','test'])
  3.  
  4. #获得标签列表
  5. label_list = train_ds.get_labels()
  6.  
  7. #看看数据长什么样子，分别打印训练集、验证集、测试集的前3条数据。
  8. print("训练集数据：{}\n".format(train_ds[0:3]))
  9. print("验证集数据:{}\n".format(dev_ds[0:3]))
  10. print("测试集数据:{}\n".format(test_ds[0:3]))
  11.  
  12. print("训练集样本个数:{}".format(len(train_ds)))
  13. print("验证集样本个数:{}".format(len(dev_ds)))
  14. print("测试集样本个数:{}".format(len(test_ds)))

3、数据预处理：

  1. #调用ppnlp.transformers.BertTokenizer进行数据处理，tokenizer可以把原始输入文本转化成模型model可接受的输入数据格式。
  2. tokenizer = ppnlp.transformers.BertTokenizer.from_pretrained("bert-base-chinese")
  3.  
  4. #数据预处理
  5. def convert_example(example,tokenizer,label_list,max_seq_length=256,is_test=False):
  6.     if is_test:
  7.         text = example
  8.     else:
  9.         text, label = example
  10.     #tokenizer.encode方法能够完成切分token，映射token ID以及拼接特殊token
  11.     encoded_inputs = tokenizer.encode(text=text, max_seq_len=max_seq_length)
  12.     input_ids = encoded_inputs["input_ids"]
  13.     segment_ids = encoded_inputs["segment_ids"]
  14.  
  15.     if not is_test:
  16.         label_map = {}
  17.         for (i, l) in enumerate(label_list):
  18.             label_map[l] = i
  19.  
  20.         label = label_map[label]
  21.         label = np.array([label], dtype="int64")
  22.         return input_ids, segment_ids, label
  23.     else:
  24.         return input_ids, segment_ids
  25.  
  26. #数据迭代器构造方法
  27. def create_dataloader(dataset, trans_fn=None, mode='train', batch_size=1, use_gpu=False, pad_token_id=0, batchify_fn=None):
  28.     if trans_fn:
  29.         dataset = dataset.apply(trans_fn, lazy=True)
  30.  
  31.     if mode == 'train' and use_gpu:
  32.         sampler = paddle.io.DistributedBatchSampler(dataset=dataset, batch_size=batch_size, shuffle=True)
  33.     else:
  34.         shuffle = True if mode == 'train' else False #如果不是训练集，则不打乱顺序
  35.         sampler = paddle.io.BatchSampler(dataset=dataset, batch_size=batch_size, shuffle=shuffle) #生成一个取样器
  36.     dataloader = paddle.io.DataLoader(dataset, batch_sampler=sampler, return_list=True, collate_fn=batchify_fn)
  37.     return dataloader
  38.  
  39. #使用partial()来固定convert_example函数的tokenizer, label_list, max_seq_length, is_test等参数值
  40. trans_fn = partial(convert_example, tokenizer=tokenizer, label_list=label_list, max_seq_length=128, is_test=False)
  41. batchify_fn = lambda samples, fn=Tuple(Pad(axis=0,pad_val=tokenizer.pad_token_id), Pad(axis=0, pad_val=tokenizer.pad_token_id), Stack(dtype="int64")):[data for data in fn(samples)]
  42. #训练集迭代器
  43. train_loader = create_dataloader(train_ds, mode='train', batch_size=64, batchify_fn=batchify_fn, trans_fn=trans_fn)
  44. #验证集迭代器
  45. dev_loader = create_dataloader(dev_ds, mode='dev', batch_size=64, batchify_fn=batchify_fn, trans_fn=trans_fn)
  46. #测试集迭代器
  47. test_loader = create_dataloader(test_ds, mode='test', batch_size=64, batchify_fn=batchify_fn, trans_fn=trans_fn)

4、模型训练：

  1. #加载预训练模型Bert用于文本分类任务的Fine-tune网络BertForSequenceClassification, 它在BERT模型后接了一个全连接层进行分类。
  2. #由于本任务中的情感分类是二分类问题，设定num_classes为2
  3. model = ppnlp.transformers.BertForSequenceClassification.from_pretrained("bert-base-chinese", num_classes=2)
  4.  
  5. #设置训练超参数
  6.  
  7. #学习率
  8. learning_rate = 1e-5 
  9. #训练轮次
  10. epochs = 20
  11. #学习率预热比率
  12. warmup_proption = 0.1
  13. #权重衰减系数
  14. weight_decay = 0.01
  15.  
  16. num_training_steps = len(train_loader) * epochs
  17. num_warmup_steps = int(warmup_proption * num_training_steps)
  18.  
  19. def get_lr_factor(current_step):
  20.     if current_step < num_warmup_steps:
  21.         return float(current_step) / float(max(1, num_warmup_steps))
  22.     else:
  23.         return max(0.0,
  24.                     float(num_training_steps - current_step) /
  25.                     float(max(1, num_training_steps - num_warmup_steps)))
  26. #学习率调度器
  27. lr_scheduler = paddle.optimizer.lr.LambdaDecay(learning_rate, lr_lambda=lambda current_step: get_lr_factor(current_step))
  28.  
  29. #优化器
  30. optimizer = paddle.optimizer.AdamW(
  31.     learning_rate=lr_scheduler,
  32.     parameters=model.parameters(),
  33.     weight_decay=weight_decay,
  34.     apply_decay_param_fun=lambda x: x in [
  35.         p.name for n, p in model.named_parameters()
  36.         if not any(nd in n for nd in ["bias", "norm"])
  37.     ])
  38.  
  39. #损失函数
  40. criterion = paddle.nn.loss.CrossEntropyLoss()
  41. #评估函数
  42. metric = paddle.metric.Accuracy()
  43.  
  44. #评估函数
  45. def evaluate(model, criterion, metric, data_loader):
  46.     model.eval()
  47.     metric.reset()
  48.     losses = []
  49.     for batch in data_loader:
  50.         input_ids, segment_ids, labels = batch
  51.         logits = model(input_ids, segment_ids)
  52.         loss = criterion(logits, labels)
  53.         losses.append(loss.numpy())
  54.         correct = metric.compute(logits, labels)
  55.         metric.update(correct)
  56.         accu = metric.accumulate()
  57.     print("eval loss: %.5f, accu: %.5f" % (np.mean(losses), accu))
  58.     model.train()
  59.     metric.reset()
  60.  
  61. #开始训练
  62. global_step = 0
  63. for epoch in range(1, epochs + 1):
  64.     for step, batch in enumerate(train_loader, start=1): #从训练数据迭代器中取数据
  65.         input_ids, segment_ids, labels = batch
  66.         logits = model(input_ids, segment_ids)
  67.         loss = criterion(logits, labels) #计算损失
  68.         probs = F.softmax(logits, axis=1)
  69.         correct = metric.compute(probs, labels)
  70.         metric.update(correct)
  71.         acc = metric.accumulate()
  72.  
  73.         global_step += 1
  74.         if global_step % 50 == 0 :
  75.             print("global step %d, epoch: %d, batch: %d, loss: %.5f, acc: %.5f" % (global_step, epoch, step, loss, acc))
  76.         loss.backward()
  77.         optimizer.step()
  78.         lr_scheduler.step()
  79.         optimizer.clear_gradients()
  80.     evaluate(model, criterion, metric, dev_loader)

5、模型预测：

在倒数第六行的data变量列表中，同之前PaddleHub中提到的方法，可以有效地传入文本内容参数，以一条消息为单个字符串元素排列此一维列表。

  1. def predict(model, data, tokenizer, label_map, batch_size=1):
  2.     examples = []
  3.     for text in data:
  4.         input_ids, segment_ids = convert_example(text, tokenizer, label_list=label_map.values(),  max_seq_length=128, is_test=True)
  5.         examples.append((input_ids, segment_ids))
  6.  
  7.     batchify_fn = lambda samples, fn=Tuple(Pad(axis=0, pad_val=tokenizer.pad_token_id), Pad(axis=0, pad_val=tokenizer.pad_token_id)): fn(samples)
  8.     batches = []
  9.     one_batch = []
  10.     for example in examples:
  11.         one_batch.append(example)
  12.         if len(one_batch) == batch_size:
  13.             batches.append(one_batch)
  14.             one_batch = []
  15.     if one_batch:
  16.         batches.append(one_batch)
  17.  
  18.     results = []
  19.     model.eval()
  20.     for batch in batches:
  21.         input_ids, segment_ids = batchify_fn(batch)
  22.         input_ids = paddle.to_tensor(input_ids)
  23.         segment_ids = paddle.to_tensor(segment_ids)
  24.         logits = model(input_ids, segment_ids)
  25.         probs = F.softmax(logits, axis=1)
  26.         idx = paddle.argmax(probs, axis=1).numpy()
  27.         idx = idx.tolist()
  28.         labels = [label_map[i] for i in idx]
  29.         results.extend(labels)
  30.     return results
  31.  
  32. #待预测文本变量
  33. data = ['有点东西啊', '这个老师讲课水平挺高的', '你在干什么']
  34. label_map = {0: '负向情绪', 1: '正向情绪'}
  35.  
  36. predictions = predict(model, data, tokenizer, label_map, batch_size=32)
  37. for idx, text in enumerate(data):
  38.     print('预测文本: {} \n情绪标签: {}'.format(text, predictions[idx]))

 6、预测结果：

预测结果类同前PaddleHub部分，此略，程序会直接给出这句话是更偏向积极的正向情绪还是消极的负向情绪。

五、PaddlePaddle层从头搭建的自然语言情绪分析（ERNIE模型）：

效果上，我们基于百度自建测试集（包含闲聊、客服）和nlpcc2014微博情绪数据集，进行评测，效果如下表所示，此外我们还开源了百度基于海量数据训练好的模型，该模型在聊天对话语料上fine-tune之后，可以得到更好的效果。

对话情绪识别任务输入是一段用户文本，输出是检测到的情绪类别，包括消极、积极、中性，这是一个经典的短文本三分类任务。数据集链接：https://aistudio.baidu.com/aistudio/datasetdetail/9740

数据集解压后生成data目录，data目录下有训练集数据（train.tsv）、开发集数据（dev.tsv）、测试集数据（test.tsv）、 待预测数据（infer.tsv）以及对应词典（vocab.txt）
训练、预测、评估使用的数据示例如下，数据由两列组成，以制表符（'\t'）分隔，第一列是情绪分类的类别（0表示消极；1表示中性；2表示积极），第二列是以空格分词的中文文本：

ERNIE：百度自研基于海量数据和先验知识训练的通用文本语义表示模型，并基于此在对话情绪分类数据集上进行 fine-tune 获得。

ERNIE 于 2019 年 3 月发布，通过建模海量数据中的词、实体及实体关系，学习真实世界的语义知识。相较于 BERT 学习原始语言信号，ERNIE 直接对先验语义知识单元进行建模，增强了模型语义表示能力。
同年 7 月，百度发布了 ERNIE 2.0。ERNIE 2.0 是基于持续学习的语义理解预训练框架，使用多任务学习增量式构建预训练任务。ERNIE 2.0 中，新构建的预训练任务类型可以无缝的加入训练框架，持续的进行语义理解学习。 通过新增的实体预测、句子因果关系判断、文章句子结构重建等语义任务，ERNIE 2.0 语义理解预训练模型从训练数据中获取了词法、句法、语义等多个维度的自然语言信息，极大地增强了通用语义表示能力，示意图如下：

 4 个 cell 定义 ErnieModel 中使用的基本网络结构，包括：

  1. multi_head_attention
  2. positionwise_feed_forward
  3. pre_post_process_layer：增加 residual connection, layer normalization 和 droput，在 multi_head_attention 和 positionwise_feed_forward 前后使用
  4. encoder_layer：调用上述三种结构生成 encoder 层
  5. encoder：堆叠 encoder_layer 生成完整的 encoder

关于 multi_head_attention 和 positionwise_feed_forward 的介绍可以参考：The Annotated Transformer

3 个 cell 定义分词代码类，包括：

  1. FullTokenizer：完整的分词，在数据读取代码中使用，调用 BasicTokenizer 和 WordpieceTokenizer 实现
  2. BasicTokenizer：基本分词，包括标点划分、小写转换等
  3. WordpieceTokenizer：单词划分

 4 个 cell 定义数据读取器和预处理代码，包括：

  1. BaseReader：数据读取器基类
  2. ClassifyReader：用于分类模型的数据读取器，重写 _readtsv 和 _pad_batch_records 方法
  3. pad_batch_data：数据预处理，给数据加 padding，并生成位置数据和 mask
  4. ernie_pyreader：生成训练、验证和预测使用的 pyreader

数据集和ERNIE网络相关配置：

  1. # 数据集相关配置
  2. data_config = {
  3.     'data_dir': 'data/data9740/data',               # Directory path to training data.
  4.     'vocab_path': 'pretrained_model/ernie_finetune/vocab.txt',   # Vocabulary path.
  5.     'batch_size': 32,   # Total examples' number in batch for training.
  6.     'random_seed': 0,   # Random seed.
  7.     'num_labels': 3,    # label number
  8.     'max_seq_len': 512, # Number of words of the longest seqence.
  9.     'train_set': 'data/data9740/data/test.tsv',   # Path to training data.
  10.     'test_set':  'data/data9740/data/test.tsv',   # Path to test data.
  11.     'dev_set':   'data/data9740/data/dev.tsv',    # Path to validation data.
  12.     'infer_set': 'data/data9740/data/infer.tsv',  # Path to infer data.
  13.     'label_map_config': None,   # label_map_path
  14.     'do_lower_case': True,      # Whether to lower case the input text. Should be True for uncased models and False for cased models.
  15. }
  16.  
  17. # Ernie 网络结构相关配置
  18. ernie_net_config = {
  19.     "attention_probs_dropout_prob": 0.1,
  20.     "hidden_act": "relu",
  21.     "hidden_dropout_prob": 0.1,
  22.     "hidden_size": 768,
  23.     "initializer_range": 0.02,
  24.     "max_position_embeddings": 513,
  25.     "num_attention_heads": 12,
  26.     "num_hidden_layers": 12,
  27.     "type_vocab_size": 2,
  28.     "vocab_size": 18000,
  29. }

训练阶段相关配置：

  1. train_config = {
  2.     'init_checkpoint': 'pretrained_model/ernie_finetune/params',
  3.     'output_dir': 'train_model',
  4.     
  5.     'epoch': 10,
  6.     'save_steps': 100,
  7.     'validation_steps': 100,
  8.     'lr': 0.00002,
  9.     
  10.     'skip_steps': 10,
  11.     'verbose': False,
  12.     
  13.     'use_cuda': True,
  14. }

预测阶段相关配置：

  1. infer_config = {
  2.     'init_checkpoint': 'train_model',
  3.     'use_cuda': True,
  4. }

具体实现的详细代码：

  1. #!/usr/bin/env python
  2. # coding: utf-8
  3.  
  4. # ### 一、项目背景介绍
  5. # 对话情绪识别（Emotion Detection，简称EmoTect），专注于识别智能对话场景中用户的情绪，针对智能对话场景中的用户文本，自动判断该文本的情绪类别并给出相应的置信度，情绪类型分为积极、消极、中性。
  6. # 
  7. # 对话情绪识别适用于聊天、客服等多个场景，能够帮助企业更好地把握对话质量、改善产品的用户交互体验，也能分析客服服务质量、降低人工质检成本。可通过 AI开放平台-对话情绪识别 线上体验。
  8. # 
  9. # 效果上，我们基于百度自建测试集（包含闲聊、客服）和nlpcc2014微博情绪数据集，进行评测，效果如下表所示，此外我们还开源了百度基于海量数据训练好的模型，该模型在聊天对话语料上fine-tune之后，可以得到更好的效果。
  10. # 
  11. # 
  12. # | 模型 | 闲聊 | 客服 | 微博 |
  13. # | -------- | -------- | -------- | -------- |
  14. # | BOW |	90.2% |	87.6% |	74.2% |
  15. # | LSTM |	91.4% |	90.1% |	73.8% |
  16. # | Bi-LSTM |	91.2% |	89.9% |	73.6% |
  17. # | CNN |	90.8% |	90.7% |	76.3% |
  18. # | TextCNN |	91.1% |	91.0% |	76.8% |
  19. # | BERT |	93.6% |	92.3% |	78.6% |
  20. # | ERNIE |	94.4% |	94.0% |	80.6% |
  21.  
  22. # ### 二、数据集介绍
  23. # 
  24. # 对话情绪识别任务输入是一段用户文本，输出是检测到的情绪类别，包括消极、积极、中性，这是一个经典的短文本三分类任务。  
  25. # 
  26. # 数据集解压后生成data目录，data目录下有训练集数据（train.tsv）、开发集数据（dev.tsv）、测试集数据（test.tsv）、 待预测数据（infer.tsv）以及对应词典（vocab.txt）  
  27. # 训练、预测、评估使用的数据示例如下，数据由两列组成，以制表符（'\t'）分隔，第一列是情绪分类的类别（0表示消极；1表示中性；2表示积极），第二列是以空格分词的中文文本：
  28. # 
  29. # 	label   text_a
  30. # 	0   谁 骂人 了 ？ 我 从来 不 骂人 ， 我 骂 的 都 不是 人 ， 你 是 人 吗 ？
  31. # 	1   我 有事 等会儿 就 回来 和 你 聊
  32. # 	2   我 见到 你 很高兴 谢谢 你 帮 我
  33.  
  34. # In[1]:
  35.  
  36.  
  37. # 解压数据集
  38. get_ipython().system('cd /home/aistudio/data/data9740 && unzip -qo 对话情绪识别.zip')
  39.  
  40.  
  41. # In[2]:
  42.  
  43.  
  44. # 各种引用库
  45. from __future__ import absolute_import
  46. from __future__ import division
  47. from __future__ import print_function
  48.  
  49. import io
  50. import os
  51. import six
  52. import sys
  53. import time
  54. import random
  55. import string
  56. import logging
  57. import argparse
  58. import collections
  59. import unicodedata
  60. from functools import partial
  61. from collections import namedtuple
  62.  
  63. import multiprocessing
  64.  
  65. import paddle
  66. import paddle.fluid as fluid
  67. import paddle.fluid.layers as layers
  68. import numpy as np
  69.  
  70.  
  71. # In[3]:
  72.  
  73.  
  74. # 统一的 logger 配置
  75.  
  76. logger = None
  77.  
  78. def init_log_config():
  79.     """
  80.     初始化日志相关配置
  81.     :return:
  82.     """
  83.     global logger
  84.     logger = logging.getLogger()
  85.     logger.setLevel(logging.INFO)
  86.     log_path = os.path.join(os.getcwd(), 'logs')
  87.     if not os.path.exists(log_path):
  88.         os.makedirs(log_path)
  89.     log_name = os.path.join(log_path, 'train.log')
  90.     sh = logging.StreamHandler()
  91.     fh = logging.FileHandler(log_name, mode='w')
  92.     fh.setLevel(logging.DEBUG)
  93.     formatter = logging.Formatter("%(asctime)s - %(filename)s[line:%(lineno)d] - %(levelname)s: %(message)s")
  94.     fh.setFormatter(formatter)
  95.     sh.setFormatter(formatter)
  96.     logger.handlers = []
  97.     logger.addHandler(sh)
  98.     logger.addHandler(fh)
  99.  
  100.  
  101. # In[4]:
  102.  
  103.  
  104. # util
  105.  
  106. def print_arguments(args):
  107.     """
  108.     打印参数
  109.     """
  110.     logger.info('-----------  Configuration Arguments -----------')
  111.     for key in args.keys():
  112.         logger.info('%s: %s' % (key, args[key]))
  113.     logger.info('------------------------------------------------')
  114.  
  115.  
  116. def init_checkpoint(exe, init_checkpoint_path, main_program):
  117.     """
  118.     加载缓存模型
  119.     """
  120.     assert os.path.exists(
  121.         init_checkpoint_path), "[%s] cann't be found." % init_checkpoint_path
  122.  
  123.     def existed_persitables(var):
  124.         """
  125.         If existed presitabels
  126.         """
  127.         if not fluid.io.is_persistable(var):
  128.             return False
  129.         return os.path.exists(os.path.join(init_checkpoint_path, var.name))
  130.  
  131.     fluid.io.load_vars(
  132.         exe,
  133.         init_checkpoint_path,
  134.         main_program=main_program,
  135.         predicate=existed_persitables)
  136.     logger.info("Load model from {}".format(init_checkpoint_path))
  137.  
  138.  
  139. def csv_reader(fd, delimiter='\t'):
  140.     """
  141.     csv 文件读取
  142.     """
  143.     def gen():
  144.         for i in fd:
  145.             slots = i.rstrip('\n').split(delimiter)
  146.             if len(slots) == 1:
  147.                 yield slots,
  148.             else:
  149.                 yield slots
  150.     return gen()
  151.  
  152.  
  153. # ### 三、网络结构构建
  154. # **ERNIE**：百度自研基于海量数据和先验知识训练的通用文本语义表示模型，并基于此在对话情绪分类数据集上进行 fine-tune 获得。
  155. # 
  156. # **ERNIE** 于 2019 年 3 月发布，通过建模海量数据中的词、实体及实体关系，学习真实世界的语义知识。相较于 BERT 学习原始语言信号，**ERNIE** 直接对先验语义知识单元进行建模，增强了模型语义表示能力。  
  157. # 同年 7 月，百度发布了 **ERNIE 2.0**。**ERNIE 2.0** 是基于持续学习的语义理解预训练框架，使用多任务学习增量式构建预训练任务。**ERNIE 2.0** 中，新构建的预训练任务类型可以无缝的加入训练框架，持续的进行语义理解学习。 通过新增的实体预测、句子因果关系判断、文章句子结构重建等语义任务，**ERNIE 2.0** 语义理解预训练模型从训练数据中获取了词法、句法、语义等多个维度的自然语言信息，极大地增强了通用语义表示能力，示意图如下：  
  158. # <img src="https://ai-studio-static-online.cdn.bcebos.com/b2c3107a955147238ecd0d44fff9850f9c47b45c189044009e688de80e7fc826" width="50%" />  
  159. # 
  160. # 参考资料：
  161. # 1. [ERNIE: Enhanced Representation through Knowledge Integration](https://arxiv.org/abs/1904.09223)
  162. # 2. [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://arxiv.org/abs/1907.12412)
  163. # 3. [ERNIE预览：百度 知识增强语义表示模型ERNIE](https://www.jianshu.com/p/fb66f444bb8c)
  164. # 4. [ERNIE 2.0 GitHub](https://github.com/PaddlePaddle/ERNIE)
  165.  
  166. # #### 3.1 ERNIE 模型定义
  167. # class ErnieModel 定义 ERNIE encoder 网络结构  
  168. # **输入** src_ids、position_ids、sentence_ids 和 input_mask  
  169. # **输出** sequence_output 和 pooled_output
  170.  
  171. # In[5]:
  172.  
  173.  
  174. class ErnieModel(object):
  175.     """Ernie模型定义"""
  176.  
  177.     def __init__(self,
  178.                  src_ids,
  179.                  position_ids,
  180.                  sentence_ids,
  181.                  input_mask,
  182.                  config,
  183.                  weight_sharing=True,
  184.                  use_fp16=False):
  185.  
  186.         # Ernie 相关参数
  187.         self._emb_size = config['hidden_size']
  188.         self._n_layer = config['num_hidden_layers']
  189.         self._n_head = config['num_attention_heads']
  190.         self._voc_size = config['vocab_size']
  191.         self._max_position_seq_len = config['max_position_embeddings']
  192.         self._sent_types = config['type_vocab_size']
  193.         self._hidden_act = config['hidden_act']
  194.         self._prepostprocess_dropout = config['hidden_dropout_prob']
  195.         self._attention_dropout = config['attention_probs_dropout_prob']
  196.         self._weight_sharing = weight_sharing
  197.  
  198.         self._word_emb_name = "word_embedding"
  199.         self._pos_emb_name = "pos_embedding"
  200.         self._sent_emb_name = "sent_embedding"
  201.         self._dtype = "float16" if use_fp16 else "float32"
  202.  
  203.         # Initialize all weigths by truncated normal initializer, and all biases
  204.         # will be initialized by constant zero by default.
  205.         self._param_initializer = fluid.initializer.TruncatedNormal(
  206.             scale=config['initializer_range'])
  207.  
  208.         self._build_model(src_ids, position_ids, sentence_ids, input_mask)
  209.  
  210.     def _build_model(self, src_ids, position_ids, sentence_ids, input_mask):
  211.         # padding id in vocabulary must be set to 0
  212.         emb_out = fluid.layers.embedding(
  213.             input=src_ids,
  214.             size=[self._voc_size, self._emb_size],
  215.             dtype=self._dtype,
  216.             param_attr=fluid.ParamAttr(
  217.                 name=self._word_emb_name, initializer=self._param_initializer),
  218.             is_sparse=False)
  219.         position_emb_out = fluid.layers.embedding(
  220.             input=position_ids,
  221.             size=[self._max_position_seq_len, self._emb_size],
  222.             dtype=self._dtype,
  223.             param_attr=fluid.ParamAttr(
  224.                 name=self._pos_emb_name, initializer=self._param_initializer))
  225.  
  226.         sent_emb_out = fluid.layers.embedding(
  227.             sentence_ids,
  228.             size=[self._sent_types, self._emb_size],
  229.             dtype=self._dtype,
  230.             param_attr=fluid.ParamAttr(
  231.                 name=self._sent_emb_name, initializer=self._param_initializer))
  232.  
  233.         emb_out = emb_out + position_emb_out
  234.         emb_out = emb_out + sent_emb_out
  235.  
  236.         emb_out = pre_process_layer(
  237.             emb_out, 'nd', self._prepostprocess_dropout, name='pre_encoder')
  238.  
  239.         if self._dtype == "float16":
  240.             input_mask = fluid.layers.cast(x=input_mask, dtype=self._dtype)
  241.         self_attn_mask = fluid.layers.matmul(
  242.             x=input_mask, y=input_mask, transpose_y=True)
  243.  
  244.         self_attn_mask = fluid.layers.scale(
  245.             x=self_attn_mask, scale=10000.0, bias=-1.0, bias_after_scale=False)
  246.         n_head_self_attn_mask = fluid.layers.stack(
  247.             x=[self_attn_mask] * self._n_head, axis=1)
  248.         n_head_self_attn_mask.stop_gradient = True
  249.  
  250.         self._enc_out = encoder(
  251.             enc_input=emb_out,
  252.             attn_bias=n_head_self_attn_mask,
  253.             n_layer=self._n_layer,
  254.             n_head=self._n_head,
  255.             d_key=self._emb_size // self._n_head,
  256.             d_value=self._emb_size // self._n_head,
  257.             d_model=self._emb_size,
  258.             d_inner_hid=self._emb_size * 4,
  259.             prepostprocess_dropout=self._prepostprocess_dropout,
  260.             attention_dropout=self._attention_dropout,
  261.             relu_dropout=0,
  262.             hidden_act=self._hidden_act,
  263.             preprocess_cmd="",
  264.             postprocess_cmd="dan",
  265.             param_initializer=self._param_initializer,
  266.             name='encoder')
  267.  
  268.     def get_sequence_output(self):
  269.         """Get embedding of each token for squence labeling"""
  270.         return self._enc_out
  271.  
  272.     def get_pooled_output(self):
  273.         """Get the first feature of each sequence for classification"""
  274.         next_sent_feat = fluid.layers.slice(
  275.             input=self._enc_out, axes=[1], starts=[0], ends=[1])
  276.         next_sent_feat = fluid.layers.fc(
  277.             input=next_sent_feat,
  278.             size=self._emb_size,
  279.             act="tanh",
  280.             param_attr=fluid.ParamAttr(
  281.                 name="pooled_fc.w_0", initializer=self._param_initializer),
  282.             bias_attr="pooled_fc.b_0")
  283.         return next_sent_feat
  284.  
  285.  
  286. # #### 3.2 基本网络结构定义
  287. # 以下 4 个 cell 定义 ErnieModel 中使用的基本网络结构，包括：
  288. # 1. multi_head_attention  
  289. # 2. positionwise_feed_forward  
  290. # 3. pre_post_process_layer：增加 residual connection, layer normalization 和 droput，在 multi_head_attention 和 positionwise_feed_forward 前后使用
  291. # 4. encoder_layer：调用上述三种结构生成 encoder 层
  292. # 5. encoder：堆叠 encoder_layer 生成完整的 encoder
  293. # 
  294. # 关于 multi_head_attention 和 positionwise_feed_forward 的介绍可以参考：[The Annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html)
  295.  
  296. # In[6]:
  297.  
  298.  
  299. def multi_head_attention(queries, keys, values, attn_bias, d_key, d_value, d_model, n_head=1, dropout_rate=0.,
  300.                          cache=None, param_initializer=None, name='multi_head_att'):
  301.     """
  302.     Multi-Head Attention. Note that attn_bias is added to the logit before
  303.     computing softmax activiation to mask certain selected positions so that
  304.     they will not considered in attention weights.
  305.     """
  306.     keys = queries if keys is None else keys
  307.     values = keys if values is None else values
  308.  
  309.     if not (len(queries.shape) == len(keys.shape) == len(values.shape) == 3):
  310.         raise ValueError(
  311.             "Inputs: quries, keys and values should all be 3-D tensors.")
  312.  
  313.     def __compute_qkv(queries, keys, values, n_head, d_key, d_value):
  314.         """
  315.         Add linear projection to queries, keys, and values.
  316.         """
  317.         q = layers.fc(input=queries,
  318.                       size=d_key * n_head,
  319.                       num_flatten_dims=2,
  320.                       param_attr=fluid.ParamAttr(
  321.                           name=name + '_query_fc.w_0',
  322.                           initializer=param_initializer),
  323.                       bias_attr=name + '_query_fc.b_0')
  324.         k = layers.fc(input=keys,
  325.                       size=d_key * n_head,
  326.                       num_flatten_dims=2,
  327.                       param_attr=fluid.ParamAttr(
  328.                           name=name + '_key_fc.w_0',
  329.                           initializer=param_initializer),
  330.                       bias_attr=name + '_key_fc.b_0')
  331.         v = layers.fc(input=values,
  332.                       size=d_value * n_head,
  333.                       num_flatten_dims=2,
  334.                       param_attr=fluid.ParamAttr(
  335.                           name=name + '_value_fc.w_0',
  336.                           initializer=param_initializer),
  337.                       bias_attr=name + '_value_fc.b_0')
  338.         return q, k, v
  339.  
  340.     def __split_heads(x, n_head):
  341.         """
  342.         Reshape the last dimension of inpunt tensor x so that it becomes two
  343.         dimensions and then transpose. Specifically, input a tensor with shape
  344.         [bs, max_sequence_length, n_head * hidden_dim] then output a tensor
  345.         with shape [bs, n_head, max_sequence_length, hidden_dim].
  346.         """
  347.         hidden_size = x.shape[-1]
  348.         # The value 0 in shape attr means copying the corresponding dimension
  349.         # size of the input as the output dimension size.
  350.         reshaped = layers.reshape(
  351.             x=x, shape=[0, 0, n_head, hidden_size // n_head], inplace=True)
  352.  
  353.         # permuate the dimensions into:
  354.         # [batch_size, n_head, max_sequence_len, hidden_size_per_head]
  355.         return layers.transpose(x=reshaped, perm=[0, 2, 1, 3])
  356.  
  357.     def __combine_heads(x):
  358.         """
  359.         Transpose and then reshape the last two dimensions of inpunt tensor x
  360.         so that it becomes one dimension, which is reverse to __split_heads.
  361.         """
  362.         if len(x.shape) == 3:
  363.             return x
  364.         if len(x.shape) != 4:
  365.             raise ValueError("Input(x) should be a 4-D Tensor.")
  366.  
  367.         trans_x = layers.transpose(x, perm=[0, 2, 1, 3])
  368.         # The value 0 in shape attr means copying the corresponding dimension
  369.         # size of the input as the output dimension size.
  370.         return layers.reshape(
  371.             x=trans_x,
  372.             shape=[0, 0, trans_x.shape[2] * trans_x.shape[3]],
  373.             inplace=True)
  374.  
  375.     def scaled_dot_product_attention(q, k, v, attn_bias, d_key, dropout_rate):
  376.         """
  377.         Scaled Dot-Product Attention
  378.         """
  379.         scaled_q = layers.scale(x=q, scale=d_key**-0.5)
  380.         product = layers.matmul(x=scaled_q, y=k, transpose_y=True)
  381.         if attn_bias:
  382.             product += attn_bias
  383.         weights = layers.softmax(product)
  384.         if dropout_rate:
  385.             weights = layers.dropout(
  386.                 weights,
  387.                 dropout_prob=dropout_rate,
  388.                 dropout_implementation="upscale_in_train",
  389.                 is_test=False)
  390.         out = layers.matmul(weights, v)
  391.         return out
  392.  
  393.     q, k, v = __compute_qkv(queries, keys, values, n_head, d_key, d_value)
  394.  
  395.     if cache is not None:  # use cache and concat time steps
  396.         # Since the inplace reshape in __split_heads changes the shape of k and
  397.         # v, which is the cache input for next time step, reshape the cache
  398.         # input from the previous time step first.
  399.         k = cache["k"] = layers.concat(
  400.             [layers.reshape(
  401.                 cache["k"], shape=[0, 0, d_model]), k], axis=1)
  402.         v = cache["v"] = layers.concat(
  403.             [layers.reshape(
  404.                 cache["v"], shape=[0, 0, d_model]), v], axis=1)
  405.  
  406.     q = __split_heads(q, n_head)
  407.     k = __split_heads(k, n_head)
  408.     v = __split_heads(v, n_head)
  409.  
  410.     ctx_multiheads = scaled_dot_product_attention(q, k, v, attn_bias, d_key,
  411.                                                   dropout_rate)
  412.  
  413.     out = __combine_heads(ctx_multiheads)
  414.  
  415.     # Project back to the model size.
  416.     proj_out = layers.fc(input=out,
  417.                          size=d_model,
  418.                          num_flatten_dims=2,
  419.                          param_attr=fluid.ParamAttr(
  420.                              name=name + '_output_fc.w_0',
  421.                              initializer=param_initializer),
  422.                          bias_attr=name + '_output_fc.b_0')
  423.     return proj_out
  424.  
  425.  
  426. # In[7]:
  427.  
  428.  
  429. def positionwise_feed_forward(x, d_inner_hid, d_hid, dropout_rate, hidden_act, param_initializer=None, name='ffn'):
  430.     """
  431.     Position-wise Feed-Forward Networks.
  432.     This module consists of two linear transformations with a ReLU activation
  433.     in between, which is applied to each position separately and identically.
  434.     """
  435.     hidden = layers.fc(input=x, size=d_inner_hid, num_flatten_dims=2, act=hidden_act,
  436.                        param_attr=fluid.ParamAttr(
  437.                            name=name + '_fc_0.w_0',
  438.                            initializer=param_initializer),
  439.                        bias_attr=name + '_fc_0.b_0')
  440.     if dropout_rate:
  441.         hidden = layers.dropout(hidden, dropout_prob=dropout_rate, dropout_implementation="upscale_in_train", is_test=False)
  442.     out = layers.fc(input=hidden, size=d_hid, num_flatten_dims=2,
  443.                     param_attr=fluid.ParamAttr(
  444.                         name=name + '_fc_1.w_0', initializer=param_initializer),
  445.                     bias_attr=name + '_fc_1.b_0')
  446.     return out
  447.  
  448.  
  449. # In[8]:
  450.  
  451.  
  452. def pre_post_process_layer(prev_out, out, process_cmd, dropout_rate=0.,
  453.                            name=''):
  454.     """
  455.     Add residual connection, layer normalization and droput to the out tensor
  456.     optionally according to the value of process_cmd.
  457.     This will be used before or after multi-head attention and position-wise
  458.     feed-forward networks.
  459.     """
  460.     for cmd in process_cmd:
  461.         if cmd == "a":  # add residual connection
  462.             out = out + prev_out if prev_out else out
  463.         elif cmd == "n":  # add layer normalization
  464.             out_dtype = out.dtype
  465.             if out_dtype == fluid.core.VarDesc.VarType.FP16:
  466.                 out = layers.cast(x=out, dtype="float32")
  467.             out = layers.layer_norm(
  468.                 out,
  469.                 begin_norm_axis=len(out.shape) - 1,
  470.                 param_attr=fluid.ParamAttr(
  471.                     name=name + '_layer_norm_scale',
  472.                     initializer=fluid.initializer.Constant(1.)),
  473.                 bias_attr=fluid.ParamAttr(
  474.                     name=name + '_layer_norm_bias',
  475.                     initializer=fluid.initializer.Constant(0.)))
  476.             if out_dtype == fluid.core.VarDesc.VarType.FP16:
  477.                 out = layers.cast(x=out, dtype="float16")
  478.         elif cmd == "d":  # add dropout
  479.             if dropout_rate:
  480.                 out = layers.dropout(
  481.                     out,
  482.                     dropout_prob=dropout_rate,
  483.                     dropout_implementation="upscale_in_train",
  484.                     is_test=False)
  485.     return out
  486.  
  487.  
  488. pre_process_layer = partial(pre_post_process_layer, None)
  489. post_process_layer = pre_post_process_layer
  490.  
  491.  
  492. # In[9]:
  493.  
  494.  
  495. def encoder_layer(enc_input, attn_bias, n_head, d_key, d_value, d_model, d_inner_hid, prepostprocess_dropout,
  496.                   attention_dropout, relu_dropout, hidden_act, preprocess_cmd="n", postprocess_cmd="da",
  497.                   param_initializer=None, name=''):
  498.     """The encoder layers that can be stacked to form a deep encoder.
  499.     This module consits of a multi-head (self) attention followed by
  500.     position-wise feed-forward networks and both the two components companied
  501.     with the post_process_layer to add residual connection, layer normalization
  502.     and droput.
  503.     """
  504.     attn_output = multi_head_attention(
  505.         pre_process_layer(enc_input, preprocess_cmd, prepostprocess_dropout, name=name + '_pre_att'),
  506.         None, None, attn_bias, d_key, d_value, d_model, n_head, attention_dropout,
  507.         param_initializer=param_initializer, name=name + '_multi_head_att')
  508.     attn_output = post_process_layer(enc_input, attn_output, postprocess_cmd, prepostprocess_dropout, name=name + '_post_att')
  509.     ffd_output = positionwise_feed_forward(
  510.         pre_process_layer(attn_output, preprocess_cmd, prepostprocess_dropout, name=name + '_pre_ffn'),
  511.         d_inner_hid, d_model, relu_dropout, hidden_act, param_initializer=param_initializer,
  512.         name=name + '_ffn')
  513.     return post_process_layer(attn_output, ffd_output, postprocess_cmd, prepostprocess_dropout, name=name + '_post_ffn')
  514.  
  515.  
  516. def encoder(enc_input, attn_bias, n_layer, n_head, d_key, d_value, d_model, d_inner_hid, prepostprocess_dropout,
  517.             attention_dropout, relu_dropout, hidden_act, preprocess_cmd="n", postprocess_cmd="da",
  518.             param_initializer=None, name=''):
  519.     """
  520.     The encoder is composed of a stack of identical layers returned by calling
  521.     encoder_layer.
  522.     """
  523.     for i in range(n_layer):
  524.         enc_output = encoder_layer(enc_input, attn_bias, n_head, d_key, d_value, d_model, d_inner_hid,
  525.             prepostprocess_dropout, attention_dropout, relu_dropout, hidden_act, preprocess_cmd,
  526.             postprocess_cmd, param_initializer=param_initializer, name=name + '_layer_' + str(i))
  527.         enc_input = enc_output
  528.     enc_output = pre_process_layer(enc_output, preprocess_cmd, prepostprocess_dropout, name="post_encoder")
  529.  
  530.     return enc_output
  531.  
  532.  
  533. # #### 3.3 编码器 和 分类器 定义
  534. # 以下 cell 定义 encoder 和 classification 的组织结构：
  535. # 1. ernie_encoder：根据 ErnieModel 组织输出 embeddings  
  536. # 2. create_ernie_model：定义分类网络，以 embeddings 为输入，使用全连接网络 + softmax 做分类
  537.  
  538. # In[10]:
  539.  
  540.  
  541. def ernie_encoder(ernie_inputs, ernie_config):
  542.     """return sentence embedding and token embeddings"""
  543.  
  544.     ernie = ErnieModel(
  545.         src_ids=ernie_inputs["src_ids"],
  546.         position_ids=ernie_inputs["pos_ids"],
  547.         sentence_ids=ernie_inputs["sent_ids"],
  548.         input_mask=ernie_inputs["input_mask"],
  549.         config=ernie_config)
  550.  
  551.     enc_out = ernie.get_sequence_output()
  552.     unpad_enc_out = fluid.layers.sequence_unpad(
  553.         enc_out, length=ernie_inputs["seq_lens"])
  554.     cls_feats = ernie.get_pooled_output()
  555.  
  556.     embeddings = {
  557.         "sentence_embeddings": cls_feats,
  558.         "token_embeddings": unpad_enc_out,
  559.     }
  560.  
  561.     for k, v in embeddings.items():
  562.         v.persistable = True
  563.  
  564.     return embeddings
  565.  
  566.  
  567. def create_ernie_model(args,
  568.                  embeddings,
  569.                  labels,
  570.                  is_prediction=False):
  571.  
  572.     """
  573.     Create Model for sentiment classification based on ERNIE encoder
  574.     """
  575.     sentence_embeddings = embeddings["sentence_embeddings"]
  576.     token_embeddings = embeddings["token_embeddings"]
  577.  
  578.     cls_feats = fluid.layers.dropout(
  579.         x=sentence_embeddings,
  580.         dropout_prob=0.1,
  581.         dropout_implementation="upscale_in_train")
  582.     logits = fluid.layers.fc(
  583.         input=cls_feats,
  584.         size=args['num_labels'],
  585.         param_attr=fluid.ParamAttr(
  586.             name="cls_out_w",
  587.             initializer=fluid.initializer.TruncatedNormal(scale=0.02)),
  588.         bias_attr=fluid.ParamAttr(
  589.             name="cls_out_b", initializer=fluid.initializer.Constant(0.)))
  590.  
  591.     ce_loss, probs = fluid.layers.softmax_with_cross_entropy(
  592.         logits=logits, label=labels, return_softmax=True)
  593.     if is_prediction:
  594.         return probs
  595.     loss = fluid.layers.mean(x=ce_loss)
  596.  
  597.     num_seqs = fluid.layers.create_tensor(dtype='int64')
  598.     accuracy = fluid.layers.accuracy(input=probs, label=labels, total=num_seqs)
  599.  
  600.     return loss, accuracy, num_seqs
  601.  
  602.  
  603. # #### 3.4 分词代码
  604. # 以下 3 个 cell 定义分词代码类，包括：
  605. # 1. FullTokenizer：完整的分词，在数据读取代码中使用，调用 BasicTokenizer 和 WordpieceTokenizer 实现  
  606. # 2. BasicTokenizer：基本分词，包括标点划分、小写转换等
  607. # 3. WordpieceTokenizer：单词划分
  608.  
  609. # In[11]:
  610.  
  611.  
  612. class FullTokenizer(object):
  613.     """Runs end-to-end tokenziation."""
  614.  
  615.     def __init__(self, vocab_file, do_lower_case=True):
  616.         self.vocab = load_vocab(vocab_file)
  617.         self.inv_vocab = {v: k for k, v in self.vocab.items()}
  618.         self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
  619.         self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
  620.  
  621.     def tokenize(self, text):
  622.         split_tokens = []
  623.         for token in self.basic_tokenizer.tokenize(text):
  624.             for sub_token in self.wordpiece_tokenizer.tokenize(token):
  625.                 split_tokens.append(sub_token)
  626.  
  627.         return split_tokens
  628.  
  629.     def convert_tokens_to_ids(self, tokens):
  630.         return convert_by_vocab(self.vocab, tokens)
  631.  
  632.  
  633. # In[12]:
  634.  
  635.  
  636. class BasicTokenizer(object):
  637.     """Runs basic tokenization (punctuation splitting, lower casing, etc.)."""
  638.  
  639.     def __init__(self, do_lower_case=True):
  640.         """Constructs a BasicTokenizer.
  641.         Args:
  642.             do_lower_case: Whether to lower case the input.
  643.         """
  644.         self.do_lower_case = do_lower_case
  645.  
  646.     def tokenize(self, text):
  647.         """Tokenizes a piece of text."""
  648.         text = convert_to_unicode(text)
  649.         text = self._clean_text(text)
  650.  
  651.         # This was added on November 1st, 2018 for the multilingual and Chinese
  652.         # models. This is also applied to the English models now, but it doesn't
  653.         # matter since the English models were not trained on any Chinese data
  654.         # and generally don't have any Chinese data in them (there are Chinese
  655.         # characters in the vocabulary because Wikipedia does have some Chinese
  656.         # words in the English Wikipedia.).
  657.         text = self._tokenize_chinese_chars(text)
  658.  
  659.         orig_tokens = whitespace_tokenize(text)
  660.         split_tokens = []
  661.         for token in orig_tokens:
  662.             if self.do_lower_case:
  663.                 token = token.lower()
  664.                 token = self._run_strip_accents(token)
  665.             split_tokens.extend(self._run_split_on_punc(token))
  666.  
  667.         output_tokens = whitespace_tokenize(" ".join(split_tokens))
  668.         return output_tokens
  669.  
  670.     def _run_strip_accents(self, text):
  671.         """Strips accents from a piece of text."""
  672.         text = unicodedata.normalize("NFD", text)
  673.         output = []
  674.         for char in text:
  675.             cat = unicodedata.category(char)
  676.             if cat == "Mn":
  677.                 continue
  678.             output.append(char)
  679.         return "".join(output)
  680.  
  681.     def _run_split_on_punc(self, text):
  682.         """Splits punctuation on a piece of text."""
  683.         chars = list(text)
  684.         i = 0
  685.         start_new_word = True
  686.         output = []
  687.         while i < len(chars):
  688.             char = chars[i]
  689.             if _is_punctuation(char):
  690.                 output.append([char])
  691.                 start_new_word = True
  692.             else:
  693.                 if start_new_word:
  694.                     output.append([])
  695.                 start_new_word = False
  696.                 output[-1].append(char)
  697.             i += 1
  698.  
  699.         return ["".join(x) for x in output]
  700.  
  701.     def _tokenize_chinese_chars(self, text):
  702.         """Adds whitespace around any CJK character."""
  703.         output = []
  704.         for char in text:
  705.             cp = ord(char)
  706.             if self._is_chinese_char(cp):
  707.                 output.append(" ")
  708.                 output.append(char)
  709.                 output.append(" ")
  710.             else:
  711.                 output.append(char)
  712.         return "".join(output)
  713.  
  714.     def _is_chinese_char(self, cp):
  715.         """Checks whether CP is the codepoint of a CJK character."""
  716.         # This defines a "chinese character" as anything in the CJK Unicode block:
  717.         #     https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
  718.         #
  719.         # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
  720.         # despite its name. The modern Korean Hangul alphabet is a different block,
  721.         # as is Japanese Hiragana and Katakana. Those alphabets are used to write
  722.         # space-separated words, so they are not treated specially and handled
  723.         # like the all of the other languages.
  724.         if ((cp >= 0x4E00 and cp <= 0x9FFF) or  #
  725.             (cp >= 0x3400 and cp <= 0x4DBF) or  #
  726.             (cp >= 0x20000 and cp <= 0x2A6DF) or  #
  727.             (cp >= 0x2A700 and cp <= 0x2B73F) or  #
  728.             (cp >= 0x2B740 and cp <= 0x2B81F) or  #
  729.             (cp >= 0x2B820 and cp <= 0x2CEAF) or
  730.             (cp >= 0xF900 and cp <= 0xFAFF) or  #
  731.             (cp >= 0x2F800 and cp <= 0x2FA1F)):  #
  732.             return True
  733.  
  734.         return False
  735.  
  736.     def _clean_text(self, text):
  737.         """Performs invalid character removal and whitespace cleanup on text."""
  738.         output = []
  739.         for char in text:
  740.             cp = ord(char)
  741.             if cp == 0 or cp == 0xfffd or _is_control(char):
  742.                 continue
  743.             if _is_whitespace(char):
  744.                 output.append(" ")
  745.             else:
  746.                 output.append(char)
  747.         return "".join(output)
  748.  
  749.  
  750. # In[13]:
  751.  
  752.  
  753. class WordpieceTokenizer(object):
  754.     """Runs WordPiece tokenziation."""
  755.  
  756.     def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=100):
  757.         self.vocab = vocab
  758.         self.unk_token = unk_token
  759.         self.max_input_chars_per_word = max_input_chars_per_word
  760.  
  761.     def tokenize(self, text):
  762.         """Tokenizes a piece of text into its word pieces.
  763.         This uses a greedy longest-match-first algorithm to perform tokenization
  764.         using the given vocabulary.
  765.         For example:
  766.             input = "unaffable"
  767.             output = ["un", "##aff", "##able"]
  768.         Args:
  769.             text: A single token or whitespace separated tokens. This should have
  770.                 already been passed through `BasicTokenizer.
  771.         Returns:
  772.             A list of wordpiece tokens.
  773.         """
  774.  
  775.         text = convert_to_unicode(text)
  776.  
  777.         output_tokens = []
  778.         for token in whitespace_tokenize(text):
  779.             chars = list(token)
  780.             if len(chars) > self.max_input_chars_per_word:
  781.                 output_tokens.append(self.unk_token)
  782.                 continue
  783.  
  784.             is_bad = False
  785.             start = 0
  786.             sub_tokens = []
  787.             while start < len(chars):
  788.                 end = len(chars)
  789.                 cur_substr = None
  790.                 while start < end:
  791.                     substr = "".join(chars[start:end])
  792.                     if start > 0:
  793.                         substr = "##" + substr
  794.                     if substr in self.vocab:
  795.                         cur_substr = substr
  796.                         break
  797.                     end -= 1
  798.                 if cur_substr is None:
  799.                     is_bad = True
  800.                     break
  801.                 sub_tokens.append(cur_substr)
  802.                 start = end
  803.  
  804.             if is_bad:
  805.                 output_tokens.append(self.unk_token)
  806.             else:
  807.                 output_tokens.extend(sub_tokens)
  808.         return output_tokens
  809.  
  810.  
  811. # #### 3.5 分词辅助代码
  812. # 以下 cell 定义分词中的辅助性代码，包括 convert_to_unicode、whitespace_tokenize 等。
  813.  
  814. # In[14]:
  815.  
  816.  
  817. def convert_to_unicode(text):
  818.     """Converts `text` to Unicode (if it's not already), assuming utf-8 input."""
  819.     if six.PY3:
  820.         if isinstance(text, str):
  821.             return text
  822.         elif isinstance(text, bytes):
  823.             return text.decode("utf-8", "ignore")
  824.         else:
  825.             raise ValueError("Unsupported string type: %s" % (type(text)))
  826.     elif six.PY2:
  827.         if isinstance(text, str):
  828.             return text.decode("utf-8", "ignore")
  829.         elif isinstance(text, unicode):
  830.             return text
  831.         else:
  832.             raise ValueError("Unsupported string type: %s" % (type(text)))
  833.     else:
  834.         raise ValueError("Not running on Python2 or Python 3?")
  835.  
  836.  
  837. def load_vocab(vocab_file):
  838.     """Loads a vocabulary file into a dictionary."""
  839.     vocab = collections.OrderedDict()
  840.     fin = io.open(vocab_file, encoding="utf8")
  841.     for num, line in enumerate(fin):
  842.         items = convert_to_unicode(line.strip()).split("\t")
  843.         if len(items) > 2:
  844.             break
  845.         token = items[0]
  846.         index = items[1] if len(items) == 2 else num
  847.         token = token.strip()
  848.         vocab[token] = int(index)
  849.     return vocab
  850.  
  851.  
  852. def convert_by_vocab(vocab, items):
  853.     """Converts a sequence of [tokens|ids] using the vocab."""
  854.     output = []
  855.     for item in items:
  856.         output.append(vocab[item])
  857.     return output
  858.  
  859.  
  860. def whitespace_tokenize(text):
  861.     """Runs basic whitespace cleaning and splitting on a peice of text."""
  862.     text = text.strip()
  863.     if not text:
  864.         return []
  865.     tokens = text.split()
  866.     return tokens
  867.  
  868.  
  869. def _is_whitespace(char):
  870.     """Checks whether `chars` is a whitespace character."""
  871.     # \t, \n, and \r are technically contorl characters but we treat them
  872.     # as whitespace since they are generally considered as such.
  873.     if char == " " or char == "\t" or char == "\n" or char == "\r":
  874.         return True
  875.     cat = unicodedata.category(char)
  876.     if cat == "Zs":
  877.         return True
  878.     return False
  879.  
  880.  
  881. def _is_control(char):
  882.     """Checks whether `chars` is a control character."""
  883.     # These are technically control characters but we count them as whitespace
  884.     # characters.
  885.     if char == "\t" or char == "\n" or char == "\r":
  886.         return False
  887.     cat = unicodedata.category(char)
  888.     if cat.startswith("C"):
  889.         return True
  890.     return False
  891.  
  892.  
  893. def _is_punctuation(char):
  894.     """Checks whether `chars` is a punctuation character."""
  895.     cp = ord(char)
  896.     # We treat all non-letter/number ASCII as punctuation.
  897.     # Characters such as "^", "$", and "`" are not in the Unicode
  898.     # Punctuation class but we treat them as punctuation anyways, for
  899.     # consistency.
  900.     if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or
  901.         (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)):
  902.         return True
  903.     cat = unicodedata.category(char)
  904.     if cat.startswith("P"):
  905.         return True
  906.     return False
  907.  
  908.  
  909. # #### 3.6 数据读取 及 预处理代码
  910. # 以下 4 个 cell 定义数据读取器和预处理代码，包括：
  911. # 1. BaseReader：数据读取器基类  
  912. # 2. ClassifyReader：用于分类模型的数据读取器，重写 _readtsv 和 _pad_batch_records 方法  
  913. # 3. pad_batch_data：数据预处理，给数据加 padding，并生成位置数据和 mask
  914. # 4. ernie_pyreader：生成训练、验证和预测使用的 pyreader
  915.  
  916. # In[15]:
  917.  
  918.  
  919. class BaseReader(object):
  920.     """BaseReader for classify and sequence labeling task"""
  921.  
  922.     def __init__(self,
  923.                  vocab_path,
  924.                  label_map_config=None,
  925.                  max_seq_len=512,
  926.                  do_lower_case=True,
  927.                  in_tokens=False,
  928.                  random_seed=None):
  929.         self.max_seq_len = max_seq_len
  930.         self.tokenizer = FullTokenizer(
  931.             vocab_file=vocab_path, do_lower_case=do_lower_case)
  932.         self.vocab = self.tokenizer.vocab
  933.         self.pad_id = self.vocab["[PAD]"]
  934.         self.cls_id = self.vocab["[CLS]"]
  935.         self.sep_id = self.vocab["[SEP]"]
  936.         self.in_tokens = in_tokens
  937.  
  938.         np.random.seed(random_seed)
  939.  
  940.         self.current_example = 0
  941.         self.current_epoch = 0
  942.         self.num_examples = 0
  943.  
  944.         if label_map_config:
  945.             with open(label_map_config) as f:
  946.                 self.label_map = json.load(f)
  947.         else:
  948.             self.label_map = None
  949.  
  950.     def _read_tsv(self, input_file, quotechar=None):
  951.         """Reads a tab separated value file."""
  952.         with io.open(input_file, "r", encoding="utf8") as f:
  953.             reader = csv_reader(f, delimiter="\t")
  954.             headers = next(reader)
  955.             Example = namedtuple('Example', headers)
  956.  
  957.             examples = []
  958.             for line in reader:
  959.                 example = Example(*line)
  960.                 examples.append(example)
  961.             return examples
  962.  
  963.     def _truncate_seq_pair(self, tokens_a, tokens_b, max_length):
  964.         """Truncates a sequence pair in place to the maximum length."""
  965.  
  966.         # This is a simple heuristic which will always truncate the longer sequence
  967.         # one token at a time. This makes more sense than truncating an equal percent
  968.         # of tokens from each, since if one sequence is very short then each token
  969.         # that's truncated likely contains more information than a longer sequence.
  970.         while True:
  971.             total_length = len(tokens_a) + len(tokens_b)
  972.             if total_length <= max_length:
  973.                 break
  974.             if len(tokens_a) > len(tokens_b):
  975.                 tokens_a.pop()
  976.             else:
  977.                 tokens_b.pop()
  978.  
  979.     def _convert_example_to_record(self, example, max_seq_length, tokenizer):
  980.         """Converts a single `Example` into a single `Record`."""
  981.  
  982.         text_a = convert_to_unicode(example.text_a)
  983.         tokens_a = tokenizer.tokenize(text_a)
  984.         tokens_b = None
  985.         if "text_b" in example._fields:
  986.             text_b = convert_to_unicode(example.text_b)
  987.             tokens_b = tokenizer.tokenize(text_b)
  988.  
  989.         if tokens_b:
  990.             # Modifies `tokens_a` and `tokens_b` in place so that the total
  991.             # length is less than the specified length.
  992.             # Account for [CLS], [SEP], [SEP] with "- 3"
  993.             self._truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
  994.         else:
  995.             # Account for [CLS] and [SEP] with "- 2"
  996.             if len(tokens_a) > max_seq_length - 2:
  997.                 tokens_a = tokens_a[0:(max_seq_length - 2)]
  998.  
  999.         # The convention in BERT/ERNIE is:
  1000.         # (a) For sequence pairs:
  1001.         #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
  1002.         #  type_ids: 0     0  0    0    0     0       0 0     1  1  1  1   1 1
  1003.         # (b) For single sequences:
  1004.         #  tokens:   [CLS] the dog is hairy . [SEP]
  1005.         #  type_ids: 0     0   0   0  0     0 0
  1006.         #
  1007.         # Where "type_ids" are used to indicate whether this is the first
  1008.         # sequence or the second sequence. The embedding vectors for `type=0` and
  1009.         # `type=1` were learned during pre-training and are added to the wordpiece
  1010.         # embedding vector (and position vector). This is not *strictly* necessary
  1011.         # since the [SEP] token unambiguously separates the sequences, but it makes
  1012.         # it easier for the model to learn the concept of sequences.
  1013.         #
  1014.         # For classification tasks, the first vector (corresponding to [CLS]) is
  1015.         # used as as the "sentence vector". Note that this only makes sense because
  1016.         # the entire model is fine-tuned.
  1017.         tokens = []
  1018.         text_type_ids = []
  1019.         tokens.append("[CLS]")
  1020.         text_type_ids.append(0)
  1021.         for token in tokens_a:
  1022.             tokens.append(token)
  1023.             text_type_ids.append(0)
  1024.         tokens.append("[SEP]")
  1025.         text_type_ids.append(0)
  1026.  
  1027.         if tokens_b:
  1028.             for token in tokens_b:
  1029.                 tokens.append(token)
  1030.                 text_type_ids.append(1)
  1031.             tokens.append("[SEP]")
  1032.             text_type_ids.append(1)
  1033.  
  1034.         token_ids = tokenizer.convert_tokens_to_ids(tokens)
  1035.         position_ids = list(range(len(token_ids)))
  1036.  
  1037.         if self.label_map:
  1038.             label_id = self.label_map[example.label]
  1039.         else:
  1040.             label_id = example.label
  1041.  
  1042.         Record = namedtuple(
  1043.             'Record',
  1044.             ['token_ids', 'text_type_ids', 'position_ids', 'label_id', 'qid'])
  1045.  
  1046.         qid = None
  1047.         if "qid" in example._fields:
  1048.             qid = example.qid
  1049.  
  1050.         record = Record(
  1051.             token_ids=token_ids,
  1052.             text_type_ids=text_type_ids,
  1053.             position_ids=position_ids,
  1054.             label_id=label_id,
  1055.             qid=qid)
  1056.         return record
  1057.  
  1058.     def _prepare_batch_data(self, examples, batch_size, phase=None):
  1059.         """generate batch records"""
  1060.         batch_records, max_len = [], 0
  1061.         for index, example in enumerate(examples):
  1062.             if phase == "train":
  1063.                 self.current_example = index
  1064.             record = self._convert_example_to_record(example, self.max_seq_len,
  1065.                                                      self.tokenizer)
  1066.             max_len = max(max_len, len(record.token_ids))
  1067.             if self.in_tokens:
  1068.                 to_append = (len(batch_records) + 1) * max_len <= batch_size
  1069.             else:
  1070.                 to_append = len(batch_records) < batch_size
  1071.             if to_append:
  1072.                 batch_records.append(record)
  1073.             else:
  1074.                 yield self._pad_batch_records(batch_records)
  1075.                 batch_records, max_len = [record], len(record.token_ids)
  1076.  
  1077.         if batch_records:
  1078.             yield self._pad_batch_records(batch_records)
  1079.  
  1080.     def get_num_examples(self, input_file):
  1081.         """return total number of examples"""
  1082.         examples = self._read_tsv(input_file)
  1083.         return len(examples)
  1084.     
  1085.     def get_examples(self, input_file):
  1086.         examples = self._read_tsv(input_file)
  1087.         return examples
  1088.  
  1089.     def data_generator(self,
  1090.                        input_file,
  1091.                        batch_size,
  1092.                        epoch,
  1093.                        shuffle=True,
  1094.                        phase=None):
  1095.         """return generator which yields batch data for pyreader"""
  1096.         examples = self._read_tsv(input_file)
  1097.  
  1098.         def _wrapper():
  1099.             for epoch_index in range(epoch):
  1100.                 if phase == "train":
  1101.                     self.current_example = 0
  1102.                     self.current_epoch = epoch_index
  1103.                 if shuffle:
  1104.                     np.random.shuffle(examples)
  1105.  
  1106.                 for batch_data in self._prepare_batch_data(
  1107.                         examples, batch_size, phase=phase):
  1108.                     yield batch_data
  1109.  
  1110.         return _wrapper
  1111.  
  1112.  
  1113. # In[16]:
  1114.  
  1115.  
  1116. class ClassifyReader(BaseReader):
  1117.     """ClassifyReader"""
  1118.  
  1119.     def _read_tsv(self, input_file, quotechar=None):
  1120.         """Reads a tab separated value file."""
  1121.         with io.open(input_file, "r", encoding="utf8") as f:
  1122.             reader = csv_reader(f, delimiter="\t")
  1123.             headers = next(reader)
  1124.             text_indices = [
  1125.                 index for index, h in enumerate(headers) if h != "label"
  1126.             ]
  1127.             Example = namedtuple('Example', headers)
  1128.  
  1129.             examples = []
  1130.             for line in reader:
  1131.                 for index, text in enumerate(line):
  1132.                     if index in text_indices:
  1133.                         line[index] = text.replace(' ', '')
  1134.                 example = Example(*line)
  1135.                 examples.append(example)
  1136.             return examples
  1137.  
  1138.     def _pad_batch_records(self, batch_records):
  1139.         batch_token_ids = [record.token_ids for record in batch_records]
  1140.         batch_text_type_ids = [record.text_type_ids for record in batch_records]
  1141.         batch_position_ids = [record.position_ids for record in batch_records]
  1142.         batch_labels = [record.label_id for record in batch_records]
  1143.         batch_labels = np.array(batch_labels).astype("int64").reshape([-1, 1])
  1144.  
  1145.         # padding
  1146.         padded_token_ids, input_mask, seq_lens = pad_batch_data(
  1147.             batch_token_ids,
  1148.             pad_idx=self.pad_id,
  1149.             return_input_mask=True,
  1150.             return_seq_lens=True)
  1151.         padded_text_type_ids = pad_batch_data(
  1152.             batch_text_type_ids, pad_idx=self.pad_id)
  1153.         padded_position_ids = pad_batch_data(
  1154.             batch_position_ids, pad_idx=self.pad_id)
  1155.  
  1156.         return_list = [
  1157.             padded_token_ids, padded_text_type_ids, padded_position_ids,
  1158.             input_mask, batch_labels, seq_lens
  1159.         ]
  1160.  
  1161.         return return_list
  1162.  
  1163.  
  1164. # In[17]:
  1165.  
  1166.  
  1167. def pad_batch_data(insts,
  1168.                    pad_idx=0,
  1169.                    return_pos=False,
  1170.                    return_input_mask=False,
  1171.                    return_max_len=False,
  1172.                    return_num_token=False,
  1173.                    return_seq_lens=False):
  1174.     """
  1175.     Pad the instances to the max sequence length in batch, and generate the
  1176.     corresponding position data and input mask.
  1177.     """
  1178.     return_list = []
  1179.     max_len = max(len(inst) for inst in insts)
  1180.     # Any token included in dict can be used to pad, since the paddings' loss
  1181.     # will be masked out by weights and make no effect on parameter gradients.
  1182.  
  1183.     inst_data = np.array(
  1184.         [inst + list([pad_idx] * (max_len - len(inst))) for inst in insts])
  1185.     return_list += [inst_data.astype("int64").reshape([-1, max_len, 1])]
  1186.  
  1187.     # position data
  1188.     if return_pos:
  1189.         inst_pos = np.array([
  1190.             list(range(0, len(inst))) + [pad_idx] * (max_len - len(inst))
  1191.             for inst in insts
  1192.         ])
  1193.  
  1194.         return_list += [inst_pos.astype("int64").reshape([-1, max_len, 1])]
  1195.  
  1196.     if return_input_mask:
  1197.         # This is used to avoid attention on paddings.
  1198.         input_mask_data = np.array([[1] * len(inst) + [0] *
  1199.                                     (max_len - len(inst)) for inst in insts])
  1200.         input_mask_data = np.expand_dims(input_mask_data, axis=-1)
  1201.         return_list += [input_mask_data.astype("float32")]
  1202.  
  1203.     if return_max_len:
  1204.         return_list += [max_len]
  1205.  
  1206.     if return_num_token:
  1207.         num_token = 0
  1208.         for inst in insts:
  1209.             num_token += len(inst)
  1210.         return_list += [num_token]
  1211.  
  1212.     if return_seq_lens:
  1213.         seq_lens = np.array([len(inst) for inst in insts])
  1214.         return_list += [seq_lens.astype("int64").reshape([-1])]
  1215.  
  1216.     return return_list if len(return_list) > 1 else return_list[0]
  1217.  
  1218.  
  1219. # In[18]:
  1220.  
  1221.  
  1222. def ernie_pyreader(args, pyreader_name):
  1223.     """define standard ernie pyreader"""
  1224.     pyreader_name += '_' + ''.join(random.sample(string.ascii_letters + string.digits, 6))
  1225.     pyreader = fluid.layers.py_reader(
  1226.         capacity=50,
  1227.         shapes=[[-1, args['max_seq_len'], 1], [-1, args['max_seq_len'], 1],
  1228.                 [-1, args['max_seq_len'], 1], [-1, args['max_seq_len'], 1], [-1, 1],
  1229.                 [-1]],
  1230.         dtypes=['int64', 'int64', 'int64', 'float32', 'int64', 'int64'],
  1231.         lod_levels=[0, 0, 0, 0, 0, 0],
  1232.         name=pyreader_name,
  1233.         use_double_buffer=True)
  1234.  
  1235.     (src_ids, sent_ids, pos_ids, input_mask, labels,
  1236.      seq_lens) = fluid.layers.read_file(pyreader)
  1237.  
  1238.     ernie_inputs = {
  1239.         "src_ids": src_ids,
  1240.         "sent_ids": sent_ids,
  1241.         "pos_ids": pos_ids,
  1242.         "input_mask": input_mask,
  1243.         "seq_lens": seq_lens
  1244.     }
  1245.     return pyreader, ernie_inputs, labels
  1246.  
  1247.  
  1248. # #### 通用参数介绍
  1249. # 1. 数据集相关配置
  1250. # ```
  1251. # data_config = {
  1252. #     'data_dir': 'data/data9740/data',
  1253. #     'vocab_path': 'data/data9740/data/vocab.txt',
  1254. #     'batch_size': 32,
  1255. #     'random_seed': 0,
  1256. #     'num_labels': 3,
  1257. #     'max_seq_len': 512,
  1258. #     'train_set': 'data/data9740/data/test.tsv',
  1259. #     'test_set':  'data/data9740/data/test.tsv',
  1260. #     'dev_set':   'data/data9740/data/dev.tsv',
  1261. #     'infer_set': 'data/data9740/data/infer.tsv',
  1262. #     'label_map_config': None,
  1263. #     'do_lower_case': True,
  1264. # }
  1265. # ```
  1266. # 参数介绍：
  1267. # * **data_dir**：数据集路径，默认 'data/data9740/data'  
  1268. # * **vocab_path**：vocab.txt所在路径，默认 'data/data9740/data/vocab.txt'  
  1269. # * **batch_size**：训练和验证的批处理大小，默认：32  
  1270. # * **random_seed**：随机种子，默认 0  
  1271. # * **num_labels**：类别数，默认 3  
  1272. # * **max_seq_len**：句子中最长词数，默认 512  
  1273. # * **train_set**：训练集路径，默认 'data/data9740/data/test.tsv'  
  1274. # * **test_set**： 测试集路径，默认 'data/data9740/data/test.tsv'  
  1275. # * **dev_set**： 验证集路径，默认 'data/data9740/data/dev.tsv'  
  1276. # * **infer_set**：预测集路径，默认 'data/data9740/data/infer.tsv'  
  1277. # * **label_map_config**：label_map路径，默认 None  
  1278. # * **do_lower_case**：是否对输入进行额外的小写处理，默认 True  
  1279. # <br><br>
  1280. # 
  1281. # 2. ERNIE 网络结构相关配置
  1282. # ```
  1283. # ernie_net_config = {
  1284. #     "attention_probs_dropout_prob": 0.1,
  1285. #     "hidden_act": "relu",
  1286. #     "hidden_dropout_prob": 0.1,
  1287. #     "hidden_size": 768,
  1288. #     "initializer_range": 0.02,
  1289. #     "max_position_embeddings": 513,
  1290. #     "num_attention_heads": 12,
  1291. #     "num_hidden_layers": 12,
  1292. #     "type_vocab_size": 2,
  1293. #     "vocab_size": 18000,
  1294. # }
  1295. # ```
  1296. # 参数介绍：
  1297. # * **attention_probs_dropout_prob**：attention块dropout比例，默认 0.1  
  1298. # * **hidden_act**：隐层激活函数，默认 'relu'  
  1299. # * **hidden_dropout_prob**：隐层dropout比例，默认 0.1  
  1300. # * **hidden_size**：隐层大小，默认 768  
  1301. # * **initializer_range**：参数初始化缩放范围，默认 0.02  
  1302. # * **max_position_embeddings**：position序列最大长度，默认 513  
  1303. # * **num_attention_heads**：attention块头部数量，默认 12  
  1304. # * **num_hidden_layers**：隐层数，默认 12  
  1305. # * **type_vocab_size**：sentence类别数，默认 2  
  1306. # * **vocab_size**：字典长度，默认 18000  
  1307.  
  1308. # In[19]:
  1309.  
  1310.  
  1311. # 数据集相关配置
  1312. data_config = {
  1313.     'data_dir': 'data/data9740/data',               # Directory path to training data.
  1314.     'vocab_path': 'pretrained_model/ernie_finetune/vocab.txt',   # Vocabulary path.
  1315.     'batch_size': 32,   # Total examples' number in batch for training.
  1316.     'random_seed': 0,   # Random seed.
  1317.     'num_labels': 3,    # label number
  1318.     'max_seq_len': 512, # Number of words of the longest seqence.
  1319.     'train_set': 'data/data9740/data/test.tsv',   # Path to training data.
  1320.     'test_set':  'data/data9740/data/test.tsv',   # Path to test data.
  1321.     'dev_set':   'data/data9740/data/dev.tsv',    # Path to validation data.
  1322.     'infer_set': 'data/data9740/data/infer.tsv',  # Path to infer data.
  1323.     'label_map_config': None,   # label_map_path
  1324.     'do_lower_case': True,      # Whether to lower case the input text. Should be True for uncased models and False for cased models.
  1325. }
  1326.  
  1327. # Ernie 网络结构相关配置
  1328. ernie_net_config = {
  1329.     "attention_probs_dropout_prob": 0.1,
  1330.     "hidden_act": "relu",
  1331.     "hidden_dropout_prob": 0.1,
  1332.     "hidden_size": 768,
  1333.     "initializer_range": 0.02,
  1334.     "max_position_embeddings": 513,
  1335.     "num_attention_heads": 12,
  1336.     "num_hidden_layers": 12,
  1337.     "type_vocab_size": 2,
  1338.     "vocab_size": 18000,
  1339. }
  1340.  
  1341.  
  1342. # ### 四、模型训练
  1343. # 用户可基于百度开源的对话情绪识别模型在自有数据上实现 Finetune 训练，以期获得更好的效果提升，百度提供 TextCNN、ERNIE 两种预训练模型，具体模型 Finetune 方法如下所示：
  1344. # 1. 下载预训练模型
  1345. # 2. 修改参数
  1346. # 	* 'init_checkpoint'：'pretrained_model/ernie_finetune/params'
  1347. # 3. 执行 “ERNIE 训练代码”
  1348. # <br><br>
  1349. # 
  1350. # #### 训练阶段相关配置
  1351. # ```
  1352. # train_config = {
  1353. #     'init_checkpoint': 'pretrained_model/ernie_finetune/params',
  1354. #     'output_dir': 'train_model',
  1355. #     
  1356. #     'epoch': 10,
  1357. #     'save_steps': 100,
  1358. #     'validation_steps': 100,
  1359. #     'lr': 0.00002,
  1360. #     
  1361. #     'skip_steps': 10,
  1362. #     'verbose': False,
  1363. #     
  1364. #     'use_cuda': True,
  1365. # }
  1366. # ```
  1367. # 参数介绍：
  1368. # * **init_checkpoint**：是否使用预训练模型，默认：'pretrained_model/ernie_finetune/params'  
  1369. # * **output_dir**：模型缓存路径，默认 'train_model'  
  1370. # * **epoch**：训练轮数，默认 10  
  1371. # * **save_steps**：模型缓存间隔，默认 100  
  1372. # * **validation_steps**：验证间隔，默认 100  
  1373. # * **lr**：学习率，默认0.00002  
  1374. # * **skip_steps**：日志输出间隔，默认 10  
  1375. # * **verbose**：是否输出详细日志，默认 False  
  1376. # * **use_cuda**：是否使用 GPU，默认 True  
  1377.  
  1378. # In[20]:
  1379.  
  1380.  
  1381. # 下载预训练模型
  1382. get_ipython().system('mkdir pretrained_model')
  1383. # 下载并解压 ERNIE 预训练模型
  1384. get_ipython().system('cd pretrained_model && wget --no-check-certificate https://baidu-nlp.bj.bcebos.com/emotion_detection_ernie_finetune-1.0.0.tar.gz')
  1385. get_ipython().system('cd pretrained_model && tar xzf emotion_detection_ernie_finetune-1.0.0.tar.gz')
  1386.  
  1387.  
  1388. # In[21]:
  1389.  
  1390.  
  1391. # ERNIE 训练代码
  1392. train_config = {
  1393.     'init_checkpoint': 'pretrained_model/ernie_finetune/params',        # Init checkpoint to resume training from.
  1394.     # 'init_checkpoint': 'None',
  1395.     'output_dir': 'train_model',    # Directory path to save checkpoints
  1396.     
  1397.     'epoch': 5,                # Number of epoches for training.
  1398.     'save_steps': 100,          # The steps interval to save checkpoints.
  1399.     'validation_steps': 100,    # The steps interval to evaluate model performance.
  1400.     'lr': 0.00002,              # The Learning rate value for training.
  1401.     
  1402.     'skip_steps': 10,   # The steps interval to print loss.
  1403.     'verbose': False,   # Whether to output verbose log
  1404.     
  1405.     'use_cuda':True,   # If set, use GPU for training.
  1406. }
  1407. train_config.update(data_config)
  1408.  
  1409.  
  1410. def evaluate(exe, test_program, test_pyreader, fetch_list, eval_phase):
  1411.     """
  1412.     Evaluation Function
  1413.     """
  1414.     test_pyreader.start()
  1415.     total_cost, total_acc, total_num_seqs = [], [], []
  1416.     time_begin = time.time()
  1417.     while True:
  1418.         try:
  1419.             # 执行一步验证
  1420.             np_loss, np_acc, np_num_seqs = exe.run(program=test_program,
  1421.                                                    fetch_list=fetch_list,
  1422.                                                    return_numpy=False)
  1423.             np_loss = np.array(np_loss)
  1424.             np_acc = np.array(np_acc)
  1425.             np_num_seqs = np.array(np_num_seqs)
  1426.             total_cost.extend(np_loss * np_num_seqs)
  1427.             total_acc.extend(np_acc * np_num_seqs)
  1428.             total_num_seqs.extend(np_num_seqs)
  1429.         except fluid.core.EOFException:
  1430.             test_pyreader.reset()
  1431.             break
  1432.     time_end = time.time()
  1433.     logger.info("[%s evaluation] avg loss: %f, ave acc: %f, elapsed time: %f s" %
  1434.         (eval_phase, np.sum(total_cost) / np.sum(total_num_seqs),
  1435.         np.sum(total_acc) / np.sum(total_num_seqs), time_end - time_begin))
  1436.  
  1437.  
  1438. def main(config):
  1439.     """
  1440.     Main Function
  1441.     """
  1442.  
  1443.     # 定义 executor
  1444.     if config['use_cuda']:
  1445.         place = fluid.CUDAPlace(0)
  1446.         dev_count = fluid.core.get_cuda_device_count()
  1447.     else:
  1448.         place = fluid.CPUPlace()
  1449.         dev_count = int(os.environ.get('CPU_NUM', multiprocessing.cpu_count()))
  1450.     exe = fluid.Executor(place)
  1451.  
  1452.     # 定义数据 reader
  1453.     reader = ClassifyReader(
  1454.         vocab_path=config['vocab_path'],
  1455.         label_map_config=config['label_map_config'],
  1456.         max_seq_len=config['max_seq_len'],
  1457.         do_lower_case=config['do_lower_case'],
  1458.         random_seed=config['random_seed'])
  1459.  
  1460.     startup_prog = fluid.Program()
  1461.     if config['random_seed'] is not None:
  1462.         startup_prog.random_seed = config['random_seed']
  1463.  
  1464.     # 训练阶段初始化
  1465.     train_data_generator = reader.data_generator(
  1466.         input_file=config['train_set'],
  1467.         batch_size=config['batch_size'],
  1468.         epoch=config['epoch'],
  1469.         shuffle=True,
  1470.         phase="train")
  1471.  
  1472.     num_train_examples = reader.get_num_examples(config['train_set'])
  1473.  
  1474.     # 通过训练集大小 * 训练轮数得出总训练步数
  1475.     max_train_steps = config['epoch'] * num_train_examples // config['batch_size'] // dev_count + 1
  1476.  
  1477.     logger.info("Device count: %d" % dev_count)
  1478.     logger.info("Num train examples: %d" % num_train_examples)
  1479.     logger.info("Max train steps: %d" % max_train_steps)
  1480.  
  1481.     train_program = fluid.Program()
  1482.  
  1483.     with fluid.program_guard(train_program, startup_prog):
  1484.         with fluid.unique_name.guard():
  1485.             # create ernie_pyreader
  1486.             train_pyreader, ernie_inputs, labels = ernie_pyreader(config, pyreader_name='train_reader')
  1487.             embeddings = ernie_encoder(ernie_inputs, ernie_config=ernie_net_config)
  1488.  
  1489.             # user defined model based on ernie embeddings
  1490.             loss, accuracy, num_seqs = create_ernie_model(config, embeddings, labels=labels, is_prediction=False)
  1491.  
  1492.             """
  1493.             sgd_optimizer = fluid.optimizer.Adagrad(learning_rate=config['lr'])
  1494.             sgd_optimizer.minimize(loss)
  1495.             """
  1496.             optimizer = fluid.optimizer.Adam(learning_rate=config['lr'])
  1497.             optimizer.minimize(loss)
  1498.  
  1499.     if config['verbose']:
  1500.         lower_mem, upper_mem, unit = fluid.contrib.memory_usage(
  1501.             program=train_program, batch_size=config['batch_size'])
  1502.         logger.info("Theoretical memory usage in training: %.3f - %.3f %s" %
  1503.             (lower_mem, upper_mem, unit))
  1504.  
  1505.     # 验证阶段初始化
  1506.     test_prog = fluid.Program()
  1507.     with fluid.program_guard(test_prog, startup_prog):
  1508.         with fluid.unique_name.guard():
  1509.             # create ernie_pyreader
  1510.             test_pyreader, ernie_inputs, labels = ernie_pyreader(config, pyreader_name='eval_reader')
  1511.             embeddings = ernie_encoder(ernie_inputs, ernie_config=ernie_net_config)
  1512.  
  1513.             # user defined model based on ernie embeddings
  1514.             loss, accuracy, num_seqs = create_ernie_model(config, embeddings, labels=labels, is_prediction=False)
  1515.  
  1516.     test_prog = test_prog.clone(for_test=True)
  1517.  
  1518.     exe.run(startup_prog)
  1519.  
  1520.     # 加载预训练模型
  1521.     # if config['init_checkpoint']:
  1522.     #     init_checkpoint(exe, config['init_checkpoint'], main_program=train_program)
  1523.  
  1524.     # 模型训练代码
  1525.     if not os.path.exists(config['output_dir']):
  1526.         os.mkdir(config['output_dir'])
  1527.     
  1528.     logger.info('Start training')
  1529.     train_pyreader.decorate_tensor_provider(train_data_generator)
  1530.     train_pyreader.start()
  1531.     steps = 0
  1532.     total_cost, total_acc, total_num_seqs = [], [], []
  1533.     time_begin = time.time()
  1534.     while True:
  1535.         try:
  1536.             steps += 1
  1537.             if steps % config['skip_steps'] == 0:
  1538.                 fetch_list = [loss.name, accuracy.name, num_seqs.name]
  1539.             else:
  1540.                 fetch_list = []
  1541.  
  1542.             # 执行一步训练
  1543.             outputs = exe.run(program=train_program, fetch_list=fetch_list, return_numpy=False)
  1544.             if steps % config['skip_steps'] == 0:
  1545.                 # 打印日志
  1546.                 np_loss, np_acc, np_num_seqs = outputs
  1547.                 np_loss = np.array(np_loss)
  1548.                 np_acc = np.array(np_acc)
  1549.                 np_num_seqs = np.array(np_num_seqs)
  1550.                 total_cost.extend(np_loss * np_num_seqs)
  1551.                 total_acc.extend(np_acc * np_num_seqs)
  1552.                 total_num_seqs.extend(np_num_seqs)
  1553.  
  1554.                 if config['verbose']:
  1555.                     verbose = "train pyreader queue size: %d, " % train_pyreader.queue.size()
  1556.                     logger.info(verbose)
  1557.  
  1558.                 time_end = time.time()
  1559.                 used_time = time_end - time_begin
  1560.                 logger.info("step: %d, avg loss: %f, "
  1561.                     "avg acc: %f, speed: %f steps/s" %
  1562.                     (steps, np.sum(total_cost) / np.sum(total_num_seqs),
  1563.                     np.sum(total_acc) / np.sum(total_num_seqs),
  1564.                     config['skip_steps'] / used_time))
  1565.                 total_cost, total_acc, total_num_seqs = [], [], []
  1566.                 time_begin = time.time()
  1567.  
  1568.             if steps % config['save_steps'] == 0:
  1569.                 # 缓存模型
  1570.                 # fluid.io.save_persistables(exe, config['output_dir'], train_program)
  1571.                 fluid.save(train_program, os.path.join(config['output_dir'], "checkpoint"))
  1572.             if steps % config['validation_steps'] == 0:
  1573.                 # 在验证集上执行验证
  1574.                 test_pyreader.decorate_tensor_provider(
  1575.                     reader.data_generator(
  1576.                         input_file=config['dev_set'],
  1577.                         batch_size=config['batch_size'],
  1578.                         phase='dev',
  1579.                         epoch=1,
  1580.                         shuffle=False))
  1581.  
  1582.                 evaluate(exe, test_prog, test_pyreader,
  1583.                         [loss.name, accuracy.name, num_seqs.name],
  1584.                         "dev")
  1585.  
  1586.         except fluid.core.EOFException:
  1587.             # 训练结束
  1588.             # fluid.io.save_persistables(exe, config['output_dir'], train_program)
  1589.             fluid.save(train_program, os.path.join(config['output_dir'], "checkpoint"))
  1590.             train_pyreader.reset()
  1591.             logger.info('Training end.')
  1592.             break
  1593.  
  1594.     # 模型验证代码
  1595.     test_pyreader.decorate_tensor_provider(
  1596.         reader.data_generator(
  1597.             input_file=config['test_set'],
  1598.             batch_size=config['batch_size'], phase='test', epoch=1,
  1599.             shuffle=False))
  1600.     logger.info("Final validation result:")
  1601.     evaluate(exe, test_prog, test_pyreader,
  1602.         [loss.name, accuracy.name, num_seqs.name], "test")
  1603.  
  1604.  
  1605. if __name__ == "__main__":
  1606.     init_log_config()
  1607.     print_arguments(train_config)
  1608.     main(train_config)
  1609.  
  1610.  
  1611. # ### 五、模型预测
  1612. # 
  1613. # 
  1614. # 预测阶段加载保存的模型，对预测集进行预测，通过修改如下参数实现
  1615. # <br><br>
  1616. # 
  1617. # #### 预测阶段相关配置
  1618. # ```
  1619. # infer_config = {
  1620. #     'init_checkpoint': 'train_model',
  1621. #     'use_cuda': True,
  1622. # }
  1623. # ```
  1624. # 参数介绍：
  1625. # * **init_checkpoint**：加载预训练模型，默认：'train_model'  
  1626. # * **use_cuda**：是否使用 GPU，默认 True  
  1627.  
  1628. # In[22]:
  1629.  
  1630.  
  1631. # ERNIE 预测代码
  1632. infer_config = {
  1633.     'init_checkpoint': 'train_model',        # Init checkpoint to resume training from.
  1634.     'use_cuda': True,   # If set, use GPU for training.
  1635. }
  1636. infer_config.update(data_config)
  1637.  
  1638.  
  1639. def init_checkpoint_infer(exe, init_checkpoint_path, main_program):
  1640.     """
  1641.     加载缓存模型
  1642.     """
  1643.     assert os.path.exists(
  1644.         init_checkpoint_path), "[%s] cann't be found." % init_checkpoint_path
  1645.  
  1646.     # fluid.io.load_vars(
  1647.     #     exe,
  1648.     #     init_checkpoint_path,
  1649.     #     main_program=main_program,
  1650.     #     predicate=existed_persitables)
  1651.     fluid.load(main_program, os.path.join(init_checkpoint_path, "checkpoint"), exe)
  1652.     logger.info("Load model from {}".format(init_checkpoint_path))
  1653.  
  1654.  
  1655. def infer(exe, infer_program, infer_pyreader, fetch_list, infer_phase, examples):
  1656.     """Infer"""
  1657.     infer_pyreader.start()
  1658.     time_begin = time.time()
  1659.     while True:
  1660.         try:
  1661.             # 进行一步预测
  1662.             batch_probs = exe.run(program=infer_program, fetch_list=fetch_list,
  1663.                                 return_numpy=True)
  1664.             for i, probs in enumerate(batch_probs[0]):
  1665.                 logger.info("Probs: %f %f %f, prediction: %d, input: %s" % (probs[0], probs[1], probs[2], np.argmax(probs), examples[i]))
  1666.         except fluid.core.EOFException:
  1667.             infer_pyreader.reset()
  1668.             break
  1669.     time_end = time.time()
  1670.     logger.info("[%s] elapsed time: %f s" % (infer_phase, time_end - time_begin))
  1671.  
  1672.  
  1673. def main(config):
  1674.     """
  1675.     Main Function
  1676.     """
  1677.  
  1678.     # 定义 executor
  1679.     if config['use_cuda']:
  1680.         place = fluid.CUDAPlace(0)
  1681.         dev_count = fluid.core.get_cuda_device_count()
  1682.     else:
  1683.         place = fluid.CPUPlace()
  1684.         dev_count = int(os.environ.get('CPU_NUM', multiprocessing.cpu_count()))
  1685.     exe = fluid.Executor(place)
  1686.  
  1687.     # 定义数据 reader
  1688.     reader = ClassifyReader(
  1689.         vocab_path=config['vocab_path'],
  1690.         label_map_config=config['label_map_config'],
  1691.         max_seq_len=config['max_seq_len'],
  1692.         do_lower_case=config['do_lower_case'],
  1693.         random_seed=config['random_seed'])
  1694.  
  1695.     startup_prog = fluid.Program()
  1696.     if config['random_seed'] is not None:
  1697.         startup_prog.random_seed = config['random_seed']
  1698.  
  1699.     # 预测阶段初始化
  1700.     test_prog = fluid.Program()
  1701.     with fluid.program_guard(test_prog, startup_prog):
  1702.         with fluid.unique_name.guard():
  1703.             infer_pyreader, ernie_inputs, labels = ernie_pyreader(config, pyreader_name='infer_reader')
  1704.             embeddings = ernie_encoder(ernie_inputs, ernie_config=ernie_net_config)
  1705.  
  1706.             probs = create_ernie_model(config, embeddings, labels=labels, is_prediction=True)
  1707.     test_prog = test_prog.clone(for_test=True)
  1708.  
  1709.     exe.run(startup_prog)
  1710.  
  1711.     # 加载预训练模型
  1712.     if not config['init_checkpoint']:
  1713.         raise ValueError("args 'init_checkpoint' should be set if"
  1714.                          "only doing validation or infer!")
  1715.     init_checkpoint_infer(exe, config['init_checkpoint'], main_program=test_prog)
  1716.  
  1717.     # 模型预测代码
  1718.     infer_pyreader.decorate_tensor_provider(
  1719.         reader.data_generator(
  1720.             input_file=config['infer_set'],
  1721.             batch_size=config['batch_size'],
  1722.             phase='infer',
  1723.             epoch=1,
  1724.             shuffle=False))
  1725.     logger.info("Final test result:")
  1726.     infer(exe, test_prog, infer_pyreader,
  1727.         [probs.name], "infer", reader.get_examples(config['infer_set']))
  1728.  
  1729. if __name__ == "__main__":
  1730.     init_log_config()
  1731.     print_arguments(infer_config)
  1732.     main(infer_config)
  1733.  
  1734.  
  1735. # ### 六、总结
  1736. # 
  1737. # ERNIE 在对话情绪识别数据集上的实际运行结果如下：
  1738. # 
  1739. # 
  1740. # | 模型 | 准确率 |
  1741. # | -------- | -------- |
  1742. # | ERNIE pretrained | 0.944981
  1743. # | ERNIE finetuned | 0.999035
  1744. # <br>
  1745. # 
  1746. # 本项目实现 ERNIE 1.0 版本，在对话情绪识别任务上表现良好，除此之外，ERNIE 还可以执行：
  1747. # * 自然语言推断任务 XNLI
  1748. # * 阅读理解任务 DRCD、DuReader、CMRC2018
  1749. # * 命名实体识别任务 MSRA-NER (SIGHAN2006)
  1750. # * 情感分析任务 ChnSentiCorp
  1751. # * 语义相似度任务 BQ Corpus、LCQMC
  1752. # * 问答任务 NLPCC2016-DBQA
  1753. # 
  1754. # 读者也可以尝试移植 ERNIE 2.0 进行对比测试。

 六、PaddlePaddle结合爬虫实现女票微博情绪长期监控：

1、imdb数据集可网络搜索，整理后灌入数据：

  1. from __future__ import print_function
  2. import sys
  3. import paddle.v2 as paddle
  4. from __future__ import print_function
  5. import sys
  6. import paddle.v2 as paddle
  7. import sys
  8. import os
  9. import json
  10. import nltk
  11. if __name__ == '__main__':
  12.     # init
  13.     paddle.init(use_gpu=False)
  14.     print('load dictionary...')
  15.     word_dict = paddle.dataset.imdb.word_dict()
  16.     print(word_dict)

2、数据读取及网络定义准备：

  1. def reader_creator(pos_pattern, neg_pattern, word_idx, buffer_size):
  2. 	# this unk is a token
  3.     UNK = word_idx['']
  4. 	# start a quen to using multi-process
  5.     qs = [Queue.Queue(maxsize=buffer_size), Queue.Queue(maxsize=buffer_size)]
  6.     def load(pattern, queue):
  7.         for doc in tokenize(pattern):
  8.             queue.put(doc)
  9.         queue.put(None)
  10.     def reader():
  11.         # Creates two threads that loads positive and negative samples
  12.         # into qs.
  13.         t0 = threading.Thread(
  14.             target=load, args=(
  15.                 pos_pattern,
  16.                 qs[0], ))
  17.         t0.daemon = True
  18.         t0.start()
  19.         t1 = threading.Thread(
  20.             target=load, args=(
  21.                 neg_pattern,
  22.                 qs[1], ))
  23.         t1.daemon = True
  24.         t1.start()
  25.         # Read alternatively from qs[0] and qs[1].
  26.         i = 0
  27.         doc = qs[i].get()
  28.         while doc != None:
  29.             yield [word_idx.get(w, UNK) for w in doc], i % 2
  30.             i += 1
  31.             doc = qs[i % 2].get()
  32.         # If any queue is empty, reads from the other queue.
  33.         i += 1
  34.         doc = qs[i % 2].get()
  35.         while doc != None:
  36.             yield [word_idx.get(w, UNK) for w in doc], i % 2
  37.             doc = qs[i % 2].get()
  38.     return reader()

这个方法其实已经内置在Paddle中，我们不需要写它，但是为了让大家能够理解，我把它单独拿出来讲解一下，这个函数执行的操作其实非常简单，那就是根据上面所得到的word dict，把文本的每一个句子转换成一维的数字向量。由于Imdb里面是一句正情绪，一句负情绪，所以或有一个 %2的操作。

好了，接下来重点来了，我们要用PaddlePaddle构建我们的模型了，我之前提到了这个embed层，我们直接embed之后，接一个CNN来构建一个简单的文本卷积分类网络：

  1. def convolution_net(input_dim, class_dim=2, emb_dim=128, hid_dim=128):
  2. 	# we are starting with a embed layer
  3.     data = paddle.layer.data("word",
  4.                              paddle.data_type.integer_value_sequence(input_dim))
  5.     emb = paddle.layer.embedding(input=data, size=emb_dim)
  6.     # this convolution is a sequence convolution
  7.     conv_3 = paddle.networks.sequence_conv_pool(
  8.         input=emb, context_len=3, hidden_size=hid_dim)
  9.     conv_4 = paddle.networks.sequence_conv_pool(
  10.         input=emb, context_len=4, hidden_size=hid_dim)
  11.     output = paddle.layer.fc(
  12.         input=[conv_3, conv_4], size=class_dim, act=paddle.activation.Softmax())
  13.     lbl = paddle.layer.data("label", paddle.data_type.integer_value(2))
  14.     cost = paddle.layer.classification_cost(input=output, label=lbl)
  15.     return cost, output

这里面有一个词嵌入操作，紧接着是两个卷积层，注意这里的卷基层并非是图片卷积，而是文本序列卷积，这个应该是PaddlePaddle中特有的一个特殊层，百度在文本序列和语音序列处理上还是有一套，等一下大家会看到，这么一个简单的模型可以在仅仅6个epoch就达到99.99%的精确度。embed的size是128，隐藏层神经元个数是128。 

3、训练：

  1. from __future__ import print_function
  2. import sys
  3. import paddle.v2 as paddle
  4. import sys
  5. import os
  6. import json
  7. import nltk
  8. def convolution_net(input_dim, class_dim=2, emb_dim=128, hid_dim=128):
  9.     data = paddle.layer.data("word",
  10.                              paddle.data_type.integer_value_sequence(input_dim))
  11.     emb = paddle.layer.embedding(input=data, size=emb_dim)
  12.     conv_3 = paddle.networks.sequence_conv_pool(
  13.         input=emb, context_len=3, hidden_size=hid_dim)
  14.     conv_4 = paddle.networks.sequence_conv_pool(
  15.         input=emb, context_len=4, hidden_size=hid_dim)
  16.     output = paddle.layer.fc(
  17.         input=[conv_3, conv_4], size=class_dim, act=paddle.activation.Softmax())
  18.     lbl = paddle.layer.data("label", paddle.data_type.integer_value(2))
  19.     cost = paddle.layer.classification_cost(input=output, label=lbl)
  20.     return cost, output
  21.     
  22. if __name__ == '__main__':
  23.     # init
  24.     paddle.init(use_gpu=False)
  25. 	# those lines are get the code
  26.     print('load dictionary...')
  27.     word_dict = paddle.dataset.imdb.word_dict()
  28.     print(word_dict)
  29.     dict_dim = len(word_dict)
  30.     class_dim = 2
  31.     train_reader = paddle.batch(
  32.         paddle.reader.shuffle(
  33.             lambda: paddle.dataset.imdb.train(word_dict), buf_size=1000),
  34.         batch_size=100)
  35.     test_reader = paddle.batch(
  36.         lambda: paddle.dataset.imdb.test(word_dict),
  37.         batch_size=100)
  38.     feeding = {'word': 0, 'label': 1}
  39. 	# get the output of the model
  40.     [cost, output] = convolution_net(dict_dim, class_dim=class_dim)
  41.     parameters = paddle.parameters.create(cost)
  42.     adam_optimizer = paddle.optimizer.Adam(
  43.         learning_rate=2e-3,
  44.         regularization=paddle.optimizer.L2Regularization(rate=8e-4),
  45.         model_average=paddle.optimizer.ModelAverage(average_window=0.5))
  46.     trainer = paddle.trainer.SGD(
  47.         cost=cost, parameters=parameters, update_equation=adam_optimizer)
  48.     def event_handler(event):
  49.         if isinstance(event, paddle.event.EndIteration):
  50.             if event.batch_id % 100 == 0:
  51.                 print("\nPass %d, Batch %d, Cost %f, %s" % (
  52.                     event.pass_id, event.batch_id, event.cost, event.metrics))
  53.             else:
  54.                 sys.stdout.write('.')
  55.                 sys.stdout.flush()
  56.         if isinstance(event, paddle.event.EndPass):
  57.             with open('./params_pass_%d.tar' % event.pass_id, 'w') as f:
  58.                 trainer.save_parameter_to_tar(f)
  59.             result = trainer.test(reader=test_reader, feeding=feeding)
  60.             print("\nTest with Pass %d, %s" % (event.pass_id, result.metrics))
  61.     inference_topology = paddle.topology.Topology(layers=output)
  62.     with open("./inference_topology.pkl", 'wb') as f:
  63.         inference_topology.serialize_for_inference(f)
  64.     trainer.train(
  65.         reader=train_reader,
  66.         event_handler=event_handler,
  67.         feeding=feeding,
  68.         num_passes=20)

训练完成后我们现在只有一个main.py，这里面就是我们训练的脚本。我们有一个inference_topology.pkl，这个是我们的网络模型保存的二进制文件。

4、预测：

  1. import numpy as np
  2. import sys
  3. import paddle.v2 as paddle
  4. from __future__ import print_function
  5. import sys
  6. import os
  7. import json
  8. import nltk
  9. def convolution_net(input_dim,
  10.                     class_dim=2,
  11.                     emb_dim=128,
  12.                     hid_dim=128,
  13.                     is_predict=False):
  14.     data = paddle.layer.data("word",
  15.                              paddle.data_type.integer_value_sequence(input_dim))
  16.     emb = paddle.layer.embedding(input=data, size=emb_dim)
  17.     conv_3 = paddle.networks.sequence_conv_pool(
  18.         input=emb, context_len=3, hidden_size=hid_dim)
  19.     conv_4 = paddle.networks.sequence_conv_pool(
  20.         input=emb, context_len=4, hidden_size=hid_dim)
  21.     output = paddle.layer.fc(input=[conv_3, conv_4],
  22.                              size=class_dim,
  23.                              act=paddle.activation.Softmax())
  24.     if not is_predict:
  25.         lbl = paddle.layer.data("label", paddle.data_type.integer_value(2))
  26.         cost = paddle.layer.classification_cost(input=output, label=lbl)
  27.         return cost
  28.     else:
  29.         return output
  30. if __name__ == '__main__':
  31.     # Movie Reviews, from imdb test
  32.     paddle.init(use_gpu=False)
  33.     word_dict = paddle.dataset.imdb.word_dict()
  34.     dict_dim = len(word_dict)
  35.     class_dim = 2
  36.     reviews = [
  37.         'Read the book, forget the movie!',
  38.         'This is a great movie.'
  39.     ]
  40.     print(reviews)
  41.     reviews = [c.split() for c in reviews]
  42.     UNK = word_dict['']
  43.     input = []
  44.     for c in reviews:
  45.         input.append([[word_dict.get(words, UNK) for words in c]])
  46.     # 0 stands for positive sample, 1 stands for negative sample
  47.     label = {0: 'pos', 1: 'neg'}
  48.     # Use the network used by trainer
  49.     out = convolution_net(dict_dim, class_dim=class_dim, is_predict=True)
  50.     
  51.     parameters = paddle.parameters.create(out)
  52.     print(parameters)
  53.     # out = stacked_lstm_net(dict_dim, class_dim=class_dim, stacked_num=3, is_predict=True)
  54.     probs = paddle.infer(output_layer=out, parameters=parameters, input=input)
  55.     print('probs:', probs)
  56.     labs = np.argsort(-probs)
  57.     print(labs)
  58.     for idx, lab in enumerate(labs):
  59.         print(idx, "predicting probability is", probs[idx], "label is", label[lab[0]])

review的内容可由微博爬虫得到。

5、微博爬虫程序爬取微博内容，作为上述review的信息：

此处感谢@cici_富贵的文章，同样深受启发。

url = 'https://m.weibo.cn'首先打开手机端微博的登陆网址，登上自己的微博。

之后就是打开你的特别关注，然后搜索你喜欢的那个人，进入她的微博主页，然后F12，到Network界面下，选择XHR，F5刷新以下就会出现Ajax响应：
然后接着下拉，找寻加载页面返回内容的Ajax请求响应，并打开对其进行分析：

在这里插入图片描述

在这里插入图片描述

很容易找到在所选响应下的data->cards->mblog中找到自己想要的内容，并发现其时以json形式返回的，那么我就开始分析其请求头和所携带的参数列表，寻找其中的不同，以构造自己的Ajax请求头：

 在这里插入图片描述

在这里插入图片描述 在这里插入图片描述

观察发现在参数列表当中，刚点进去微博主页刷新得到的参数列表中没有since_id这一项，而下拉内容再返回的响应参数列表中却出现的不同的since_id，可以发现再请求响应过程中只有since_id在改变，其它参数都没有变（since_id是用来控制页面的增加的）

在搞清楚这些后我们就可以构造自己的头部，模拟Ajax请求从而爬取内容：

  1. from urllib.parse import urlencode
  2. from pyquery import PyQuery as py
  3. import requests
  4.  
  5. def get_information(since_id = '',uid = 0,):
  6.     #X-Requested-With 用来标识Ajax请求，必须得有
  7.     #Referer 用来指明请求来源  必须有
  8.     #User-Agent  伪装浏览器，必须有
  9.     headers = {'Referer': 'https://m.weibo.cn',
  10.                 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36',
  11.                 'X-Requested-With': 'XMLHttpRequest'}
  12.  
  13.     params = {
  14.                 'uid': uid,
  15.                 'luicode': 10000011,
  16.                 'lfid': '231093_-_selffollowed',
  17.                 'type': int(uid),
  18.                 'value': 5768045317,
  19.                 'containerid': '107603' + str(uid),
  20.                 'since_id': since_id
  21.     }
  22.  
  23.     # urlencode() 方法将参数转化为URL的GET请求参数
  24.     url = 'https://m.weibo.cn/api/container/getIndex?' + urlencode(params)
  25.     response = requests.get(url,headers = headers)
  26.     #获得返回的 json 内容，做分析
  27.     json_text = response.json()
  28.     #获得 since_id 为增加页面做准备
  29.     since_id = json_text.get('data').get('cardlistInfo').get('since_id')
  30.     return json_text,since_id
  31.  
  32. def parse_json(json):
  33.     items = json.get('data').get('cards')
  34.  
  35.  
  36.  
  37.     for item in items:
  38.         item = item.get('mblog')
  39.         weibo = {}
  40.         weibo['发表时间'] = item.get('created_at')
  41.         weibo['手机类型'] = item.get('source')
  42.         weibo['内容'] = py(item.get('text')).text()
  43.         weibo['图片链接'] = item.get('bmiddle_pic')
  44.         weibo['点赞数'] = item.get('attitudes_count')
  45.         weibo['评论数'] = item.get('comments_count')
  46.         yield weibo
  47.  
  48.  
  49. if __name__ == '__main__':
  50.     #uid 你所要爬取的微博的ID，在响应的参数列表中可以得到，图中可以找到
  51.     uid = 5768045317
  52.     #p 爬取的页数
  53.     p = 3
  54.     #获得返回的 JSON 内容 和 since_id
  55.     s = get_information(since_id = '',uid = uid)
  56.     #解析 JSON
  57.     parse_json(s[0])
  58.     #输出解析后的内容
  59.     for i in parse_json(s[0]):
  60.         print(i)
  61.     '''
  62.     #多页爬取
  63.     for i in range(p):
  64.         s = get_information(since_id = s[1],uid = uid)
  65.         for i in parse_json(s[0]):
  66.             print(i)
  67.     '''

 

笔者首先需要澄清，本文的核心目的是学习自然语言处理预训练模型的调用和Finetune等方法，并且利用一个有趣的现实问题进行实战。虽然笔者找到了这几种测试对话情绪的方式，同时也利用飞桨框架实现了对对话内容的预测，但是笔者不会用于真实生活中。同样作为工程师or程序员的你不要忘记用自己对待工作的认真和拼命的态度去对待自己心爱的女孩哦。
不要忘记那句歌词：“女孩儿的心思男孩你别猜，你猜来猜去也猜不明白。不知道她为什么掉眼泪，也不知她为什么笑开怀……”

 

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/小蓝xlanll/article/detail/311463?site
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号