python短文本相似度计算_预训练Bert句向量_flask_python bert-base-chinese 文本相似度

1. 任务介绍(预训练模型sentence_transformers来做文本相似度匹配任务)

从一堆候选短文本数据集中筛选出与输入问句最相似的短文本（即短文本相似度匹配）

1.1 首先输入比如是:“我要出国”

1.2 候选数据集有10条短文本，如下：

1.3 最终输出相似度最高的文本，如下：

外国人出入境证签发

附录一：代码实现：

from sentence_transformers import SentenceTransformer, util
import pandas as pd
from flask import Flask, request, jsonify
import json
import numpy as np
import time
 
app = Flask(__name__)
model = SentenceTransformer('bert-base-chinese')
model.eval()
content = pd.read_table("data.txt", header=None)[0].values.tolist()
passage_embedding = model.encode(content, convert_to_numpy=True)
 
def cosine_similarity(x, y):
    num = x.dot(y.T)
    denom = np.linalg.norm(x) * np.linalg.norm(y, axis=1)
    return num / denom
 
def sim(ask, level):
    time1 = time.time()
    ask = model.encode([ask])
    time2 = time.time()
    spend_time = time2 - time1
    print(spend_time)
    cos_matrix = cosine_similarity(ask, passage_embedding)[0]
    index = np.argsort(-cos_matrix)[level]
    return [content[index], cos_matrix[index]]
 
@app.route('/top_ask', methods=['GET', 'POST'])
def get_synonyms():
    json_data = request.get_data()
    comment_json = json.loads(json_data)
    ask = comment_json['ask']
    json_data = sim2(ask, 0)
    return jsonify({"text": str(json_data[0]), "相似度": str(json_data[1])})
 
if __name__ == '__main__':  #
    app.run(host="0.0.0.0", port=1089, debug=True)