垂域LLM评测指标个人记录

bertscore

1.安装

pip install bert-score

2.如果报错

OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like roberta-large is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

解决办法：改为hf镜像网站

pip install -U huggingface_hub
export HF_ENDPOINT=https://hf-mirror.com

代码

import os
os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'
from huggingface_hub import hf_hub_download
 
from bert_score import score
# 定义参考句子和生成句子
refs = ["The cat sat on the mat.", "It was raining outside."]
cands = ["The cat sat on the mat.", "It was pouring outside."]
 
# 使用bert_score计算分数
P, R, F1 = score(cands, refs, lang='en', model_type="roberta-large", verbose=True)
 
# 打印结果
print("Precision:", P)
print("Recall:", R)
print("F1 score:", F1)