HuggingFace 模型离线使用最佳方法！_huggingface上的模型和数据如何离线安装使用

以：Helsinki-NLP/opus-mt-en-zh （translation）模型为例：

如图，假设我们的运行环境基于python，那么我们通常只需要以下文件：

"config.json",
"pytorch_model.bin",
"tokenizer_config.json",
"vocab.json",
"source.spm",
"target.spm" 

其余文件如：rust,我们并不需要下载， 不使用tesorflow ,tf_model.h5也不需下载/

huggingface 官方提供以下几种下载方式：

git lfs install

git clone https://huggingface.co/Helsinki-NLP/opus-mt-en-zh

# If you want to clone without large files - just their pointers
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Helsinki-NLP/opus-mt-en-zh

有完全克隆库，也有只克隆指针文件，但都无法根据我们的语言需求指定下载我们需要的部分文件：

所以本文推荐使用以下代码进行模型下载：

import requests
import os
 
def download_file(url, dest_folder, file_name):
    """
    下载文件并保存到指定的文件夹中。
    """
    response = requests.get(url, allow_redirects=True)
    if response.status_code == 200:
        with open(os.path.join(dest_folder, file_name), 'wb') as file:
            file.write(response.content)
    else:
        print(f"Failed to download {file_name}. Status code: {response.status_code}")
 
def download_model_files(model_name):
    """
    根据模型名称下载模型文件。
    """
    # 文件列表
    files_to_download = [
        "config.json",
        "pytorch_model.bin",
        "tokenizer_config.json",
        "vocab.json",
        "source.spm",
        "target.spm"  # 如果模型不使用SentencePiece，这两个文件可能不需要
    ]
 
    # 创建模型文件夹
    model_folder = model_name.split('/')[-1]  # 从模型名称中获取文件夹名称
    if not os.path.exists(model_folder):
        os.makedirs(model_folder)
 
    # 构建下载链接并下载文件
    base_url = f"https://huggingface.co/{model_name}/resolve/main/"
    for file_name in files_to_download:
        download_url = base_url + file_name
        print(f"Downloading {file_name}...")
        download_file(download_url, model_folder, file_name)
 
# 示例使用
model_name = "Helsinki-NLP/opus-mt-zh-en"
download_model_files(model_name)

如果需要增添模型文件，可在文件列表进行修改。

同时：当我们使用：

pipeline 本地运行模型时，虽然会下载python_model， 但是仍需要联网访问config.json不能实现离线运行，所以，当使用本文上述下载代码完成模型的下载后：

运行以下案例代码：

from transformers import pipeline, AutoTokenizer
import os
 
os.environ['TRANSFORMERS_OFFLINE']="1"
# 示例文本列表
texts_to_translate = [
    "你好",
    "你好啊",
    # 添加更多文本以确保总token数超过400
]
 
# 指定模型的本地路径
model_name = "./opus-mt-zh-en"  # 请替换为实际路径
# 创建翻译管道，指定本地模型路径
pipe = pipeline("translation", model=model_name)
# 获取tokenizer，指定本地模型路径
tokenizer = AutoTokenizer.from_pretrained(model_name)
 
result = pipe(texts_to_translate)
print(result)

建议保留：

os.environ['TRANSFORMERS_OFFLINE']="1"，放弃向huggingface联网访问。

结果如下：

2024-03-29 15:57:52.211565: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-03-29 15:57:53.251227: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[{'translation_text': 'Hello.'}, {'translation_text': 'Hello.'}]