Huggingface镜像网站下载语言模型方法

通常通过镜像网站下载https://hf-mirror.com/。
在链接页面有介绍方法，对于不大的模型可以直接下载。这里介绍比较常用且方便的下载方法。

使用huggingface 官方提供的 huggingface-cli 命令行工具

安装（huggingface_hub、hf_transfer安装可以使用-i命令从镜像网站下载）

pip install -U huggingface_hub hf_transfer
1

基本命令（每次打开远程链接都要输入）

export HF_HUB_ENABLE_HF_TRANSFER=1
export HF_ENDPOINT=https://hf-mirror.com
1
2

下载模型（下载NousResearch/Llama-2-13b-chat-hf，放在当前目录文件夹Llama-2-13b-chat-hf下）

huggingface-cli download --resume-download NousResearch/Llama-2-13b-chat-hf --local-dir Llama-2-13b-chat-hf --local-dir-use-symlinks False
1

携带参数下载

huggingface-cli download --token hf_*** --resume-download meta-llama/Llama-2-7b-hf --local-dir Llama-2-7b-hf --local-dir-use-symlinks False
1

对于下载中断的问题，由于huggingface-cli本身能断点重传，但下载界面关闭后只能重新下载，因而采用选择性下载

huggingface-cli download --resume-download NousResearch/Llama-2-70b-chat-hf --local-dir Llama-2-70b-chat-hf --local-dir-use-symlinks False --include "model-00013-of-00015.safetensors"

huggingface-cli download --resume-download NousResearch/Llama-2-70b-chat-hf --local-dir Llama-2-70b-chat-hf --local-dir-use-symlinks False --include "pytorch_model*.bin"
1
2
3

其他用法

huggingface-cli download --helps

usage: huggingface-cli <command> [<args>] download [-h] [--repo-type {model,dataset,space}]
                                                   [--revision REVISION] [--include [INCLUDE ...]]
                                                   [--exclude [EXCLUDE ...]] [--cache-dir CACHE_DIR]  
                                                   [--local-dir LOCAL_DIR]   
                                                   [--local-dir-use-symlinks {auto,True,False}]   
                                                   [--force-download] [--resume-download]
                                                   [--token TOKEN]                                                         
                                                   [--quiet]                                                                                                      
                                                   repo_id [filenames ...]
positional arguments:                                                                                            repo_id               ID of the repo to download from (e.g. `username/repo-name`).
  filenames             Files to download (e.g. `config.json`, `data/metadata.jsonl`).

options:
  -h, --help            show this help message and exit
  --repo-type {model,dataset,space}
                        Type of repo to download from (e.g. `dataset`).
  --revision REVISION   An optional Git revision id which can be a branch name, a tag, or a commit hash.
  --include [INCLUDE ...]
                        Glob patterns to match files to download.
  --exclude [EXCLUDE ...]
                        Glob patterns to exclude from files to download.
  --cache-dir CACHE_DIR
                        Path to the directory where to save the downloaded files.
  --local-dir LOCAL_DIR
                        If set, the downloaded file will be placed under this directory either as a symlink
                        (default) or a regular file. Check out
                        https://huggingface.co/docs/huggingface_hub/guides/download#download-files-to-local-
                        folder for more details.
  --local-dir-use-symlinks {auto,True,False}
                        To be used with `local_dir`. If set to 'auto', the cache directory will be used and
                        the file will be either duplicated or symlinked to the local directory depending on
                        its size. It set to `True`, a symlink will be created, no matter the file size. If
                        set to `False`, the file will either be duplicated from cache (if already exists) or
                        downloaded from the Hub and not cached.
  --force-download      If True, the files will be downloaded even if they are already cached.
  --resume-download     If True, resume a previously interrupted download.
  --token TOKEN         A User Access Token generated from https://huggingface.co/settings/tokens
  --quiet               If True, progress bars are disabled and only the path to the download files is
                        printed.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41