【safetensor】Debug_with safe_open(checkpoint_file, framework="pt") as

更多信息详见我的另一篇文档:
介绍

加载权重错误

从本地加载 stable-diffusion 时，

model_id = 'D:\code\git_diffusion_repo\sd\stable-diffusion-v1-5'
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16, revision="fp16")
1
2

出现 加载文件错误：

Loading pipeline components...:   0%|          | 0/7 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "D:\code\Github_code\sd\stable-diffusion-main\加载权重.py", line 7, in <module>
    pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16, revision="fp16")
  File "C:\conda\envs\ldm\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 1105, in from_pretrained
    loaded_sub_model = load_sub_model(
  File "C:\conda\envs\ldm\lib\site-packages\diffusers\pipelines\pipeline_utils.py", line 472, in load_sub_model
    loaded_sub_model = load_method(os.path.join(cached_folder, name), **loading_kwargs)
  File "C:\conda\envs\ldm\lib\site-packages\transformers\modeling_utils.py", line 3019, in from_pretrained
    state_dict = load_state_dict(resolved_archive_file)
  File "C:\conda\envs\ldm\lib\site-packages\transformers\modeling_utils.py", line 463, in load_state_dict
    with safe_open(checkpoint_file, framework="pt") as f:
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

Process finished with exit code 1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

出现原因：

HeaderTooLarge 说明头解析错误

• LFS在git下载中被略过，文件是空指针
• 文档损坏
• 不是safetensor格式的权重
• 权重读取时被修改

解决

• 从github或者hugging face原链接手动下载，注意标有LFS文件
• 使用flag，强制下载LFS
• 重新下载权重
• 不使用软连接
• 查看文件大小、检查md5 code