git clone git lfs 出现错误 fatal: model-00001-of-00002.safetensors: smudge filter lfs failed_error: external filter 'git-lfs filter-process' fa

git clone git lfs 出现错误 fatal: model-00001-of-00002.safetensors: smudge filter lfs failed
error: external filter ‘git-lfs filter-process’ failed
fatal: model-00001-of-00002.safetensors: smudge filter lfs failed。

---

Error downloading object: model-00001-of-00002.safetensors (66dec18): Smudge error: Error downloading model-00001-of-00002.safetensors (66dec18c9f1705b9387d62f8485f4e7d871ca388718786737ed3c72dbfaac9fb): batch response: Repository or object not found: https://huggingface.co/meta-llama/Llama-2-7b-chat-hf.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it

Errors logged to /root/meta-llama/Llama-2-7b-chat-hf/.git/lfs/logs/20240221T024656.481462682.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: model-00001-of-00002.safetensors: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'
1
2
3
4
5
6
7

详细错误如下：

git-lfs/2.11.0 (GitHub; linux amd64; go 1.13.4)
git version 2.27.0

$ git-lfs filter-process
Error downloading object: model-00001-of-00002.safetensors (66dec18): Smudge error: Error downloading model-00001-of-00002.safetensors (66dec18c9f1705b9387d62f8485f4e7d871ca388718786737ed3c72dbfaac9fb): batch response: Repository or object not found: https://huggingface.co/meta-llama/Llama-2-7b-chat-hf.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it

Repository or object not found: https://huggingface.co/meta-llama/Llama-2-7b-chat-hf.git/info/lfs/objects/batch
Check that it exists and that you have proper access to it
github.com/git-lfs/git-lfs/errors.Errorf
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/errors/errors.go:69
github.com/git-lfs/git-lfs/lfshttp.defaultError
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfshttp/errors.go:128
github.com/git-lfs/git-lfs/lfshttp.(*Client).handleResponse
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfshttp/errors.go:51
github.com/git-lfs/git-lfs/lfshttp.(*Client).DoWithRedirect
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfshttp/client.go:326
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithCreds
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfsapi/auth.go:103
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithAuth
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfsapi/auth.go:67
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoWithAuth
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfsapi/auth.go:25
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoAPIRequestWithAuth
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfsapi/auth.go:56
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/tq/api.go:74
github.com/git-lfs/git-lfs/tq.Batch
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/tq/api.go:40
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/tq/transfer_queue.go:559
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/tq/transfer_queue.go:453
runtime.goexit
/usr/lib/golang/src/runtime/asm_amd64.s:1357
batch response
github.com/git-lfs/git-lfs/errors.newWrappedError
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/errors/types.go:198
github.com/git-lfs/git-lfs/errors.Wrap
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/errors/errors.go:74
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/tq/api.go:77
github.com/git-lfs/git-lfs/tq.Batch
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/tq/api.go:40
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/tq/transfer_queue.go:559
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/tq/transfer_queue.go:453
runtime.goexit
/usr/lib/golang/src/runtime/asm_amd64.s:1357
Error downloading model-00001-of-00002.safetensors (66dec18c9f1705b9387d62f8485f4e7d871ca388718786737ed3c72dbfaac9fb)
github.com/git-lfs/git-lfs/errors.newWrappedError
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/errors/types.go:198
github.com/git-lfs/git-lfs/errors.Wrapf
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/errors/errors.go:85
github.com/git-lfs/git-lfs/lfs.(*GitFilter).downloadFile
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:115
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:76
github.com/git-lfs/git-lfs/commands.smudge
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.filterCommand
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:120
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:766
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).ExecuteC
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:852
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:800
github.com/git-lfs/git-lfs/commands.Run
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/commands/run.go:103
main.main
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/git-lfs.go:33
runtime.main
/usr/lib/golang/src/runtime/proc.go:203
runtime.goexit
/usr/lib/golang/src/runtime/asm_amd64.s:1357
Smudge error
github.com/git-lfs/git-lfs/errors.newWrappedError
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/errors/types.go:198
github.com/git-lfs/git-lfs/errors.NewSmudgeError
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/errors/types.go:284
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/lfs/gitfilter_smudge.go:85
github.com/git-lfs/git-lfs/commands.smudge
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.filterCommand
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/commands/command_filter_process.go:120
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).execute
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:766
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).ExecuteC
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:852
github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra.(*Command).Execute
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/vendor/github.com/spf13/cobra/command.go:800
github.com/git-lfs/git-lfs/commands.Run
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/commands/run.go:103
main.main
/builddir/build/BUILD/git-lfs-2.11.0/_build/src/github.com/git-lfs/git-lfs/git-lfs.go:33
runtime.main
/usr/lib/golang/src/runtime/proc.go:203
runtime.goexit
/usr/lib/golang/src/runtime/asm_amd64.s:1357

Current time in UTC:
2024-02-21 02:46:56

ENV:
LocalWorkingDir=/root/meta-llama/Llama-2-7b-chat-hf
LocalGitDir=/root/meta-llama/Llama-2-7b-chat-hf/.git
LocalGitStorageDir=/root/meta-llama/Llama-2-7b-chat-hf/.git
LocalMediaDir=/root/meta-llama/Llama-2-7b-chat-hf/.git/lfs/objects
LocalReferenceDirs=
TempDir=/root/meta-llama/Llama-2-7b-chat-hf/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/root/meta-llama/Llama-2-7b-chat-hf/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file
UploadTransfers=basic,lfs-standalone-file
GIT_DIR=/root/meta-llama/Llama-2-7b-chat-hf/.git
GIT_EXEC_PATH=/usr/libexec/git-core

Client IP addresses:
172.17.0.2

---

解决方法：

// Skip smudge - We'll download binary files later in a faster batch
git lfs install --skip-smudge
// Do git clone here
git clone ...
// Fetch all the binary files in the new clone
cd project
git lfs pull
// Reinstate smudge
git lfs install --force
1
2
3
4
5
6
7
8
9

如果是国外的大文件，比如 大模型，最好配置git的代理，如下：

git config --global http.sslVerify "false"
git config --global advice.detachedHead false

#建议socks5 和http 都配置一下代理
#配置socks5
#git config --global http.proxy socks5 127.0.0.1:7890
#git config --global https.proxy socks5 127.0.0.1:789
#配置http
#git config --global http.proxy 127.0.0.1:7890
#git config --global https.proxy 127.0.0.1:7890
#查看设置：
#git config --global --get http.proxy
#git config --global --get https.proxy
#取消办法：
#git config --global --unset http.proxy
#git config --global --unset https.proxy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

就可以正常下载了：

Downloading LFS objects:  20% (1/5), 298 MB | 3.1 MB/s
1