apline无法向gitlab上传git lfs问题_fatal error:server error

1 背景

在k8s中基于alpine做底层系统的容器进行git lfs push操作时，发现报错无法上传成功

Fatal error: Server error: http://git.ops.xxxxx.com/xxxx/yyyy.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3                                                                
error: failed to push some refs to 'http://git.ops.xxxx.com/xxxx/yyyy.git'

而git clone、commit等操作均正常，并且git lfs push的操作在另一个机器A上能正常执行。

2 探索思路

2.1 从报错入手

先根据报错信息搜索，基本回答是此报错是git server(gitlab)侧发生错误，不过可以使用如下命令来打印trace跟踪信息(via: https://github.com/git-lfs/git-lfs/issues/3613#issuecomment-483852691)

GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push origin xxx/xxx

确实打印了更多信息，包括与Server的请求过程，如下：

07:18:13.442791 trace git-lfs: HTTP: POST http://xxxx:xxxx@git.ops.yunnex.com/suncard/packages.git/info/lfs/objects/batch
> POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1
> Host: git.ops.yunnex.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 158
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)
> 
{"operation":"upload","objects":[{"oid":"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e","size":3}],"ref":{"name":"refs/heads/test-branch"}}07:18:13.618655 trace git-lfs: HTTP: 200
 
 
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Cache-Control: max-age=0, private, must-revalidate
< Connection: keep-alive
< Content-Type: application/json; charset=utf-8
< Date: Thu, 24 Sep 2020 07:18:13 GMT
< Etag: W/"0fb39c6aeeda2850fdcf244083961713"
< Server: nginx
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Request-Id: 570154fd-b15f-4e8a-a735-b034e98dfbfd
< X-Runtime: 0.165423
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
< 
07:18:13.619071 trace git-lfs: HTTP: {"objects":[{"oid":"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e","size":3,"actions":{"upload":{"href":"http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3","header":{"Authorization":"Basic cHJvZHVjdGlvbjpVc2VfMTk5MA=="}}}}]}
{"objects":[{"oid":"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e","size":3,"actions":{"upload":{"href":"http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3","header":{"Authorization":"Basic cHJvZHVjdGlvbjpVc2VfMTk5MA=="}}}}]}07:18:13.619307 trace git-lfs: tq: starting transfer adapter "basic"
07:18:13.619345 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
07:18:13.619383 trace git-lfs: xfer: adapter "basic" started
07:18:13.619413 trace git-lfs: xfer: adapter "basic" worker 1 starting
07:18:13.619537 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
07:18:13.619549 trace git-lfs: xfer: adapter "basic" worker 5 starting
07:18:13.619585 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
07:18:13.619642 trace git-lfs: xfer: adapter "basic" worker 0 starting
07:18:13.619673 trace git-lfs: xfer: adapter "basic" worker 6 starting
07:18:13.619681 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e"
07:18:13.619702 trace git-lfs: xfer: adapter "basic" worker 7 starting
07:18:13.619722 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
07:18:13.619732 trace git-lfs: xfer: adapter "basic" worker 2 starting
07:18:13.619775 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
07:18:13.619723 trace git-lfs: xfer: adapter "basic" worker 4 starting
07:18:13.619775 trace git-lfs: xfer: adapter "basic" worker 3 starting
07:18:13.619805 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
07:18:13.619788 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
07:18:13.619697 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
07:18:13.621028 trace git-lfs: HTTP: PUT http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3
> PUT /suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3 HTTP/1.1
> Host: git.ops.yunnex.com
> Authorization: Basic * * * * *
> Content-Length: 3
> Content-Type: text/plain; charset=utf-8
> User-Agent: git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)
> 
07:18:13.621492 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
dd
07:18:13.621527 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
07:18:13.621536 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
07:18:13.621563 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
07:18:13.621571 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
07:18:13.621579 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
07:18:13.621522 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
07:18:13.786463 trace git-lfs: HTTP: 500
 
 
< HTTP/1.1 500 Internal Server Error
< Content-Length: 2911
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Connection: keep-alive
< Content-Type: text/html; charset=utf-8
< Date: Thu, 24 Sep 2020 07:18:13 GMT
< Expires: Fri, 01 Jan 1990 00:00:00 GMT
< Pragma: no-cache
< Server: nginx
< X-Request-Id: 0574e8c0-8ee8-4667-8141-1982d67c0bdc
< X-Runtime: 0.162882
< 
07:18:13.786829 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e"
07:18:13.786927 trace git-lfs: tq: retrying object b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e: Fatal error: Server error: http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3
07:18:13.787030 trace git-lfs: tq: enqueue retry #1 for "b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e" (size: 3)
07:18:13.787091 trace git-lfs: tq: sending batch of size 1
07:18:13.787212 trace git-lfs: api: batch 1 files
07:18:13.787330 trace git-lfs: HTTP: POST http://xxxxx:xxxxx@git.ops.yunnex.com/suncard/packages.git/info/lfs/objects/batch
...不断重试重复相同日志

其中关键请求如下

200 POST /suncard/packages.git/info/lfs/objects/batch
500 PUT /suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3

同时到gitlab server执行tail -f /var/log/gitlab//.log，日志如下

==> /var/log/gitlab/gitlab-rails/production.log <==
 
Started POST "/suncard/packages.git/info/lfs/objects/batch" for 10.10.51.195 at 2020-09-24 15:23:16 +0800
Processing by Projects::LfsApiController#batch as JSON
  Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e", "size"=>3}], "ref"=>{"name"=>"refs/heads/test-branch"}, "namespace_id"=>"suncard", "project_id"=>"packages.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e", "size"=>3}], "ref"=>{"name"=>"refs/heads/test-branch"}}}
Completed 200 OK in 138ms (Views: 0.3ms | ActiveRecord: 4.7ms)
 
Started PUT "/suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3" for 10.10.51.195 at 2020-09-24 15:23:17 +0800
Processing by Projects::LfsStorageController#upload_finalize as HTML
  Parameters: {"namespace_id"=>"suncard", "project_id"=>"packages.git", "oid"=>"b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e", "size"=>"3"}
Completed 500 Internal Server Error in 104ms (ActiveRecord: 2.6ms)
 
NoMethodError (undefined method `include?' for nil:NilClass):
  app/controllers/projects/lfs_storage_controller.rb:55:in `tmp_filename'
  app/controllers/projects/lfs_storage_controller.rb:23:in `upload_finalize'
  lib/gitlab/i18n.rb:45:in `with_locale'
  lib/gitlab/i18n.rb:51:in `with_user_locale'
  app/controllers/application_controller.rb:294:in `set_locale'
  lib/gitlab/performance_bar/peek_performance_bar_with_rack_body.rb:16:in `call'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/request_context.rb:18:in `call'
  lib/gitlab/metrics/connection_rack_middleware.rb:31:in `call'
...不断重试重复相同日志
==> /var/log/gitlab/nginx/gitlab_access.log <==
10.10.51.195 - - [24/Sep/2020:15:23:16 +0800] "GET /suncard/packages.git/info/refs?service=git-upload-pack HTTP/1.1" 200 701 "-" "git/2.20.2"
10.10.51.195 - production [24/Sep/2020:15:23:16 +0800] "POST /suncard/packages.git/info/lfs/locks/verify HTTP/1.1" 404 1489 "-" "git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)"
10.10.51.195 - production [24/Sep/2020:15:23:17 +0800] "POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1" 200 251 "-" "git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)"
10.10.51.195 - production [24/Sep/2020:15:23:17 +0800] "PUT /suncard/packages.git/gitlab-lfs/objects/b6f9dd313cde39ae1b87e63b9b457029bcea6e9520b5db5de20d3284e4c0259e/3 HTTP/1.1" 500 2911 "-" "git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)"
10.10.51.195 - production [24/Sep/2020:15:23:17 +0800] "POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1" 200 251 "-" "git-lfs/2.5.1 (GitHub; linux amd64; go 1.11.4; git 7c940069fa)"	

可见与client对应，500的报错信息如下，但是搜索后发现没有相关线索

NoMethodError (undefined method `include?' for nil:NilClass):
  app/controllers/projects/lfs_storage_controller.rb:55:in `tmp_filename'
  app/controllers/projects/lfs_storage_controller.rb:23:in `upload_finalize'
  lib/gitlab/i18n.rb:45:in `with_locale'

2.2 对比测试

线索中断，想到机器A可以正常push，所以进行对比观察，执行相同操作，client日志如下

23:15:15.348869 trace git-lfs: HTTP: POST http://production:Use_1990@git.ops.yunnex.com/suncard/packages.git/info/lfs/objects/batch
> POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1
> Host: git.ops.yunnex.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 158
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)
> 
{"operation":"upload","objects":[{"oid":"9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25","size":3}],"ref":{"name":"refs/heads/test-branch"}}23:15:15.525308 trace git-lfs: HTTP: 200
 
 
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Cache-Control: max-age=0, private, must-revalidate
< Connection: keep-alive
< Content-Type: application/json; charset=utf-8
< Date: Thu, 24 Sep 2020 15:15:15 GMT
< Etag: W/"486d174eca874b88d375fecca47236e2"
< Server: nginx
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Request-Id: b87656c2-5593-4e19-bdc0-ea522657b9fc
< X-Runtime: 0.172955
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
< 
23:15:15.525557 trace git-lfs: HTTP: {"objects":[{"oid":"9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25","size":3,"actions":{"upload":{"href":"http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3","header":{"Authorization":"Basic cHJvZHVjdGlvbjpVc2VfMTk5MA=="}}}}]}
{"objects":[{"oid":"9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25","size":3,"actions":{"upload":{"href":"http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3","header":{"Authorization":"Basic cHJvZHVjdGlvbjpVc2VfMTk5MA=="}}}}]}23:15:15.525647 trace git-lfs: tq: starting transfer adapter "basic"
23:15:15.525662 trace git-lfs: xfer: adapter "basic" Begin() with 8 workers
23:15:15.525669 trace git-lfs: xfer: adapter "basic" started
23:15:15.525700 trace git-lfs: xfer: adapter "basic" worker 0 starting
23:15:15.525705 trace git-lfs: xfer: adapter "basic" worker 0 processing job for "9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25"
23:15:15.525723 trace git-lfs: xfer: adapter "basic" worker 4 starting
23:15:15.525736 trace git-lfs: xfer: adapter "basic" worker 4 waiting for Auth
23:15:15.525759 trace git-lfs: xfer: adapter "basic" worker 6 starting
23:15:15.525770 trace git-lfs: xfer: adapter "basic" worker 6 waiting for Auth
23:15:15.525773 trace git-lfs: xfer: adapter "basic" worker 7 starting
23:15:15.525780 trace git-lfs: xfer: adapter "basic" worker 7 waiting for Auth
23:15:15.525777 trace git-lfs: xfer: adapter "basic" worker 5 starting
23:15:15.525789 trace git-lfs: xfer: adapter "basic" worker 5 waiting for Auth
23:15:15.525754 trace git-lfs: xfer: adapter "basic" worker 3 starting
23:15:15.525796 trace git-lfs: xfer: adapter "basic" worker 3 waiting for Auth
23:15:15.525731 trace git-lfs: xfer: adapter "basic" worker 2 starting
23:15:15.525803 trace git-lfs: xfer: adapter "basic" worker 2 waiting for Auth
23:15:15.525821 trace git-lfs: xfer: adapter "basic" worker 1 starting
23:15:15.525836 trace git-lfs: xfer: adapter "basic" worker 1 waiting for Auth
23:15:15.525855 trace git-lfs: HTTP: PUT http://git.ops.yunnex.com/suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3
> PUT /suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3 HTTP/1.1
> Host: git.ops.yunnex.com
> Authorization: Basic * * * * *
> Content-Length: 3
> Content-Type: application/octet-stream
> User-Agent: git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)
> 
23:15:15.525963 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
23:15:15.525968 trace git-lfs: xfer: adapter "basic" worker 6 auth signal received
23:15:15.525969 trace git-lfs: xfer: adapter "basic" worker 3 auth signal received
23:15:15.525970 trace git-lfs: xfer: adapter "basic" worker 2 auth signal received
23:15:15.525972 trace git-lfs: xfer: adapter "basic" worker 4 auth signal received
23:15:15.526006 trace git-lfs: xfer: adapter "basic" worker 5 auth signal received
23:15:15.526027 trace git-lfs: xfer: adapter "basic" worker 1 auth signal received
23:15:15.931456 trace git-lfs: HTTP: 200
 
 
< HTTP/1.1 200 OK
< Content-Length: 0
< Cache-Control: no-cache
< Connection: keep-alive
< Content-Type: text/html
< Date: Thu, 24 Sep 2020 15:15:15 GMT
< Server: nginx
< Strict-Transport-Security: max-age=31536000
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-Request-Id: e0ce367a-adce-44bd-b1ed-29a33f756bb5
< X-Runtime: 0.245919
< X-Ua-Compatible: IE=edge
< X-Xss-Protection: 1; mode=block
< 
23:15:15.931592 trace git-lfs: xfer: adapter "basic" worker 0 finished job for "9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25"
23:15:15.931664 trace git-lfs: xfer: adapter "basic" End()                                                                                                                                                                          
23:15:15.931671 trace git-lfs: xfer: adapter "basic" worker 7 stopping
23:15:15.931675 trace git-lfs: xfer: adapter "basic" worker 2 stopping
23:15:15.931684 trace git-lfs: xfer: adapter "basic" worker 6 stopping
23:15:15.931685 trace git-lfs: xfer: adapter "basic" worker 3 stopping
23:15:15.931689 trace git-lfs: xfer: adapter "basic" worker 5 stopping
23:15:15.931692 trace git-lfs: xfer: adapter "basic" worker 1 stopping
23:15:15.931688 trace git-lfs: xfer: adapter "basic" worker 0 stopping
23:15:15.931682 trace git-lfs: xfer: adapter "basic" worker 4 stopping
23:15:15.931726 trace git-lfs: xfer: adapter "basic" stopped
23:15:15.931787 trace git-lfs: filepathfilter: rewrite ".git" as "**/.git/**"                                                                                                                                                       
23:15:15.931801 trace git-lfs: filepathfilter: rewrite "**/.git" as "**/.git"
23:15:15.931819 trace git-lfs: filepathfilter: rejecting "tmp" via []
23:15:15.931824 trace git-lfs: filepathfilter: accepting "tmp"
Uploading LFS objects: 100% (1/1), 3 B | 0 B/s, done

其中关键请求如下，其中500请求已变成200

200 POST /suncard/packages.git/info/lfs/objects/batch
200 PUT /suncard/packages.git/gitlab-lfs/objects/9a92adbc0cee38ef658c71ce1b1bf8c65668f166bfb213644c895ccb1ad07a25/3

而gitalb server对应日志如下

==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/suncard/packages.git/info/lfs/objects/batch" for 192.168.1.25 at 2020-09-23 18:36:27 +0800
Processing by Projects::LfsApiController#batch as JSON
  Parameters: {"operation"=>"upload", "objects"=>[{"oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>4}], "ref"=>{"name"=>"refs/heads/test-branch"}, "namespace_id"=>"suncard", "project_id"=>"packages.git", "lfs_api"=>{"operation"=>"upload", "objects"=>[{"oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>4}], "ref"=>{"name"=>"refs/heads/test-branch"}}}
Completed 200 OK in 195ms (Views: 0.3ms | ActiveRecord: 6.1ms)
 
Started PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4/authorize" for 192.168.1.25 at 2020-09-23 18:36:28 +0800
Processing by Projects::LfsStorageController#upload_authorize as HTML
  Parameters: {"namespace_id"=>"suncard", "project_id"=>"packages.git", "oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>"4"}
Completed 200 OK in 114ms (Views: 0.2ms | ActiveRecord: 3.0ms)
 
Started PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4" for 192.168.1.25 at 2020-09-23 18:36:28 +0800
Processing by Projects::LfsStorageController#upload_finalize as HTML
  Parameters: {"namespace_id"=>"suncard", "project_id"=>"packages.git", "oid"=>"edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb", "size"=>"4"}
Completed 200 OK in 169ms (ActiveRecord: 51.1ms)
Updating statistics for project 32
 
==> /var/log/gitlab/nginx/gitlab_access.log <==
192.168.1.25 - production [23/Sep/2020:18:36:28 +0800] "POST /suncard/packages.git/info/lfs/objects/batch HTTP/1.1" 200 251 "-" "git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)"
192.168.1.25 - production [23/Sep/2020:18:36:28 +0800] "PUT /suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4 HTTP/1.1" 200 0 "-" "git-lfs/2.4.2 (GitHub; linux amd64; go 1.9.4)"

变化如下

# 多了1个200 请求
PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4/authorize"
# 下列请求从500变为200
200 PUT "/suncard/packages.git/gitlab-lfs/objects/edeaaff3f1774ad2888673770c6d64097e391bc362d7d6fb34982ddf0efd18cb/4"

gitlab server相同、操作的仓库分支相同的情况下，这样的差异推测是client客户端导致的了，先来看版本

git --version     1.8.3.1(正常)  2.20.4(异常)
git lfs version   2.4.2(正常)    2.5.1(异常)

其中正常的机器A是物理机centos用yum安装的git和git-lfs；异常的是跑在k8s集群中基于alpine的pod，是通过apk安装的，遂尝试apk降级git和git-lfs到机器A正常vb.net教程的版本。apk降级需要到其市场查找目标版本https://pkgs.alpinelinux.org/packages?name=git-lfs&branch=v3.9 。

再通过降通过降repository版本来安装低版本的软c#教程件包，原alpine是v3.9，现在尝试改为v3.8

apk add --no-cache --repository http://mirrors.aliyun.com/alpine/v3.8/main --repository http://mirrors.aliyun.com/alpine/v3.8/community 'git-lfs=2.4.2-r1'

再次执行原push等操作，终于正常了！以防万一，再到原本正常的机器A把git-lfs升级一下，看是否也异常。

安装过程：
mkdir /tmp/test
wget https://github.com/git-lfs/git-lfs/releases/download/v2.5.1/git-lfs-linux-amd64-v2.5.1.tar.gz
tar -xvf git-lfs-linux-amd64-v2.5.1.tar.gz 
# 这样就可以在不影响原git-lfs安装的情况下进行测试，后续用/tmp/test/git-lfs进行push等操作
/tmp/test/git-lfs version

果不其然，也重现了相同异常。

3 结论

首先异常与git版本没有关系，只和git-lfs版本有关。之后又尝试了更低的2.3.x、更高的2.6.x~2.11.x。确定是从2.5.x开始的版本有异常。另外还需python基础教程要关注gitlab的版本,目前为9.4.1

sudo gitlab-rake gitlab:env:info
 
System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	10.5.0
Redis Version:	3.2.5
Git Version:	2.13.0
Sidekiq Version:5.0.0
Go Version:	unknown
 
GitLab information
Version:	9.4.1
Revision:	6678b4e
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		http://git.ops.yunnex.com
HTTP Clone URL:	http://git.ops.yunnex.com/some-group/some-project.git
SSH Clone URL:	git@git.ops.yunnex.com:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	no
 
GitLab Shell
Version:	5.3.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks
Git:		/opt/gitlab/embedded/bin/git

目前只能推测该gitlab/lfs server版本与git lfs client push不兼容，所以采取的java基础教程方案是lfs降为2.4.x，alpine使用v3.8 repository重新sql教程安装git-lfs。