There are 7 missing blocks. The following files may be corrupted_there are 1007 missing blocks. the following files

作者：繁依Fanyi0 | 2024-05-27 16:10:57

踩

there are 1007 missing blocks. the following files may be corrupted:

断电、hdfs web提示blocks missing


[root@nufront-master-01 ~]# hdfs fsck /
Connecting to namenode via http://nufront-master-02:50070/fsck?ugi=root&path=%2F
FSCK started by root (auth:SIMPLE) from /172.16.34.121 for path / at Fri May 21 16:36:27 CST 2021
 
/tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309041
 
/tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz: CORRUPT 1 blocks of total size 8702 B.
/tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10010/d77543dd-435e-4ef7-b87f-5383c7959499/map.xml: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309042
 
/tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10010/d77543dd-435e-4ef7-b87f-5383c7959499/map.xml: CORRUPT 1 blocks of total size 9518 B.
/tmp/logs/root/logs/application_1621214977998_5719/nufront-worker-03_8041: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309087
 
/tmp/logs/root/logs/application_1621214977998_5719/nufront-worker-03_8041: CORRUPT 1 blocks of total size 155671 B.
/user/history/done/2021/05/21/000005/job_1621214977998_5719-1621570859432-root-insert+into+table+village...t1.area_id%3Dt2.id+%28-1621570872686-1-0-SUCCEEDED-root.users.root-1621570865414.jhist: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309084
 
/user/history/done/2021/05/21/000005/job_1621214977998_5719-1621570859432-root-insert+into+table+village...t1.area_id%3Dt2.id+%28-1621570872686-1-0-SUCCEEDED-root.users.root-1621570865414.jhist: CORRUPT 1 blocks of total size 17998 B.
/user/history/done/2021/05/21/000005/job_1621214977998_5719_conf.xml: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309085
 
/user/history/done/2021/05/21/000005/job_1621214977998_5719_conf.xml: CORRUPT 1 blocks of total size 374485 B.
/user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/-ext-10001/tmpstats-0: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309081
 
/user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/-ext-10001/tmpstats-0: CORRUPT 1 blocks of total size 111 B.
/user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/_tmp.-ext-10000/000000_0: CORRUPT blockpool BP-1719687234-172.16.34.123-1607417729793 block blk_1076309080
 
/user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/_tmp.-ext-10000/000000_0: CORRUPT 1 blocks of total size 3837 B.
Status: CORRUPT
 Number of data-nodes:  3
 Number of racks:               1
 Total dirs:                    20757
 Total symlinks:                0
 
Replicated Blocks:
 Total size:    119664216828 B
 Total files:   45863 (Files currently being written: 6)
 Total blocks (validated):      46465 (avg. block size 2575362 B) (Total open file blocks (not validated): 4)
  ********************************
  UNDER MIN REPL'D BLOCKS:      7 (0.015065103 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:        7
  CORRUPT BLOCKS:       7
  CORRUPT SIZE:         570322 B
  ********************************
 Minimally replicated blocks:   46458 (99.98493 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.999505
 Missing blocks:                0
 Corrupt blocks:                7
 Missing replicas:              0 (0.0 %)
 Blocks queued for replication: 0
Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Fri May 21 16:36:28 CST 2021 in 837 milliseconds
The filesystem under path '/' is CORRUPT
[root@nufront-master-01 ~]# hdfs fsck -list-corruptfileblocks
Connecting to namenode via http://nufront-master-02:50070/fsck?ugi=root&listcorruptfileblocks=1&path=%2F
The list of corrupt files under path '/' are:
blk_1076309042  /tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10010/d77543dd-435e-4ef7-b87f-5383c7959499/map.xml
blk_1076309080  /user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/_tmp.-ext-10000/000000_0
blk_1076309081  /user/hive/warehouse/villagemonitor.db/dws_nanxiong_routerflow/partday=20200929/.hive-staging_hive_2021-05-21_12-06-58_867_4007960590733956456-1/-ext-10001/tmpstats-0
blk_1076309084  /user/history/done/2021/05/21/000005/job_1621214977998_5719-1621570859432-root-insert+into+table+village...t1.area_id%3Dt2.id+%28-1621570872686-1-0-SUCCEEDED-root.users.root-1621570865414.jhist
blk_1076309085  /user/history/done/2021/05/21/000005/job_1621214977998_5719_conf.xml
blk_1076309087  /tmp/logs/root/logs/application_1621214977998_5719/nufront-worker-03_8041
blk_1076309041  /tmp/hive/root/62665454-411d-4de2-b177-d5daca2cba98/hive_2021-05-21_12-06-58_867_4007960590733956456-1/-mr-10004/HashTable-Stage-5/Stage-5.tar.gz
The filesystem under path '/' has 7 CORRUPT files
[root@nufront-master-01 ~]# hdfs fsck -delete /xxx

解决方法：

如果文件不重要，可以直接删除此文件(hdfs fsck -delete)；或删除后重新复制一份到集群中
如果不能删除，需要从上面命令中找到发生在哪台机器上，然后到此机器上查看日志。

常用命令介绍：

1、查看节点、hdfs、丢失的数据块命令：hadoop dfsadmin -report

2、查看文件系统的健康状况：hdfs fsck --<path> [-options]


[root@nufront-master-01 ~]# hdfs fsck -help
Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks | -replicaDetails | -upgradedomains]]]] [-includeSnapshots] [-showprogress] [-storagepolicies] [-maintenance] [-blockId <blk_Id>] [-replicate]
        <path>  start checking from this path
        -move   move corrupted files to /lost+found
        -delete delete corrupted files
        -files  print out files being checked
        -openforwrite   print out files opened for write
        -includeSnapshots       include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it
        -list-corruptfileblocks print out list of missing blocks and files they belong to
        -files -blocks  print out block report
        -files -blocks -locations       print out locations for every block
        -files -blocks -racks   print out network topology for data-node locations
        -files -blocks -replicaDetails  print out each replica details
        -files -blocks -upgradedomains  print out upgrade domains for every block
        -storagepolicies        print out storage policy summary for the blocks
        -maintenance    print out maintenance state node details
        -showprogress   show progress in output. Default is OFF (no progress)
        -blockId        print out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc)
        -replicate initiate replication work to make mis-replicated
 blocks satisfy block placement policy

hadoop fsck / 用这个命令可以检查整个文件系统的健康状况，但是要注意它不会主动恢复备份缺失的block，这个是由NameNode单独的线程异步处理的。

3、手动修复 hdfs debug


[root@nufront-master-01 ~]# hdfs debug --help
Usage: hdfs debug <command> [arguments]
 
These commands are for advanced users only.
 
Incorrect usages may result in data loss. Use at your own risk.
 
verifyMeta -meta <metadata-file> [-block <block-file>]
computeMeta -block <block-file> -out <output-metadata-file>
recoverLease -path <path> [-retries <num-retries>]
[root@nufront-master-01 ~]# hdfs debug recoverLease -path /xxx -retries 10

4、如果hadoop不能自动恢复，则只能删除 corrupted blocks;

hdfs fsck -delete

参考：https://blog.csdn.net/lingbo229/article/details/81128316

https://blog.51cto.com/xiaolanlan/2071642

https://blog.csdn.net/qq_40337206/article/details/99900310

https://www.cnblogs.com/tesla-turing/p/11487899.html

https://hadoop.apache.org/docs/r3.2.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/繁依Fanyi0/article/detail/632818