【From】https://www.cnblogs.com/quchunhui/p/9583746.html
==版本信息==
HBase：2.7.1
Storm：1.0.1
RocketMQ：3.4.6（阿里版）
==问题描述==
2018年9月3号晚上23点左右，例行巡检系统运行状况时，
发现Storm消费RocketMQ入HBase的过程中，RocketMQ的TPS极低，甚至为0了。
（Storm从RocketMQ消费消息自后，读写HBase）
==问题定位==
查看了Storm的日志，发现有如下异常信息：
2018-09-03 20:29:43.696 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] java.net.SocketTimeoutException: callTimeout=1200000, callDuration=1536845: row '97390002332830' on table 'DBN_ANE' at region=DBN_ANE,97,1507795766296.69724c5bd4e82bae4a3ec8a95c8f9c65., hostname=dscn33,16020,1529215379398, seqNum=17296637
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:862)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HTable.get(HTable.java:828)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at com.dscn.topology.BasicBolt.execute(BasicBolt.java:125)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.topology.BasicBoltExecutor.execute(BasicBoltExecutor.java:50)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$fn__5030$tuple_action_fn__5032.invoke(executor.clj:729)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$mk_task_receiver$fn__4951.invoke(executor.clj:461)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.disruptor$clojure_handler$reify__4465.onEvent(disruptor.clj:40)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:482)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:460)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.daemon.executor$fn__5030$fn__5043$fn__5096.invoke(executor.clj:848)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.storm.util$async_loop$fn__557.invoke(util.clj:484)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at clojure.lang.AFn.run(AFn.java:22)
2018-09-03 20:29:43.697 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at java.lang.Thread.run(Thread.java:745)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] Caused by: java.io.IOException: HRegionInfo was null in DBN_ANE, row=keyvalues={DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:seqnumDuringOpen/1535968873207/Put/vlen=8/seqid=0, DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:server/1535968873207/Put/vlen=12/seqid=0, DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf./info:serverstartcode/1535968873207/Put/vlen=8/seqid=0}
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1289)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1181)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1152)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1136)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:957)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:83)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
2018-09-03 20:29:43.698 STDIO Thread-8-BasicBolt-executor[6 6] [ERROR] ... 14 more
然后查看HBase集群日志，里面也出现错误，日志如下：
2018-09-03 17:58:57,603 WARN [MemStoreFlusher.0] regionserver.HStore: Failed flushing store file, retrying num=0
java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
2018-09-03 17:58:57,606 INFO [regionserver/dscn33/10.11.2.31:16020.logRoller] wal.FSHLog: Slow sync cost: 16261 ms, current pipeline: [10.11.2.31:50010]
2018-09-03 17:58:57,606 ERROR [regionserver/dscn33/10.11.2.31:16020.logRoller] wal.FSHLog: Failed close of WAL writer hdfs://ns/hbase/WALs/dscn33,16020,1529215379398/dscn33%2C16020%2C1529215379398.default.1535968646203, unflushedEntries=30
org.apache.hadoop.hbase.regionserver.wal.FailedSyncBeforeLogCloseException: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WAL
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$SafePointZigZagLatch.waitSafePoint(FSHLog.java:1615)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.replaceWriter(FSHLog.java:833)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:699)
at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:148)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WAL
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Caused by: java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
2018-09-03 17:58:41,358 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Cache flush failed for region DBN_JBD,30270769371,1515408107652.fca3d8ee774898302a377535e3f0ccac.
org.apache.hadoop.hbase.regionserver.wal.DamagedWALException: Append sequenceId=41639867, requesting roll of WAL
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.append(FSHLog.java:1898)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1750)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1672)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: All datanodes 10.11.2.31:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
检查HBase集群状态（hbase hbck），发现异常，定位为HBase集群问题：
[root@dscn12 ~]# hbase hbck
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadmin/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBaseFsck command line options:
2018-09-04 00:01:54,217 INFO [main] util.HBaseFsck: Launching hbck
2018-09-04 00:01:54,456 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x57576994 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:01:54,471 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2018-09-04 00:01:54,471 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=dscn12
2018-09-04 00:01:54,471 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_73
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_73/jre
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/home/hadmin/hbase/conf:/usr/java/default/lib/tools.jar:/home/hadmin/hbase:/home/hadmin/hbase/lib/activation-1.1.jar:/home/hadmin/hbase/lib/antisamy-1.4.3.jar:/home/hadmin/hbase/lib/aopalliance-1.0.jar:/home/hadmin/hbase/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadmin/hbase/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadmin/hbase/lib/api-asn1-api-1.0.0-M20.jar:/home/hadmin/hbase/lib/api-util-1.0.0-M20.jar:/home/hadmin/hbase/lib/asm-3.1.jar:/home/hadmin/hbase/lib/avro-1.7.4.jar:/home/hadmin/hbase/lib/batik-css-1.7.jar:/home/hadmin/hbase/lib/batik-ext-1.7.jar:/home/hadmin/hbase/lib/batik-util-1.7.jar:/home/hadmin/hbase/lib/bsh-core-2.0b4.jar:/home/hadmin/hbase/lib/commons-beanutils-1.7.0.jar:/home/hadmin/hbase/lib/commons-beanutils-core-1.7.0.jar:/home/hadmin/hbase/lib/commons-cli-1.2.jar:/home/hadmin/hbase/lib/commons-codec-1.9.jar:/home/hadmin/hbase/lib/commons-collections-3.2.2.jar:/home/hadmin/hbase/lib/commons-compress-1.4.1.jar:/home/hadmin/hbase/lib/commons-configuration-1.6.jar:/home/hadmin/hbase/lib/commons-daemon-1.0.13.jar:/home/hadmin/hbase/lib/commons-digester-1.8.jar:/home/hadmin/hbase/lib/commons-el-1.0.jar:/home/hadmin/hbase/lib/commons-fileupload-1.2.jar:/home/hadmin/hbase/lib/commons-httpclient-3.1.jar:/home/hadmin/hbase/lib/commons-io-2.4.jar:/home/hadmin/hbase/lib/commons-lang-2.6.jar:/home/hadmin/hbase/lib/commons-logging-1.2.jar:/home/hadmin/hbase/lib/commons-math-2.2.jar:/home/hadmin/hbase/lib/commons-math3-3.1.1.jar:/home/hadmin/hbase/lib/commons-net-3.1.jar:/home/hadmin/hbase/lib/disruptor-3.3.0.jar:/home/hadmin/hbase/lib/esapi-2.1.0.jar:/home/hadmin/hbase/lib/findbugs-annotations-1.3.9-1.jar:/home/hadmin/hbase/lib/guava-12.0.1.jar:/home/hadmin/hbase/lib/guice-3.0.jar:/home/hadmin/hbase/lib/guice-servlet-3.0.jar:/home/hadmin/hbase/lib/hadoop-annotations-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-auth-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-client-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-hdfs-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-app-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-core-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-api-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-client-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-common-2.5.1.jar:/home/hadmin/hbase/lib/hadoop-yarn-server-common-2.5.1.jar:/home/hadmin/hbase/lib/hbase-annotations-1.2.1.jar:/home/hadmin/hbase/lib/hbase-annotations-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-client-1.2.1.jar:/home/hadmin/hbase/lib/hbase-common-1.2.1.jar:/home/hadmin/hbase/lib/hbase-common-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-examples-1.2.1.jar:/home/hadmin/hbase/lib/hbase-external-blockcache-1.2.1.jar:/home/hadmin/hbase/lib/hbase-hadoop2-compat-1.2.1.jar:/home/hadmin/hbase/lib/hbase-hadoop-compat-1.2.1.jar:/home/hadmin/hbase/lib/hbase-it-1.2.1.jar:/home/hadmin/hbase/lib/hbase-it-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-prefix-tree-1.2.1.jar:/home/hadmin/hbase/lib/hbase-procedure-1.2.1.jar:/home/hadmin/hbase/lib/hbase-protocol-1.2.1.jar:/home/hadmin/hbase/lib/hbase-resource-bundle-1.2.1.jar:/home/hadmin/hbase/lib/hbase-rest-1.2.1.jar:/home/hadmin/hbase/lib/hbase-server-1.2.1.jar:/home/hadmin/hbase/lib/hbase-server-1.2.1-tests.jar:/home/hadmin/hbase/lib/hbase-shell-1.2.1.jar:/home/hadmin/hbase/lib/hbase-thrift-1.2.1.jar:/home/hadmin/hbase/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hbase/lib/httpclient-4.2.5.jar:/home/hadmin/hbase/lib/httpcore-4.4.1.jar:/home/hadmin/hbase/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hbase/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hbase/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hbase/lib/jackson-xc-1.9.13.jar:/home/hadmin/hbase/lib/jamon-runtime-2.4.1.jar:/home/hadmin/hbase/lib/jasper-compiler-5.5.23.jar:/home/hadmin/hbase/lib/jasper-runtime-5.5.23.jar:/home/hadmin/hbase/lib/javax.inject-1.jar:/home/hadmin/hbase/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hbase/lib/jaxb-api-2.2.2.jar:/home/hadmin/hbase/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hbase/lib/jcodings-1.0.8.jar:/home/hadmin/hbase/lib/jersey-client-1.9.jar:/home/hadmin/hbase/lib/jersey-core-1.9.jar:/home/hadmin/hbase/lib/jersey-guice-1.9.jar:/home/hadmin/hbase/lib/jersey-json-1.9.jar:/home/hadmin/hbase/lib/jersey-server-1.9.jar:/home/hadmin/hbase/lib/jets3t-0.9.0.jar:/home/hadmin/hbase/lib/jettison-1.3.3.jar:/home/hadmin/hbase/lib/jetty-6.1.26.jar:/home/hadmin/hbase/lib/jetty-sslengine-6.1.26.jar:/home/hadmin/hbase/lib/jetty-util-6.1.26.jar:/home/hadmin/hbase/lib/joni-2.1.2.jar:/home/hadmin/hbase/lib/jruby-complete-1.6.8.jar:/home/hadmin/hbase/lib/jsch-0.1.42.jar:/home/hadmin/hbase/lib/jsp-2.1-6.1.14.jar:/home/hadmin/hbase/lib/jsp-api-2.1-6.1.14.jar:/home/hadmin/hbase/lib/jsr305-1.3.9.jar:/home/hadmin/hbase/lib/junit-4.12.jar:/home/hadmin/hbase/lib/leveldbjni-all-1.8.jar:/home/hadmin/hbase/lib/libthrift-0.9.3.jar:/home/hadmin/hbase/lib/log4j-1.2.17.jar:/home/hadmin/hbase/lib/metrics-core-2.2.0.jar:/home/hadmin/hbase/lib/nekohtml-1.9.12.jar:/home/hadmin/hbase/lib/netty-all-4.0.23.Final.jar:/home/hadmin/hbase/lib/paranamer-2.3.jar:/home/hadmin/hbase/lib/protobuf-java-2.5.0.jar:/home/hadmin/hbase/lib/servlet-api-2.5-6.1.14.jar:/home/hadmin/hbase/lib/servlet-api-2.5.jar:/home/hadmin/hbase/lib/slf4j-api-1.7.7.jar:/home/hadmin/hbase/lib/slf4j-log4j12-1.7.5.jar:/home/hadmin/hbase/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hbase/lib/spymemcached-2.11.6.jar:/home/hadmin/hbase/lib/xalan-2.7.0.jar:/home/hadmin/hbase/lib/xml-apis-1.3.03.jar:/home/hadmin/hbase/lib/xml-apis-ext-1.3.04.jar:/home/hadmin/hbase/lib/xmlenc-0.52.jar:/home/hadmin/hbase/lib/xom-1.2.5.jar:/home/hadmin/hbase/lib/xz-1.0.jar:/home/hadmin/hbase/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/etc/hadoop:/home/hadmin/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-auth-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hadoop-annotations-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/hadmin/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/hadmin/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/hadmin/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/hadmin/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/hadmin/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-common-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/common/hadoop-nfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/hadmin/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.1-tests.jar:/home/hadmin/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.1.jar:.::/home/hadmin/hadoop/contrib/capacity-scheduler/*.jar:/home/hadmin/hadoop/bin
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadmin/hadoop/lib/native
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2018-09-04 00:01:54,472 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-229.el7.x86_64
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2018-09-04 00:01:54,473 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/root
2018-09-04 00:01:54,474 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x575769940x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:01:54,499 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:01:54,506 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:01:54,564 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9a, negotiated timeout = 40000
Version: 1.2.1
Number of live region servers: 25
Number of dead region servers: 0
Master: dscn11,16000,1529215352324
Number of backup masters: 1
Average load: 305.88
Number of requests: 2507
Number of regions: 7647
Number of regions in transition: 0
2018-09-04 00:01:56,473 INFO [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 2
2018-09-04 00:01:57,276 INFO [main] util.HBaseFsck: getHTableDescriptors == tableNames => [DBN_BSHTIndex, DBN_TTKDIndex, SJYB_UPS, SJYB_ANE, DBN_SUR, DBN_CAEIndex, SJYB_BSHT, DBN_STOIndex, DBN_YTO, DBN_TTKD, DBN_FEDEX, SJYB_ZJS, DBN_ANEIndex, DBN_TNTIndex, SJYB_TNT, SJYB_DHL, DBN_ZJS, DBN_KJIndex, BOX, DBN_DEPPONIndex, DBN_ZGYZIndex, DBN_YUNDA, hbase:namespace, DBN_TNT, DBN_FEDEXIndex, DBN_YTOIndex, DBN_PJIndex, SJYB_EMS, SJYB_KJ, SJYB_ZGYZ, DBN_JBD, DBN_DEPPON, SJYB_KYE, SJYB_SUR, SJYB_PJ, SJYB_UC, DBN_SURIndex, DBN_EMS, DBN_STO, SJYB_TTKD, SJYB_SF, SJYB_JBD, SJYB_APEX, DBN_ZTOIndex, DBN_APEX, DBN_PJ, DBN_UPS, DBN_ZJSIndex, DBN_UPSIndex, DBN_UC, DBN_GTO, DBN_BSHT, DBN_EMSIndex, DBN_APEXIndex, SJYB_CAE, DBN_ZTO, DBN_CAE, DBN_SF, DBN_UCIndex, DBN_SFIndex, SJYB_YUNDA, SJYB_FEDEX, SJYB_STO, DBN_JBDIndex, DBN_ANE, DBN_DHLIndex, DBN_KYE, DBN_KJ, SJYB_GTO, DBN_ZGYZ, SJYB_YTO, SJYB_DEPPON, DBN_YUNDAIndex, SJYB_ZTO, DBN_DHL]
2018-09-04 00:01:57,280 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x51e8e6e6 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:01:57,280 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x51e8e6e60x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:01:57,281 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:01:57,281 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:01:57,283 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bd9c, negotiated timeout = 40000
2018-09-04 00:01:57,296 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2018-09-04 00:01:57,296 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9c
2018-09-04 00:01:57,297 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9c closed
2018-09-04 00:01:57,298 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
Number of Tables: 75
2018-09-04 00:01:57,308 INFO [main] util.HBaseFsck: Loading region directories from HDFS
..............................................................................................................................................................................................................................................................................................................
2018-09-04 00:01:59,904 INFO [main] util.HBaseFsck: Loading region information from HDFS
............................................................................
2018-09-04 00:02:02,282 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7792d851 connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:02,282 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hconnection-0x7792d8510x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:02,286 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:02,287 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:02:02,289 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda1, negotiated timeout = 40000
2018-09-04 00:02:02,303 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:02,303 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:02,304 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:02,305 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:02:02,306 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda2, negotiated timeout = 40000
2018-09-04 00:02:02,352 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda2 closed
2018-09-04 00:02:02,352 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-09-04 00:02:02,352 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bda1
2018-09-04 00:02:02,354 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda1 closed
2018-09-04 00:02:02,354 INFO [main] util.HBaseFsck: Checking and fixing region consistency
2018-09-04 00:02:02,354 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/95a27c1ae2df5db14fa66d13966a8daf, deployed => dscn33,16020,1535968743109;DBN_ANE,97290002238561,1535968864242.95a27c1ae2df5db14fa66d13966a8daf., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109
ERROR: Region { meta => null, hdfs => hdfs://ns/hbase/data/default/DBN_ANE/e051ccd37678ff95fdd7652106b1fd5b, deployed => dscn33,16020,1535968743109;DBN_ANE,97,1535968864242.e051ccd37678ff95fdd7652106b1fd5b., replicaId => 0 } not in META, but deployed on dscn33,16020,1535968743109
2018-09-04 00:02:02,733 INFO [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: There is a hole in the region chain between 97 and 97581111303371. You need to create a new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table DBN_ANE
2018-09-04 00:02:02,876 INFO [main] util.HBaseFsck: Computing mapping of all store files
...................................................................................................................................................................................................................................................................................................................................................................................
2018-09-04 00:02:10,477 INFO [main] util.HBaseFsck: Validating mapping using HDFS state
2018-09-04 00:02:10,662 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:10,662 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:10,664 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn11/10.11.2.4:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:10,665 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Socket connection established to dscn11/10.11.2.4:2181, initiating session
2018-09-04 00:02:10,666 INFO [main-SendThread(dscn11:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn11/10.11.2.4:2181, sessionid = 0x656ba3c1b1c04f, negotiated timeout = 40000
2018-09-04 00:02:10,714 INFO [main] zookeeper.ZooKeeper: Session: 0x656ba3c1b1c04f closed
2018-09-04 00:02:10,714 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2018-09-04 00:02:10,714 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=dscn11:2181,dscn12:2181,dscn13:2181
2018-09-04 00:02:10,715 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=dscn11:2181,dscn12:2181,dscn13:2181 sessionTimeout=90000 watcher=hbase Fsck0x0, quorum=dscn11:2181,dscn12:2181,dscn13:2181, baseZNode=/hbase
2018-09-04 00:02:10,717 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Opening socket connection to server dscn13/10.11.2.6:2181. Will not attempt to authenticate using SASL (unknown error)
2018-09-04 00:02:10,717 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Socket connection established to dscn13/10.11.2.6:2181, initiating session
2018-09-04 00:02:10,719 INFO [main-SendThread(dscn13:2181)] zookeeper.ClientCnxn: Session establishment complete on server dscn13/10.11.2.6:2181, sessionid = 0x2656ba391b6bda7, negotiated timeout = 40000
2018-09-04 00:02:10,752 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bda7 closed
Summary:
Table DBN_ANE is okay.
Number of regions: 147
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_CAE is okay.
Number of regions: 100
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_DHL is okay.
Number of regions: 100
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496
Table DBN_EMS is okay.
Number of regions: 226
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,15342212024962018-09-04 00:02:10,754 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
Table DBN_GTO is okay.
Number of regions: 51
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn35,16020,1534221202496
Table DBN_JBD is okay.
Number of regions: 238
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_KYE is okay.
Number of regions: 51
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_STO is okay.
Number of regions: 440
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_SUR is okay.
Number of regions: 100
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_TNT is okay.
Number of regions: 100
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_UPS is okay.
Number of regions: 100
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_YTO is okay.
Number of regions: 638
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_ZJS is okay.
Number of regions: 102
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_ZTO is okay.
Number of regions: 867
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ANE is okay.
Number of regions: 43
Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_CAE is okay.
Number of regions: 21
Deployed on: dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_DHL is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_EMS is okay.
Number of regions: 90
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_GTO is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table SJYB_JBD is okay.
Number of regions: 46
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn27,16020,1535964784402 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn35,16020,1534221202496
Table SJYB_KYE is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table SJYB_STO is okay.
Number of regions: 111
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_SUR is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn35,16020,1534221202496
Table SJYB_TNT is okay.
Number of regions: 21
Deployed on: dscn11,16020,1535969281236 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn26,16020,1535018986330 dscn31,16020,1529215376767 dscn34,16020,1535675408409
Table SJYB_UPS is okay.
Number of regions: 21
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn29,16020,1535540080040 dscn31,16020,1529215376767
Table SJYB_YTO is okay.
Number of regions: 111
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ZJS is okay.
Number of regions: 21
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ZTO is okay.
Number of regions: 137
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_TNTIndex is okay.
Number of regions: 1
Deployed on: dscn34,16020,1535675408409
Table SJYB_KJ is okay.
Number of regions: 21
Deployed on: dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn34,16020,1535675408409
Table SJYB_PJ is okay.
Number of regions: 33
Deployed on: dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table SJYB_SF is okay.
Number of regions: 92
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_UC is okay.
Number of regions: 36
Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn33,16020,1535968743109
Table DBN_DEPPONIndex is okay.
Number of regions: 1
Deployed on: dscn20,16020,1529215363444
Table DBN_SURIndex is okay.
Number of regions: 1
Deployed on: dscn14,16020,1535088098764
Table DBN_BSHTIndex is okay.
Number of regions: 2
Deployed on: dscn31,16020,1529215376767
Table DBN_APEX is okay.
Number of regions: 200
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_BSHT is okay.
Number of regions: 439
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_TTKD is okay.
Number of regions: 267
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_ZGYZ is okay.
Number of regions: 302
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table hbase:meta is okay.
Number of regions: 1
Deployed on: dscn35,16020,1534221202496
Table DBN_ANEIndex is okay.
Number of regions: 1
Deployed on: dscn13,16020,1529215824290
Table DBN_DEPPON is okay.
Number of regions: 145
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_DHLIndex is okay.
Number of regions: 1
Deployed on: dscn26,16020,1535018986330
Table DBN_SFIndex is okay.
Number of regions: 2
Deployed on: dscn17,16020,1534932216753
Table DBN_YTOIndex is okay.
Number of regions: 2
Deployed on: dscn30,16020,1533181074642
Table DBN_FEDEX is okay.
Number of regions: 100
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_YUNDA is okay.
Number of regions: 499
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_APEX is okay.
Number of regions: 21
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn32,16020,1535798049804
Table SJYB_BSHT is okay.
Number of regions: 118
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_TTKD is okay.
Number of regions: 86
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table SJYB_ZGYZ is okay.
Number of regions: 38
Deployed on: dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn28,16020,1535128210905 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409
Table DBN_CAEIndex is okay.
Number of regions: 1
Deployed on: dscn20,16020,1529215363444
Table BOX is okay.
Number of regions: 141
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_JBDIndex is okay.
Number of regions: 2
Deployed on: dscn14,16020,1535088098764
Table DBN_TTKDIndex is okay.
Number of regions: 1
Deployed on: dscn20,16020,1529215363444
Table hbase:namespace is okay.
Number of regions: 1
Deployed on: dscn12,16020,1533367433799
Table DBN_KJIndex is okay.
Number of regions: 1
Deployed on: dscn21,16020,1529215364453
Table DBN_ZJSIndex is okay.
Number of regions: 1
Deployed on: dscn18,16020,1535022020180
Table DBN_UCIndex is okay.
Number of regions: 1
Deployed on: dscn14,16020,1535088098764
Table DBN_PJIndex is okay.
Number of regions: 1
Deployed on: dscn18,16020,1535022020180
Table DBN_FEDEXIndex is okay.
Number of regions: 2
Deployed on: dscn26,16020,1535018986330
Table DBN_ZGYZIndex is okay.
Number of regions: 1
Deployed on: dscn32,16020,1535798049804
Table DBN_ZTOIndex is okay.
Number of regions: 2
Deployed on: dscn23,16020,1529215366710 dscn33,16020,1535968743109
Table SJYB_DEPPON is okay.
Number of regions: 33
Deployed on: dscn12,16020,1533367433799 dscn14,16020,1535088098764 dscn16,16020,1534933465007 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn35,16020,1534221202496
Table DBN_APEXIndex is okay.
Number of regions: 1
Deployed on: dscn16,16020,1534933465007
Table DBN_EMSIndex is okay.
Number of regions: 2
Deployed on: dscn15,16020,1529371666615
Table DBN_YUNDAIndex is okay.
Number of regions: 2
Deployed on: dscn13,16020,1529215824290
Table SJYB_FEDEX is okay.
Number of regions: 21
Deployed on: dscn11,16020,1535969281236 dscn13,16020,1529215824290 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn34,16020,1535675408409
Table DBN_STOIndex is okay.
Number of regions: 2
Deployed on: dscn24,16020,1535104303466
Table SJYB_YUNDA is okay.
Number of regions: 102
Deployed on: dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_KJ is okay.
Number of regions: 119
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_PJ is okay.
Number of regions: 164
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_SF is okay.
Number of regions: 457
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn33,16020,1535968743109 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_UC is okay.
Number of regions: 210
Deployed on: dscn11,16020,1535969281236 dscn12,16020,1533367433799 dscn13,16020,1529215824290 dscn14,16020,1535088098764 dscn15,16020,1529371666615 dscn16,16020,1534933465007 dscn17,16020,1534932216753 dscn18,16020,1535022020180 dscn19,16020,1535882140163 dscn20,16020,1529215363444 dscn21,16020,1529215364453 dscn22,16020,1534130493880 dscn23,16020,1529215366710 dscn24,16020,1535104303466 dscn25,16020,1529215369016 dscn26,16020,1535018986330 dscn27,16020,1535964784402 dscn28,16020,1535128210905 dscn29,16020,1535540080040 dscn30,16020,1533181074642 dscn31,16020,1529215376767 dscn32,16020,1535798049804 dscn34,16020,1535675408409 dscn35,16020,1534221202496
Table DBN_UPSIndex is okay.
Number of regions: 2
Deployed on: dscn15,16020,1529371666615
5 inconsistencies detected.
Status: INCONSISTENT
2018-09-04 00:02:10,770 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2018-09-04 00:02:10,771 INFO [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x2656ba391b6bd9a
2018-09-04 00:02:10,772 INFO [main] zookeeper.ZooKeeper: Session: 0x2656ba391b6bd9a closed
2018-09-04 00:02:10,773 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
[root@dscn12 ~]#
==原因分析==
从hbase hbck检查结果中可以看到以下关键信息
ERROR: Empty REGIONINFO_QUALIFIER found in hbase:meta
ERROR: There is a hole in the region chain between 97 and 97581111303371. You need to create a new .regioninfo and region dir in hdfs to plug the hole.
错误提示了hbase meta数据存在问题，缺失的rowkey范围边界，也就是hole（Region之间有空隙，正常的Region之间是无交叉、无缝隙的）。
导致HBase整个集群状态出现问题，影响Storm的读写处理。
==解决过程==
以前有位大神给了在线hbck修复的运维命令
前提：HDFS fsck确保hbase根目录下文件没有损坏丢失，如果有，则先进行corrupt block移除。 切记：一定要在所有Region都上线之后再修复，否则修复之后可能出现重复Region。 步骤1. hbase hbck 检查输出所以ERROR信息，每个ERROR都会说明错误信息。 步骤2. hbase hbck -fixTableOrphans 先修复tableinfo缺失问题，根据内存cache或者hdfs table 目录结构，重新生成tableinfo文件。 步骤3. hbase hbck -fixHdfsOrphans 修复regioninfo缺失问题，根据region目录下的hfile重新生成regioninfo文件。 步骤4. hbase hbck -fixHdfsOverlaps 修复region重叠问题，merge重叠的region为一个region目录，并从新生成一个regioninfo。 步骤5. hbase hbck -fixHdfsHoles 修复region缺失，利用缺失的rowkey范围边界，生成新的region目录以及regioninfo填补这个空洞。 步骤6. hbase hbck -fixMeta 修复meta表信息，利用regioninfo信息，重新生成对应meta row填写到meta表中，并为其填写默认的分配regionserver。 步骤7. hbase hbck -fixAssignments 把这些offline的region触发上线，当region开始重新open 上线的时候，会被重新分配到真实的RegionServer上 , 并更新meta表上对应的行信息。
我尝试执行了两个命令：
hbase hbck -fixHdfsHoles hbase hbck -fixAssignments -fixMete
重新运行hbase hbck之后，集群恢复了正常。
==经验总结==
hbase hbck检查发现集群不健康的时候，
尝试确诊不健康的原因（有hole？regioninfo缺失？或是tableinfo缺失？）
选择适合的在线修复命令修复HBase，
修复之后，再次执行hbase hbck命令，
当检查结果为下面这种状态时，hbase为健康状态：
（HBase集群不需要重启，因为是在线修复命令）
--END--
