hadoop 错误1 journalnode与namenode超时_waited 84069 ms (timeout=120000 ms) for a response

2016-06-04 18:27:49,728 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 7002 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]
2016-06-04 18:27:49,955 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ut01/10.80.248.11:8485. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-06-04 18:27:50,729 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 8003 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:27:51,730 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 9004 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:27:52,731 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 10005 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:27:53,732 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 11006 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:27:54,733 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 12007 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:27:55,734 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 13008 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:27:56,735 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 14010 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:27:57,737 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 15011 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:27:58,738 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 16012 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:27:59,739 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 17014 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:28:00,741 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 18015 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:28:01,742 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 19016 ms (timeout=20000 ms) for a response for sendEdits. Succeeded so far: [10.80.248.12:8485,10.80.248.14:8485]. Exceptions so far: [10.80.248.11:8485: Journal disabled until next roll]
2016-06-04 18:28:02,727 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: flush failed for required journal (JournalAndStream(mgr=QJM to [10.80.248.11:8485, 10.80.248.12:8485, 10.80.248.13:8485, 10.80.248.14:8485], stream=QuorumOutputStream starting at txid 3608269))
java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond.
	at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137)
	at org.apache.hadoop.hdfs.qjournal.client.QuorumOutputStream.flushAndSync(QuorumOutputStream.java:107)
	at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:113)
	at org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:107)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream$8.apply(JournalSet.java:533)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet.access$100(JournalSet.java:57)
	at org.apache.hadoop.hdfs.server.namenode.JournalSet$JournalSetOutputStream.flush(JournalSet.java:529)
	at org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:647)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3659)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename(NameNodeRpcServer.java:867)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename(ClientNamenodeProtocolServerSideTranslatorPB.java:575)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
2016-06-04 18:28:02,727 WARN org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Aborting QuorumOutputStream starting at txid 3608269
2016-06-04 18:28:02,733 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2016-06-04 18:28:02,736 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ut02/10.80.248.12
************************************************************/
2016-06-05 07:39:20,380 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ut02/10.80.248.12
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 2.7.2
STARTUP_MSG:   classpath = /home/ut/hadoop/hadoop-2.7.2/etc/hadoop:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/jetty-6.1.26.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/paranamer-2.3.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/jersey-core-1.9.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/junit-4.11.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/activation-1.1.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/jersey-json-1.9.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/commons-io-2.4.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/jettison-1.1.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/hadoop-auth-2.7.2.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/hadoop-annotations-2.7.2.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/gson-2.2.4.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/log4j-1.2.17.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/ut/hadoop/hadoop-2.7.2/share/hadoop/common/lib/commons-logging-