hadoop2.6解决 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... 问题_debug security.usergroupinformation: privilegedact

今天hadoop在启动时候，报

 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

错误，然后增加调试信息，执行命令

export HADOOP_ROOT_LOGGER=DEBUG,console
hadoop fs -text /test/data/origz/access.log.gz

于是打印出报错信息，由于太多，只选关键的

16/05/26 14:24:15 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
16/05/26 14:24:15 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
16/05/26 14:24:15 DEBUG security.Groups:  Creating new Groups object
16/05/26 14:24:15 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
16/05/26 14:24:15 DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path
16/05/26 14:24:15 DEBUG util.NativeCodeLoader: java.library.path=/usr/local/hadoop/lib/
16/05/26 14:24:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/05/26 14:24:15 DEBUG util.PerformanceAdvisory: Falling back to shell based
16/05/26 14:24:15 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping

因为之前参考了

http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-warning

http://www.xuebuyuan.com/2178383.html

的解决方案，在/HADOOP_HOME/etc/hadoop/中的hadoop_env.sh头部添加了如下信息

export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/usr/local/hadoop/lib/"

于是就出现了上述

java.library.path=/usr/local/hadoop/lib/

的错误信息，修改

export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/usr/local/hadoop/lib/native/"

再次执行./start-dfs.sh

16/05/26 14:50:24 DEBUG util.Shell: setsid exited with exit code 0
16/05/26 14:50:24 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
16/05/26 14:50:24 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
16/05/26 14:50:24 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
16/05/26 14:50:24 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
16/05/26 14:50:24 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
16/05/26 14:50:24 DEBUG security.Groups:  Creating new Groups object
16/05/26 14:50:24 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
16/05/26 14:50:24 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
16/05/26 14:50:24 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
16/05/26 14:50:24 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
16/05/26 14:50:24 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
16/05/26 14:50:24 DEBUG security.UserGroupInformation: hadoop login
16/05/26 14:50:24 DEBUG security.UserGroupInformation: hadoop login commit
16/05/26 14:50:24 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: hadoop
16/05/26 14:50:24 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: hadoop" with name hadoop
16/05/26 14:50:24 DEBUG security.UserGroupInformation: User entry: "hadoop"
16/05/26 14:50:24 DEBUG security.UserGroupInformation: UGI loginUser:hadoop (auth:SIMPLE)
16/05/26 14:50:24 DEBUG security.UserGroupInformation: PrivilegedAction as:hadoop (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)
Starting namenodes on [hmaster]
hadoop@hmaster's password: 
hmaster: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-hmaster.out
hslave1: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-hslave1.out
hslave2: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-hslave2.out
Starting secondary namenodes [hmaster]
hadoop@hmaster's password: 
hmaster: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-hmaster.out
16/05/26 14:50:50 DEBUG util.Shell: setsid exited with exit code 0
16/05/26 14:50:51 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
16/05/26 14:50:51 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
16/05/26 14:50:51 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(sampleName=Ops, about=, always=false, type=DEFAULT, valueName=Time, value=[GetGroups])
16/05/26 14:50:51 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
16/05/26 14:50:51 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty
16/05/26 14:50:51 DEBUG security.Groups:  Creating new Groups object
16/05/26 14:50:51 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
16/05/26 14:50:51 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
16/05/26 14:50:51 DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution
16/05/26 14:50:51 DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping
16/05/26 14:50:51 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
16/05/26 14:50:51 DEBUG security.UserGroupInformation: hadoop login
16/05/26 14:50:51 DEBUG security.UserGroupInformation: hadoop login commit
16/05/26 14:50:51 DEBUG security.UserGroupInformation: using local user:UnixPrincipal: hadoop
16/05/26 14:50:51 DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: hadoop" with name hadoop
16/05/26 14:50:51 DEBUG security.UserGroupInformation: User entry: "hadoop"
16/05/26 14:50:51 DEBUG security.UserGroupInformation: UGI loginUser:hadoop (auth:SIMPLE)
16/05/26 14:50:51 DEBUG security.UserGroupInformation: PrivilegedAction as:hadoop (auth:SIMPLE) from:org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:314)

发现没有报错了。

注意：每人出的错可能不一样，我只提供一种参考，学会看报错log是解决问题的关键。