解决hadoop执行MapReduce程序时Ha和yarn的冲突 YarnRuntimeException: java.lang.NullPointerException

解决hadoop执行MapReduce程序时Ha和yarn的冲突

错误信息：YarnRuntimeException: java.lang.NullPointerException

在hadoop高可用环境下执行MR程序报如下错误：

2020-04-09 22:56:58,827 ERROR [Listener at 0.0.0.0/45871] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.NullPointerException
    at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:280)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:979)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
    at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1293)
    at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$6.run(MRAppMaster.java:1761)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1757)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1691)
Caused by: java.lang.NullPointerException
    at org.apache.hadoop.mapreduce.v2.app.client.MRClientService.getHttpPort(MRClientService.java:177)
    at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:159)
    ... 14 more
2020-04-09 22:56:58,818 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
2020-04-09 22:56:58,819 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

解决方案一：

显示配置yarn的访问地址

在yarn-site.xml配置文件中添加

<property>
   <name>yarn.resourcemanager.webapp.address.rm1</name>
   <value>hdp14:8088</value>
</property>
<property>
   <name>yarn.resourcemanager.webapp.address.rm2</name>
   <value>hdp15:8088</value>
</property>
1
2
3
4
5
6
7
8

分发配置文件yarn-site.xml

[along@hdp14 hadoop]$ axync yarn-site.xml 
1

重启yarn，问题解决

解决方案二：

修改hadoop源码

阅读源代码，会发现这是由于org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer.java这个类中的一段代码引起的

if (rmIds != null) {
      List<String> urls = new ArrayList<>();
      for (String rmId : rmIds) {
        String url = getUrlByRmId(yarnConf, rmId);
        // urls.add(url); // 注释掉这端代码，修改为下面的if语句判断是否为null
        if (url != null) {
          urls.add(url);
        }        
      }
      if (!urls.isEmpty()) {
        params.put(RM_HA_URLS, StringUtils.join(",", urls));
      }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13

这与yarn-site.xml配置HA的兼容性有关，取决于 yarn.resourcemanager.webapp.address 和 yarn.resourcemanager.webapp.https.address 是否为空。

修改源码的方法：

1、首先将hadoop-yarn-server-web-proxy-3.2.1.jar从服务器上拿下来。

2、使用JD-GUI打开，复制org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer.java类中代码；

3、在eclipse新建一个工程，然后新建一个类插入复制的代码，并进行修改。

4、有报错的话就build class，把本地的hadoop相关的包导入

5、将该工程导出为jar包。

6、分别打开hadoop-yarn-server-web-proxy-3.2.1.jar 和刚导出的jar包用解压软件打开，替换原来的org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer.class文件。

修改完后替换原来的jar包，重启hdfs和yarn.job运行成功！

adoop.yarn.server.webproxy.amfilter.AmFilterInitializer.class文件。

修改完后替换原来的jar包，重启hdfs和yarn.job运行成功！

第二种方案，没有验证，参考地址
https://www.cnblogs.com/syq816/p/12667277.html