在添加tez引擎之后，2.6 GB of 2.1 GB virtual memory used. Killing container

在添加tez引擎之后，2.6 GB of 2.1 GB virtual memory used. Killing containe

	
Application application_1642949186322_0005 failed 2 times due to AM Container for appattempt_1642949186322_0005_000002 exited with exitCode: -103
Failing this attempt.Diagnostics: [2022-01-23 22:54:55.170]Container [pid=17888,containerID=container_1642949186322_0005_02_000001] is running 570644992B beyond the 'VIRTUAL' memory limit. Current usage: 157.9 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1642949186322_0005_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 17888 17887 17888 17888 (bash) 0 0 11964416 348 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yingzi/appcache/application_1642949186322_0005/container_1642949186322_0005_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1024m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1642949186322_0005/container_1642949186322_0005_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1642949186322_0005/container_1642949186322_0005_02_000001/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1642949186322_0005/container_1642949186322_0005_02_000001/stderr
|- 17900 17888 17888 17888 (java) 493 35 2813538304 40062 /opt/module/jdk1.8.0_212/bin/java -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yingzi/appcache/application_1642949186322_0005/container_1642949186322_0005_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1024m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1642949186322_0005/container_1642949186322_0005_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel= org.apache.tez.dag.app.DAGAppMaster --session
[2022-01-23 22:54:55.268]Container killed on request. Exit code is 143
[2022-01-23 22:54:55.277]Container exited with a non-zero exit code 143.
For more detailed output, check the application tracking page: http://hadoop103:8088/cluster/app/application_1642949186322_0005 Then click on links to logs of each attempt.
. Failing the application.
1
2
3
4
5
6
7
8
9
10
11

我试着修改map,reduce端参数,如下:

set mapreduce.map.memory.mb=4096;
set mapreduce.map.java.opts=-Xmx3276M;
set mapreduce.reduce.memory.mb=4096;
set mapreduce.reduce.java.opts=-Xmx3276M;
1
2
3
4

可还是运行失败。另寻它解

解决方法如下：

关闭虚拟内存检查：即在yarn-site.xml文件添加如下配置

<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
1
2
3
4

重启集群，终于运行出来了！！！！

参考资料：

https://zhuanlan.zhihu.com/p/220197000

https://blog.csdn.net/weixin_46584887/article/details/121900677