【Flink】FLink session yarn kerberos认证报错 PortUnreachableException: ICMP Port Unreachable

1.概述

本篇博客是在上一篇博客： 【Flink】FLink session yarn kerberos认证报错 KrbException: Cannot locate default realm 还没找到原因的时候做的一次尝试。

因为项目需要，我们需要对接华为认证的yarn集群，运行flink任务，我们的任务分为两种，yarn pre-job模式和flink session模式。我们在配置文件中配置如下

[root@1 flink]# cat conf/flink-conf.yaml 

# kerberos认证需要的东西
security.kerberos.login.use-ticket-cache: true
security.kerberos.login.keytab: /usr/keytab/mr.keytab
security.kerberos.login.principal: mrr@XXX.COM
security.kerberos.login.contexts: KafkaClient,Client

classloader.check-leaked-classloader: false


1
2
3
4
5
6
7
8
9
10
11

提交任务的脚本如下

   jvm_options="-Xmn${nm}m -XX:SurvivorRatio=5 -XX:-UseAdaptiveSizePolicy -Dfastjson.parser.safeMode=true  "
    echo "开启kerberos认证,配置krb5.conf"
    # 这里没有配置  -Djava.security.auth.login.config=/usr/keytab/jaas.conf
    jvm_options="$jvm_options   -Djava.security.krb5.conf=/usr/keytab/krb5.conf"
    jvm_options="$jvm_options   -Djava.security.auth.login.config=/usr/keytab/jaas.conf "
    jvm_options="$jvm_options   -Dzookeeper.sasl.client.username=zookeeper  "
#    jvm_options="$jvm_options   -Des.kerberos.jaas.appname=EsClient  "
    echo "部分现场需要使用EsClient进行认证"
    jvm_options="$jvm_options   -Dzookeeper.sasl.client=true  -Dzookeeper.sasl.clientconfig=Client "
    # 日志可能乱码的问题
    jvm_options="$jvm_options -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8"

     su - hdfs -c "export HADOOP_CONF_DIR=/opt/hadoopclient/HDFS/hadoop/etc/hadoop/ && export HADOOP_CLASSPATH=`hadoop classpath` && $basepath/bin/flink run 
1
2
3
4
5
6
7
8
9
10
11
12
13

然后就能认证成功，pre-job模式运行的flink成功。但是我们的flink-session认证不成功

export USERDNSDOMAIN=XXX.COM
JVM_ARGS="$JVM_ARGS -Xmx512m"

JVM_ARGS=" $JVM_ARGS -XX:SurvivorRatio=5 -XX:-UseAdaptiveSizePolicy -Dfastjson.parser.safeMode=true  "
echo "开启kerberos认证,配置krb5.conf"
# 这里没有配置  -Djava.security.auth.login.config=/usr/keytab/jaas.conf
JVM_ARGS="$JVM_ARGS   -Djava.security.krb5.conf=/usr/keytab/krb5.conf"
JVM_ARGS="$JVM_ARGS   -Djava.security.auth.login.config=/usr/keytab/jaas.conf "
JVM_ARGS="$JVM_ARGS   -Dzookeeper.sasl.client.username=zookeeper  "
#    JVM_ARGS="$JVM_ARGS   -Des.kerberos.jaas.appname=EsClient  "
echo "部分现场需要使用EsClient进行认证"
JVM_ARGS="$JVM_ARGS   -Dzookeeper.sasl.client=true  -Dzookeeper.sasl.clientconfig=Client "
# 日志可能乱码的问题
JVM_ARGS="$JVM_ARGS -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8  "
#JVM_ARGS="$JVM_ARGS -Djava.security.krb5.realm=XXX.COM -Djava.security.krb5.kdc=xxx.xx.xx.xx6:42732 "
JVM_ARGS="$JVM_ARGS -Djava.security.krb5.realm=XXX.COM  -Djava.security.krb5.kdc=xxx.xx.xx.xx8:42732 "



CC_CLASSPATH=`manglePathList $(constructFlinkClassPath):$INTERNAL_HADOOP_CLASSPATHS`

log=$FLINK_LOG_DIR/flink-$FLINK_IDENT_STRING-yarn-session-$HOSTNAME.log
log_setting="-Dlog.file="$log" -Dlog4j.configuration=file:"$FLINK_CONF_DIR"/log4j-session.properties -Dlog4j.configurationFile=file:"$FLINK_CONF_DIR"/log4j-session.properties -Dlogback.configurationFile=file:"$FLINK_CONF_DIR"/logback-session.xml"

$JAVA_RUN $JVM_ARGS -classpath "$CC_CLASSPATH" $log_setting org.apache.flink.yarn.cli.FlinkYarnSessionCli -j "$FLINK_LIB_DIR"/submit/flink-dist*.jar "$@"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

这个启动报错如下

org.apache.flink.runtime.security.modules.SecurityModule$SecurityInstallException: Unable to set the Hadoop login user
        at org.apache.flink.runtime.security.modules.HadoopModule.install(HadoopModule.java:138)
        at org.apache.flink.runtime.security.SecurityUtils.installModules(SecurityUtils.java:76)
        at org.apache.flink.runtime.security.SecurityUtils.install(SecurityUtils.java:57)
        at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:858)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: mrr@XXX.COM from keytab /usr/keytab/mr.keytab javax.security.auth.login.LoginException: ICMP Port Unreachable
        at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1846)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1214)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1007)
        at org.apache.flink.runtime.security.modules.HadoopModule.install(HadoopModule.java:77)
        ... 3 more
Caused by: javax.security.auth.login.LoginException: ICMP Port Unreachable
        at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:808)
        at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
        at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
        at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
        at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
        at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1924)
        at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1836)
        ... 6 more
Caused by: java.net.PortUnreachableException: ICMP Port Unreachable
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
        at java.net.DatagramSocket.receive(DatagramSocket.java:812)
        at sun.security.krb5.internal.UDPClient.receive(NetClient.java:206)
        at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:411)
        at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:364)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.krb5.KdcComm.send(KdcComm.java:348)
        at sun.security.krb5.KdcComm.sendIfPossible(KdcComm.java:253)
        at sun.security.krb5.KdcComm.send(KdcComm.java:229)
        at sun.security.krb5.KdcComm.send(KdcComm.java:200)
        at sun.security.krb5.KrbAsReqBuilder.send(KrbAsReqBuilder.java:316)
        at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:361)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

查看端口确实是不通畅的

[root@1 flink]# telnet xxx.xx.xx.xx8 42732
Trying xxx.xx.xx.xx8...
telnet: connect to address xxx.xx.xx.xx8: Connection refused
[root@1 flink]# telnet xxx.xx.xx.xx6 42732
Trying xxx.xx.xx.xx6...
telnet: connect to address xxx.xx.xx.xx6: Connection refused
[root@1 flink]# 

1
2
3
4
5
6
7
8

查看krb5文件如下

[root@1 flink]# cat ../keytab/krb5.conf

[realms]
XXX.COM = {
kdc = xxx.xx.xx.xx6:42732
kdc = xxx.xx.xx.xx8:42732
admin_server = xxx.xx.xx.xx6:42730
admin_server = xxx.xx.xx.xx8:42730
kpasswd_server = xxx.xx.xx.xx6:42731
kpasswd_server = xxx.xx.xx.xx8:42731

1
2
3
4
5
6
7
8
9
10
11

测试下面端口也不行

[root@1 flink]# telnet xxx.xx.xx.xx6 42730
Trying xxx.xx.xx.xx6...
Connected to xxx.xx.xx.xx6.
Escape character is '^]'.


1
2
3
4
5
6

经过尝试这几个端口都不行。