赞
踩
最近快要被kerberos这个东西搞疯了,Kerberos认证服务是一种网络身份验证协议,目的是提供安全的用户身份验证和授权服务。关于kereberos认证原理及详情我这里不做过多赘述,只讲在实际场景中是怎么应用的。
在kerberos服务端和客户端都安装好,客户端用kinit命令生成对应的keytab后,hadoop的配置文件也要添加上kerberos相关的配置,可以参考该链接;hadoop、hbase、zookeeper集成kerberos认证_hadoop_昨夜雨疏风zhou-华为云开发者联盟:
1、hbase-site.xml中,需要添加以下配置项来启用Kerberos认证
确保/path/to/hbase.keytab为正确的HBase Keytab文件路径,hbase/_HOST@EXAMPLE.COM为正确的HBase主体。
- <property>
- <name>hbase.security.authentication</name>
- <value>kerberos</value>
- </property>
- <property>
- <name>hbase.master.keytab.file</name>
- <value>/path/to/hbase.keytab</value>
- </property>
- <property>
- <name>hbase.master.kerberos.principal</name>
- <value>hbase/_HOST@EXAMPLE.COM</value>
- </property>
- <property>
- <name>hbase.regionserver.keytab.file</name>
- <value>/path/to/hbase.keytab</value>
- </property>
- <property>
- <name>hbase.regionserver.kerberos.principal</name>
- <value>hbase/_HOST@EXAMPLE.COM</value>
- </property>
2、core-site.xml中,需要添加以下配置项来启用Kerberos认证
- <property>
- <name>hadoop.security.authentication</name>
- <value>kerberos</value>
- </property>
- <property>
- <name>hadoop.security.authorization</name>
- <value>true</value>
- </property>
3、hbase-site.xml中,添加以下配置项:
- <property>
- <name>hbase.zookeeper.property.authProvider.1</name>
- <value>org.apache.zookeeper.server.auth.SASLAuthenticationProvider</value>
- </property>
在环境及各种配置都添加完毕后,接下来就是在代码中实现了。
刚开始我也以为很简单,就是在获取hbase链接前进行一下kerberos登录就行了,代码如下
- //添加kerberos信息
- Configuration conf = HBaseConfiguration.create();
- System.setProperty("java.security.krb5.conf", "/etc/krb5.conf");
- conf.set("hadoop.security.authentication", "Kerberos");
- conf.set("keytab.file","/home/xdy/smokeuser.headless.keytab");
- conf.set("kerberos.principal","bcdp@HADOOP.COM");
- UserGroupInformation.setConfiguration(conf);
- UserGroupInformation.loginUserFromKeytab("bcdp@HADOOP.COM", "/home/xdy/smokeuser.headless.keytab");
-
- conn = ConnectionFactory.createConnection(conf);
日志中出现Login successful for user 这一行就说明kerberos认证成功:
- INFO[org.apache.hadoop.security.UserGroupInformation:1069]- Login successful for user bcdp-bjlzxqcluster@HADOOP.COM using
- keytab file /home/xdy/smokeuser.headless.keytab
结果也确实没问题,可以成功访问hadoop集群,并进行hbase的查询和入库操作,但没想到过了24小时后却收到报错:
- [org.apache.hadoop.hbase.ipc.BlockingRpcConnection$1:398]- Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
- ERROR[org.apache.hadoop.hbase.ipc.BlockingRpcConnection$1:406]- SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
- javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
总体就是说kerberos认证无效了,由于自己也是第一次接触kerberos,所以在网上查找了大量的资料,尝试了很多方法,可参考该链接:
java连接hbase Kerberos 24小时过期问题_kerbros 24小时后断-CSDN博客
上面提到的各种方法也都试过了,各种版本最后也和他的版本保持一致hbase-client-1.5.0,hadoop-common-2.8.5版本,定时任务也加了,但还是没用,还是过24小时就失效,然后重启,就这样循环重启了好几天,最后请教专家终于是把这个问题解决了。
下面直接上代码:
- //循环执行kerberos认证登录
-
- public class ScheduleCheckTGTAndReloginFromKeyTab {
- private static Timer timer;
- private static ScheduleCheckTGTAndReloginFromKeyTab scheduleCheckTGTAndReloginFromKeyTab;
- private static final Logger logger = Logger.getLogger(ScheduleCheckTGTAndReloginFromKeyTab.class);
-
- private ScheduleCheckTGTAndReloginFromKeyTab(String serverName) {
- scheduleCheckTGTAndReloginFromKeyTab = this;
- timer = new Timer();
-
- timer.schedule(new RemindTask(serverName), 0, 2 * 3600 * 1000);
- }
-
- public synchronized static ScheduleCheckTGTAndReloginFromKeyTab getTimer(String serverName) {
- if (scheduleCheckTGTAndReloginFromKeyTab == null) {
- scheduleCheckTGTAndReloginFromKeyTab = new ScheduleCheckTGTAndReloginFromKeyTab(serverName);
- }
- return scheduleCheckTGTAndReloginFromKeyTab;
- }
-
- class RemindTask extends TimerTask {
- String server = "";
-
- public RemindTask(String serverName) {
- server = serverName;
- }
-
- public void run() {
- try {
- Connection connection = HbaseConnection.connectionMap.get(this.server);
- if (connection != null && !connection.isClosed()) {
- logger.info("start to check TGT and relogin from keytab at " + new Date().toString());
-
- UserGroupInformation.getCurrentUser().checkTGTAndReloginFromKeytab();
-
- HbaseConnection.getHbaseConnection(server);
- }
-
- System.gc();
- } catch (Exception e) {
- logger.error(e);
- }
- }
- }
- }
- //获取hbase链接
- public class HbaseConnection {
-
- public static Map<String, Connection> connectionMap = new HashMap<String, Connection>();
- private static final Logger logger = Logger.getLogger(HbaseConnection.class);
-
- public synchronized static Connection getHbaseConnection(String serverName) {
- Connection connection = connectionMap.get(serverName);
- if (connection != null && !connection.isClosed()) {
- return connection;
- } else {
- Configuration configuration = HBaseConfiguration.create();
- configuration.set("hbase.zookeeper.quorum", "lzxq-cdata01,lzxq-cdata02,lzxq-cdata03,lzxq-cdata04,lzxq-cmanager01,lzxq-cmanager02,lzxq-cmanager03");
- configuration.set("hbase.zookeeper.property.clientPort", "2181");
- configuration.set("zookeeper.znode.parent", "/hbase-secure");
- configuration.set("dfs.socket.timeout", "180000");
- configuration.set("keytab.file","/home/xdy/smokeuser.headless.keytab");
- configuration.set("kerberos.principal","bcdp@HADOOP.COM");
- configuration.set("hbase.master.port", "16000");
- configuration.set("hadoop.security.authentication", "kerberos");
- configuration.set("hbase.security.authentication", "kerberos");
-
- try {
-
- UserGroupInformation.setConfiguration(configuration);
- UserGroupInformation.loginUserFromKeytab("bcdp@HADOOP.COM", "/home/xdy/smokeuser.headless.keytab");
-
- ScheduleCheckTGTAndReloginFromKeyTab.getTimer(serverName);
-
- connection = ConnectionFactory.createConnection(configuration);
-
- if (connection != null) {
- connectionMap.put(serverName, connection);
- }
-
- }
-
- catch (IOException e) {
- logger.error(e);
- }
-
- return connection;
- }
- }
-
目前项目已经稳定运行了三天,没有出现什么问题。
总结:Java认证kerberos连接hbase成功后,24小时失效这个问题,如果客户端使用的hadoop版本在3.0以下,则使用我上面的这个方法,如果是3.0以上,则不需要循环认证,hadoop3.x版本里面自带重新认证登录kerberos的方法,更加的方便。
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。