CDH5.15部署flink1.13_cdh 5.15 集成flink

建议在cm节点操作，git、maven已安装并可以访问外网，httpd服务已开通

一、flink1.13.0-parcel包制作

• 下载制作包flink-parcel：git clone https://gitee.com/abram-guo/flink-parcel-master.git

• 修改flink-parcel.properties配置文件（cdh5.15，flink1.13，scala2.11）

#FLINk 下载地址
FLINK_URL=https://archive.apache.org/dist/flink/flink-1.13.0/flink-1.13.0-bin-scala_2.11.tgz
#flink版本号
FLINK_VERSION=1.13.0
#扩展版本号
EXTENS_VERSION=BIN-SCALA_2.11
#操作系统版本，以centos为例
OS_VERSION=7
#CDH 小版本
CDH_MIN_FULL=5.13
CDH_MAX_FULL=6.0
#CDH大版本
CDH_MIN=5
CDH_MAX=53
1
2
3
4
5
6
7
8
9
10
11
12
13
14

• 生成parcel（flink安装包）和csd（flink服务），并启动httpd服务访问parcel文件

#生成parcel文件夹：FLINK-1.13.0-BIN-SCALA_2.11_build
./build.sh parcel
# FLINK-1.13.0-BIN-SCALA_2.11_build目录下文件copy到httpd服务文件目录下
mkdir -p /var/www/html/cloudera-repos/flink-parcel/
cp -R FLINK-1.13.0-BIN-SCALA_2.11_build/* /var/www/html/cloudera-repos/flink-parcel/
# 启动httpd服务，没有httpd自行安装 yum install httpd
#  通过 http://{host_ip}/cloudera-repos/flink-parcel/ 可以访问生成的parcel文件
sudo systemctl start httpd

# 生成csd jar包：FLINK_ON_YARN-1.13.0.jar
./build.sh csd
# 将jar包copy到cm节点的 /opt/cloudera/csd/目录下
mkdir -p /opt/cloudera/csd/
cp FLINK_ON_YARN-1.13.0.jar /opt/cloudera/csd/

# 重启cm服务后，访问web添加服务中可以看到flink的服务,根据自己安装目录重启
# /usr/local/cloudera-manager/cm-5.15.0/etc/init.d/cloudera-scm-server
/opt/cloudera-manager/cm-5.15.0/etc/init.d/cloudera-scm-server restart 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

二、CDH部署flink服务

• 添加编译parcel包的http地址

• 重启cloudera Management服务

• 下载、分配、激活flink组件

• 添加服务中选择flink，一步步安装
  

三、CDH flink服务配置

• flink-conf/log4j.properties 的 Flink 客户端高级配置代码段（安全阀）配置：

monitorInterval=30

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender

# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = RollingFile
appender.main.append = true
appender.main.fileName = ${sys:log.file}
appender.main.filePattern = ${sys:log.file}.%i
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
appender.main.policies.type = Policies
appender.main.policies.size.type = SizeBasedTriggeringPolicy
appender.main.policies.size.size = 100MB
appender.main.policies.startup.type = OnStartupTriggeringPolicy
appender.main.strategy.type = DefaultRolloverStrategy
appender.main.strategy.max = ${env:MAX_LOG_FILE_NUMBER:-10}

# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

• High Availability Service high-availability 设置为none
• flink-sql-client连接hive：把依赖包放到 /opt/cloudera/parcels/FLINK/lib/flink/lib下，并且设置enable_hive_catalog和sql_current_catalog