热门标签
热门文章
- 1FPGA学习——实现任意倍分频器（奇数/偶数倍分频器均可实现）_fpga分频器
- 2Apache Spark SQL 章节六00_+----+-------+------+ | age| name|salary| +----+--
- 3解决cv2.imwrite存储带有中文路径或名称的图片乱码问题_cv2.imwrite 中文乱码
- 4【python小知识】python同一个函数并行计算_python 并行计算
- 5centos（或openEuler系统）安装kafka集群
- 6Apache RocketMQ ACL 2.0 全新升级_provider.newproducerbuilder() .settopics
- 7Facebook数据仓库的变迁与启示
- 8宿舍管理系统带文档java项目基于springboot+vue的宿舍管理系统_宿舍管理系统项目
- 9万能电视遥控器代码表_cdab107k,vcd遥控器新闻
- 10消息中间件--RabbitMQ学习（二）_消息中心rabbit mo
当前位置: article > 正文
1.paimon与flink环境整合_flink整合paimon
作者：神奇cpp | 2024-07-26 18:34:28
赞
踩
flink整合paimon
概述
paimon 0.5 ,flink 1.17.1版本, 目标flink与paimon进行整合,测试的环境搞起
相关链接
阅读之前，可以先浏览一下
下面的链接是paimon官方文档地址,有兴趣的同学可以浏览一下
官方快速入门链接
jar包
整合需要添加一些jar包
上传
上传至
linux机器
上传之后如下图
重启一下 yarn session ,如果还没有yarn session环境,请参照下面链接
yarn session环境搭建
测试
测试flink与paimon整合的环境是否有问题
yarn session
命令行连接
yarn session环境
# 连接上 yarn session环境
[root@hadoop01 flink]# bin/sql-client.sh
- 1
- 2
Flink SQL> CREATE CATALOG my_catalog WITH ( > 'type'='paimon', > 'warehouse'='hdfs:///data/hive/warehouse/paimon' > ); [INFO] Execute statement succeed. Flink SQL> USE CATALOG my_catalog; [INFO] Execute statement succeed. Flink SQL> CREATE TABLE word_count ( > word STRING PRIMARY KEY NOT ENFORCED, > cnt BIGINT > ); [INFO] Execute statement succeed. Flink SQL> CREATE TEMPORARY TABLE word_table ( > word STRING > ) WITH ( > 'connector' = 'datagen', > 'fields.word.length' = '1' > ); [INFO] Execute statement succeed. Flink SQL> SET 'execution.checkpointing.interval' = '10 s'; [INFO] Execute statement succeed. Flink SQL> INSERT INTO word_count SELECT word, COUNT(*) FROM word_table GROUP BY word;2023-10-19 09:44:34,955 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/data/soft/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. 2023-10-19 09:44:34,992 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at hadoop01/10.32.36.142:8032 2023-10-19 09:44:35,081 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar 2023-10-19 09:44:35,082 WARN org.apache.flink.yarn.YarnClusterDescriptor [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN. 2023-10-19 09:44:35,106 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface hadoop02:42563 of application 'application_1697598809136_0008'. [INFO] Submitting SQL update statement to the cluster... [INFO] SQL update statement has been successfully submitted to the cluster: Job ID: 13414ca4b36fd7adf20aa0b59e5fba0e Flink SQL> SET 'sql-client.execution.result-mode' = 'tableau'; [INFO] Execute statement succeed. Flink SQL> RESET 'execution.checkpointing.interval'; [INFO] Execute statement succeed. Flink SQL> SET 'execution.runtime-mode' = 'batch'; [INFO] Execute statement succeed. Flink SQL> SELECT * FROM word_count;2023-10-19 09:45:04,314 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/data/soft/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. 2023-10-19 09:45:04,357 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at hadoop01/10.32.36.142:8032 2023-10-19 09:45:04,358 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar 2023-10-19 09:45:04,358 WARN org.apache.flink.yarn.YarnClusterDescriptor [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN. 2023-10-19 09:45:04,361 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface hadoop02:42563 of application 'application_1697598809136_0008'. +------+-------+ | word | cnt | +------+-------+ | 0 | 13114 | | 1 | 13090 | | 2 | 13223 | | 3 | 12934 | | 4 | 13266 | | 5 | 12952 | | 6 | 13113 | | 7 | 13051 | | 8 | 13070 | | 9 | 13193 | | a | 13212 | | b | 13032 | | c | 13054 | | d | 13277 | | e | 13252 | | f | 13167 | +------+-------+ 16 rows in set Flink SQL> SET 'execution.runtime-mode' = 'streaming'; [INFO] Execute statement succeed. Flink SQL> SELECT `interval`, COUNT(*) AS interval_cnt FROM > (SELECT cnt / 10000 AS `interval` FROM word_count) GROUP BY `interval`;2023-10-19 09:50:49,672 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/data/soft/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file. 2023-10-19 09:50:49,695 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at hadoop01/10.32.36.142:8032 2023-10-19 09:50:49,696 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar 2023-10-19 09:50:49,696 WARN org.apache.flink.yarn.YarnClusterDescriptor [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set.The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN. 2023-10-19 09:50:49,699 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface hadoop02:42563 of application 'application_1697598809136_0008'. +----+----------------------+----------------------+ | op | interval | interval_cnt | +----+----------------------+----------------------+ | +I | 22 | 1 | | -U | 22 | 1 | | +U | 22 | 2 | | -U | 22 | 2 | | +U | 22 | 3 | | -U | 22 | 3 | | +U | 22 | 4 | | -U | 22 | 4 | | +U | 22 | 5 | | -U | 22 | 5 | | +U | 22 | 6 | | -U | 22 | 6 | | +U | 22 | 7 | | -U | 22 | 7 | | +U | 22 | 8 | | -U | 22 | 8 | | +U | 22 | 9 | | -U | 22 | 9 | | +U | 22 | 10 | | -U | 22 | 10 | | +U | 22 | 11 | | -U | 22 | 11 | | +U | 22 | 12 | | -U | 22 | 12 | | +U | 22 | 13 | | -U | 22 | 13 | | +U | 22 | 14 | | -U | 22 | 14 | | +U | 22 | 15 | | -U | 22 | 15 | | +U | 22 | 16 | | -U | 22 | 16 | | +U | 22 | 15 | | +I | 23 | 1 | | -U | 22 | 15 | | +U | 22 | 14 | | -U | 23 | 1 | | +U | 23 | 2 | | -U | 22 | 14 | | +U | 22 | 13 | | -U | 23 | 2 | | +U | 23 | 3 | | -U | 22 | 13 | | +U | 22 | 12 | | -U | 23 | 3 | | +U | 23 | 4 | | -U | 22 | 12 | | +U | 22 | 11 | | -U | 23 | 4 | | +U | 23 | 5 | | -U | 22 | 11 | | +U | 22 | 10 | | -U | 23 | 5 | | +U | 23 | 6 | | -U | 22 | 10 | | +U | 22 | 9 | | -U | 23 | 6 | | +U | 23 | 7 | | -U | 22 | 9 | | +U | 22 | 8 | | -U | 23 | 7 | | +U | 23 | 8 | | -U | 22 | 8 | | +U | 22 | 7 | | -U | 23 | 8 | | +U | 23 | 9 | | -U | 22 | 7 | | +U | 22 | 6 | | -U | 23 | 9 | | +U | 23 | 10 | | -U | 22 | 6 | | +U | 22 | 5 | | -U | 23 | 10 | | +U | 23 | 11 | | -U | 22 | 5 | | +U | 22 | 4 | | -U | 23 | 11 | | +U | 23 | 12 | | -U | 22 | 4 | | +U | 22 | 3 | | -U | 23 | 12 | | +U | 23 | 13 | | -U | 22 | 3 | | +U | 22 | 2 | | -U | 23 | 13 | | +U | 23 | 14 | | -U | 22 | 2 | | +U | 22 | 1 | | -U | 23 | 14 | | +U | 23 | 15 | | -D | 22 | 1 | | -U | 23 | 15 | | +U | 23 | 16 | ^CQuery terminated, received a total of 93 rows Flink SQL>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
- 72
- 73
- 74
- 75
- 76
- 77
- 78
- 79
- 80
- 81
- 82
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- 90
- 91
- 92
- 93
- 94
- 95
- 96
- 97
- 98
- 99
- 100
- 101
- 102
- 103
- 104
- 105
- 106
- 107
- 108
- 109
- 110
- 111
- 112
- 113
- 114
- 115
- 116
- 117
- 118
- 119
- 120
- 121
- 122
- 123
- 124
- 125
- 126
- 127
- 128
- 129
- 130
- 131
- 132
- 133
- 134
- 135
- 136
- 137
- 138
- 139
- 140
- 141
- 142
- 143
- 144
- 145
- 146
- 147
- 148
- 149
- 150
- 151
- 152
- 153
- 154
- 155
- 156
- 157
- 158
- 159
- 160
- 161
- 162
- 163
- 164
- 165
- 166
- 167
- 168
- 169
- 170
- 171
- 172
- 173
- 174
- 175
- 176
- 177
- 178
- 179
- 180
- 181
- 182
- 183
效果
结束
paimon与flink的整合环境ok
本文内容由网友自发贡献，转载请注明出处：【wpsshop博客】
推荐阅读
相关标签
