devbox
羊村懒王
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Flink 系例 之 搭建开发环境与数据_通过properties 构建flink数据源

作者：羊村懒王 | 2024-05-20 14:25:48

赞

踩

通过properties 构建flink数据源

本内容为后续系例示例的演示基础，所有相关示例、数据源、依赖均在此文内的工程与数据之上进行演示。后续如有变动，会在本文中变更。如有错误之处，敬请指出。

 示例环境

  1. java.version: 1.8.x
  2. flink.version: 1.11.1

创建工程

idea中创建项目，项目名称：flink-examples

pom.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <project xmlns="http://maven.apache.org/POM/4.0.0"
  3.          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  4.          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  5.     <modelVersion>4.0.0</modelVersion>
  6.     <groupId>com.flink.examples</groupId>
  7.     <artifactId>flink-examples</artifactId>
  8.     <packaging>pom</packaging>
  9.     <version>1.0-SNAPSHOT</version>
  10.     <modules>
  11.         <module>stream</module>
  12.         <module>connectors</module>
  13.     </modules>
  14.  
  15.     <properties>
  16.         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  17.         <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  18.         <java.version>1.8</java.version>
  19.         <flink.version>1.11.1</flink.version>
  20.         <compiler.version>8</compiler.version>
  21.     </properties>
  22.  
  23.     <dependencies>
  24.         <dependency>
  25.             <groupId>org.apache.flink</groupId>
  26.             <artifactId>flink-streaming-java_2.11</artifactId>
  27.             <version>${flink.version}</version>
  28.         </dependency>
  29.         <dependency>
  30.             <groupId>org.apache.flink</groupId>
  31.             <artifactId>flink-clients_2.11</artifactId>
  32.             <version>${flink.version}</version>
  33.         </dependency>
  34.         <dependency>
  35.             <groupId>org.apache.flink</groupId>
  36.             <artifactId>flink-table-api-java-bridge_2.11</artifactId>
  37.             <version>${flink.version}</version>
  38.         </dependency>
  39.         <dependency>
  40.             <groupId>org.apache.flink</groupId>
  41.             <artifactId>flink-table-planner-blink_2.11</artifactId>
  42.             <version>${flink.version}</version>
  43.         </dependency>
  44.         <dependency>
  45.             <groupId>org.apache.flink</groupId>
  46.             <artifactId>flink-table-planner_2.11</artifactId>
  47.             <version>${flink.version}</version>
  48.             <!--<scope>provided</scope>-->
  49.         </dependency>
  50.         <dependency>
  51.             <groupId>org.apache.flink</groupId>
  52.             <artifactId>flink-json</artifactId>
  53.             <version>${flink.version}</version>
  54.             <!--<scope>test</scope>-->
  55.         </dependency>
  56.         <dependency>
  57.             <groupId>com.google.code.gson</groupId>
  58.             <artifactId>gson</artifactId>
  59.             <version>2.8.6</version>
  60.         </dependency>
  61.         <dependency>
  62.             <groupId>org.apache.logging.log4j</groupId>
  63.             <artifactId>log4j-core</artifactId>
  64.             <version>2.11.1</version>
  65.         </dependency>
  66.     </dependencies>
  67.  
  68.     <build>
  69.         <plugins>
  70.             <plugin>
  71.                 <groupId>org.apache.maven.plugins</groupId>
  72.                 <artifactId>maven-compiler-plugin</artifactId>
  73.                 <configuration>
  74.                     <encoding>UTF-8</encoding>
  75.                     <source>${compiler.version}</source>
  76.                     <target>${compiler.version}</target>
  77.                 </configuration>
  78.             </plugin>
  79.  
  80.             <plugin>
  81.                 <groupId>org.apache.maven.plugins</groupId>
  82.                 <artifactId>maven-shade-plugin</artifactId>
  83.                 <executions>
  84.                     <execution>
  85.                         <phase>package</phase>
  86.                         <goals>
  87.                             <goal>shade</goal>
  88.                         </goals>
  89.                     </execution>
  90.                 </executions>
  91.             </plugin>
  92.         </plugins>
  93.     </build>
  94.  
  95. </project>

项目分两个模块

  1. flink-examples
  2. |----connectors(中件间连接器示例模块)
  3. |----examples(模拟电商订单数据并推送到kafka中，以及flink核心数据流处理客户端)
  4. |----stream(数据流与算子、方法、窗口等示例模块)
  5. |----tableapi(table&sql与中件间的使用示例模块)
  6. |----web(获取flink算子计算后的存储结果，提供给前端展示模块)

connectors模块

pom.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <project xmlns="http://maven.apache.org/POM/4.0.0"
  3.          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  4.          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  5.     <parent>
  6.         <groupId>com.flink.examples</groupId>
  7.         <artifactId>flink-examples</artifactId>
  8.         <version>1.0-SNAPSHOT</version>
  9.     </parent>
  10.     <modelVersion>4.0.0</modelVersion>
  11.     <artifactId>connectors</artifactId>
  12.  
  13.     <dependencies>
  14.         <!-- Flink jdbc依赖 -->
  15.         <dependency>
  16.             <groupId>org.apache.flink</groupId>
  17.             <artifactId>flink-jdbc_2.11</artifactId>
  18.             <version>1.10.1</version>
  19.         </dependency>
  20.  
  21.         <!-- mysql驱动包 -->
  22.         <dependency>
  23.             <groupId>mysql</groupId>
  24.             <artifactId>mysql-connector-java</artifactId>
  25.             <version>5.1.47</version>
  26.         </dependency>
  27.         <!-- kafka依赖 -->
  28.         <dependency>
  29.             <groupId>org.apache.flink</groupId>
  30.             <artifactId>flink-connector-kafka_2.11</artifactId>
  31.             <version>${flink.version}</version>
  32.         </dependency>
  33.         <!-- redis依赖 -->
  34.         <dependency>
  35.             <groupId>org.apache.bahir</groupId>
  36.             <artifactId>flink-connector-redis_2.11</artifactId>
  37.             <version>1.0</version>
  38.         </dependency>
  39.         <!-- rabbitMq依赖 -->
  40.         <dependency>
  41.             <groupId>org.apache.flink</groupId>
  42.             <artifactId>flink-connector-rabbitmq_2.11</artifactId>
  43.             <version>${flink.version}</version>
  44.         </dependency>
  45.         <!-- elasticsearch6依赖 -->
  46.         <dependency>
  47.             <groupId>org.apache.flink</groupId>
  48.             <artifactId>flink-connector-elasticsearch6_2.11</artifactId>
  49.             <version>${flink.version}</version>
  50.         </dependency>
  51.         <dependency>
  52.             <groupId>org.apache.flink</groupId>
  53.             <artifactId>flink-sql-connector-elasticsearch6_2.11</artifactId>
  54.             <version>${flink.version}</version>
  55.         </dependency>
  56.     </dependencies>
  57.  
  58. </project>

stream模块

pom.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <project xmlns="http://maven.apache.org/POM/4.0.0"
  3.          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  4.          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  5.     <parent>
  6.         <artifactId>flink-examples</artifactId>
  7.         <groupId>com.flink.examples</groupId>
  8.         <version>1.0-SNAPSHOT</version>
  9.     </parent>
  10.     <modelVersion>4.0.0</modelVersion>
  11.     <artifactId>stream</artifactId>
  12.     <dependencies>
  13.         <dependency>
  14.             <groupId>org.apache.flink</groupId>
  15.             <artifactId>flink-table-common</artifactId>
  16.             <version>${flink.version}</version>
  17.             <!--<scope>provided</scope>-->
  18.         </dependency>
  19.     </dependencies>
  20.  
  21. </project>

项目全局示例图

示例数据类

后续所有算子演示的数据，均来自此类来提供公共数据。

  1. import org.apache.flink.api.java.tuple.Tuple3;
  2. import java.util.Arrays;
  3. import java.util.List;
  4.  
  5. /**
  6.  * @Description 公共示例数据
  7.  */
  8. public class DataSource {
  9.  
  10.     /**
  11.      * 示例数据集合
  12.      * Tuple3 是一个固定3个属性变量的实体类，分别用f0,f1,f2表示三个构造传参与变量
  13.      * @return
  14.      */
  15.     public static List<Tuple3<String,String,Integer>> getTuple3ToList(){
  16.         //Tuple3<f0,f1,f2> = Tuple3<姓名，性别（man男，girl女），年龄>
  17.         return Arrays.asList(
  18.                 new Tuple3<>("张三", "man", 20),
  19.                 new Tuple3<>("李四", "girl", 24),
  20.                 new Tuple3<>("王五", "man", 29),
  21.                 new Tuple3<>("刘六", "girl", 32),
  22.                 new Tuple3<>("伍七", "girl", 18),
  23.                 new Tuple3<>("吴八", "man", 30)
  24.         );
  25.     }
  26.  
  27. }

源码下载

Gitee：flink-examples: 基于flink.1.11.1版本的工程示例，此示例包含大部份算子、窗口、中间件连接器、tables&sql的用法，适合新人学习使用；

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/羊村懒王/article/detail/598007
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。