赞
踩
我们在flink开发的过程中,在idea开发调试的过程中flink的一些依赖需要声明maven依赖范围为compile;而在flink程序开发完成打包发布的时候需要声明maven的依赖范围为provided,从而避免依赖冲突或者冗余的打包。这需要在idea开发调试环境和代码打包环境来回切换,以下配置可以很轻松的完成此功能。
pom.xml文件配置示例:
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.pony</groupId> <artifactId>flink-demo01</artifactId> <version>1.0-SNAPSHOT</version> <properties> <java.version>1.8</java.version> <maven.compiler.source>${java.version}</maven.compiler.source> <maven.compiler.target>${java.version}</maven.compiler.target> <flink.version>1.12.2</flink.version> <hadoop.version>2.7.3</hadoop.version> <scala.version>2.11</scala.version> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> </properties> <dependencies> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-statebackend-rocksdb_2.11</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka_2.11</artifactId> <version>${flink.version}</version> <scope>compile</scope> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.25</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <version>1.7.25</version> <scope>provided</scope> </dependency> <!--<dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-core</artifactId> <version>2.15.0</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-api</artifactId> <version>2.15.0</version> </dependency> <!–用于与slf4j保持桥接–> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-slf4j-impl</artifactId> <version>2.15.0</version> </dependency>--> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-shaded-hadoop-2-uber</artifactId> <version>2.7.5-9.0</version> <scope>provided</scope> </dependency> <!--为了能在本地IDE启动看到flinkwebUI--> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-runtime-web_2.11</artifactId> <version>${flink.version}</version> <scope>provided</scope> </dependency> </dependencies> <build> <plugins> <!-- 编译插件 --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.6.0</version> <configuration> <source>${java.version}</source> <target>${java.version}</target> <encoding>${project.build.sourceEncoding}</encoding> </configuration> </plugin> <!-- scala编译插 打包时注释掉--> <!--<plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>3.1.6</version> <configuration> <scalaCompatVersion>2.11</scalaCompatVersion> <scalaVersion>2.11.12</scalaVersion> <encoding>UTF-8</encoding> </configuration> <executions> <execution> <id>compile-scala</id> <phase>compile</phase> <goals> <goal>add-source</goal> <goal>compile</goal> </goals> </execution> <execution> <id>test-compile-scala</id> <phase>test-compile</phase> <goals> <goal>add-source</goal> <goal>testCompile</goal> </goals> </execution> </executions> </plugin>--> <!-- 使用 maven-shade打包依賴(推荐)--> <!-- We use the maven-shade plugin to create a fat jar that contains all necessary dependencies. --> <!-- Change the value of <mainClass>...</mainClass> if your program entry point changes. --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.0.0</version> <executions> <!-- Run shade goal on package phase --> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <excludes> <exclude>org.apache.flink:force-shading</exclude> <exclude>com.google.code.findbugs:jsr305</exclude> <exclude>org.slf4j:*</exclude> <exclude>log4j:*</exclude> </excludes> </artifactSet> <filters> <filter> <!-- Do not copy the signatures in the META-INF folder. Otherwise, this might cause SecurityExceptions when using the JAR. --> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass>com.pony.watermark.StreamingWindowWatermarkKafkaNewApi</mainClass> </transformer> </transformers> <!-- 将“com.pony.kafka”重命名为“com.shade.kafka"--> <relocations> <relocation> <pattern>com.pony.kafka</pattern> <shadedPattern>com.shade.kafka</shadedPattern> <excludes> <exclude>com.pony.kafka.KeyedStateFunction</exclude> <exclude>com.pony.kafka.StreamingWithKafka*</exclude> </excludes> </relocation> </relocations> <!-- 修改包的后缀名--> <shadedArtifactAttached>true</shadedArtifactAttached> <shadedClassifierName>pony-shade</shadedClassifierName> </configuration> </execution> </executions> </plugin> <!-- 使用assembly打包依賴,打jar包插件(会包含所有依赖) --> <!--<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <version>2.6</version> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> <archive> <manifest> <!– 可以设置jar包的入口类(可选) –> <mainClass>xuwei.tech.streaming.SocketWindowWordCountJava</mainClass> </manifest> </archive> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin>--> </plugins> </build> <profiles> <profile> <id>add-dependencies-for-IDEA</id> <activation> <property> <name>idea.version</name> </property> <activeByDefault>true</activeByDefault> </activation> <dependencies> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-streaming-java_2.11</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-clients_2.11</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-statebackend-rocksdb_2.11</artifactId> <version>${flink.version}</version> </dependency> <!--https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/kafka.html--> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-connector-kafka_2.11</artifactId> <version>${flink.version}</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.25</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> <version>1.7.25</version> </dependency> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-shaded-hadoop-2-uber</artifactId> <version>2.7.5-9.0</version> </dependency> <!--为了能在本地IDE启动看到flinkwebUI--> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-runtime-web_2.11</artifactId> <version>${flink.version}</version> </dependency> </dependencies> </profile> </profiles> </project>
打包使用说明:
打包:
idea开发使用说明:
需要勾选Profiles的【add-dependencies-for-IDEA】选项
注意: 需要重新加载maven依赖
否则会报错:
编写flink jar包程序的依赖问题
在编写flink jar包程序时,一定要特别注意运行时存在的jar包,设置其依赖scope为provided,否则可能会有冲突。特别是在应用引用低版本flink库,而
镜像使用高版本时,应用可能不能平滑迁移。
同时,在idea中调试,又需要将其scope设置为compile,否则本地调试可能会有问题。
为了解决这个问题,可以在pom中使用profile区分,示例如下:
<properties> ... <!-- components exist online scope define --> <libs.online.scope>provided</libs.online.scope> ... </properties> <profiles> <profile> <id>add-dependencies-for-IDEA</id> <activation> <property> <name>idea.maven.embedder.version</name> </property> </activation> <properties> <libs.online.scope>compile</libs.online.scope> </properties> </profile> </profiles>
之后,将在运行时存在的jar包的依赖按照如下方式设置即可,比如
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-java</artifactId>
<version>${flink.version}</version>
<scope>${libs.online.scope}</scope>
</dependency>
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。