Dolphinscheduler集成Flink任务踩坑记录

1、关于Flink打包

flink任务编写完成，在本地运行调试正常后，我打包提交到Dolphinscheduler平台进行测试。运行后没多久就报错：

 [taskAppId=TASK-10-108-214]:[138] -  -> java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer
		at com.bigdata.flink.FlinkKafka.main(FlinkKafka.java:30)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer
1
2
3

看日志详情，我们可以看到任务已经开始准备提交了，但是在环境里找不到FlinkKafkaConsumer。

flink run -m yarn-cluster -ys 1 -yjm 1G -ytm 2G -yqu default -p 1 -sae -c com.bigdata.flink.FlinkKafka flink-job/bigdata-flink.jar
1

我的打包方式是通过IDEA 菜单的Build Artfacts…打出的jar文件很小，而且依赖包也都没有被打进去。
于是换了一种方式：

• 配置pom文件

<build>
        <!--<pluginManagement>-->
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>build-helper-maven-plugin</artifactId>
                <version>1.8</version>
                <executions>
                    <execution>
                        <id>add-source</id>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>add-source</goal>
                        </goals>
                        <configuration>
                            <sources>
                                <source>src/main/scala</source>
                                <source>src/test/scala</source>
                            </sources>
                        </configuration>
                    </execution>
                    <execution>
                        <id>add-test-source</id>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>add-test-source</goal>
                        </goals>
                        <configuration>
                            <sources>
                                <source>src/test/scala</source>
                            </sources>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <version>3.1.5</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scalaVersion>2.11.8</scalaVersion>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>utf-8</encoding>
                </configuration>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <appendAssemblyId>false</appendAssemblyId>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <!-- 此处指定main方法入口的class -->
                            <mainClass>com.bigdata.flink.FlinkKafka</mainClass>
                        </manifest>
                    </archive>
                </configuration>

                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>assembly</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
        <!--</pluginManagement>-->
    </build>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97

• 执行maven下的package命令；可以看到这种打包方式执行后的jar包里包含了所需的依赖包。
• 
• 

2、HADOOP_CLASSPATH环境变量配置

环境依赖的问题解决后，在执行中又出现了HADOOP_CLASSPATH环境变量的报错。

taskAppId=TASK-10-112-218]:[138] -  -> 
	------------------------------------------------------------
	 The program finished with the following exception:
	
	org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath. For more information refer to the "Deployment" section of the official Apache Flink documentation.
		at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
		at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
		at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
		at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
		at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
		at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
		at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
		at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
		at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
	Caused by: java.lang.IllegalStateException: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath. For more information refer to the "Deployment" section of the official Apache Flink documentation.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

这次需要修改worker所在节点的flink文件。

• 找到 flink/bin/flink文件
• 在第一行加入

export HADOOP_CLASSPATH=`hadoop classpath`
1

• 
• 无需重启集群机器，直接重跑flink任务即可。