【Flink部署】java部署、本地部署(maven-shade工具)、平台部署（windows部署、 linux部署）；端口号修改_flink单机java项目部署

1.flink的java部署

01.直接main方法启动

public class FlinkEnvironment01 {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();

        DataStreamSource<String> streamSource = environment.addSource(new SourceFunction<String>() {
            @Override
            public void run(SourceContext<String> sourceContext) throws Exception {
                while (true) {
                    sourceContext.collect(String.valueOf(System.currentTimeMillis()));
                    TimeUnit.MILLISECONDS.sleep(500);
                }
            }

            @Override
            public void cancel() {

            }
        });

        streamSource.print();

        environment.execute();
    }
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

2> 1692668361022
3> 1692668361537
4> 1692668362039
5> 1692668362542
1
2
3
4

02.idea中可以直接创建web界面并提交直接启动（了解）

        <!--idea中可以直接创建web界面并提交直接启动的依赖-->
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-runtime-web_2.12</artifactId>
            <version>1.12.7</version>
        </dependency>
1
2
3
4
5
6

Configuration configuration = new Configuration();
StreamExecutionEnvironment environment = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(configuration);
1
2

http://localhost:8081

2.flink的本地部署(打包成jar包，在黑窗口中运行)

01.maven-shade工具进行打包操作*

需要加插件，不然默认的没有依赖(外部库)
pom.xml（直接加在</dependencies>后面）

	<!--打包-->
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.1.1</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <excludes>
                                    <exclude>com.google.code.findbugs:jsr305</exclude>
                                    <exclude>log4j:*</exclude>
                                </excludes>
                            </artifactSet>
                            <filters>
                                <filter>
                                    <!-- Do not copy the signatures in the META-INF folder.
                                    Otherwise, this might cause SecurityExceptions when using the JAR. -->
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>com.itszt23.flink.FlinkEnvironment01</mainClass>
                                </transformer>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                    <resource>reference.conf</resource>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

需要指明主类（全名称） <mainClass>com.itszt23.flink.FlinkEnvironment01</mainClass>

java -jar 直接运行jar包

3.flink的平台部署

01.windows部署（了解）

需要有start-cluster.bat

先启动.bat，会弹出两个黑窗口，一个Job，一个Task
然后访问页面http://localhost:8081/

上传jar包

运行后：job窗口不变化、task窗口显示运行内容

ctrl+c退出

02.linux部署（掌握）*

wget https://repo.huaweicloud.com/apache/flink/flink-1.12.7/flink-1.12.7-bin-scala_2.12.tgz

tar -xf flink-1.12.7-bin-scala_2.12.tgz

cd flink-1.12.7/

cd bin
1
2
3
4
5
6
7

1、前台启动./start-cluster.sh

./start-cluster.sh启动

访问http://服务器IP地址:8081 （记得开放防火墙端口8081）
上传jar包

日志可以在log文件夹里查看

比较小的文件可以直接用vi编辑器查看vi flink-ubuntu-taskexecutor-0-VM-8-4-ubuntu.out

tail -f 动态输出大文件查看日志

1. 使用 ./stop-cluster.sh关闭

2. 使用kill -9 关闭flink

首先使用ps -ef | grep flink 查找进程ID为15256、15533

kill -9 15256
kill -9 15533
ps -ef |grep flink
1
2
3

2、后台启动 nohup

scp命令将jar包复制到云服务器中

 nohup java -jar demo-flink-1.0-SNAPSHOT.jar > myout.txt 2>&1 &
1

输出内容到myout.txt中

4.端口修改

1、IDEA代码中直接配置

Configuration configuration = new Configuration();
configuration.setInteger("rest.port", 9000);
StreamExecutionEnvironment environment = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(configuration);
1
2
3

2、Windows本机修改flink配置文件flink-conf.yaml 中端口设置

3、云服务器中修改配置文件flink-conf.yaml

vi flink-conf.yaml

使用查找命令/rest.port或者?rest.port，同样修改即可