docker-compose 部署flink集群_docker-compose flink

1. docker-compose常用命令

详见
https://blog.csdn.net/qq_42267173/article/details/124687804
https://blog.csdn.net/m0_72838865/article/details/126421685
https://www.cnblogs.com/cw2blog/p/16689584.html 容器启停相关的

# 前台启动
docker-compose up
# 后台启动
docker-compose up -d

logs
# docker-compose logs查看服务容器的输出日志。默认情况下，docker-compose将对不同的服务输出使用不同的颜色来区分。可以通过--no-color来关闭颜色。
# 输出日志，不同的服务输出使用不同的颜色来区分
docker-compose logs
# 跟踪日志输出
docker-compose logs -f
# 关闭颜色
docker-compose logs --no-color

run
docker-compose run在指定服务容器上执行一个命令。
docker-compose run nginx echo "helloworld" # 在工程中指定服务的容器上执行 echo "helloworld"

exec
docker-compose exec进入服务容器。
docker-compose exec nginx bash # 进入工程中指定服务的容器
docker-compose exec --index=1 nginx bash # 当一个服务拥有多个容器时，可通过 --index 参数进入到该服务下的任何容器

pause
docker-compose pause暂停服务容器
docker-compose pause # 暂停工程中所有服务的容器
docker-compose pause nginx # 暂停工程中指定服务的容器

unpause
docker-compose unpause恢复服务容器。
docker-compose unpause # 恢复工程中所有服务的容器
docker-compose unpause nginx # 恢复工程中指定服务的容器

restart
docker-compose restart重启服务容器。
docker-compose restart # 重启工程中所有服务的容器
docker-compose restart nginx # 重启工程中指定服务的容器

start
docker-compose start启动服务容器。
docker-compose start # 启动工程中所有服务的容器
docker-compose start nginx # 启动工程中指定服务的容器

stop
docker-compose stop停止服务容器。
docker-compose stop # 停止工程中所有服务的容器
docker-compose stop nginx # 停止工程中指定服务的容器


sudo docker-compose ps
sudo docker-compose stop jobmanager
sudo docker-compose stop taskmanager
sudo docker-compose rm -f jobmanager
sudo docker-compose rm -f taskmanager

sudo docker-compose exec jobmanager ./bin/sql-client.sh -f sql/flink_kafka1.sql
sudo docker-compose exec jobmanager ./bin/flink list
sudo docker logs -f -t --since="2023-05-08" --tail=200 flink_taskmanager_1
sudo docker-compose logs -f taskmanager

sudo docker-compose exec jobmanager bash
sudo docker-compose exec jobmanager ./bin/flink cancel 8d8cc94d73f7bd0c4cdc557264553a04

sudo docker-compose scale taskmanager=5
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

2. 准备工作

sudo yum install docker
sudo yum install docker-compose
1
2

3. 创建目录

mkdir flink
1

4. vim docker-compose.yml

version: "2.1"
services:
  jobmanager:
    image: flink:1.14
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
  taskmanager:
    image: flink:1.14
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

5. 启动

启动报错

$ docker-compose up
ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
1
2
3
4

解决参考：https://blog.csdn.net/xiojing825/article/details/79494408

# 加上sudo，这个会输出详细日志到窗口
flink]$ sudo docker-compose up
Creating network "flink_default" with the default driver
Pulling jobmanager (flink:1.14)...
1.14: Pulling from library/flink
301a8b74f71f: Pull complete
4688df200b56: Pull complete
317eff501ed4: Pull complete
1
2
3
4
5
6
7
8

docker-compose up -d 后台运行，不输出日志到窗口

5. 启动多个taskmanager

# 启用5个taskmanager
docker-compose scale taskmanager=5 
1
2

6. 浏览器观看

http://IP:8082
1

想web访问，但是只能访问localhost，本地ip都无法访问。
解决方法：https://blog.csdn.net/as350144/article/details/125654394

7. 停止容器

flink]$ sudo docker-compose ps
       Name                      Command               State                              Ports
---------------------------------------------------------------------------------------------------------------------------
flink_jobmanager_1    /docker-entrypoint.sh jobm ...   Up      6123/tcp, 8081/tcp, 0.0.0.0:8082->8082/tcp,:::8082->8082/tcp
flink_taskmanager_1   /docker-entrypoint.sh task ...   Up      6121/tcp, 6122/tcp, 6123/tcp, 8081/tcp

看下yml配置文件的服务名字，jobmanager和taskmanager
[xx@sandbox-test flink]$ sudo docker-compose stop jobmanager
Stopping flink_jobmanager_1 ... done
[xx@sandbox-test flink]$ sudo docker-compose stop taskmanager
Stopping flink_taskmanager_1 ... done
1
2
3
4
5
6
7
8
9
10
11

8. 完整的

# 启动flink
[xx@sandbox-test flink]$ sudo docker-compose up -d
Starting flink_jobmanager_1 ... done
Starting flink_taskmanager_1 ... done

# 查看容器
[xx@sandbox-test flink]$ sudo docker-compose ps
       Name                      Command               State                              Ports
---------------------------------------------------------------------------------------------------------------------------
flink_jobmanager_1    /docker-entrypoint.sh jobm ...   Up      6123/tcp, 8081/tcp, 0.0.0.0:8082->8082/tcp,:::8082->8082/tcp
flink_taskmanager_1   /docker-entrypoint.sh task ...   Up      6121/tcp, 6122/tcp, 6123/tcp, 8081/tcp

# 查看日志
[xx@sandbox-test flink]$ sudo docker-compose logs
或者 docker-compose logs -f taskmanager

# 扩展taskmanager
[xx@sandbox-test flink]$ sudo docker-compose scale taskmanager=5
WARNING: The scale command is deprecated. Use the up command with the --scale flag instead.
Starting flink_taskmanager_1 ... done
Creating flink_taskmanager_2 ... done
Creating flink_taskmanager_3 ... done
Creating flink_taskmanager_4 ... done
Creating flink_taskmanager_5 ... done
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

9. 扩展（修改flink的conf）

如果要修改flink的conf呢？
参考:https://www.dandelioncloud.cn/article/details/1589983093958324225
https://www.clzg.cn/article/432005.html

flink下创建以下目录并将flink-conf.yaml拷贝到两个目录下
├── conf
│   ├── job
│   │   └── flink-conf.yaml
│   └── task
│       └── flink-conf.yaml
1
2
3
4
5
6

docker-compose.yml修改如下：主要是增加了volumes，表示使用本地的conf文件，不使用容器里面自带的conf。

version: "2.1"
services:
  jobmanager:
    image: flink:1.14
    expose:
      - "6123"
    ports:
      - "8082:8082"
    command: jobmanager
    volumes:
      - /home/disk1/xx/soft/flink/conf/job/flink-conf.yaml:/opt/flink/conf/flink-conf.yaml
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
  taskmanager:
    image: flink:1.14
    expose:
      - "6121"
      - "6122"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
    volumes:
      - /home/disk1/xx/soft/flink/conf/task/flink-conf.yaml:/opt/flink/conf/flink-conf.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

flink-conf.yaml需要做一些修改，修改内容参考上面的链接。
就可以启动flink了。

sudo docker-compose up -d
sudo docker-compose scale taskmanager=5
1
2

11. lib jar包

参考 （修改flink的conf）

12. 执行sql 语句

flink sql执行参考：https://www.cnblogs.com/isuning/p/16214356.html
报错如下，解决参考：https://help.aliyun.com/document_detail/468412.html#section-jtg-wow-qey

sandbox-test flink]$ sudo docker-compose exec jobmanager ./bin/sql-client.sh -f flink_kafka1.sql


Exception in thread "main" org.apache.flink.table.client.SqlClientException: Unexpected exception. This is a bug. Please consider filing an issue.
	at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:201)
	at org.apache.flink.table.client.SqlClient.main(SqlClient.java:161)
Caused by: java.lang.NoSuchMethodError: org.apache.flink.table.catalog.FunctionCatalog.<init>(Lorg/apache/flink/configuration/ReadableConfig;Lorg/apache/flink/table/catalog/CatalogManager;Lorg/apache/flink/table/module/ModuleManager;)V
	at org.apache.flink.table.client.gateway.context.SessionContext.create(SessionContext.java:237)
	at org.apache.flink.table.client.gateway.local.LocalContextUtils.buildSessionContext(LocalContextUtils.java:87)
	at org.apache.flink.table.client.gateway.local.LocalExecutor.openSession(LocalExecutor.java:87)
	at org.apache.flink.table.client.SqlClient.start(SqlClient.java:88)
	at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:187)
	... 1 more
1
2
3
4
5
6
7
8
9
10
11
12
13

flink版本和jar包之间不兼容。
将flink版本改为1.16，重新启动docker

sudo docker-compose exec jobmanager ./bin/sql-client.sh embedded -f sql/flink_kafka2.sql报错如下

Flink SQL> [INFO] Submitting SQL update statement to the cluster...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/opt/flink/lib/flink-dist-1.16.0.jar) to field java.lang.String.value
WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
1
2
3
4
5
6

解决方法：https://blog.csdn.net/LIZHONGPING00/article/details/127947431
https://blog.csdn.net/qq_41538097/article/details/129113866

docker容器中安装包

0、首先更新源
# apt-get update
1、安装wget
# apt-get install wget
2、安装ping，ifconfig
# apt-get install inetutils-ping
# apt-get install net-tools
3、安装gcc
# apt-get install gcc
4、安装各种依赖包
# apt-get install zlib* bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel tcl-devel gdbm-devel db4-devel libpcap-devel xz-devel libffi-dev
1
2
3
4
5
6
7
8
9
10
11

详见https://www.cnblogs.com/aliensir/articles/15838834.html