devbox
weixin_40725706
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Datasophon添加第三方组件--FLINKSTANDALONE

作者：weixin_40725706 | 2024-04-20 05:26:36

赞

踩

datasophon

Datasophon添加第三方组件--FLINKSTANDALONE

Datasophon简介

DataSophon（点击访问官网）是致力于自动化监控、运维、管理大数据基础组件和节点的，帮助您快速构建起稳定，高效的大数据集群服务，具有极易部署、兼容开源生态、兼容复杂环境、方便运维管控的特点。
DataSophon支持非常方便的集成自己需要的组件，目前官方提供了Flink客户端的安装，但是有些时候没有Hadoop环境，运行Flink作业需要部署Flink Standalone,下面演示如何自定义添加FLINKSTANDALONE组件。DataSophon版本号1.1.1，Flink 1.16.2。

制作Flink Standalone安装包

参考官网提供的服务集成协议，

  1. 我们需要准备一个Flink的安装包，首先下载一个自己需要的版本的Flink，以flink-1.16.2为例，到官网下载flink-1.16.2-bin-scala_2.12.tgz并解压缩，
  2. 提供Jobmanager和Taskmanager服务的start、stop、status和restart的脚本，下面是我自己写的一个control-flink.sh脚本，需要将该脚本放入flink-1.16.2/bin/ 目录下,
SCRIPT_PATH="$( cd "$( dirname "$0" )" >/dev/null 2>&1 && pwd )"
cd "$SCRIPT_PATH/../"

usage() {
    echo "Usage: bash control-flink.sh [jobmanager|taskmanager] [status|restart]"
    exit 1
}

APP_NAME="$1"
case "$APP_NAME" in
  "jobmanager")
    program="StandaloneSessionClusterEntrypoint"
    ;;
  "taskmanager")
    program="TaskManagerRunner"
    ;;
  *)
    usage
    exit 1
    ;;
esac

is_exist(){
  pid=`jps | grep ${program} | awk '{print $1}'`
  #如果不存在返回1，存在返回0
  if [ -z "${pid}" ]; then
   return 1
  else
    return 0
  fi
}

#输出运行状态
status(){
  is_exist
  if [ $? -eq "0" ]; then
    echo "${APP_NAME} is running. Pid is ${pid}"
  else
    echo "${APP_NAME} is NOT running."
    exit 1
  fi
}

#启动方法
start(){
  is_exist
  if [ $? -eq "0" ]; then
    echo "${APP_NAME} is already running. pid=${pid} ."
  else
    bash bin/${APP_NAME}.sh start
    echo "${APP_NAME} start success"
  fi
}

#停止方法
stop(){
  is_exist
  if [ $? -eq "0" ]; then
    bash bin/${APP_NAME}.sh stop
  else
    echo "${APP_NAME} is not running"
  fi
}

#重启
restart(){
  stop
  start
}

#根据输入参数，选择执行对应方法，不输入则执行使用说明
case "$2" in
  "start")
    start
    ;;
  "stop")
    stop
    ;;
  "status")
    status
    ;;
  "restart")
    restart
    ;;
  *)
    usage
    ;;
esac

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  1. 另外我把我常用的jar包flink-doris-connector-1.16-1.4.0.jar和flink-sql-connector-mysql-cdc-2.4.0.jar也放到了flink-1.16.2/lib/目录下。
    目录
  2. 打包
tar czf flink-1.16.2-with-cdc.tar.gz flink-1.16.2

  • 1
  1. 将制作好的包上传到datasophon所在机器的/opt/datasophon/DDP/packages目录下，并将包的md5值写入flink-1.16.2-with-cdc.tar.gz.md5文件：
md5sum flink-1.16.2-with-cdc.tar.gz
echo 'a9e11924a9e0408d025f85f21fb577ea' > flink-1.16.2-with-cdc.tar.gz.md5

  • 1
  • 2

编写配置文件

在meta/DDP-1.0.0目录下创建FLINKSTANDALONE目录和service_ddl.json服务集成文件。（注意在创建文件夹的时候中间不能有‘-’，例如如果是FLINK-STANDALONE，worker在安装插件时工作目录会按‘-’进行拆分成Flink/STANDALONE，导致安装失败）。

{
  "name": "FLINKSTANDALONE",
  "label": "FlinkStandalone",
  "description": "实时计算引擎Standalone",
  "version": "1.16.2",
  "sortNum": 23,
  "dependencies":[],
  "packageName": "flink-1.16.2-with-cdc.tar.gz",
  "decompressPackageName": "flink-1.16.2",
  "roles": [
    {
      "name": "JobManager",
      "label": "JobManager",
      "roleType": "master",
      "cardinality": "1+",
      "logFile": "log/flink-root-standalonesession-0-localhost.localdomain.log",
      "jmxPort": 12356,
      "startRunner": {
        "timeout": "60",
        "program": "bin/control-flink.sh",
        "args": [
          "jobmanager",
          "start"
        ]
      },
      "stopRunner": {
        "timeout": "600",
        "program": "bin/control-flink.sh",
        "args": [
          "jobmanager",
          "stop"
        ]
      },
      "statusRunner": {
        "timeout": "60",
        "program": "bin/control-flink.sh",
        "args": [
          "jobmanager",
          "status"
        ]
      },
      "restartRunner": {
        "timeout": "60",
        "program": "bin/control-flink.sh",
        "args": [
          "jobmanager",
          "restart"
        ]
      },
      "externalLink": {
        "name": "flink-standalone",
        "label": "flink-standalone",
        "url": "http://${host}:8083/"
      }
    },
    {
      "name": "TaskManager",
      "label": "TaskManager",
      "roleType": "worker",
      "cardinality": "1+",
      "logFile": "log/flink-root-taskexecutor-0-localhost.localdomain.log",
      "jmxPort": 12358,
      "startRunner": {
        "timeout": "60",
        "program": "bin/control-flink.sh",
        "args": [
          "taskmanager",
          "start"
        ]
      },
      "stopRunner": {
        "timeout": "600",
        "program": "bin/control-flink.sh",
        "args": [
          "taskmanager",
          "stop"
        ]
      },
      "statusRunner": {
        "timeout": "60",
        "program": "bin/control-flink.sh",
        "args": [
          "taskmanager",
          "status"
        ]
      },
      "restartRunner": {
        "timeout": "60",
        "program": "bin/control-flink.sh",
        "args": [
          "taskmanager",
          "restart"
        ]
      }
    }
  ],
  "configWriter": {
    "generators": [
      {
        "filename": "flink-conf.yaml",
        "configFormat": "properties3",
        "outputDirectory": "conf",
        "includeParams": [
          "rest.port",
          "rest.bind-address",
          "jobmanager.bind-host",
          "jobmanager.memory.process.size",
          "taskmanager.bind-host",
          "taskmanager.host",
          "taskmanager.memory.process.size",
          "taskmanager.numberOfTaskSlots",
          "parallelism.default",
          "jobmanager.execution.failover-strategy",
          "rest.address",
          "classloader.resolve-order",
          "state.backend",
          "state.checkpoints.dir: file",
          "state.savepoints.dir: file",
          "jobmanager.rpc.port",
          "jobmanager.rpc.address",
          "custom.flink.conf"
        ]
      },
      {
        "filename": "masters",
        "configFormat": "custom",
        "outputDirectory": "conf",
        "templateName": "properties_value.ftl",
        "includeParams": [
          "masters"
        ]
      },
      {
        "filename": "workers",
        "configFormat": "custom",
        "outputDirectory": "conf",
        "templateName": "properties_value.ftl",
        "includeParams": [
          "workers"
        ]
      }
    ]
  },
  "parameters": [
    {
      "name": "high-availability",
      "label": "高可用选择",
      "description": "NONE 或者 zookeeper",
      "configType": "input",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "NONE"
    },
    {
      "name": "high-availability.zookeeper.quorum",
      "label": "高可用zk地址",
      "description": "",
      "configType": "input",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "${zkUrls}"
    },
    {
      "name": "rest.port",
      "label": "flink web端口号",
      "description": "",
      "configType": "input",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "8083"
    },
    {
      "name": "rest.bind-address",
      "label": "flink web绑定地址",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "0.0.0.0"
    },
    {
      "name": "jobmanager.bind-host",
      "label": "jobmanager.bind-host",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "0.0.0.0"
    },
    {
      "name": "jobmanager.memory.process.size",
      "label": "jobmanager.memory.process.size",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "1600m"
    },
    {
      "name": "taskmanager.bind-host",
      "label": "taskmanager.bind-host",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "0.0.0.0"
    },
    {
      "name": "taskmanager.host",
      "label": "taskmanager.host",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "localhost"
    },
    {
      "name": "taskmanager.memory.process.size",
      "label": "taskmanager.memory.process.size",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "1728m"
    },{
      "name": "taskmanager.numberOfTaskSlots",
      "label": "taskmanager.numberOfTaskSlots",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "4"
    },{
      "name": "parallelism.default",
      "label": "parallelism.default",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "1"
    },{
      "name": "jobmanager.execution.failover-strategy",
      "label": "jobmanager.execution.failover-strategy",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "region"
    },{
      "name": "rest.address",
      "label": "rest.address",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "localhost"
    },{
      "name": "classloader.resolve-order",
      "label": "classloader.resolve-order",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "parent-first"
    },{
      "name": "state.backend",
      "label": "state.backend",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "rocksdb"
    },{
      "name": "state.checkpoints.dir",
      "label": "state.checkpoints.dir",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "file:///opt/datasophon/flink-1.16.2/flink-checkpoints"
    },{
      "name": "state.savepoints.dir",
      "label": "state.savepoints.dir",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "file:///opt/datasophon/flink-1.16.2/flink-savepoints"
    },{
      "name": "jobmanager.rpc.port",
      "label": "jobmanager.rpc.port",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "6123"
    },{
      "name": "jobmanager.rpc.address",
      "label": "jobmanager.rpc.address",
      "description": "",
      "required": true,
      "type": "input",
      "value": "",
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": "localhost"
    },
    {
      "name": "custom.flink.conf",
      "label": "custom.flink.conf",
      "description": "自定义配置flink-conf.yaml",
      "configType": "custom",
      "required": false,
      "type": "multipleWithKey",
      "value": [],
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": ""
    },
    {
      "name": "masters",
      "label": "masters",
      "description": "masters机器的IP:Port",
      "configType": "custom",
      "required": true,
      "type": "multipleWithKey",
      "value": [],
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": ""
    },
    {
      "name": "workers",
      "label": "workers",
      "description": "workers机器的IP",
      "configType": "custom",
      "required": true,
      "type": "multipleWithKey",
      "value": [],
      "configurableInWizard": true,
      "hidden": false,
      "defaultValue": ""
    }
  ]
}

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118
  • 119
  • 120
  • 121
  • 122
  • 123
  • 124
  • 125
  • 126
  • 127
  • 128
  • 129
  • 130
  • 131
  • 132
  • 133
  • 134
  • 135
  • 136
  • 137
  • 138
  • 139
  • 140
  • 141
  • 142
  • 143
  • 144
  • 145
  • 146
  • 147
  • 148
  • 149
  • 150
  • 151
  • 152
  • 153
  • 154
  • 155
  • 156
  • 157
  • 158
  • 159
  • 160
  • 161
  • 162
  • 163
  • 164
  • 165
  • 166
  • 167
  • 168
  • 169
  • 170
  • 171
  • 172
  • 173
  • 174
  • 175
  • 176
  • 177
  • 178
  • 179
  • 180
  • 181
  • 182
  • 183
  • 184
  • 185
  • 186
  • 187
  • 188
  • 189
  • 190
  • 191
  • 192
  • 193
  • 194
  • 195
  • 196
  • 197
  • 198
  • 199
  • 200
  • 201
  • 202
  • 203
  • 204
  • 205
  • 206
  • 207
  • 208
  • 209
  • 210
  • 211
  • 212
  • 213
  • 214
  • 215
  • 216
  • 217
  • 218
  • 219
  • 220
  • 221
  • 222
  • 223
  • 224
  • 225
  • 226
  • 227
  • 228
  • 229
  • 230
  • 231
  • 232
  • 233
  • 234
  • 235
  • 236
  • 237
  • 238
  • 239
  • 240
  • 241
  • 242
  • 243
  • 244
  • 245
  • 246
  • 247
  • 248
  • 249
  • 250
  • 251
  • 252
  • 253
  • 254
  • 255
  • 256
  • 257
  • 258
  • 259
  • 260
  • 261
  • 262
  • 263
  • 264
  • 265
  • 266
  • 267
  • 268
  • 269
  • 270
  • 271
  • 272
  • 273
  • 274
  • 275
  • 276
  • 277
  • 278
  • 279
  • 280
  • 281
  • 282
  • 283
  • 284
  • 285
  • 286
  • 287
  • 288
  • 289
  • 290
  • 291
  • 292
  • 293
  • 294
  • 295
  • 296
  • 297
  • 298
  • 299
  • 300
  • 301
  • 302
  • 303
  • 304
  • 305
  • 306
  • 307
  • 308
  • 309
  • 310
  • 311
  • 312
  • 313
  • 314
  • 315
  • 316
  • 317
  • 318
  • 319
  • 320
  • 321
  • 322
  • 323
  • 324
  • 325
  • 326
  • 327
  • 328
  • 329
  • 330
  • 331
  • 332
  • 333
  • 334
  • 335
  • 336
  • 337
  • 338
  • 339
  • 340
  • 341
  • 342
  • 343
  • 344
  • 345
  • 346
  • 347
  • 348
  • 349
  • 350
  • 351
  • 352
  • 353
  • 354
  • 355
  • 356
  • 357
  • 358
  • 359
  • 360
  • 361
  • 362
  • 363
  • 364
  • 365
  • 366
  • 367
  • 368
  • 369
  • 370
  • 371
  • 372
  • 373
  • 374
  • 375
  • 376
  • 377
  • 378
  • 379
  • 380
  • 381
  • 382
  • 383
  • 384

服务器上的目录：
配置文件放置目录

编写properties_value.ftl

由于flink conf文件夹下的masters和slaves，不是key,value格式，没有找到合适的生成方式，
在这里插入图片描述
需要编写properties_value.ftl，并放置到每个worker的/opt/datasophon/datasophon-worker/conf/templates/目录下，

<#list itemList as item>
${item.value}
</#list>

  • 1
  • 2
  • 3

然后重启datasophon-worker服务。

bin/datasophon-worker.sh restart worker

  • 1

重启datasophon-manager服务

bin/datasophon-api.sh restart api

  • 1

重启完成后会在数据库表t_ddh_frame_service和t_ddh_frame_service_role中生成对应的记录

t_ddh_frame_service表：
在这里插入图片描述
t_ddh_frame_service_role表：
在这里插入图片描述
如果对service.json文件修改的话，要想使其生效，需要将这俩表里对应的数据删除，然后重启

页面安装

到这儿的话不出意外打开页面是可以看到添加过后的组件，然后就可以愉快的安装了（当然大概率是会出一些意外的，哈哈，不过根据日志报错很容易排查

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/weixin_40725706/article/detail/455585
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。