devbox
很楠不爱3
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Dolphinscheduler集群搭建_dolphinscheduler创建集群

作者：很楠不爱3 | 2024-04-11 15:17:57

赞

踩

dolphinscheduler创建集群

目录

背景

前置准备工作

准备 DolphinScheduler 启动环境

配置用户免密及权限

配置机器SSH免密登陆

启动zookeeper

修改相关配置

初始化数据库

启动 DolphinScheduler

登录 DolphinScheduler

启停服务

背景

集群部署目的是在多台机器部署 DolphinScheduler 服务，用于运行大量任务情况。

前置准备工作

伪分布式部署 DolphinScheduler 需要有外部软件的支持

  • JDK：下载JDK (1.8+)，并将 JAVA_HOME 配置到以及 PATH 变量中。如果你的环境中已存在，可以跳过这步。
  • 二进制包：在下载页面下载 DolphinScheduler 二进制包
  • 数据库：PostgreSQL (8.2.15+) 或者 MySQL (5.7+)，两者任选其一即可，如 MySQL 则需要 JDBC Driver 8.0.16
  • 注册中心：ZooKeeper (3.4.6+)，下载地址
  • 进程树分析
    • macOS安装pstree
    • Fedora/Red/Hat/CentOS/Ubuntu/Debian安装psmisc

注意: DolphinScheduler 本身不依赖 Hadoop、Hive、Spark，但如果你运行的任务需要依赖他们，就需要有对应的环境支持

准备 DolphinScheduler 启动环境

配置用户免密及权限

创建部署用户，并且一定要配置 sudo 免密。以创建 dolphinscheduler 用户为例

  1. # 创建用户需使用 root 登录
  2. useradd dolphinscheduler
  3.  
  4. # 添加密码
  5. echo "dolphinscheduler" | passwd --stdin dolphinscheduler
  6.  
  7. # 配置 sudo 免密
  8. sed -i '$adolphinscheduler  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' /etc/sudoers
  9. sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers
  10.  
  11. # 修改目录权限，使得部署用户对二进制包解压后的 apache-dolphinscheduler-*-bin 目录有操作权限
  12. chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin

注意:

  • 因为任务执行服务是以 sudo -u {linux-user} 切换不同 linux 用户的方式来实现多租户运行作业，所以部署用户需要有 sudo 权限，而且是免密的。初学习者不理解的话，完全可以暂时忽略这一点
  • 如果发现 /etc/sudoers 文件中有 "Defaults requirett" 这行，也请注释掉

配置机器SSH免密登陆

        由于安装的时候需要向不同机器发送资源，所以要求各台机器间能实现SSH免密登陆。配置免密登陆的步骤如下

  1. su dolphinscheduler
  2.  
  3. ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
  4. cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  5. chmod 600 ~/.ssh/authorized_keys

注意: 配置完成后，可以通过运行命令 ssh localhost 判断是否成功，如果不需要输入密码就能ssh登陆则证明成功

启动zookeeper

   前面讲过 zk建的搭建及启动，参考这篇文章：CSDNicon-default.png?t=N7T8https://mp.csdn.net/mp_blog/creation/editor/134516112

修改相关配置

        完成了基础环境的准备后，在运行部署命令前，还需要根据环境修改配置文件。配置文件在路径在conf/config/install_config.conf下，一般部署只需要修改INSTALL MACHINE、DolphinScheduler ENV、Database、Registry Server部分即可完成部署，下面对必须修改参数进行说明

  1. #
  2. # Licensed to the Apache Software Foundation (ASF) under one or more
  3. # contributor license agreements.  See the NOTICE file distributed with
  4. # this work for additional information regarding copyright ownership.
  5. # The ASF licenses this file to You under the Apache License, Version 2.0
  6. # (the "License"); you may not use this file except in compliance with
  7. # the License.  You may obtain a copy of the License at
  8. #
  9. #     http://www.apache.org/licenses/LICENSE-2.0
  10. #
  11. # Unless required by applicable law or agreed to in writing, software
  12. # distributed under the License is distributed on an "AS IS" BASIS,
  13. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  14. # See the License for the specific language governing permissions and
  15. # limitations under the License.
  16. #
  17.  
  18. # ---------------------------------------------------------
  19. # INSTALL MACHINE
  20. # ---------------------------------------------------------
  21. # A comma separated list of machine hostname or IP would be installed DolphinScheduler,
  22. # including master, worker, api, alert. If you want to deploy in pseudo-distributed
  23. # mode, just write a pseudo-distributed hostname
  24. # Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: 
  25. #  ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
  26. ips="dsj-node231,dsj-node232,dsj-node233,dsj-node234,dsj-node236"
  27.  
  28. # Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
  29. # modify it if you use different ssh port
  30. sshPort="22"
  31.  
  32. # A comma separated list of machine hostname or IP would be installed Master server, it
  33. # must be a subset of configuration `ips`.
  34. # Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
  35.  
  36. masters="dsj-node231,dsj-node232"
  37.  
  38. # A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
  39. # subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
  40. # Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: 
  41. # workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
  42.  
  43. workers="dsj-node231:default,dsj-node232:default,dsj-node233:default,dsj-node234:default,dsj-node236:default"
  44.  
  45. # A comma separated list of machine hostname or IP would be installed Alert server, it
  46. # must be a subset of configuration `ips`.
  47. # Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
  48. alertServer="dsj-node234"
  49.  
  50. # A comma separated list of machine hostname or IP would be installed API server, it
  51. # must be a subset of configuration `ips`.
  52. # Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
  53. apiServers="dsj-node236"
  54.  
  55. # A comma separated list of machine hostname or IP would be installed Python gateway server, it
  56. # must be a subset of configuration `ips`.
  57. # Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"
  58. pythonGatewayServers="dsj-node236"
  59.  
  60. # The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
  61. # Do not set this configuration same as the current path (pwd)
  62.  
  63. installPath="/data/dolphinscheduler"
  64.  
  65. # The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
  66. # script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
  67. # to be created by this user
  68. deployUser="root"
  69.  
  70. # The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.
  71. dataBasedirPath="/tmp/dolphinscheduler"
  72.  
  73. # ---------------------------------------------------------
  74. # DolphinScheduler ENV
  75. # ---------------------------------------------------------
  76. # JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
  77. # and this configuration only support one parameter so far.
  78.  
  79. javaHome="/usr/lib/jvm/java-8-openjdk-amd64"
  80.  
  81. # DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345
  82.  
  83. apiServerPort="12345"
  84.  
  85. # ---------------------------------------------------------
  86. # Database
  87. # NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
  88. # ---------------------------------------------------------
  89. # The type for the metadata database
  90. # Supported values: ``postgresql``, ``mysql`, `h2``.
  91.  
  92. DATABASE_TYPE=${DATABASE_TYPE:-"mysql"}
  93.  
  94. # Spring datasource url, following <HOST>:<PORT>/<database>?<parameter> format, If you using mysql, you could use jdbc
  95. # string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
  96.  
  97. SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:mysql://172.16.10.235:13306/dolphinscheduler?useSSL=false"}
  98.  
  99. # Spring datasource username
  100.  
  101. SPRING_DATASOURCE_USERNAME=${SPRING_DATASOURCE_USERNAME:-"root"}
  102.  
  103. # Spring datasource password
  104.  
  105. SPRING_DATASOURCE_PASSWORD=${SPRING_DATASOURCE_PASSWORD:-"Whdsj@!db0521"}
  106.  
  107. # ---------------------------------------------------------
  108. # Registry Server
  109. # ---------------------------------------------------------
  110. # Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
  111. registryPluginName="zookeeper"
  112.  
  113. # Registry Server address.
  114. registryServers="dsj-node231:2181,dsj-node232:2181,dsj-node233:2181,dsj-node234:2181,dsj-node236:2181"
  115.  
  116. # Registry Namespace
  117. registryNamespace="dolphinscheduler"
  118.  
  119. # ---------------------------------------------------------
  120. # Worker Task Server
  121. # ---------------------------------------------------------
  122. # Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
  123.  
  124. taskPluginDir="lib/plugin/task"
  125.  
  126. # resource storage type: HDFS, S3, NONE
  127.  
  128. resourceStorageType="HDFS"
  129.  
  130. # resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
  131.  
  132. resourceUploadPath="/dolphinscheduler"
  133.  
  134. # if resourceStorageType is HDFS，defaultFS write namenode address，HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
  135. # if S3，write S3 address，HA，for example ：s3a://dolphinscheduler，
  136. # Note，S3 be sure to create the root directory /dolphinscheduler
  137.  
  138. defaultFS="hdfs:dsj-node231:8020"
  139.  
  140. # if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
  141. s3Endpoint="http://192.168.xx.xx:9010"
  142. s3AccessKey="xxxxxxxxxx"
  143. s3SecretKey="xxxxxxxxxx"
  144.  
  145. # resourcemanager port, the default value is 8088 if not specified
  146. resourceManagerHttpAddressPort="8088"
  147.  
  148. # if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
  149. yarnHaIps="192.168.xx.xx,192.168.xx.xx"
  150.  
  151. # if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
  152. singleYarnIp="yarnIp1"
  153.  
  154. # who has permission to create directory under HDFS/S3 root path
  155. # Note: if kerberos is enabled, please config hdfsRootUser=
  156. hdfsRootUser="root"
  157.  
  158. # kerberos config
  159. # whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
  160. kerberosStartUp="false"
  161. # kdc krb5 config file path
  162. krb5ConfPath="$installPath/conf/krb5.conf"
  163. # keytab username,watch out the @ sign should followd by \\
  164. keytabUserName="hdfs-mycluster\\@ESZ.COM"
  165. # username keytab path
  166. keytabPath="$installPath/conf/hdfs.headless.keytab"
  167. # kerberos expire time, the unit is hour
  168. kerberosExpireTime="2"
  169.  
  170. # use sudo or not
  171. sudoEnable="true"
  172.  
  173. # worker tenant auto create
  174. workerTenantAutoCreate="false"

迁移Hive和Hadoop配置(集群部署需要)

注意，将core-site.xml 和 hdfs-site.xml 文件 从hadoop中复制到dolphinscheduler中。

否则可能会遇到 HadoopUtils.createTenateDir 空指针的问题。

  1. # 转发文件
  2. rsync -av /data/hadoop/etc/hadoop/hdfs-site.xml root@dsj-node6:/data/dolphinscheduler/conf/ && \
  3. rsync -av /data/hadoop/etc/hadoop/hdfs-site.xml root@dsj-node7:/data/dolphinscheduler/conf/ && \
  4. rsync -av /data/hadoop/etc/hadoop/core-site.xml root@dsj-node6:/data/dolphinscheduler/conf/ && \
  5. rsync -av /data/hadoop/etc/hadoop/core-site.xml root@dsj-node7:/data/dolphinscheduler/conf/

初始化数据库

        DolphinScheduler 元数据存储在关系型数据库中，目前支持 PostgreSQL 和 MySQL，如果使用 MySQL 则需要手动下载 mysql-connector-java 驱动 (8.0.16) 并移动到 DolphinScheduler 的 lib目录下。下面以 MySQL 为例，说明如何初始化数据库

  1. mysql -uroot -p
  2.  
  3. mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
  4.  
  5. # 修改 {user} 和 {password} 为你希望的用户名和密码
  6. mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'%' IDENTIFIED BY '{password}';
  7. mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{user}'@'localhost' IDENTIFIED BY '{password}';
  8.  
  9. mysql> flush privileges;

        完成上述步骤后，您已经为 DolphinScheduler 创建一个新数据库，现在你可以通过快速的 Shell 脚本来初始化数据库

sh script/create-dolphinscheduler.sh

启动 DolphinScheduler

        使用上面创建的部署用户运行以下命令完成部署，部署后的运行日志将存放在 logs 文件夹内

sh install.sh

注意: 第一次部署的话，可能出现 5 次sh: bin/dolphinscheduler-daemon.sh: No such file or directory相关信息，次为非重要信息直接忽略即可

登录 DolphinScheduler

        浏览器访问地址 http://localhost:12345/dolphinscheduler 即可登录系统UI。默认的用户名和密码是 admin/dolphinscheduler123

启停服务

  1. # 一键停止集群所有服务
  2. sh ./bin/stop-all.sh
  3.  
  4. # 一键开启集群所有服务
  5. sh ./bin/start-all.sh
  6.  
  7. # 启停 Master
  8. sh ./bin/dolphinscheduler-daemon.sh stop master-server
  9. sh ./bin/dolphinscheduler-daemon.sh start master-server
  10.  
  11. # 启停 Worker
  12. sh ./bin/dolphinscheduler-daemon.sh start worker-server
  13. sh ./bin/dolphinscheduler-daemon.sh stop worker-server
  14.  
  15. # 启停 Api
  16. sh ./bin/dolphinscheduler-daemon.sh start api-server
  17. sh ./bin/dolphinscheduler-daemon.sh stop api-server
  18.  
  19. # 启停 Logger
  20. sh ./bin/dolphinscheduler-daemon.sh start logger-server
  21. sh ./bin/dolphinscheduler-daemon.sh stop logger-server
  22.  
  23. # 启停 Alert
  24. sh ./bin/dolphinscheduler-daemon.sh start alert-server
  25. sh ./bin/dolphinscheduler-daemon.sh stop alert-server
  26.  
  27. # 启停 Python Gateway
  28. sh ./bin/dolphinscheduler-daemon.sh start python-gateway-server
  29. sh ./bin/dolphinscheduler-daemon.sh stop python-gateway-server

本文内容由网友自发贡献，转载请注明出处：https://www.wpsshop.cn/w/很楠不爱3/article/detail/405750
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号