devbox
酷酷是懒虫
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

StreamPark集成k8s运行Flink实战_streampark配置flink集群

作者：酷酷是懒虫 | 2024-07-21 06:41:38

赞

踩

streampark配置flink集群

集群配置(Centos7)

node1(4vCpu,16G)node2(4vCpu,16G)node3(4vCpu,16G)
dockerdockerdocker
k8s-masterk8s-node1k8s-node2
java-8
mysql-57
maven-3.9.6(可选)
flink-1.16.3
streampark-2.1.3_2.12

Docker安装(三台机器)

  1. #卸载自带docker
  2. sudo yum remove docker*
  3. #安装工具类
  4. sudo yum install -y yum-utils
  5.  
  6. #配置docker的yum地址
  7. sudo yum-config-manager \
  8. --add-repo \
  9. http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
  10.  
  11.  
  12. #安装docker
  13. sudo yum install -y docker-ce docker-ce-cli containerd.io
  14.  
  15. #	启动&开机启动docker
  16. systemctl enable docker --now
  17.  
  18. # docker加速配置
  19. sudo mkdir -p /etc/docker
  20. sudo tee /etc/docker/daemon.json <<-'EOF'
  21. {
  22.   "registry-mirrors": ["https://82m9ar63.mirror.aliyuncs.com"],
  23.   "exec-opts": ["native.cgroupdriver=systemd"],
  24.   "log-driver": "json-file",
  25.   "log-opts": {
  26.     "max-size": "100m"
  27.   },
  28.   "storage-driver": "overlay2"
  29. }
  30. EOF
  31. sudo systemctl daemon-reload
  32. sudo systemctl restart docker

安装完成后可以使用docker ps 查看docker运行状态

docker ps

k8s安装

基本环境配置(三台机器)：

确保每个机器使用内网ip互通

  1. #设置每个机器自己的hostname
  2. hostnamectl set-hostname xxx(更换为自己的主机名)
  3.  
  4. # 将 SELinux 设置为 permissive 模式（相当于将其禁用）
  5. sudo setenforce 0
  6. sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
  7.  
  8. #关闭swap
  9. swapoff -a  
  10. sed -ri 's/.*swap.*/#&/' /etc/fstab
  11.  
  12. #允许 iptables 检查桥接流量
  13. cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
  14. br_netfilter
  15. EOF
  16.  
  17. cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
  18. net.bridge.bridge-nf-call-ip6tables = 1
  19. net.bridge.bridge-nf-call-iptables = 1
  20. EOF
  21. sudo sysctl --system

安装kubelet、kubeadm、kubectl(三台机器)

  1. #配置k8s的yum源地址
  2. cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
  3. [kubernetes]
  4. name=Kubernetes
  5. baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
  6. enabled=1
  7. gpgcheck=0
  8. repo_gpgcheck=0
  9. gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
  10.    http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  11. EOF
  12.  
  13.  
  14. #安装 kubelet，kubeadm，kubectl
  15. sudo yum install -y kubelet-1.20.9 kubeadm-1.20.9 kubectl-1.20.9
  16.  
  17. #启动kubelet
  18. sudo systemctl enable --now kubelet
  19.  
  20. #所有机器配置master域名
  21. echo "<你的k8s-master主机ip>  k8s-master" >> /etc/hosts
  23. #ping一下检测连通性
  24. ping k8s-master

初始化master节点(k8s-master机器)

  1. kubeadm init \
  2. --apiserver-advertise-address=<你的k8s-master主机ip> \
  3. --control-plane-endpoint=k8s-master \
  4. --image-repository registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images \
  5. --kubernetes-version v1.20.9 \
  6. --service-cidr=10.96.0.0/16 \
  7. --pod-network-cidr=192.168.0.0/16

记录打印信息并执行3-5行(k8s-master机器)

  1. Your Kubernetes control-plane has initialized successfully!
  2.  
  3. To start using your cluster, you need to run the following as a regular user:
  4.  
  5.   mkdir -p $HOME/.kube
  6.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  7.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  8.  
  9. Alternatively, if you are the root user, you can run:
  10.  
  11.   export KUBECONFIG=/etc/kubernetes/admin.conf
  12.  
  13. You should now deploy a pod network to the cluster.
  14. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  15.   https://kubernetes.io/docs/concepts/cluster-administration/addons/
  16.  
  17. You can now join any number of control-plane nodes by copying certificate authorities
  18. and service account keys on each node and then running the following as root:
  19.  
  20.   kubeadm join k8s-master:6443 --token 3d05gm.ayjl70fgtg4egst2 \
  21.     --discovery-token-ca-cert-hash sha256:a3d6e581c493d2e02ca2e4736eeb755af671a9df26095c700bc288a514800b39 \
  22.     --control-plane 
  23.  
  24. Then you can join any number of worker nodes by running the following on each as root:
  25.  
  26. kubeadm join k8s-master:6443 --token 3d05gm.ayjl70fgtg4egst2 \
  27.     --discovery-token-ca-cert-hash sha256:a3d6e581c493d2e02ca2e4736eeb755af671a9df26095c700bc288a514800b39

安装Calico网络插件(注意版本问题:使用v3.20)(k8s-master机器)

  1. curl https://docs.projectcalico.org/v3.20/manifests/calico.yaml -O
  2. kubectl apply -f calico.yaml
  3.  
  4. #检测安装情况(都在runing状态再向下进行)
  5. kubectl get pod -A

加入worker节点(复制打印信息中的内容)(k8s-worker机器)

  1. kubeadm join k8s-master:6443 --token 3d05gm.ayjl70fgtg4egst2 \
  2.     --discovery-token-ca-cert-hash sha256:a3d6e581c493d2e02ca2e4736eeb755af671a9df26095c700bc288a514800b39

检测节点运行情况(k8s-master机器)

  1. #都在readly状态证明安装完成
  2. kubectl get node 

java安装

上传jdk-8u401-linux-x64.tar.gz到/opt/software目录下

  1. #解压
  2. tar -zxvf jdk-8u401-linux-x64.tar.gz -C /opt/module
  3.  
  4. #创建环境变量
  5. vim /etc/profile.d/myenv.sh 
  6.  
  7. #设置JAVA_HOME
  8. #JAVA_HOME
  9. export JAVA_HOME=/opt/module/jdk1.8.0_401
  10. export PATH=${JAVA_HOME}/bin:$PATH
  11.  
  12. #刷新生效
  13. source /etc/profile.d/myenv.sh 
  14.  
  15. #检验
  16. java -version

mysql安装

  1. #检查是否安装过MySQL
  2. rpm -qa | grep mysql 
  3. #检查是否存在 mariadb 数据库（内置的MySQL数据库），有则强制删除
  4. rpm -qa | grep mariadb 
  5. #强制删除
  6. rpm -e --nodeps mariadb-libs-5.5.68-1.el7.x86_64 
  7. #强制删除
  8. rpm -e --nodeps mariadb-5.5.68-1.el7.x86_64 
  9.  
  10.  
  11. #下载mysql源
  12. curl -O https://repo.mysql.com//mysql57-community-release-el7-11.noarch.rpm
  13.  
  14. #安装MySQL源
  15. yum localinstall mysql57-community-release-el7-11.noarch.rpm
  16.  
  17. #安装密钥
  18. rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
  19.  
  20. #安装mysql
  21. yum install mysql-community-server
  22.  
  23. #启动mysql
  24. systemctl start mysqld
  25.  
  26. #获取默认密码
  27. grep 'temporary password' /var/log/mysqld.log
  28.  
  29. #登录mysql
  30. mysql -uroot -p显示的默认密码
  31.  
  32. #修改密码策略
  33. set global validate_password_policy=0;
  34. set global validate_password_length=1;
  35.  
  36. #修改密码
  37. ALTER USER 'root'@'localhost' IDENTIFIED BY '新密码';
  38.  
  39. #授予root用户远程访问权限，并刷新权限使生效(可选)
  40. grant all privileges on *.* to 'root' @'%' identified by '新密码';

maven安装(用于依赖项下载，可选)

  1. #下载安装包
  2. curl -O https://archive.apache.org/dist/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz
  3.  
  4. #解压
  5. tar -zxvf apache-maven-3.9.6-bin.tar.gz -C /opt/module

flink安装

  1. #下载安装包
  2. curl -O https://archive.apache.org/dist/flink/flink-1.16.3/flink-1.16.3-bin-scala_2.12.tgz
  3.  
  4. #解压
  5. tar -zxvf flink-1.16.3-bin-scala_2.12.tgz -C /opt/module

Streampark安装

官网手册：安装部署 | Apache StreamPark (incubating)

上传apache-streampark_2.12-2.1.3-incubating-bin.tar.gz到/opt/software目录下

  1. #解压
  2. tar -zxvf apache-streampark_2.12-2.1.3-incubating-bin.tar.gz -C /opt/module
  3.  
  4. #初始化表结构、登录mysql
  5. source streampark安装位置/script/schema/mysql-schema.sql
  6. source streampark安装位置/script/data/mysql-data.sql
  7.  
  8.  
  9. #修改连接信息,进入到 conf 下，修改 conf/application.yml,找到 spring 这一项，找到 profiles.active 的配置，修改成mysql即可
  10.  
  11. spring:
  12.   profiles.active: mysql #[h2,pgsql,mysql]
  13.  
  14.  
  15. #修改 config/application-mysql.yml
  16. spring:
  17.   datasource:
  18.     username: root
  19.     password: 你的密码如果是纯数字需要加单引号'000000'
  20.     
    声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/酷酷是懒虫/article/detail/859894
    推荐阅读
    相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。