devbox
盐析白兔
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

【Ambari】Docker 安装Ambari 大数据单机版本

作者：盐析白兔 | 2024-05-30 14:55:27

赞

踩

【Ambari】Docker 安装Ambari 大数据单机版本

目录

一、前期准备

1.1 部署 docker

1.2 部署 docker-compose

1.3 版本说明

二 、镜像构建启动

2.1 系统镜像构建

2.2 安装包源镜像构建

2.3 kdc镜像构建

2.4 集群安装

2.5 容器导出为镜像

三、Ubuntu环境安装测试

3.1 环境准备

3.2 集群容器启动

一、前期准备

1.1 部署 docker

  1. # 安装yum-config-manager配置工具
  2. yum -y install yum-utils
  3.  
  4. # 建议使用阿里云yum源：（推荐）
  5. #yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
  6. yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
  7.  
  8. # 安装docker-ce版本
  9. yum install -y docker-ce
  10.  
  11. # 启动并开机启动
  12. systemctl enable --now docker
  13. docker --version
  14. docker info

1.2 部署 docker-compose

  1. curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
  2.  
  3. chmod +x /usr/local/bin/docker-compose
  4. docker-compose --version

创建网络

docker network create hadoop-network

1.3 版本说明

组件

版本

操作系统

Centos7.6/Ubuntu20.04.5

ambari

2.7.4

HDP

3.1.4

HDP-GPL

3.1.4

HDP-UTILS

1.1.0.22

JDK

jdk-8u162-linux-x64

如下安装只做测试。

二 、镜像构建启动

2.1 系统镜像构建

  1. # 拉取 Centos 镜像
  2. FROM centos:7.6.1810
  3.  
  4. # 作者信息
  5. LABEL  kangll 
  6.  
  7. # 安装基本工具
  8. RUN yum update -y && yum install -y \
  9.     net-tools \
  10.     vim \
  11.     wget \
  12.     curl \
  13.     lrzsz \ 
  14.     nc \
  15.     openssl \
  16.     sed \
  17.     telnet \
  18.     openssh-server \
  19.     httpd \
  20.     chronyd \
  21.     zip unzip \
  22.     openssh-clients \
  23.     expect \
  24.     krb5-server \
  25.     krb5-libs \ 
  26.     krb5-workstation \
  27.     && yum clean all
  28.  
  29. # JDK 
  30. RUN mkdir -p /usr/java 
  31. ADD ./config/jdk-8u162-linux-x64.tar.gz /usr/java/
  32. ENV JAVA_HOME=/usr/java/jdk1.8.0_162
  33. ENV CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
  34. ENV PATH=$JAVA_HOME/bin:$PATH
  35.  
  36.  
  37. # 添加一个名为myuser的用户
  38. RUN useradd -m winner_spark
  39. # 设置myuser用户的密码，这里设置为mypassword
  40. RUN echo 'winner_spark:123456' | chpasswd
  41. # 设置myuser用户为sudoer
  42. RUN echo 'winner_spark ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
  43.  
  44.  
  45. # SSH
  46. RUN sed -i '/^HostKey/'d /etc/ssh/sshd_config
  47. RUN echo 'HostKey /etc/ssh/ssh_host_rsa_key' >> /etc/ssh/sshd_config
  48. RUN mkdir /var/run/sshd/
  49. RUN echo 'UseDNS no' >> /etc/ssh/sshd_config
  50. RUN sed -i -e '/pam_loginuid.so/d' /etc/pam.d/sshd
  51.  
  52. COPY ./config/hosts    /etc/
  53.  
  54. #set default password
  55. RUN echo 'root:123456' | chpasswd
  56. RUN /usr/bin/ssh-keygen -A
  57.  
  58. COPY ./config/krb5.conf   /etc
  59.  
  60. #声明22端口 
  61. EXPOSE 22
  62.  
  63. RUN mkdir -p /opt && mkdir -p /var/log/supervisord
  64. RUN echo '/usr/sbin/sshd -D' >> /opt/run.sh
  65. RUN chmod +x /opt/run.sh
  66. CMD ["/opt/run.sh"]
  67.  
  68. #
  69. WORKDIR /opt/

系统镜像构建

docker build -f  dockerfile-centos -t centos-jdk:1.0  .

2.2 安装包源镜像构建

dockerfile-httpd

  1. FROM centos-jdk:1.0
  2.  
  3. # centos7.6 JDK1.8
  4. LABEL kangll
  5.  
  6. RUN yum -y install httpd
  7. RUN echo "crushlinux" >/var/www/html/index.html
  8. RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
  9.  
  10.  
  11. # 添加yum 源
  12. COPY ./repo/ambari.repo   /etc/yum.repos.d/
  13. COPY ./repo/hdp.repo      /etc/yum.repos.d/
  14. COPY ./repo/hdp.gpl.repo  /etc/yum.repos.d/
  15.  
  16.  
  17. RUN mkdir -p /hadoop/hdp
  18. ADD ./HDP/ambari.tar.gz    /var/www/html
  19. ADD ./HDP/HDP-GPL.tar.gz   /var/www/html
  20. ADD ./HDP/HDP.tar.gz       /var/www/html
  21. ADD ./HDP/HDP-UTILS.tar.gz /var/www/html
  22.  
  23.  
  24. EXPOSE 80
  25.  
  26. CMD ["httpd","-DFOREGROUND"]

源镜像构建

docker build -t ambari:2.7.4  -f dockerfile-ambari  .

2.3 kdc镜像构建

dockerfile-kerberos

  1. FROM centos:7.6.1810
  2.  
  3. RUN yum install -y krb5-server krb5-libs krb5-auth-dialog krb5-workstation
  4.  
  5. RUN yum install -y net-tools telnet
  6.  
  7. COPY ./config/krb5.conf   /etc
  8. COPY ./config/kdc.conf    /var/kerberos/krb5kdc/
  9. COPY ./config/kadm5.acl   /var/kerberos/krb5kdc/
  10.  
  11.  
  12. CMD ["/usr/sbin/init"]

服务启动

  1. # 进入容器 启动服务
  2. Systemctl restart krb5kdc
  3. Systemctl restart kadmin

docker-compose-base.yml 启动基础环境

  1. version: '3'
  2. services:
  3.   mysql:
  4.     image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7
  5.     restart: always
  6.     networks:
  7.       - hadoop-network
  8.     container_name: mysql
  9.     hostname: mysql-test
  10.     environment:
  11.       MYSQL_ROOT_PASSWORD: Winner001
  12.       TZ: Asia/Shanghai
  13.     ports:
  14.       - 3306:3306
  15.     volumes:
  16.       -  /data:/var/lib/mysql
  17.       - ./conf/my.cnf:/etc/mysql/my.cnf
  18.     command:
  19.       --max_connections=1000
  20.       --character-set-server=utf8mb4
  21.       --collation-server=utf8mb4_general_ci
  22.       --default-authentication-plugin=mysql_native_password
  23.     healthcheck:
  24.       test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"]
  25.       interval: 10s
  26.       timeout: 10s
  27.   httpd:
  28.     container_name: httpd
  29.     hostname: httpd
  30.     image: httpd:1.0
  31.     networks:
  32.       - hadoop-network
  33.     ports:
  34.       - "80:80"
  35.     privileged: true
  36.     restart: always
  37.     volumes:
  38.       - "/etc/hosts:/etc/hosts"
  39.   kdc:
  40.     image: kerberos:1.0
  41.     restart: always
  42.     networks:
  43.       - hadoop-network
  44.     container_name: kerberos
  45.     hostname: kerberos.winner
  46.     ports:
  47.       - 88:88
  48.       - 749:749
  49.       - 750:750
  50.     volumes:
  51.       - /etc/hosts:/etc/hosts
  52.       - /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc
  53.       - /hadoop/kerberos/keytabs:/etc/security/keytabs
  54.     command:
  55.       - /bin/sh
  56.       - -c
  57.       - |
  58.         /opt/apache/kerberosStart.sh
  59.  
  60. networks:
  61.   hadoop-network:
  62.     external:
  63.       name: hadoop-network

Ambari启动

  1. # 启动基础环境
  2. docker-compose -f docker-compose-base.yaml up -d
  3.  
  4. # 启动ambari , 在此容器中安装集群
  5. docker-compose -f docker-compose-ambari.yaml up -d

2.4 集群安装

repo地址

节点和秘钥

agent安装

服务安装

由于是单机直接下一步

安装组件

输入密码

数据目录确认

用户确认下一步

服务参数确定

发布

安装完成

2.5 容器导出为像

  1.  
  2.  
  3. docker ps -a
  4. docker commit  容器ID ambari-1.0.tar.gz

三、Ubuntu环境安装测试

3.1 环境准备

我们准备Ubuntu 环境，在Ubuntu环境安装Ambari单机，相当于在Centos服务器安装的docker化的Ambari 导出为镜像 然后在Ubutntu 环境尝试启动。 

准备Ubuntu 环境，版本：Ubuntu 20.04.5 LTS

将镜像 ambari-1.0.tar.gz 加载到Ubuntu 环境

docker load -i ambari-1.0.tar.gz

MySQL元数据库备份还原（Ambari、hive、ranger）

HDFS、kerberos 备份的数据拷贝到Ubuntu服务器并保持数据文件路径不变。

将在centos 安装完成的集群备份元数据库写入到新的库中 , 也就是Ubuntu环境要使用的元数据库。

  1. # 备份的元数据写入新的数据库中
  2. mysql -hmysql-test -uroot -pWinner001 ambari < ambari.sql
  3. mysql -hmysql-test -uroot -pWinner001 ranger < ranger.sql
  4. mysql -hmysql-test -uroot -pWinner001 hive   < hive.sql

备份的kerberos和安装包解压

  1. # 解压到 /usr 路径下，保存的是集群安装的数据
  2. hdp.tar.gz
  3.  
  4. # 在根路径解压保留的kerberos 和HDFS 初始化的元数据
  5. hadoop.tar.gz

基础环境启动docker-compose-base.yaml

  1. version: '3'
  2. services:
  3.   mysql:
  4.     image: registry.cn-hangzhou.aliyuncs.com/bigdata_cloudnative/mysql-test:5.7
  5.     restart: always
  6.     networks:
  7.       - hadoop-network
  8.     container_name: mysql
  9.     hostname: mysql-test
  10.     environment:
  11.       MYSQL_ROOT_PASSWORD: Winner001
  12.       TZ: Asia/Shanghai
  13.     ports:
  14.       - 3306:3306
  15.     volumes:
  16.       -  /data:/var/lib/mysql
  17.       - ./conf/my.cnf:/etc/mysql/my.cnf
  18.     command:
  19.       --max_connections=1000
  20.       --character-set-server=utf8mb4
  21.       --collation-server=utf8mb4_general_ci
  22.       --default-authentication-plugin=mysql_native_password
  23.     healthcheck:
  24.       test: ["CMD-SHELL", "curl --silent localhost:3306 >/dev/null || exit 1"]
  25.       interval: 10s
  26.       timeout: 10s
  27.   httpd:
  28.     container_name: httpd
  29.     hostname: httpd
  30.     privileged: true
  31.     image: httpd:1.0
  32.     networks:
  33.       - hadoop-network
  34.     ports:
  35.       - "80:80"
  36.     privileged: true
  37.     restart: always
  38.     volumes:
  39.       - "/etc/hosts:/etc/hosts"
  40.   kdc:
  41.     image: kerberos:1.0
  42.     restart: always
  43.     privileged: true
  44.     networks:
  45.       - hadoop-network
  46.     container_name: kerberos
  47.     hostname: kerberos.winner
  48.     ports:
  49.       - 888:88
  50.       - 8749:749
  51.       - 8750:750
  52.     volumes:
  53.       - /etc/hosts:/etc/hosts
  54.       - /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc
  55.       - /hadoop/kerberos/keytabs:/etc/security/keytabs
  56.  
  57. networks:
  58.   hadoop-network:
  59.     external:
  60.       name: hadoop-network

启动

  1. # 启动
  2. docker-compose -f docker-compose-base.yaml up -d
  3.  
  4. # 停止
  5. docker-compose -f docker-compose-base.yaml down

3.2 集群容器启动

镜像

启动容器

  1. version: '3'
  2. services:
  3.   ambari:
  4.     image: ambari:2.7.4  # 修改为新的容器
  5.     container_name: ambari
  6.     hostname: ambari-server
  7.     restart: always
  8.     privileged: true
  9.     networks:
  10.       - hadoop-network
  11.     ports:
  12.       - 8080:8080
  13.       - 8088:8088
  14.       - 50070:50070
  15.       - 50075:50075
  16.       - 8188:8188
  17.       - 8042:8042
  18.       - 10000:10000
  19.       - 9083:9083
  20.       - 6080:6080
  21.       - 16010:16010
  22.     volumes:
  23.       - /etc/hosts:/etc/hosts
  24.       - /usr/hdp:/usr/hdp
  25.       - /hadoop/hdfs:/hadoop/hdfs
  26.       - /hadoop/etc/hadoop/:/etc/hadoop/
  27.       - /hadoop/etc/hbase/:/etc/hbase/
  28.       - /hadoop/etc/hive/:/etc/hive/
  29.       - /hadoop/etc/livy2/:/etc/livy2/
  30.       - /hadoop/etc/spark2/:/etc/spark2/
  31.       - /hadoop/etc/tez/:/etc/tez/
  32.       - /hadoop/etc/zookeeper/:/etc/zookeeper/
  33.       - /hadoop/yarn:/hadoop/yarn
  34.       - /hadoop/kerberos/krb5kdc:/var/kerberos/krb5kdc
  35.       - /hadoop/kerberos/keytabs:/etc/security/keytabs
  36.     command:
  37.       - /bin/sh
  38.       - -c
  39.       - |
  40.         /opt/apache/bootstrap.sh
  41. networks:
  42.   hadoop-network:
  43.     external:
  44.       name: hadoop-network

启动

docker-compose -f docker-compose-ambari.yaml up -d

下次安装直接加载镜像，然后执行yaml文件即可

  1. docker load -i ambari-1.0.tar.gz
  2. docker load -i kerberos.tar.gz
  3. docker load -i mysql.tar.gz
  4. docker load -i HDP-REPO.tar.gz

脚本

bootstrap.sh

  1. #!/bin/bash
  2.  
  3. set -x
  4.  
  5. ambari_db=ambari
  6. myurl=mysql-test
  7. myuser=root
  8. mypwd=Winner001
  9. myport=3306
  10. mydb=ipvacloud
  11. java_home=/usr/java/jdk1.8.0_162
  12. mysql_driver=/usr/share/java/mysql-connector-java.jar
  13.  
  14.  
  15. BASEDIR=$(cd "$(dirname "$0")"; pwd)
  16.  
  17. wait_for() {
  18.     echo Waiting for $1 to listen on $2...
  19.     while ! nc -z $1 $2; do echo waiting...; sleep 1s; done
  20. }
  21.  
  22. setup_ambari() {
  23.   # 强制替换文件
  24.   cp -f /tmp/ambari.properties /etc/ambari-server/conf/ 
  25.   cp -f /tmp/password.dat     /etc/ambari-server/conf/
  26.   cp -f /tmp/ambari-agent.ini    /etc/ambari-agent/conf/
  27. }
  28.  
  29.  
  30. # wait for mysql and httpd
  31. #wait_for windp-aio 3306
  32. #wait_for httpd 80
  33.  
  34. # 开始ambari安装
  35. yum -y install ambari-server
  36. yum -y install ambari-agent
  37.  
  38. re=`mysql -h${myurl} -u${myuser} -p${mypwd} -e "show databases"|grep ambari|wc -l`
  39. if [ $re -eq 0 ];then
  40.    # sql 初始化
  41.    mysql -h${myurl} -u${myuser} -p${mypwd} < /tmp/init_db.sql 
  42.    mysql -h${myurl} -u${myuser} -p${mypwd} ambari  < /tmp/Ambari-DDL-MySQL-CREATE.sql
  43. fi
  44.  
  45. # 配置
  46. setup_ambari
  47.  
  48. # 启动ambari服务
  49. /usr/sbin/ambari-server start
  50. /usr/sbin/ambari-agent start
  51.  
  52. /usr/sbin/sshd -D
  53.  
  54. tail -f /var/log/ambari-server/ambari-server.log

参考文章：https://www.cnblogs.com/liugp/p/17489616.html

Ubuntu 环境安装docker: 【云计算】Ubuntu多种安装docker方式_ubuntu安装docker-CSDN博客

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/盐析白兔/article/detail/647434
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号