赞
踩
docker 安装
狂神说docker(最全笔记)_狂神说docker笔记_烟霞畔的博客-CSDN博客
阿里云镜像加速地址 docker 下载镜像加速
- sudo mkdir -p /etc/docker
- sudo tee /etc/docker/daemon.json <<-'EOF'
- {
- "registry-mirrors": ["https://80ifgx2s.mirror.aliyuncs.com"]
- }
- EOF
- sudo systemctl daemon-reload
- sudo systemctl restart docker
下面这个网址有hadoop hbase phoenix 等等的安装包资源
在此页面搜索hadoop 或hbase 或 phoenix可以找到相关链接
如果对phoenix的sql语法有疑问,可以进入官网查找语法说明
1.1创建容器
docker pull centos:8
- docker images
- docker run -itd --name=hadoop --privileged centos:8 /usr/sbin/init
- docker exec -it hadoop bash
1.2 解决容器无法下载vim 等工具的问题。创建一个有java ssh 基本环境的镜像。参考链接:【已解决】Error: Failed to download metadata for repo ‘appstream‘: Cannot prepare internal mirrorlist_ReadThroughLife的博客-CSDN博客
- cd /etc/yum.repos.d/
- sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
- sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-*
- yum makecache
运维(18) 解决Docker容器内无法访问外网问题_容器内无法访问外部ip-CSDN博客
yum update -y
- yum -y install vim net-tools
- #java 路径/usr 下,这在后面的hadoop hbase集群要配置
- yum install -y java-1.8.0-openjdk-devel openssh-clients openssh-server rsync
- systemctl enable sshd && systemctl start sshd
- java -version
- netstat -lnpt
- exit
- #配置了一个有java ssh vim 等工具的容器,生成镜像使用
- docker stop hadoop
- docker commit hadoop java_ssh
- docker network create --driver=bridge hadoop
- docker network ls
配置docker 网卡后,让docker 容器之间的通信通过网卡连接,这样做部署hadoop集群会比较简单。
docker run -d --name=hadoop_single --privileged java_ssh /usr/sbin/init
- #复制集群安装包到容器
- docker cp /opt/software/hadoop-3.1.3.tar.gz hadoop_single:/root
- docker cp /opt/software/apache-zookeeper-3.5.7-bin.tar.gz hadoop_single:/root
- docker cp /opt/software/hbase-2.2.6-bin.tar.gz hadoop_single:/root
- docker exec -it hadoop_single bash
- cd /root
- tar -zvxf hadoop-3.1.3.tar.gz
- tar -zvxf apache-zookeeper-3.5.7-bin.tar.gz
- tar -zvxf hbase-2.2.6-bin.tar.gz
-
- mv hadoop-3.1.3 /usr/local/hadoop-3.1.3
- mv apache-zookeeper-3.5.7-bin /usr/local/zookeeper-3.5.7
- mv hbase-2.2.6 /usr/local/hbase-2.2.6
-
- #配置hadoop环境变量
- echo "export HADOOP_HOME=/usr/local/hadoop-3.1.3" >> /etc/profile
- echo "export ZOOKEEPER_HOME=/usr/local/zookeeper-3.5.7" >> /etc/profile
- echo "export HBASE_HOME=/usr/local/hbase-2.2.6" >> /etc/profile
- echo "export PHOENIX_HOME=/usr/local/phoenix" >> /etc/profile
- source /etc/profile
- echo "export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$PHOENIX_HOME/bin" >> /etc/profile
-
- source /etc/profile #自己新增的,否则后面配置bashrc文件中的hadoop变量会不生效
-
-
- echo "export HADOOP_HOME=/usr/local/hadoop-3.1.3" >> /etc/bashrc
- echo "export ZOOKEEPER_HOME=/usr/local/zookeeper-3.5.7" >> /etc/bashrc
- echo "export HBASE_HOME=/usr/local/hbase-2.2.6" >> /etc/bashrc
- echo "export PHOENIX_HOME=/usr/local/phoenix" >> /etc/bashrc
- echo "export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$PHOENIX_HOME/bin" >> /etc/bashrc
-
-
- #退出容器后bashrc配置生效
- echo "export JAVA_HOME=/usr" >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh
- echo "export HADOOP_HOME=/usr/local/hadoop-3.1.3" >> $HADOOP_HOME/etc/hadoop/hadoop-env.sh
- #ssh 免密登陆
- cd ~
- ssh-keygen -t rsa -P ""
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
4.1 在目录 /usr/local/hadoop/etc/hadoop 下
- cd /usr/local/hadoop-3.1.3/etc/hadoop/
- vim hadoop-env.sh
修改 hadoop-env.sh 文件,在文件末尾添加一下信息
- export HDFS_NAMENODE_USER=root
- export HDFS_DATANODE_USER=root
- export HDFS_SECONDARYNAMENODE_USER=root
- export YARN_RESOURCEMANAGER_USER=root
- export YARN_NODEMANAGER_USER=root
4.2 修改 core-site.xml
- cd /usr/local/hadoop-3.1.3/etc/hadoop/
- vim core-site.xml
修改为
- <configuration>
- <!-- 指定NameNode的地址 -->
- <property>
- <name>fs.defaultFS</name>
- <value>hdfs://hadoop01:8020</value>
- </property>
- <!-- 指定hadoop数据的存储目录 -->
- <property>
- <name>hadoop.tmp.dir</name>
- <value>/home/hadoop3/hadoop/tmp</value>
- </property>
- </configuration>
4.3修改 hdfs-site.xml
vim hdfs-site.xml
修改为
<configuration> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/home/hadoop3/hadoop/hdfs/name</value> </property> <property> <name>dfs.namenode.data.dir</name> <value>/home/hadoop3/hadoop/hdfs/data</value> </property> <!-- nn web端访问地址--> <property> <name>dfs.namenode.http-address</name> <value>hadoop01:9870</value> </property> <!-- 2nn web端访问地址--> <property> <name>dfs.namenode.secondary.http-address</name> <value>hadoop03:9868</value> </property> </configuration>
4.4 修改 mapred-site.xml
vim mapred-site.xml
修改为
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.application.classpath</name> <value> /usr/local/hadoop-3.3.1/etc/hadoop, /usr/local/hadoop-3.3.1/share/hadoop/common/*, /usr/local/hadoop-3.3.1/share/hadoop/common/lib/*, /usr/local/hadoop-3.3.1/share/hadoop/hdfs/*, /usr/local/hadoop-3.3.1/share/hadoop/hdfs/lib/*, /usr/local/hadoop-3.3.1/share/hadoop/mapreduce/*, /usr/local/hadoop-3.3.1/share/hadoop/mapreduce/lib/*, /usr/local/hadoop-3.3.1/share/hadoop/yarn/*, /usr/local/hadoop-3.3.1/share/hadoop/yarn/lib/* </value> </property> </configuration>
4.5 修改 yarn-site.xml
vim yarn-site.xml
修改为
- <configuration>
- <property>
- <name>yarn.resourcemanager.hostname</name>
- <value>hadoop02</value>
- </property>
- <property>
- <name>yarn.nodemanager.aux-services</name>
- <value>mapreduce_shuffle</value>
- </property>
- </configuration>
4.6 修改 workers 文件
#之前这里写错了,写到新的works文件里面,导致启动启动hdfs总只有一个datanode节点,而不是三个,因为hadoop读workers内容只有一个localhost#
vim workers
删除localhost,添加三个节点的名称
- hadoop01
- hadoop02
- hadoop03
- docker commit -m "haddop" -a "hadoop" hadoop_single newhadoop
- docker images
- docker stop hadoop_single
启动 3 个终端,分别执行这几个命令
第一条命令启动的是 h01 是做 master 节点的,所以暴露了端口,以供访问 web 页面
- docker run -itd --network hadoop -h "hadoop01" --name "hadoop01" -p 9870:9870 -p 8088:8088 -p 2181:2181 --privileged newhadoop /sbin/init
-
- docker exec -it hadoop01 bash
创建其他两个容器启动
- docker run -itd --network hadoop -h "hadoop02" --name "hadoop02" --privileged newhadoop /sbin/init
- docker exec -it hadoop02 bash
- docker run -itd --network hadoop -h "hadoop03" --name "hadoop03" --privileged newhadoop /sbin/init
- docker exec -it hadoop03 bash
或者开放所有端口,直接在创建容器的时候设置ip映射关系,但要保证映射关系正确
- #hadoop01 这里默认hadoop01容器ip是172.18.0.2,hadoop02是172.18.0.3,hadoop03是172.18.0.4,真实情况ip可能不同
-
- docker run -itd --network hadoop --add-host='hadoop02:172.18.0.3' --add-host='hadoop03:172.18.0.4' -h "hadoop01" --name "hadoop01" -p 9870:9870 -p 2181:2181 -p 16100:16100 -p 16110:16110 -p 16120:16120 -p 16130:16130 --privileged newhadoop /sbin/init
-
- docker run -itd --network hadoop --add-host='hadoop01:172.18.0.2' --add-host='hadoop03:172.18.0.4' -h "hadoop02" --name "hadoop02" -p 8088:8088 -p 16101:16101 -p 16111:16111 -p 16121:16121 -p 16131:16131 --privileged newhadoop /sbin/init
-
- docker run -itd --network hadoop --add-host='hadoop01:172.18.0.2' --add-host='hadoop02:172.18.0.3' -h "hadoop03" --name "hadoop03" -p 16102:16102 -p 16112:16112 -p 16122:16122 -p 16132:16132 --privileged newhadoop /sbin/init
我的hadoop01容器的ip和映射关系如图,其他两个容器的host文件和hadoop01的内容一样,但顺序可以不一样。
- #hadoop01 文件格式化
- hadoop namenode -format
- ssh hadoop01
- start-dfs.sh
- ssh hadoop02
- start-yarn.sh
停止脚本
- ssh hadoop02
- stop-yarn.sh
- ssh hadoop01
- stop-dfs.sh
hadoop各个节点和服务规划部署如下图
实际启动后服务如下图,和规划的一样,启动成功。(jpsall脚本看第12点)
可以使用命令 ./hadoop dfsadmin -report 可查看分布式文件系统的状态
- cd /usr/local/zookeeper-3.5.7/conf/
- cp zoo_sample.cfg zoo.cfg
- vim zoo.cfg
在末尾追加
- server.1=hadoop01:2888:3888
- server.2=hadoop02:2888:3888
- server.3=hadoop03:2888:3888
修改
dataDir=/usr/local/zookeeper-3.5.7/data
- #进入hadoop01容器 在相关目录下输入
- ssh hadoop01
- cd /usr/local/zookeeper-3.5.7/
- mkdir data
- cd data
- touch myid
- echo 1 >>myid
- cat myid
-
- #进入hadoop02容器 在相关目录下输入
- ssh hadoop02
- cd /usr/local/zookeeper-3.5.7/
- mkdir data
- cd data
- touch myid
- echo 2 >>myid
- cat myid
-
- #进入hadoop03容器 在相关目录下输入
- ssh hadoop03
- cd /usr/local/zookeeper-3.5.7/
- mkdir data
- cd data
- touch myid
- echo 3 >>myid
- cat myid
启动脚本
- #三个节点都要启动
- zkServer.sh start
- #查看状态
- zkServer.sh status
进入 /usr/local/hbase/conf/hbase-env
文件,将最后一行不允许注释
修改 hbase-env.sh,追加
- echo "export JAVA_HOME=/usr" >> /usr/local/hbase-2.2.6/conf/hbase-env.sh
- echo "export HBASE_MANAGES_ZK=false" >> /usr/local/hbase-2.2.6/conf/hbase-env.sh
-
修改 hbase-site.xml 为
<property> <name>hbase.rootdir</name> <value>hdfs://hadoop01:8020/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>hadoop01,hadoop02,hadoop03</value> </property> <property> <name>hbase.regionserver.wal.codec</name> <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value> </property> <property> <name>hbase.unsafe.stream.capability.enforce</name> <value>false</value> </property>
修改 regionservers 文件为
- hadoop01
- hadoop02
- hadoop03
hbase 安装要记得 去除log4j的jar包和hadoop的冲突
mv /usr/local/hbase-2.2.6/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar /usr/local/hbase-2.2.6/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar.bak
其次要解决docker安装的端口映射问题
参考链接 : 【HBase之轨迹】(1)使用 Docker 搭建 HBase 集群_docker hbase集群_寒冰小澈IceClean的博客-CSDN博客
hbase-site.xml 文件要添加如下,其中每个节点的各个value值都要修改,修改的值参考上面的链接
<!-- master 连接端口 --> <property> <name>hbase.master.port</name> <value>16100</value> </property> <!-- master web 界面端口 --> <property> <name>hbase.master.info.port</name> <value>16110</value> </property> <!-- regionserver 连接端口 --> <property> <name>hbase.regionserver.port</name> <value>16120</value> </property> <!-- regionserver web 界面端口 --> <property> <name>hbase.regionserver.info.port</name> <value>16130</value> </property>
打开防火墙映射
启动 start-hbase.sh
tar -zxvf phoenix-hbase-2.2-5.1.3-bin.tar.gz -C /usr/local/phoenix
cp /usr/local/phoenix/phoenix-server-hbase-2.2-5.1.3.jar /usr/local/hbase-2.2.6/lib/
- cd /usr/local/hbase-2.2.6/lib
- xsync phoenix-server-hbase-2.2-5.1.3.jar
vim /usr/local/hbase-2.2.6/conf/hbase-site.xml
在hbase-site.xml 文件中添加phoenix相关配置,然后分发给其他节点
- <property>
- <name>phoenix.schema.isNamespaceMappingEnabled</name>
- <value>true</value>
- </property>
- <property>
- <name>phoenix.schema.mapSystemTablesToNamespace</name>
- <value>true</value>
- </property>
xsync /usr/local/hbase-2.2.6/conf/hbase-site.xml
dnf install python2
- #建议软链接
- ln -s /usr/bin/python2.7 /usr/bin/python
参考资料
如何在 CentOS 8 上安装 Python-腾讯云开发者社区-腾讯云
sqlline.py
执行下面的测试sql,验证phoenix数据库可以使用
- !table
- create schema "testPHOENIX";
- create table "testPHOENIX".test(id integer primary key,name varchar);
- upsert into "testPHOENIX".test values(1,'zs');
- select * from "testPHOENIX".test;
- systemctl stop docker.socket
- systemctl stop docker
cd /var/lib/docker/containers
vim hostconfig.json
找到 ExtraHosts 配置项,修改:
<code class="language-plaintext hljs">["hadoop01:172.20.0.2","hadoop02:172.20.0.3","hadoop03:172.20.0.4"]</code>
或者在创建docker 容器的时候使用 --add-host参数 添加hosts
添加hosts例子:
docker run --add-host='hadoop01:172.18.0.3' --add-host='hadoop02:172.18.0.4' --add-host='hadoop03:172.18.0.5' --name hello-docker -it 192.168.0.1:5002/lybgeek/hello-docker:1.0
参考链接 Docker容器迁移到其他服务器的5种方法详解_docker-安全小天地
- #1.将hadoop容器 commit成镜像
- docker commit hadoop image-name
-
- #2.使用“docker save”压缩镜像并将其迁移到新服务器。
- docker save image-name > image-name.tar
-
- #3.在新服务器中,使用“docker load”将压缩镜像文件用于创建新镜像。
- cat image-name.tar | docker load
- cd /usr/local/bin
- vim jpsall
- #!/bin/bash
- for host in hadoop01 hadoop02 hadoop03
- do
- echo =============== $host ===============
- ssh $host jps
- done
chmod 777 jpsall
- cd /usr/local/bin
- vim xsync
- #!/bin/bash
-
- #1. 判断参数个数
- if [ $# -lt 1 ]
- then
- echo Not Enough Arguement!
- exit;
- fi
- #2. 遍历集群所有机器
- for host in hadoop01 hadoop02 hadoop03
- do
- echo ==================== $host ====================
- #3. 遍历所有目录,挨个发送
-
- for file in $@
- do
- #4. 判断文件是否存在
- if [ -e $file ]
- then
- #5. 获取父目录
- pdir=$(cd -P $(dirname $file); pwd)
-
- #6. 获取当前文件的名称
- fname=$(basename $file)
- ssh $host "mkdir -p $pdir"
- rsync -av $pdir/$fname $host:$pdir
- else
- echo $file does not exists!
- fi
- done
- done
chmod 777 xsync
- cd /usr/local/bin
- vim myhadoop.sh
- #!/bin/bash
-
- if [ $# -lt 1 ]
- then
- echo "No Args Input..."
- exit ;
- fi
-
- case $1 in
- "start")
- echo " =================== 启动 hadoop集群 ==================="
-
- echo " --------------- 启动 hdfs ---------------"
- ssh hadoop01 "/usr/local/hadoop-3.1.3/sbin/start-dfs.sh"
- echo " --------------- 启动 yarn ---------------"
- ssh hadoop02 "/usr/local/hadoop-3.1.3/sbin/start-yarn.sh"
- echo " --------------- 启动 historyserver ---------------"
- ssh hadoop01 "/usr/local/hadoop-3.1.3/bin/mapred --daemon start historyserver"
- ;;
- "stop")
- echo " =================== 关闭 hadoop集群 ==================="
-
- echo " --------------- 关闭 historyserver ---------------"
- ssh hadoop01 "/usr/local/hadoop-3.1.3/bin/mapred --daemon stop historyserver"
- echo " --------------- 关闭 yarn ---------------"
- ssh hadoop02 "/usr/local/hadoop-3.1.3/sbin/stop-yarn.sh"
- echo " --------------- 关闭 hdfs ---------------"
- ssh hadoop01 "/usr/local/hadoop-3.1.3/sbin/stop-dfs.sh"
- ;;
- *)
- echo "Input Args Error..."
- ;;
- esac
chmod 777 myhadoop.sh
- cd /usr/local/bin
- vim zk.sh
- #!/bin/bash
-
- if [ $# -lt 1 ]
- then
- echo "No Args Input..."
- exit ;
- fi
-
- case $1 in
- "start")
- echo " =================== 启动 zookeeper集群 ==================="
- echo " --------------- hadoop01 ---------------"
- ssh hadoop01 "/usr/local/zookeeper-3.5.7/bin/zkServer.sh start"
- echo " --------------- hadoop02 ---------------"
- ssh hadoop02 "/usr/local/zookeeper-3.5.7/bin/zkServer.sh start"
- echo " --------------- hadoop03 ---------------"
- ssh hadoop03 "/usr/local/zookeeper-3.5.7/bin/zkServer.sh start"
- ;;
- "stop")
- echo " =================== 关闭 zookeeper集群 ==================="
- echo " --------------- hadoop01 ---------------"
- ssh hadoop01 "/usr/local/zookeeper-3.5.7/bin/zkServer.sh stop"
- echo " --------------- hadoop02 ---------------"
- ssh hadoop02 "/usr/local/zookeeper-3.5.7/bin/zkServer.sh stop"
- echo " --------------- hadoop03 ---------------"
- ssh hadoop03 "/usr/local/zookeeper-3.5.7/bin/zkServer.sh stop"
- ;;
- "status")
- echo " =================== 查看 zookeeper集群状态 ==================="
- echo " --------------- hadoop01 ---------------"
- ssh hadoop01 "/usr/local/zookeeper-3.5.7/bin/zkServer.sh status"
- echo " --------------- hadoop02 ---------------"
- ssh hadoop02 "/usr/local/zookeeper-3.5.7/bin/zkServer.sh status"
- echo " --------------- hadoop03 ---------------"
- ssh hadoop03 "/usr/local/zookeeper-3.5.7/bin/zkServer.sh status"
- ;;
- *)
- echo "Input Args Error..."
- ;;
- esac
chmod 777 zk.sh
我将以上操作自己本地打包成3个docker镜像压缩包,开箱即用。分享到夸克网盘中,普通用户无法一次保存,可以分批存储下载。
链接:https://pan.quark.cn/s/a00c2b0aaa0d
提取码:APSp
Copyright © 2003-2013 www.wpsshop.cn 版权所有,并保留所有权利。