小蓝xlanll

这个屌丝很懒，什么也没留下！

热门标签

OBD搭建OceanBase三副本集群(1-1-1)docker版

作者：小蓝xlanll | 2024-02-29 20:09:47

踩

1. 目的

本次使用OBD方式搭建一个OceanBase三副本集群，由于服务器资源有限，选择较为轻量级的docker，运行三个容器来实现OceanBase(1-1-1)+OBProxy集群架构的搭建，ODC开发工具客户端远程连接做简单查询。

2. 环境规划

IP	服务器	资源规格	软件及版本	宿主机:容器端口映射	宿主机:容器目录映射
10.0.0.11	docker01	20C15g512G	OceanBase(4.2.1)	1881:2881 1882:2882 1883:2883	/mnt/db/oceanbase/ob01:/opt /sys/fs/cgroup:/sys/fs/cgroup
10.0.0.12	docker02	20C15g512G	OceanBase(4.2.1) OBProxy(4.2.1) OBD(2.5.0)	2881:2881 2882:2882 2883:2883 2884:2884	/mnt/db/oceanbase/ob02:/opt /sys/fs/cgroup:/sys/fs/cgroup
10.0.0.11	docker03	20C15g512G	OceanBase(4.2.1)	3881:2881 3882:2882 3883:2883	/mnt/db/oceanbase/ob03:/opt /sys/fs/cgroup:/sys/fs/cgroup
192.168.0.113	Ubantu (宿主机)	20C64g512G	docker	/	/
192.168.0.21	Windows (客户端)	8C16g512G	ODC(4.2.2) 等开发工具	/	/

3. 搭建步骤

3.1. 初始化服务器

1）docker运行三个CentOS7.6容器：


## docker拉取一个基础镜像：
root@system:~# docker pull vitotp/centos7.6
 
## 基于基础镜像构建一个自己的基础镜像：
root@system:~# vim Dockerfile
root@system:~# cat Dockerfile
FROM vitotp/centos7.6
RUN echo "net.ipv4.ip_forward  = 1" >> /etc/sysctl.conf && yum -y update && yum -y install openssh-server openssh-clients lrzsz sudo && yum -y clean all && echo "root:qwerty"|chpasswd && echo "root ALL=(ALL) ALL" >> /etc/sudoers && mkdir /var/run/sshd 
EXPOSE 22
CMD ["systemctl","start","sshd.server"]
root@system:~# docker build -t myos-el7:1.0 .
root@system:~# docker images
REPOSITORY               TAG         IMAGE ID       CREATED       SIZE
myos-el7                 1.0         6510f68c38d4   3 weeks ago   771MB
vitotp/centos7.6         latest      0429a3daccd0   4 years ago   433MB
 
## 创建ob01容器：
root@system:~# docker run -d -m 15g -p 122:22 -p 1881:2881 -p 1882:2882 -p 1883:2883 -v /mnt/db/oceanbase/ob01:/opt -v /mnt/hub/obadmin01:/home -h OBSERVER01 --name=ob01 --privileged=true 6510f68c38d4 /usr/sbin/init
 
## 创建ob02容器：
root@system:~# docker run -d -m 15g -p 222:22 -p 2881:2881 -p 2882:2882 -p 2883:2883 -p 2884:2884 -p 8080:8080 -v /mnt/db/oceanbase/ob02:/opt -v /mnt/hub/obadmin02:/home -h OBSERVER02 --name=ob02 --privileged=true 6510f68c38d4 /usr/sbin/init
 
## 创建ob03容器：
root@system:~# docker run -d -m 15g -p 322:22 -p 3881:2881 -p 3882:2882 -p 3883:2883 -v /mnt/db/oceanbase/ob03:/opt -v /mnt/hub/obadmin03:/home -h OBSERVER03 --name=ob03 --privileged=true 6510f68c38d4 /usr/sbin/init

2）创建容器桥接网络：


## 创建桥接网卡：
root@system:~# docker network create --subnet 10.0.0.0/24 --gateway 10.0.0.1 innet
 
## 设置固定IP追加到各个容器：
root@system:~# docker network create --subnet 10.0.0.0/24 --gateway 10.0.0.1 innet
root@system:~# docker network connect innet --ip "10.0.0.11" ob01
root@system:~# docker network connect innet --ip "10.0.0.12" ob02
root@system:~# docker network connect innet --ip "10.0.0.13" ob03

docker基础命令可浏览博客：

Docker基础(小白篇)-CSDN博客https://blog.csdn.net/weixin_61894388/article/details/130708198?spm=1001.2014.3001.5501 3）初始化操作系统环境：三个OB容器均执行；


## 添加一个初始化操作系统脚本：
root@system:~# cd /mnt/db/oceanbase
root@system:oceanbase# cat initos.sh
echo '               _
           _ _|_|        ___
         _|_|_|_|_       \  \___
       _|_|_|_|_|_|___    \   __\
      /===============\___/  /
 ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~~/~~ ~~~
      \_____o           ___/
       \   \          __/
         \__\________/
        docker - centos7.6' > /root/logo.txt
echo '#!/bin/bash
echo -e "\033[1;36m$(cat /root/logo.txt)\033[0m\n登录主机: [\033[1;35m$(hostname -I)\033[0m],时间: [\033[1;33m$(date +%F_%T)\033[0m]"
pg(){
    ps -ef | grep "$1" | grep -v grep
}
ns(){
	netstat -unlpt | grep "$1"
}
clockdiff 10.0.0.11
clockdiff 10.0.0.12
clockdiff 10.0.0.13
' > /root/cmd
chmod 775 /root/cmd
echo 'source /root/cmd' >> /root/.bash_profile
source /root/.bash_profile
 
## 关闭透明大页：
echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/enabled
 
## 修改可用最小内存值，小于该值则清理缓存：
echo 3 > /proc/sys/vm/drop_caches
 
## 设置资源限制大小：
echo '
root soft nofile 655350
root hard nofile 655350
* soft nofile 655350
* hard nofile 655350
* soft nproc 655360
* hard nproc 655360
* soft stack unlimited
* hard stack unlimited
* soft core unlimited
* hard core unlimited
' >> /etc/security/limits.conf
ulimit -a
 
## 修改操作系统内核参数：
echo 'fs.aio-max-nr=1048576
net.core.somaxconn = 2048
net.core.netdev_max_backlog = 10000 
net.core.rmem_default = 16777216 
net.core.wmem_default = 16777216 
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216
net.ipv4.ip_local_port_range = 3500 65535 
net.ipv4.conf.default.rp_filter = 1 
net.ipv4.conf.default.accept_source_route = 0 
net.ipv4.tcp_syncookies = 0 
net.ipv4.tcp_rmem = 4096 87380 16777216 
net.ipv4.tcp_wmem = 4096 65536 16777216 
net.ipv4.tcp_max_syn_backlog = 16384 
net.ipv4.tcp_fin_timeout = 15 
net.ipv4.tcp_max_syn_backlog = 16384 
net.ipv4.tcp_tw_reuse = 1 
net.ipv4.tcp_tw_recycle = 1 
net.ipv4.tcp_slow_start_after_idle=0
vm.swappiness = 0
vm.min_free_kbytes = 2097152
fs.file-max = 6573688
vm.max_map_count = 655360
kernel.core_pattern = /opt/core-%e-%p-%t
' >> /etc/sysctl.conf
sysctl -p
 
## 创建操作系统用户：
useradd -g root -m -d /home/admin -s /bin/bash admin
echo 'Admin123' | passwd --stdin admin
id admin
## 配置sodu权限：
echo "admin       ALL=(ALL)       NOPASSWD: ALL" >> /etc/sudoers
 
## 目录规划：
mkdir -p /opt/{data,redolog}
 
## 配置服务域名：
echo '
10.0.0.11       OBSVR01
10.0.0.12       OBSVR02
10.0.0.13       OBSVR03
' >> /etc/hosts
 
## 时钟校正：
clockdiff OBSVR01
clockdiff OBSVR02
clockdiff OBSVR03
 
## 修改环变量：
echo 'export OB_HOME=/home/admin/oceanbase
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$OB_HOME/lib
export PATH=$PATH:$OB_HOME/bin
' >> /home/admin/.bash_profile
 
## 设置免密登陆：
su - admin -c "ssh-keygen -t rsa -N ''"
 
## 各个容器执行初始化：
root@system:oceanbase# docker exec -it ob01 /bin/bash -c "/opt/initos.sh"
root@system:oceanbase# docker exec -it ob02 /bin/bash -c "/opt/initos.sh"
root@system:oceanbase# docker exec -it ob03 /bin/bash -c "/opt/initos.sh"

4）docker集群管理脚本：


## 简化docker命令集：
root@system:~# cd /var/lib/docker/dockercmd
#!/bin/bash
dls(){
    docker images
}
dps(){
    docker ps -a
}
dexec(){
    docker exec -it "$1" /bin/bash -c "sudo -i"
}
drm(){
    docker rm -f "$1"
}
drmi(){
    docker rmi -f "$1"
}
OBCService(){
OBSVRLIST=(ob01 ob02 ob03)
mode="${1:-stats}"
for svr in ${OBSVRLIST[@]};do echo "正在${mode}容器${svr}..."&&echo -e "[\033[1;32mOK\033[0m]: $(docker ${mode} ${svr})容器${mode}成功.";done
}
complete -W "start stop restart stats" OBCService
root@system:~# source /var/lib/docker/dockercmd
 
## 一键重启OceanBase集群：
root@system:~# OBCService restart
正在restart容器ob01...
[OK]: ob01容器restart成功.
正在restart容器ob02...
[OK]: ob02容器restart成功.
正在restart容器ob03...
[OK]: ob03容器restart成功.
 
## 进入ob1容器：
root@system:~# dexec ob01
               _
           _ _|_|        ___
         _|_|_|_|_       \  \___
       _|_|_|_|_|_|___    \   __\
      /===============\___/  /
 ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~~/~~ ~~~
      \_____o           ___/
       \   \          __/
         \__\________/
        docker - centos7.6
登录主机: [128.128.0.3 10.0.0.11 ],时间: [2024-01-20_13:18:54]
.
host=10.0.0.11 rtt=750(187)ms/0ms delta=0ms/0ms Sat Jan 20 13:18:54 2024
.
host=10.0.0.12 rtt=750(187)ms/0ms delta=0ms/0ms Sat Jan 20 13:18:54 2024
.
host=10.0.0.13 rtt=750(187)ms/0ms delta=0ms/0ms Sat Jan 20 13:18:54 2024
[root@OBSERVER01 ~]# vim /etc/security/limits.d/20-nproc.conf
[root@OBSERVER01 ~]# su - admin
Last login: Sat Jan 20 12:25:28 CST 2024 on pts/2
[admin@OBSERVER01 ~]$

执行初始化OS脚本后，注意观察clockdiff时间延时情况，由于我这里是同在一个宿主机里的三个docker容器，所以这三个容器时间是一致的，延时为零；如果是三台物理机，各个节点是需要配置时钟源来让各节点时间保持延时最小，具体配置时钟源方法可查看OceanBase官网文档：

（可选）配置时钟源-OceanBase 数据库 -OceanBase文档中心-分布式数据库使用文档https://www.oceanbase.com/docs/common-oceanbase-database-cn-1000000000220840

3.2. OBD下载安装rpm包

1）下载OB安装包：（在容器ob02执行即可）

服务器如果不能连外网，可在官网下载，上传到服务器进行安装，官网链接：

OceanBase 社区版下载 - 开源数据库下载 - OceanBase 数据库下载中心https://www.oceanbase.com/softwarecenter 服务器如果可以连外网，可以直接下载：


## 进入ob02容器交互：
root@system:~# docker exec -it ob02 /bin/bash 
[root@OBSERVER02 ~]$ cd /opt/obinstall
[root@OBSERVER02 obinstall]$ wget https://obbusiness-private.oss-cn-shanghai.aliyuncs.com/download-center/opensource/observer/4.2.1_CE_BP2/oceanbase-ce-4.2.1.2-102000042023120514.el7.x86_64.rpm
[root@OBSERVER02 obinstall]$ wget https://obbusiness-private.oss-cn-shanghai.aliyuncs.com/download-center/opensource/observer/4.2.1_CE_BP2/oceanbase-ce-libs-4.2.1.2-102000042023120514.el7.x86_64.rpm
[root@OBSERVER02 obinstall]$ wget https://obbusiness-private.oss-cn-shanghai.aliyuncs.com/download-center/opensource/obdeploy/2.5.0/ob-deploy-2.5.0-2.el7.x86_64.rpm
[root@OBSERVER02 obinstall]$ wget https://obbusiness-private.oss-cn-shanghai.aliyuncs.com/download-center/opensource/obproxy/v4.2.1_CE/obproxy-ce-4.2.1.0-11.el7.x86_64.rpm
[root@OBSERVER02 obinstall]$ ll
total 285096
-rw-r--r-- 1 root root  11936672 Jan  3 18:36 obclient-2.2.3-1.el7.x86_64.rpm
-rw-r--r-- 1 root root  65033488 Jan  3 18:36 ob-deploy-2.5.0-2.el7.x86_64.rpm
-rw-r--r-- 1 root root 123411192 Jan  2 16:14 obproxy-ce-4.2.1.0-11.el7.x86_64.rpm
-rw-r--r-- 1 root root  91384448 Jan  2 16:14 oceanbase-ce-4.2.1.2-102000042023120514.el7.x86_64.rpm
-rw-r--r-- 1 root root    158316 Jan  2 16:14 oceanbase-ce-libs-4.2.1.2-102000042023120514.el7.x86_64.rpm
[root@OBSERVER02 obinstall]$ chown -R admin:root /opt/obinstall

2）安装OBD：


## 安装OBD：
[admin@OBSERVER02 obinstall]$ sudo rpm -ivh ob-deploy-2.4.1-2.el7.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:ob-deploy-2.4.1-2.el7            ################################# [100%]
Installation of obd finished successfully
Please source /etc/profile.d/obd.sh to enable it
[admin@OBSERVER02 obinstall]$ source /etc/profile.d/obd.sh
[admin@OBSERVER02 obinstall]$ obd --help
Usage: obd <command> [options]
 
Available commands:
 
cluster        Deploy and manage a cluster.
 
demo           Quickly start
 
display-trace  display trace_id log.
 
mirror         Manage a component repository for OBD.
 
obdiag         Oceanbase Diagnostic Tool
 
repo           Manage local repository for OBD.
 
test           Run test for a running deployment.
 
update         Update OBD.
 
web            Start obd deploy application as web.
 
 
Options:
  --version      show program's version number and exit
  -h, --help     Show help and exit.
  -v, --verbose  Activate verbose output.

3）禁用远程仓库：（建议OBD离线方式安装rpm包，速度比OBD在线下载安装步骤要快一些）


## (可选)删除或移走远程yum源包：
[admin@OBSERVER02 ~]$ rm -rf ~/.obd/mirror/remote/OceanBase.repo
[admin@OBSERVER02 ~]$ obd mirror list
Update OceanBase-community-stable-el7 ok
Update OceanBase-development-kit-el7 ok
+-----------------------------------------------------------------------------+
|                            Mirror Repository List                           |
+----------------------------+--------+---------+----------+------------------+
| SectionName                | Type   | Enabled | Avaiable | Update Time      |
+----------------------------+--------+---------+----------+------------------+
| oceanbase.community.stable | remote | True    | True     | 2024-01-03 10:31 |
| oceanbase.development-kit  | remote | True    | True     | 2024-01-03 10:31 |
| local                      | local  | -       | True     | 2024-01-03 10:37 |
+----------------------------+--------+---------+----------+------------------+
Use `obd mirror list <section name>` for more details
Trace ID: 083bc7ec-a9e1-11ee-8b6b-0242ac110003
If you want to view detailed obd logs, please run: obd display-trace 083bc7ec-a9e1-11ee-8b6b-0242ac110003
 
## OBD禁用远程仓库：
[admin@OBSERVER02 ~]$ obd mirror disable remote
Disable remote ok
Trace ID: 14b436b2-a9e1-11ee-bd85-0242ac110003
If you want to view detailed obd logs, please run: obd display-trace 14b436b2-a9e1-11ee-bd85-0242ac110003
[admin@OBSERVER02 ~]$ obd mirror list
+-----------------------------------------------------------------------------+
|                            Mirror Repository List                           |
+----------------------------+--------+---------+----------+------------------+
| SectionName                | Type   | Enabled | Avaiable | Update Time      |
+----------------------------+--------+---------+----------+------------------+
| oceanbase.community.stable | remote | False   | False    | 2024-01-03 10:37 |
| oceanbase.development-kit  | remote | False   | False    | 2024-01-03 10:37 |
| local                      | local  | -       | True     | 2024-01-03 10:39 |
+----------------------------+--------+---------+----------+------------------+
Use `obd mirror list <section name>` for more details
Trace ID: 50f681a2-a9e1-11ee-9248-0242ac110003
If you want to view detailed obd logs, please run: obd display-trace 50f681a2-a9e1-11ee-9248-0242ac110003

4）将本地rpm包加入OBD：


[admin@OBSERVER02 obinstall]$ obd mirror clone ./*.rpm
name: obproxy-ce
version: 4.2.1.0
release:11.el7
arch: x86_64
md5: 0aed4b782120e4248b749f67be3d2cc82cdcb70d
add ./obproxy-ce-4.2.1.0-11.el7.x86_64.rpm to local mirror
name: oceanbase-ce
version: 4.2.1.2
release:102000042023120514.el7
arch: x86_64
md5: bf178e82c99ca1324a3df9e1a21cbbb8f8c4d46c
add ./oceanbase-ce-4.2.0.0-102000042023120514.el7.x86_64.rpm to local mirror
name: oceanbase-ce-libs
version: 4.2.1.2
release:102000042023120514.el7
arch: x86_64
md5: f77ba7e678acf0645889967391c847ca9cf684b6
add ./oceanbase-ce-libs-4.2.0.0-102000042023120514.el7.x86_64.rpm to local mirror
Trace ID: 03fa748c-a3bd-11ee-acb3-0242ac110003
If you want to view detailed obd logs, please run: obd display-trace 03fa748c-a3bd-11ee-acb3-0242ac110003
 
## 查看本地仓库：
[admin@OBSERVER02 ~]$ obd mirror list local
+----------------------------------------------------------------------------------------------------------+
|                                            local Package List                                            |
+-------------------+---------+------------------------+--------+------------------------------------------+
| name              | version | release                | arch   | md5                                      |
+-------------------+---------+------------------------+--------+------------------------------------------+
| obproxy-ce        | 4.2.1.0 | 11.el7                 | x86_64 | 0aed4b782120e4248b749f67be3d2cc82cdcb70d |
| oceanbase-ce      | 4.2.1.2 | 102000042023120514.el7 | x86_64 | b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4 |
| oceanbase-ce-libs | 4.2.1.2 | 102000042023120514.el7 | x86_64 | b4ae00ee729404557fa858d4cdd87250bca1aa63 |
+-------------------+---------+------------------------+--------+------------------------------------------+
Trace ID: ccbaf870-ac77-11ee-8848-0242ac110003
If you want to view detailed obd logs, please run: obd display-trace ccbaf870-ac77-11ee-8848-0242ac110003
[admin@OBSERVER02 ~]$

3.3. 搭建OceaBase集群

1）配置yaml文件：


[admin@OBSERVER02 obinstall]$ cat myobc.yaml
user:
    username: admin
    password: Admin123
oceanbase-ce:
 servers:
   - name: obsvr01
     ip: 10.0.0.11
   - name: obsvr02
     ip: 10.0.0.12
   - name: obsvr03
     ip: 10.0.0.13
 global:
   devname: eth1
   mysql_port: 2881
   rpc_port: 2882
   home_path: /home/admin/oceanbase
   data_dir: /opt/data
   redo_dir: /opt/redolog
   cluster_id: 202401
   production_mode: false
   memory_limit: 8G
   system_memory: 2G
   stack_size: 512K
   cpu_count: 16
   datafile_size: 15G
   log_disk_size: 5G
   cache_wash_threshold: 1G
   __min_full_resource_pool_memory: 2147483648
   workers_per_cpu_quota: 10
   schema_history_expire_time: 1d
   net_thread_count: 4
   major_freeze_duty_time: Disable
   minor_freeze_times: 10
   enable_separate_sys_clog: 0
   enable_merge_by_turn: false
   syslog_level: WARN
   enable_syslog_wf: false
   enable_syslog_recycle: true
   enable_rich_error_msg: true
   max_syslog_file_count: 4
   appname: myobc
   root_password: 1qazxsw2
   proxyro_password:
 obsvr01:
   zone: zone1
 obsvr02:
   zone: zone2
 obsvr03:
   zone: zone3
[admin@OBSERVER02 obinstall]$

2）OBD自动搭建OB集群：


[admin@OBSERVER02 ~]$ obd cluster deploy myobc -c /opt/obinstall/myobc.yaml -f 
Package oceanbase-ce-4.2.1.2-102000042023120514.el7 is available.
install oceanbase-ce-4.2.1.2 for local ok
+--------------------------------------------------------------------------------------------+
|                                          Packages                                          |
+--------------+---------+------------------------+------------------------------------------+
| Repository   | Version | Release                | Md5                                      |
+--------------+---------+------------------------+------------------------------------------+
| oceanbase-ce | 4.2.1.2 | 102000042023120514.el7 | b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4 |
+--------------+---------+------------------------+------------------------------------------+
Repository integrity check ok
Parameter check ok
Initializes observer work home ok
Remote oceanbase-ce-4.2.1.2-102000042023120514.el7-b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4 repository install ok
Remote oceanbase-ce-4.2.1.2-102000042023120514.el7-b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4 repository lib check !!
Try to get lib-repository
Package oceanbase-ce-libs-4.2.1.2-102000042023120514.el7 is available.
install oceanbase-ce-libs-4.2.1.2 for local ok
Remote oceanbase-ce-libs-4.2.1.2-102000042023120514.el7-29557bf57e13a46975c28ca5ce222ee3d776a8a9 repository install ok
Remote oceanbase-ce-4.2.1.2-102000042023120514.el7-b2ccb524f200a9ef0fad2cddf59d309ddaa2e3e4 repository lib check ok
myobc deployed
Please execute ` obd cluster start myobc ` to start
Trace ID: 76232158-b74a-11ee-99a2-024280800002
If you want to view detailed obd logs, please run: obd display-trace 76232158-b74a-11ee-99a2-024280800002

-f 参数可以强制覆盖原有的安装目录文件，避免安装过程出现安装目录不为空产生的报错；如果是第一次安装搭建，可以不加-f。

3）启动集群：


[admin@OBSERVER02 ~]$ obd cluster start myobc
Get local repositories ok
Search plugins ok
Load cluster param plugin ok
Open ssh connection ok
Check before start observer ok
[WARN] OBD-1012: (10.0.0.11) clog and data use the same disk (/opt)
[WARN] OBD-1012: (10.0.0.12) clog and data use the same disk (/opt)
[WARN] OBD-1012: (10.0.0.13) clog and data use the same disk (/opt)
 
Start observer ok
observer program health check ok
Connect to observer 10.0.0.11:2881 ok
Initialize oceanbase-ce ok
Wait for observer init ok
+---------------------------------------------+
|                   observer                  |
+-----------+---------+------+-------+--------+
| ip        | version | port | zone  | status |
+-----------+---------+------+-------+--------+
| 10.0.0.11 | 4.2.1.2 | 2881 | zone1 | ACTIVE |
| 10.0.0.12 | 4.2.1.2 | 2881 | zone2 | ACTIVE |
| 10.0.0.13 | 4.2.1.2 | 2881 | zone3 | ACTIVE |
+-----------+---------+------+-------+--------+
obclient -h10.0.0.11 -P2881 -uroot -p'1qazxsw2' -Doceanbase -A
 
myobc running
Trace ID: ffe97290-aaa5-11ee-86dc-0242ac110003
If you want to view detailed obd logs, please run: obd display-trace ffe97290-aaa5-11ee-86dc-0242ac110003

4）安装obclient客户端工具：


[admin@OBSERVER02 opt]$ sudo rpm -ivh obclient-2.2.3-1.el7.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:obclient-2.2.3-1.el7             ################################# [100%]
[admin@OBSERVER02 opt]$

3.4. 安装运行OBProxy

OBD方式也可以在安装配置文件myobc.yaml上添加OBProxy的配置项，由OBD一键执行下载安装并启停服务，但这一步在本次的搭建试验中老是存在问题，问题详情可看第5章节《问题处理》会讲到，为了绕开问题，采用手动完成OBProxy的搭建，并写到脚本将OBProxy与OBD合并一处一键启停。

1）安装OBProxy并拉起服务：


[admin@OBSERVER02 obinstall]$ sudo rpm -ivh obproxy-ce-4.2.1.0-11.el7.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:obproxy-ce-4.2.1.0-11.el7        ################################# [100%]
[admin@OBSERVER02 obinstall]$ cd ~/obproxy
[admin@OBSERVER02 obproxy]$ /home/admin/obproxy/bin/obproxy -o 'enable_strict_kernel_release=false,enable_cluster_checkout=false,skip_proxy_sys_private_check=true,enable_metadb_used=false' --listen_port 2883 --prometheus_listen_port 2884 --rs_list '10.0.0.11:2881;10.0.0.12:2881;10.0.0.13:2881' --cluster_name myobc
/home/admin/obproxy/bin/obproxy -o enable_strict_kernel_release=false,enable_cluster_checkout=false,skip_proxy_sys_private_check=true,enable_metadb_used=false --listen_port 2883 --prometheus_listen_port 2884 --rs_list 10.0.0.11:2881;10.0.0.12:2881;10.0.0.13:2881 --cluster_name myobc
optstr: enable_strict_kernel_release=false,enable_cluster_checkout=false,skip_proxy_sys_private_check=true,enable_metadb_used=false
listen port: 2883
prometheus listen port: 2884
[admin@OBSERVER02 obproxy]$ ps -ef|grep obproxy
admin        855       1  3 18:14 ?        00:00:23 /home/admin/obproxy/bin/obproxy -o enable_strict_kernel_release=false,enable_cluster_checkout=false,skip_proxy_sys_private_check=true,enable_metadb_used=false --listen_port 2883 --prometheus_listen_port 2884 --rs_list 10.0.0.11:2881;10.0.0.12:2881;10.0.0.13:2881 --cluster_name myobc
[admin@OBSERVER02 obproxy]$ ss -unlpt|grep 2883
tcp    LISTEN     0      1024      *:2883                  *:*                   users:(("obproxy",pid=2062,fd=92))

2）设置OBProxy进程系统密码：


[admin@OBSERVER02 ~]$ obclient -h10.0.0.12 -P2883 -uroot@proxysys -p -c -A
Enter password:
Welcome to the OceanBase.  Commands end with ; or \g.
Your OceanBase connection id is 5
Server version: 5.6.25
 
Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
obclient [(none)]> show processlist;
+------+----------+------+-----------------+------+-------------+-------------------+-------------------+------+------+
| Id   | Tenant   | User | Host            | db   | trans_count | svr_session_count | state             | tid  | pid  |
+------+----------+------+-----------------+------+-------------+-------------------+-------------------+------+------+
|    5 | proxysys | root | 10.0.0.12:52932 | NULL |           0 |                 0 | MCS_ACTIVE_READER | 2062 | 2062 |
+------+----------+------+-----------------+------+-------------+-------------------+-------------------+------+------+
1 row in set (0.002 sec)
 
obclient [(none)]> show proxyconfig like '%sys_password%';
+------------------------+-------+--------------------------------+-------------+---------------+
| name                   | value | info                           | need_reboot | visible_level |
+------------------------+-------+--------------------------------+-------------+---------------+
| observer_sys_password1 |       | password for observer sys user | false       | SYS           |
| observer_sys_password  |       | password for observer sys user | false       | SYS           |
| obproxy_sys_password   |       | password for obproxy sys user  | false       | SYS           |
+------------------------+-------+--------------------------------+-------------+---------------+
3 rows in set (0.002 sec)
 
obclient [(none)]> alter proxyconfig set observer_sys_password='1QAZXSW2';
Query OK, 0 rows affected (0.017 sec)
 
obclient [(none)]> alter proxyconfig set obproxy_sys_password='1QAZXSW2';
Query OK, 0 rows affected (0.016 sec)
 
obclient [(none)]> show proxyconfig like '%sys_password%';
+------------------------+------------------------------------------+--------------------------------+-------------+---------------+
| name                   | value                                    | info                           | need_reboot | visible_level |
+------------------------+------------------------------------------+--------------------------------+-------------+---------------+
| observer_sys_password1 |                                          | password for observer sys user | false       | SYS           |
| observer_sys_password  | 51045cddb1a4447ece609d181d80833cbcd361c6 | password for observer sys user | false       | SYS           |
| obproxy_sys_password   | 51045cddb1a4447ece609d181d80833cbcd361c6 | password for obproxy sys user  | false       | SYS           |
+------------------------+------------------------------------------+--------------------------------+-------------+---------------+
3 rows in set (0.003 sec)
 
obclient [(none)]> exit
Bye

3）创建OBProxy与OBServer的通信用户(proxyro)：设置proxyro用户密码要与observer_sys_password值一致。


## 进入sys租户修改proxyro用户密码：
[admin@OBSERVER02 ~]$ obclient -h10.0.0.12 -P2881 -uroot@sys -p'1qazxsw2' -c -A oceanbase
Welcome to the OceanBase.  Commands end with ; or \g.
Your OceanBase connection id is 3221780363
Server version: OceanBase_CE 4.2.1.2 (r102000042023120514-ccdde7d34de421336c5362483d64bf2b73348bd4) (Built Dec  5 2023 14:34:01)
 
Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
obclient [oceanbase]> show tenant;
+---------------------+
| Current_tenant_name |
+---------------------+
| sys                 |
+---------------------+
1 row in set (0.003 sec)
 
obclient [oceanbase]> select user,host,password from mysql.user;
+---------+------+-------------------------------------------+
| user    | host | password                                  |
+---------+------+-------------------------------------------+
| root    | %    | *79b36e3d5f430af5b15934d61d71c031b6502834 |
| proxyro | %    |                                           |
+---------+------+-------------------------------------------+
2 rows in set (0.009 sec)
 
obclient [oceanbase]> grant select on oceanbase.* to proxyro identified by '1QAZXSW2';
Query OK, 0 rows affected (0.097 sec)
 
obclient [oceanbase]> grant usage on *.* to proxyro;
Query OK, 0 rows affected (0.095 sec)
 
obclient [oceanbase]> select user,host,password from mysql.user;
+---------+------+-------------------------------------------+
| user    | host | password                                  |
+---------+------+-------------------------------------------+
| root    | %    | *79b36e3d5f430af5b15934d61d71c031b6502834 |
| proxyro | %    | *64ac39155222d296839dc451f38bba28b5a33a12 |
+---------+------+-------------------------------------------+
2 rows in set (0.000 sec)
 
obclient [oceanbase]> show full processlist;
+------------+---------+--------+-----------------+-----------+---------+------+--------+-----------------------+-----------+------+----------------------+
| Id         | User    | Tenant | Host            | db        | Command | Time | State  | Info                  | Ip        | Port | Proxy_sessid         |
+------------+---------+--------+-----------------+-----------+---------+------+--------+-----------------------+-----------+------+----------------------+
| 3221780363 | root    | sys    | 10.0.0.12:43370 | oceanbase | Query   |    0 | ACTIVE | show full processlist | 10.0.0.12 | 2881 |                 NULL |
| 3221687230 | proxyro | sys    | 10.0.0.12:57964 | oceanbase | Sleep   |    1 | SLEEP  | NULL                  | 10.0.0.11 | 2881 | 12398691262201528324 |
| 3221661587 | proxyro | sys    | 10.0.0.12:47908 | oceanbase | Sleep   | 1463 | SLEEP  | NULL                  | 10.0.0.11 | 2881 | 12398691262201528323 |
+------------+---------+--------+-----------------+-----------+---------+------+--------+-----------------------+-----------+------+----------------------+
3 rows in set (0.011 sec)
 
obclient [oceanbase]> quit
Bye

4）连接2883端口登录数据库：


[admin@OBSERVER02 ~]$ obclient -h10.0.0.12 -P2883 -uroot@sys#myobc -p'1qazxsw2' -Doceanbase -c -A
Welcome to the OceanBase.  Commands end with ; or \g.
Your OceanBase connection id is 10
Server version: OceanBase_CE 4.2.1.2 (r102000042023120514-ccdde7d34de421336c5362483d64bf2b73348bd4) (Built Dec  5 2023 14:34:01)
 
Copyright (c) 2000, 2018, OceanBase and/or its affiliates. All rights reserved.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
obclient [oceanbase]> select svr_ip,svr_port,zone,inner_port,with_rootserver,status,stop_time,start_service_time,last_offline_time from __all_server;
+-----------+----------+-------+------------+-----------------+--------+-----------+--------------------+-------------------+
| svr_ip    | svr_port | zone  | inner_port | with_rootserver | status | stop_time | start_service_time | last_offline_time |
+-----------+----------+-------+------------+-----------------+--------+-----------+--------------------+-------------------+
| 10.0.0.11 |     2882 | zone1 |       2881 |               1 | ACTIVE |         0 |   1704536095795754 |                 0 |
| 10.0.0.12 |     2882 | zone2 |       2881 |               0 | ACTIVE |         0 |   1704536100355839 |                 0 |
| 10.0.0.13 |     2882 | zone3 |       2881 |               0 | ACTIVE |         0 |   1704536096955836 |                 0 |
+-----------+----------+-------+------------+-----------------+--------+-----------+--------------------+-------------------+
3 rows in set (0.004 sec)
 
obclient [oceanbase]> quit
Bye

大功告成！！

5）启停数据库集群脚本：


[admin@OBSERVER02 bin]$ cat OBService
#!/bin/bash 
OPTMODE="${1:-status}"
OBCLU_NAME=myobc
OB_HOME=/home/admin/oceanbase
ODP_HOME=/home/admin/obproxy
ODP_PID=`ps -ef|grep -w $ODP_HOME|grep -v grep|awk '{print $2}'`
LAB="\033[36m[$(date +%F_%T)]\033[0m"
if [[ -n "$ODP_PID" ]];then RMD="obproxy服务已开启.";else RMD="obproxy服务已关闭.";fi
if [[ "$OPTMODE" == "start" && -z "$ODP_PID" ]];then
	obd cluster start "$OBCLU_NAME"
	cd $ODP_HOME && $ODP_HOME/bin/obproxy 
	LAB="[\033[1;32mOK\033[0m]";RMD="obproxy服务已开启."
elif [[ "$OPTMODE" == "stop" && -n "$ODP_PID" ]];then 
	obd cluster stop "$OBCLU_NAME"
	kill -9 "$ODP_PID"
	LAB="[\033[1;32mOK\033[0m]";RMD="obproxy服务已关闭."
elif [[ "$OPTMODE" == "restart" ]];then 
	obd cluster restart "$OBCLU_NAME"
	kill -9 "$ODP_PID"
	cd $ODP_HOME && $ODP_HOME/bin/obproxy 
	LAB="[\033[1;32mOK\033[0m]";RMD="obproxy服务已重启."
elif [[ "$OPTMODE" == "status" ]];then 
	ps -ef|grep -w $ODP_HOME|grep -v grep
	ss -unlpt|grep -w 2883
else
	LAB="[\033[1;31mERROR\033[0m]";RMD="参数选项有误,正确执行方法: $0 start/stop/restart/status."
fi 
obd cluster list
echo -e "${LAB}: ${RMD}"
 
[admin@OBSERVER02 bin]$ chmod 775 OBService

一键启动OB集群及OBProxy：

一键关闭OB集群及OBProxy：

4. ODC开发工具客户端连接

ODC远程连接宿主机里docker的数据库，IP填写宿主机IP，端口填容器ob02与宿主机映射的2883端口既可以连接上OceanBase数据库，用图形化界面开心的玩耍了。

5. 报错处理

5.1. 安装失败后卸载重装

出现安装失败后，需要重新将环境回退到安装前，可参考以下做法：


## kill掉数据库进程：
[admin@OBSERVER02 ~]$ pidof observer|xargs kill -9
 
## kill掉obproxy进程：
[admin@OBSERVER02 ~]$ pidof observer|xargs kill -9
 
## 清空数据文件目录和日志文件目录：
[admin@OBSERVER02 ~]$ rm -rf /home/admin/oceanbase /opt/data/* /opt/redolog/*
 
## 移除掉rpm包：
[admin@OBSERVER02 ~]$ yum list installed|grep -iE 'oceanbase|obproxy'|awk '{print $1}'|xargs sudo yum remove -y
 
## obd重新安装部署集群：
[admin@OBSERVER02 ~]$ obd cluster deploy myobc -c /opt/obinstall/myobc.yaml -f

5.2. 安装用户可使用线程数小于要求

1）现象：

2）排错过程：


## 先查看/etc/security/limits.conf配置：
[admin@OBSERVER02 ~]$ tail -15 /etc/security/limits.conf
#@student        -       maxlogins       4
 
# End of file
 
root soft nofile 655350
root hard nofile 655350
* soft nofile 655350
* hard nofile 655350
* soft nproc 655360
* hard nproc 655360
* soft stack unlimited
* hard stack unlimited
* soft core unlimited
* hard core unlimited

发现在这个文件里是配置上了，但显然admin用户没有生效，检查admin用户的ulimit值：


## 查看admin用户ulimit值：
[admin@OBSERVER02 ~]$ ulimit -u 
4096
## 查看root用户ulimit值：
[root@OBSERVER02 ~]# ulimit -u
655360

明显在root用户生效了，但admin用户没有生效；

3）解决办法：修改/etc/security/limits.d/20-nproc.conf文件，nproc.conf文件随OS发行版的不同命名会略有区别；


## 将普通用户soft nproc值改成655360；
[root@OBSERVER02 ~]$ ll /etc/security/limits.d/*-nproc.conf
-rw-r--r-- 1 root root 193 Jan 20 12:49 /etc/security/limits.d/20-nproc.conf
[admin@OBSERVER02 ~]$ cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
 
*          soft    nproc     655360
root       soft    nproc     unlimited
[root@OBSERVER02 ~]$
[root@OBSERVER02 ~]# su - admin
Last login: Sat Jan 20 12:33:47 CST 2024 on pts/0
[admin@OBSERVER02 ~]$ ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256628
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 655350
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 655360
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

重新登录admin用户执行ulimit -a看到max user processes值变成655360了，同样的，在其他节点修改，后再重新执行启动集群的操作。

5.3. 空间资源不足产生的报错

1）现象：

2）日志定位报错：

查看OBD跟踪日志：发现数据库进程先是被OBD拉起来过，后面不知道啥原因数据库进程自动kill掉，导致OBD跟踪日志显示Failed to connect to oceanbase-ce；

挑其中一个数据库节点查看数据库服务日志，检索所有关键字error，发现报剩余的磁盘空间大小全给了日志空间占用，导致observer无空间可用，数据库服务自动退出；

3）解决办法：

对日志使用空间上限进行限制，在OBD安装配置文件myobc.yaml里添加全局参数：log_disk_size: 5G，大小自定义，重新执行安装即可。

5.4. OBD拉起OBProxy服务失败

1）现象：

2）日志排错：

查看OBD跟踪日志，也是未发现有效报错的信息，查看OBProxy日志，也未能找到有效的信息，后面是在OceanBase官网搜有关的帖子，发现这个问题与帖子里的几乎一样，给的解释是OBD存在BUG，在资源给得不是很大的情况下，OBD拉起OBProxy是会报错的，至于Why，我也不知道了；

OBD部署obproxy集群失败 - #14，来自妙妙 - OceanBase - 社区问答- OceanBase社区-分布式数据库【使用环境】测试环境【 OB or 其他组件】OBD 2.0.1 【使用版本】【问题描述】OBD集群部署异常：期望按照此架构进行部署测试机器*3：CPU 8core+ Memory 16GB + Disk …https://ask.oceanbase.com/t/topic/35603305/143）解决办法：

既然OBD方式没法拉起OBProxy，那就改手动启停服务，将原来myobc.yaml文件里有关obroxy-ce下的参数项全去掉，只留oceanbase-ce的参数再次使用OBD安装，安装OK后，手动拉起OBProxy服务即可，启停可自写脚本一键执行。

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/小蓝xlanll/article/detail/168065?site