OceanBase学习笔记（一）——使用OBD自动部署多节点集群_obd-2008: cluster clocks are out of sync

部署安排：三副本三节点集群，OBD自动部署

操作系统：Centos7.2

参考官方文档：OceanBase 社区版

1. 部署准备

节点安排：四台阿里云服务器，均为32C64G，其中一台安装OBD自动部署软件，并安装obclient作为客户端，使用该机OBD自动部署三节点集群。

在中控机上安装相关软件：

yum install -y yum-utils
yum-config-manager --add-repo https://mirrors.aliyun.com/oceanbase/OceanBase.repo
yum -y install ob-deploy oceanbase obclient

2. 服务器初始化

在四台服务器上，均做如下初始化（必做，否则后续部署集群会报很多错误）：

2.1 内核参数修改

执行：

vim /etc/sysctl.conf

将打开的文件内容改为： 

net.core.somaxconn = 2048
net.core.netdev_max_backlog = 10000
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
 
net.ipv4.ip_local_port_range = 3500 65535
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_slow_start_after_idle=0
 
vm.swappiness = 0
vm.min_free_kbytes = 2097152
vm.max_map_count=655360
fs.aio-max-nr=1048576

保存并退出后，执行：

sysctl -p

2.2 修改会话变量设置

执行：

vi /etc/security/limits.conf

 在最后加上（前两行可能已经存在，修改数字即可）：

* soft nofile 655360
* hard nofile 655360
* soft nproc 655360
* hard nproc 655360
* soft core unlimited
* hard core unlimited
* soft stack unlimited
* hard stack unlimited

 2.3 配置时间同步服务

使用如下代码，判断是否使用ntpd服务：

systemctl status ntpd

如果出现如下结果， 跳到2.4：

Unit ntpd.service could not be found.

否则卸载ntpd：

yum autoremove ntp

安装chrony：

yum -y install chrony

修改chrony配置文件：

vi /etc/chrony.conf

将打开的文件改为：

# server 后面跟时间同步服务器
# 使用 pool.ntp.org 项目中的公共服务器。按 server 配置，理论上您想添加多少时间服务器都可以。
# 或者使用 阿里云的 ntp 服务器
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server ntp.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst
server ntp.aliyun.com minpoll 4 maxpoll 10 iburst
server ntp1.aliyun.com minpoll 4 maxpoll 10 iburst
server ntp1.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst
server ntp10.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst
 
# 如果是测试环境，没有时间同步服务器，那就选取一台配置为时间同步服务器。
# 如果选中的是本机，则取消下面 server 注释
#server 127.127.1.0
 
# 根据实际时间计算出服务器增减时间的比率，然后记录到一个文件中，在系统重启后为系统做出最佳时间补偿调整。
driftfile /var/lib/chrony/drift
 
# chronyd 根据需求减慢或加速时间调整，
# 在某些情况下系统时钟可能漂移过快，导致时间调整用时过长。
# 该指令强制 chronyd 调整时期，大于某个阀值时步进调整系统时钟。
# 只有在因 chronyd 启动时间超过指定的限制时（可使用负值来禁用限制）没有更多时钟更新时才生效。
makestep 10 3
 
# 将启用一个内核模式，在该模式中，系统时间每 11 分钟会拷贝到实时时钟（RTC）。
rtcsync
 
# Enable hardware timestamping on all interfaces that support it.
# 通过使用 hwtimestamp 指令启用硬件时间戳
#hwtimestamp eth0
#hwtimestamp eth1
#hwtimestamp *
 
# Increase the minimum number of selectable sources required to adjust
# the system clock.
#minsources 2
 
# 指定一台主机、子网，或者网络以允许或拒绝 NTP 连接到扮演时钟服务器的机器
#allow 192.168.0.0/16
 
# 即使没有同步到时间源，也要服务时间
local stratum 10
 
# 指定包含 NTP 验证密钥的文件。
keyfile /etc/chrony.keys
 
# 指定日志文件的目录。
logdir /var/log/chrony
 
 
 
# Select which information is logged.
#log measurements statistics tracking

对该文件的一些说明：

①选择中控机作为时间同步服务器

②第13行，如果此时做中控机初始化，改为：（去掉#即可）

server 127.127.1.0

否则，改为：

server 中控机IP地址

③# 指定一台主机、子网，或者网络以允许或拒绝 NTP 连接到扮演时钟服务器的机器

在这一行下面改为：

allow 第一个节点子网/24
allow 第一个节点子网/24
allow 第一个节点子网/24
 
例：
allow 172.28.240.0/24
allow 172.28.137.0/24
allow 172.28.59.0/24

保存退出。

执行命令，使该配置文件生效：

systemctl start chronyd.service
systemctl enable chronyd.service

 检查是否完成时间同步设置，在中控机使用clockdiff查各个节点：

 当delta值为0ms或1ms可基本视为成功，如果还是很大建议多使用clockdiff查几次，如果没有变小，再检查是否出错。

2.4 配置 SSH 免密登录

在中控机执行：

ssh-keygen -t rsa

上面命令会在用户的 HOME 目录中生成文件夹 .ssh 

复制 RSA 或 DSA 公钥到目标节点，打通到三个节点的 SSH 免密登录。

在中控机执行：

ssh-copy-id 用户名@IP地址
 
例：
ssh-copy-id root@172.24.280.125
ssh-copy-id root@172.24.280.126
ssh-copy-id root@172.24.280.126

在四台服务器上均做此初始化即可。

3. 部署

3.1准备配置文件

下载配置文件模板：

obdeploy/mini-single-with-obproxy-example.yaml at master · oceanbase/obdeploy · GitHub

修改该配置文件：

## Only need to configure when remote login is required
user:
   username: root
   password: 123465
   key_file: /root/.ssh/id_rsa.pub
   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
oceanbase-ce:
  servers:
    - name: obce01
      # Please don't use hostname, only IP can be supported
      ip: 172.17.237.61
    - name: obce02
      ip: 172.20.126.115
    - name: obce03
      ip: 172.28.240.128
  global:
    # Please set devname as the network adaptor's name whose ip is  in the setting of severs.
    # if set severs as "127.0.0.1", please set devname as "lo"
    # if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"
    devname: eth0
    cluster_id: 1
    # please set memory limit to a suitable value which is matching resource. 
    memory_limit: 8G # The maximum running memory for an observer
    system_memory: 4G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
    stack_size: 512K
    cpu_count: 16
    cache_wash_threshold: 1G
    __min_full_resource_pool_memory: 268435456
    workers_per_cpu_quota: 10
    schema_history_expire_time: 1d
    # The value of net_thread_count had better be same as cpu's core number. 
    net_thread_count: 4
    major_freeze_duty_time: Disable
    minor_freeze_times: 10
    enable_separate_sys_clog: 0
    enable_merge_by_turn: FALSE
    datafile_disk_percentage: 20 # The percentage of the data_dir space to the total disk space. This value takes effect only when datafile_size is 0. The default value is 90.
    datafile_size: 50G
    syslog_level: WRAN # System log level. The default value is INFO.
    enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
    enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
    max_syslog_file_count: 10 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
    # observer cluster name, consistent with obproxy's cluster_name
    appname: obce-3zones
    root_password: 12346 # root user password, can be empty
    proxyro_password: 123456 # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty
  obce01:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /root/obce-3zones
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data/obce-3zones
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo/obce-3zones
    zone: zone1
  obce02:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /root/obce-3zones
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data/obce-3zones
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo/obce-3zones
    zone: zone2
  obce03:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /root/obce-3zones
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data/obce-3zones
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo/obce-3zones
    zone: zone3
obproxy:
  # Set dependent components for the component.
  # When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.
  depends:
    - oceanbase-ce
  servers:
    - 172.17.237.62
  global:
    listen_port: 2883 # External port. The default value is 2883.
    prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
    home_path: /root/obproxy
    # oceanbase root server list
    # format: ip:mysql_port;ip:mysql_port. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
    rs_list: 172.17.237.61:2881;172.20.126.115:2881;172.28.240.128:2881
    enable_cluster_checkout: false
    # observer cluster name, consistent with oceanbase-ce's appname. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
    cluster_name: obce-3zones
    skip_proxy_sys_private_check: true
    obproxy_sys_password: 123456 # obproxy sys user password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
    observer_sys_password: 123465 # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.

将用户名，IP地址等修改为自己所对应的值即可。

保存后重命名为集群名：obce-3zones.yaml

3.2 部署

部署使用命令：obd cluster deploy [集群名] -c 集群配置文件 。

obd cluster deploy obce-3zones -c obce-3zones.yaml

检查集群部署状态，为deployed。

obd cluster list

启动集群

obd cluster start obce-3zones

确定集群启动成功，为running： 

obd cluster list

3.3 部署可能会出现的问题

open ssh connection报错问题

在部署时，可能会出现在open ssh connection处报错：

这是前面初始化没有做好，没有打通ssh的问题，回到2.4 配置ssh免密登录再试一试。

Cluster NTP is out of sync 报错问题 

服务器初始化时的时间同步没有处理好：用clockdiff查相应节点，Oceanbase集群要求delta不能超过100ms。处理方法在本文2.3 时间同步处理。

最终部署成功：