OceanBase在线扩容缩容

IT那活儿 2021-08-22

783

一. 基本概述

现有集群 1-1-1架构,每个zone里面只有一个observer

扩容后2-2-2，每个zone里面有两个observer

二. observer服务器准备

配置建议和原集群中的observer一模一样

1. 所有主机配置主机名及/etc/hosts

hostnamectl set-hostname ob5hostnamectl set-hostname ob6hostnamectl set-hostname ob7配置/etc/hosts文件192.168.8.147 ob5 192.168.8.148 ob6 192.168.8.149 ob7

复制

2. 所有主机关闭防火墙与selinux

1) 关闭防火墙,并开机禁用systemctl stop firewalld.servicesystemctl disable firewalld.servicesystemctl status firewalld.service2) 禁用selinuxsed -i 's@SELINUX=enforcing@SELINUX=disabled@g' /etc/selinux/configgrep "^SELINUX=" /etc/selinux/config3) 重启reboot4) 检查systemctl status firewalld.serviceiptables -Lgetenforce
复制

3. 所有主机配置NTP

vi /etc/ntp.conf注释文件中的以下4行#server 0.centos.pool.ntp.org iburst#server 1.centos.pool.ntp.org iburst#server 2.centos.pool.ntp.org iburst#server 3.centos.pool.ntp.org iburst在末行添加NTP服务器IP地址，我这里是本地NTP服务器,就使用OCP主机最为时钟源server 192.168.8.130重启NTP服务，预计5~10分钟就会同步，可手动拉同步systemctl restart ntpd.servicentpq -p--检查时间误差clockdiff 192.168.8.130

复制

4. 允许root SSH

vi /etc/ssh/sshd_config注释#PermitRootLogin nosystemctl restart sshd.service

复制

5. 关闭numa

判断系统是否开启了numa 功能grep -i numa var/log/dmesg如果出现了："No NUMA configuration found" ，则证明是关闭了numa 功能，如果是其他内容，则为开启了numa--关闭numavi etc/default/grub在 GRUB_CMDLINE_LINUX 参数的末尾增加：numa=off[root@ob6 clonescripts]# vi etc/default/grubGRUB_TIMEOUT=5GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' etc/system-release)"GRUB_DEFAULT=savedGRUB_DISABLE_SUBMENU=trueGRUB_TERMINAL_OUTPUT="console"GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=vg0/lv_root rd.lvm.lv=vg0/lv_swap rhgb quiet numa=off elevator=deadline numa=off"GRUB_DISABLE_RECOVERY="true"--重建grub 配置文件grub2-mkconfig -o boot/grub2/grub.cfg--重启操作系统reboot

复制

三. 部署Observer

使用自动化部署工具部署Observer

1.安装自动化部署工具

rpm –ivh t-oceanbase-antman-1.3.6-1919351.alios7.x86_64.rpmcd /root/t-oceanbase-antman/clonescripts./clone.sh –h 获取帮助

复制

2. 添加admin用户

./clone.sh –u

3. 修改属主属组

chown admin:admin -R /data/1chown admin:admin -R /data/log1

复制

4. 对每台服务器的操作系统内核进行设置

cd /root/t-oceanbase-antman/clonescripts./clone.sh -r ob –c

复制

5. 安装依赖包

cd /root/t-oceanbase-antman/clonescripts./clone.sh -r ob –m

复制

6. 部署前环境检查

统一检查脚本

cd /root/t-oceanbase-antman/clonescripts./clone.sh –t

复制

也可以使用如下检查方法：

sh precheck.sh –m ob          ob主机使用
复制

关注failed项，处理好后继续检查，直到所有的检查项都通过。

四. 在OCP扩容

1. 添加主机

直到新添加的observer的状态为空闲。

2. 添加observer

3. 注意事项

添加遇到问题，日志如下：


2021-03-10 10:43:20.703  INFO 67 --- [ task-2,0abda3d4bc1b4206,14188bec0edd] c.a.o.c.metadb.task.TaskInstanceEntity : task 35868 run with context Context(parallelIdx=-1, stringMap={cluster_name=cluster, cluster_id=2, service_type=OB_CLUSTER, rpm_name=oceanbase-2.2.73-20201117154111.el7.x86_64.rpm, service_name=cluster:1, target_operate_status=NORMAL, zone_name=zone3, server_port=2882, service_version=2.2.73, ob_cluster_id=1}, listMap={root_server_ips=[192.168.8.132, 192.168.8.131, 192.168.8.133], host_ids=[8], server_ips=[192.168.8.147]})

2021-03-10 10:43:20.705  INFO 67 --- [ task-2,0abda3d4bc1b4206,14188bec0edd] c.a.o.c.metadb.task.TaskInstanceEntity : executor node ip:192.168.8.130

2021-03-10 10:43:20.707 ERROR 67 --- [ task-2,0abda3d4bc1b4206,14188bec0edd] com.alipay.ocp.core.util.ExceptionUtils : Checked Exception: com.alipay.ocp.core.exception.IllegalArgumentException occurred with code error.file.meta.not.found, and args [sys-package, oceanbase-2.2.73-20201117154111.el7.x86_64.rpm]

2021-03-10 10:43:20.710 ERROR 67 --- [ task-2,0abda3d4bc1b4206,14188bec0edd] c.a.o.core.task.manager.TaskManagerImpl : got exception while execute task:
com.alipay.ocp.core.exception.IllegalArgumentException: [OCP IllegalArgumentException]: status=400 BAD_REQUEST, errorCode=FILE_META_NOT_FOUND, args=sys-package,oceanbase-2.2.73-20201117154111.el7.x86_64.rpm at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_152]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_152]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_152]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_152]
at com.alipay.ocp.core.util.ExceptionUtils.newException(ExceptionUtils.java:96) ~[ocp-core-2.5.0-20210105.jar!/:2.5.0-20210105]
at com.alipay.ocp.core.util.ExceptionUtils.throwException(ExceptionUtils.java:90) ~[ocp-core-2.5.0-20210105.jar!/:2.5.0-20210105]
at com.alipay.ocp.core.util.ExceptionUtils.illegalArgs(ExceptionUtils.java:29) ~[ocp-core-2.5.0-20210105.jar!/:2.5.0-20210105]
at com.alipay.ocp.service.compute.softwarepackage.SoftwarePackageService.getPackageByName(SoftwarePackageService.java:233) ~[ocp-service-2.5.0-20210105.jar!/:2.5.0-20210105]
at com.alipay.ocp.service.task.business.host.ReserveHostTask.getSoftwarePackageId(ReserveHostTask.java:73) ~[ocp-service-2.5.0-20210105.jar!/:2.5.0-20210105]
at com.alipay.ocp.service.task.business.host.ReserveHostTask.run(ReserveHostTask.java:45) ~[ocp-service-2.5.0-20210105.jar!/:2.5.0-20210105]
at com.alipay.ocp.core.metadb.task.TaskInstanceEntity.run(TaskInstanceEntity.java:192) ~[ocp-core-2.5.0-20210105.jar!/:2.5.0-20210105]
at com.alipay.ocp.core.task.manager.TaskManagerImpl.lambda$null$1(TaskManagerImpl.java:177) ~[ocp-core-2.5.0-20210105.jar!/:2.5.0-20210105]
at com.alipay.ocp.core.task.manager.TaskManagerImpl.redirectOutputIfNotSysSchedule(TaskManagerImpl.java:221) ~[ocp-core-2.5.0-20210105.jar!/:2.5.0-20210105]
at com.alipay.ocp.core.task.manager.TaskManagerImpl.lambda$executeTask$2(TaskManagerImpl.java:149) ~[ocp-core-2.5.0-20210105.jar!/:2.5.0-20210105] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_152]
at com.alipay.ocp.core.thread.TraceDecorator.lambda$decorate$0(TraceDecorator.java:28) ~[ocp-core-2.5.0-20210105.jar!/:2.5.0-20210105]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) ~[na:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627) ~[na:1.8.0_152]
at java.lang.Thread.run(Thread.java:882) ~[na:1.8.0_152]
复制