0068.O 删除重建observer_[bootstrap]ERROR 4015 (HY000) System error

rundba 2021-07-16

1980

计划在单台主机上安装6个observer，前3个observer进行bootstrap，后续3个计划进行扩容，但执行bootstrap时失败，文中通过清理重建，完成bootstrap操作。

只对observer1-3进行操作，observer4-6相关的步骤可不进行操作。

0.ENV

CentOS7.6;

OceanBase 2.2.75

以下操作OceanBase社区版同样适用。

1. 现象

执行bootstrap报错：ERROR 4015 (HY000): System error

[admin@ob6 ~]$ obclient -h127.1 -uroot -P2881 -p
Enter password: 
Welcome to OceanBase.  Commands end with ; or \g.
Your OceanBase connection id is 3221225483
Server version: OceanBase 2.2.75 (r20210108170301-c8c037dd6352fa0d00bd8a7105a1b566a5bc790d) (Built Jan  8 2021 17:31:33)


Copyright (c) 2000, 2020, OceanBase and/or its affiliates. All rights reserved.


Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
obclient> set session ob_query_timeout=1000000000; alter system bootstrap ZONE 'zone1' SERVER '172.18.33.109:2882', ZONE 'zone2' SERVER '172.18.33.109:3882', ZONE 'zone3' SERVER '172.18.33.109:4882';
Query OK, 0 rows affected (0.00 sec)

ERROR 4015 (HY000): System error

2. 问题说明

这一步是关键，不成功，就推到重来。看看机器环境或者操作步骤错误，通过下述步骤，确保各项均正常配置，一切都对的情况下一定能成功。

关键点：目录、IP、端口号、Zone名称、语法。

3. 问题处理

1) 清理observer原有进程和目录（第一次不用）

清理进程：

[root@ob6 ~]# su - admin
[admin@ob6 ~]$ pidof observer
121499 116728 105822 2092 1479 908

[admin@ob6 ~]$ kill -9 121499 116728 105822 2092 1479 908

[admin@ob6 ~]$ sleep 3

删除目录:

[admin@ob6 ~]$ rm -rf data/1/* /data/2/* /data/3/* /data/4/* /data/5/* /data/6/*
[admin@ob6 ~]$ rm -rf data/log1/[1-6]      #也可rm -rf data/log1/1/* ...
[admin@ob6 ~]$ rm -rf home/admin/oceanbase/[1-6]  #也可cd home/admin/oceanbase && rm -rf 1/* 2/* 3/*

2) 初始化目录

准备observer安装目录：

先安装oceanbase软件，目录在/home/admin/oceanbase上。然后复制该软件目录为oceanbase/[1-6]

[root@ob6 ~]# su - admin
[admin@ob6 ~]$ mv oceanbase oceanbase_orig
[admin@ob6 ~]$ mkdir -p oceanbase/1 oceanbase/2 oceanbase/3 oceanbase/4 oceanbase/5 oceanbase/6
cp -rf oceanbase_orig/* oceanbase/1/
cp -rf oceanbase_orig/* oceanbase/2/
cp -rf oceanbase_orig/* oceanbase/3/
cp -rf oceanbase_orig/* oceanbase/4/
cp -rf oceanbase_orig/* oceanbase/5/
cp -rf oceanbase_orig/* oceanbase/6/

[root@ob6 ~]# mkdir data/1 data/2 data/3 data/4 data/5 data/6
[root@ob6 ~]# cd data/log1/
[root@ob6 ~]# mkdir 1 2 3 4 5 6
[root@ob6 ~]# chown -R admin:admin data/1 data/2 data/3 data/4 data/5 data/6

3) 创建实例等必备目录(admin用户)

bootstrap以zone1、zone2、zone3中的observer1-3进行创建，observer5-6可忽略。

#OB1-zone1目录:
mkdir -p data/1/obdemo/{etc3,sort_dir,sstable}
mkdir -p /data/log1/1/obdemo/{clog,etc2,ilog,slog,oob_clog}
mkdir -p /home/admin/oceanbase/1/store/obdemo
for t in {etc3,sort_dir,sstable};do ln -s /data/1/obdemo/$t /home/admin/oceanbase/1/store/obdemo/$t; done
for t in {clog,etc2,ilog,slog,oob_clog};do ln -s /data/log1/1/obdemo/$t /home/admin/oceanbase/1/store/obdemo/$t; done
#OB2-zone2目录:
mkdir -p /data/2/obdemo/{etc3,sort_dir,sstable}
mkdir -p /data/log1/2/obdemo/{clog,etc2,ilog,slog,oob_clog}
mkdir -p /home/admin/oceanbase/2/store/obdemo
for t in {etc3,sort_dir,sstable};do ln -s /data/2/obdemo/$t /home/admin/oceanbase/2/store/obdemo/$t; done
for t in {clog,etc2,ilog,slog,oob_clog};do ln -s /data/log1/2/obdemo/$t /home/admin/oceanbase/2/store/obdemo/$t; done
#OB3-zone3目录:
mkdir -p /data/3/obdemo/{etc3,sort_dir,sstable}
mkdir -p /data/log1/3/obdemo/{clog,etc2,ilog,slog,oob_clog}
mkdir -p /home/admin/oceanbase/3/store/obdemo
for t in {etc3,sort_dir,sstable};do ln -s /data/3/obdemo/$t /home/admin/oceanbase/3/store/obdemo/$t; done
for t in {clog,etc2,ilog,slog,oob_clog};do ln -s /data/log1/3/obdemo/$t /home/admin/oceanbase/3/store/obdemo/$t; done
#OB4-zone1目录:
mkdir -p /data/4/obdemo/{etc3,sort_dir,sstable}
mkdir -p /data/log1/4/obdemo/{clog,etc2,ilog,slog,oob_clog}
mkdir -p /home/admin/oceanbase/4/store/obdemo
for t in {etc3,sort_dir,sstable};do ln -s /data/4/obdemo/$t /home/admin/oceanbase/4/store/obdemo/$t; done
for t in {clog,etc2,ilog,slog,oob_clog};do ln -s /data/log1/4/obdemo/$t /home/admin/oceanbase/4/store/obdemo/$t; done
#OB5-zone2目录:
mkdir -p /data/5/obdemo/{etc3,sort_dir,sstable}
mkdir -p /data/log1/5/obdemo/{clog,etc2,ilog,slog,oob_clog}
mkdir -p /home/admin/oceanbase/5/store/obdemo
for t in {etc3,sort_dir,sstable};do ln -s /data/5/obdemo/$t /home/admin/oceanbase/5/store/obdemo/$t; done
for t in {clog,etc2,ilog,slog,oob_clog};do ln -s /data/log1/5/obdemo/$t /home/admin/oceanbase/5/store/obdemo/$t; done
#OB6-zone3目录:
mkdir -p /data/6/obdemo/{etc3,sort_dir,sstable}
mkdir -p /data/log1/6/obdemo/{clog,etc2,ilog,slog,oob_clog}
mkdir -p /home/admin/oceanbase/6/store/obdemo
for t in {etc3,sort_dir,sstable};do ln -s /data/6/obdemo/$t /home/admin/oceanbase/6/store/obdemo/$t; done
for t in {clog,etc2,ilog,slog,oob_clog};do ln -s /data/log1/6/obdemo/$t /home/admin/oceanbase/6/store/obdemo/$t; done

查看目录：

tree /data/[1-6]
tree /data/log1/[1-6]
tree /home/admin/oceanbase/[1-6]

4) 启动 3 个 observer 进程

启动 3 个 observer 进程,observer 4-6可在扩容前启动。

注意：ip、网卡名都要根据实际情况调整。

OB1-zone1:
$ cd /home/admin/oceanbase/1 && /home/admin/oceanbase/1/bin/observer -i ens33 -P 2882 -p 2881 -z zone1 -d /home/admin/oceanbase/1/store/obdemo -r '172.18.33.109:2882:2881' -c 20210713 -n obdemo -o "memory_limit=15G,system_memory=5G,cpu_count=32,__min_full_resource_pool_memory=2147483648,cache_wash_threashold=2G,net_thread_count=8,workers_per_cpu_quota=4,stack_size=512K,config_additional_dir=/data/1/obdemo/etc3;/data/log1/1/obdemo/etc2"


OB2-zone2:
$ cd /home/admin/oceanbase/2 && /home/admin/oceanbase/2/bin/observer -i ens33 -P 3882 -p 3881 -z zone2 -d /home/admin/oceanbase/2/store/obdemo -r '172.18.33.109:3882:3881' -c 20210713 -n obdemo -o "memory_limit=15G,system_memory=5G,cpu_count=32,__min_full_resource_pool_memory=2147483648,cache_wash_threashold=2G,net_thread_count=8,workers_per_cpu_quota=4,stack_size=512K,config_additional_dir=/data/2/obdemo/etc3;/data/log1/2/obdemo/etc2"


OB3-zone3:
$ cd /home/admin/oceanbase/3 && /home/admin/oceanbase/3/bin/observer -i ens33 -P 4882 -p 4881 -z zone3 -d /home/admin/oceanbase/3/store/obdemo -r '172.18.33.109:4882:4881' -c 20210713 -n obdemo -o "memory_limit=15G,system_memory=5G,cpu_count=32,__min_full_resource_pool_memory=2147483648,cache_wash_threashold=2G,net_thread_count=8,workers_per_cpu_quota=4,stack_size=512K,config_additional_dir=/data/3/obdemo/etc3;/data/log1/3/obdemo/etc2"


OB4-zone1(后续扩容加入zone1):
cd /home/admin/oceanbase/4 && /home/admin/oceanbase/4/bin/observer -i ens33 -P 5882 -p 5881 -z zone1 -d /home/admin/oceanbase/4/store/obdemo -r '172.18.33.109:5882:5881' -c 20210713 -n obdemo -o "memory_limit=15G,system_memory=5G,cpu_count=32,__min_full_resource_pool_memory=2147483648,cache_wash_threashold=2G,net_thread_count=8,workers_per_cpu_quota=4,stack_size=512K,config_additional_dir=/data/4/obdemo/etc3;/data/log1/4/obdemo/etc2"


OB5-zone2(后续扩容加入zone2):
cd /home/admin/oceanbase/5 && /home/admin/oceanbase/5/bin/observer -i ens33 -P 6882 -p 6881 -z zone2 -d /home/admin/oceanbase/5/store/obdemo -r '172.18.33.109:6882:6881' -c 20210713 -n obdemo -o "memory_limit=15G,system_memory=5G,cpu_count=32,__min_full_resource_pool_memory=2147483648,cache_wash_threashold=2G,net_thread_count=8,workers_per_cpu_quota=4,stack_size=512K,config_additional_dir=/data/5/obdemo/etc3;/data/log1/5/obdemo/etc2"


OB6-zone3(后续扩容加入zone3):
cd /home/admin/oceanbase/6 && /home/admin/oceanbase/6/bin/observer -i ens33 -P 7882 -p 7881 -z zone3 -d /home/admin/oceanbase/6/store/obdemo -r '172.18.33.109:7882:7881' -c 20210713 -n obdemo -o "memory_limit=15G,system_memory=5G,cpu_count=32,__min_full_resource_pool_memory=2147483648,cache_wash_threashold=2G,net_thread_count=8,workers_per_cpu_quota=4,stack_size=512K,config_additional_dir=/data/6/obdemo/etc3;/data/log1/6/obdemo/etc2"

5) 安装客户端

[root@ob6 t-oceanbase-antman]# rpm -ivh obclient-1.2.3-20200814145858.el7.alios7.x86_64.rpm

6）bootstrap集群

登录sys租户

[admin@ob6 ~]$ obclient -h127.1 -uroot@sys#obdemo -P2881 -p        #密码为空时，可去掉-p参数
Enter password:                  #回车,默认密码为空

执行bootstrap成功：

obclient> set session ob_query_timeout=1000000000; alter system bootstrap ZONE 'zone1' SERVER '172.18.33.109:2882', ZONE 'zone2' SERVER '172.18.33.109:3882', ZONE 'zone3' SERVER '172.18.33.109:4882';
Query OK, 0 rows affected (0.00 sec)


Query OK, 0 rows affected (47.29 sec)

7）查看数据库

bootstrap成功后，可查询到数据库，否则无数据库。

obclient> show databases;
+--------------------+
| Database           |
+--------------------+
| oceanbase          |
| information_schema |
| mysql              |
| SYS                |
| LBACSYS            |
| ORAAUDITOR         |
| test               |
+--------------------+
7 rows in set (0.02 sec)

4. 小结

bootstrap失败，通过上述步骤先进行清理，然后重建，在目录、IP、端口号、Zone名称、语法等步骤正确时，参考上述步骤，bootstrap会成功的。

- 完 -

不足之处，还望抛砖。

作者:王坤，微信公众号:rundba，欢迎转载，转载请注明出处。

如需公众号转发，请联系wx: landnow。

文章转载自rundba，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

0068.O 删除重建observer_[bootstrap]ERROR 4015 (HY000) System error

评论