本文以3节点集群（1-1-1）,进行节点故障现象模拟。租户副本数量为2，模拟单节点故障，双节点故障现象。
三节点搭建文章地址：
https://www.modb.pro/db/190765
https://www.modb.pro/db/190799
1 系统信息
sys租户副本信息：为2分布在zone1,zone.
sys租户副本数量为2：
MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G;
*************************** 1. row ***************************gmt_create: 2021-12-07 16:42:12.523525gmt_modified: 2021-12-07 16:42:12.523525tenant_id: 1tenant_name: sysreplica_num: -1zone_list: zone1;zone2primary_zone: zone1;zone2locked: 0collation_type: 0info: system tenantread_only: 0rewrite_merge_version: 0locality: FULL{1}@zone1, FULL{1}@zone2logonly_replica_num: 0previous_locality:storage_format_version: 0
storage_format_work_version: 0default_tablegroup_id: -1compatibility_mode: 0drop_tenant_time: -1status: TENANT_STATUS_NORMALin_recyclebin: 0
1 row in set (0.001 sec)
ERROR: No query specified
复制2 observer信息
集群信息（1-1-1）
zone1 192.168.5.200 2881 2882
zone1 192.168.5.200 3881 3882
zone1 192.168.5.200 4881 4882
MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-07 15:52:48.009135 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-07 16:17:55.557006 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863788395674 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-07 16:24:29.110670 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.002 sec)复制
3 故障模拟：2个节点kill
对外不能提供服务，只有zone1,zone2正常才能正常提供服务。
3.1 kill zone2,zone2
[admin@db01 ~]$ ps -ef|grep observer |grep -v grep admin 727 1 99 02:52 ? 20:43:00 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/observer01/store -i ens192 -l INFO admin 1559 1 99 02:55 ? 19:04:39 /home/admin/oceanbase/bin/observer -r 192.168.5.200:3882:3881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO admin 2141 1 80 02:55 ? 13:37:51 /home/admin/oceanbase/bin/observer -r 192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone3 -p 4881 -P 4882 -c 1 -d /data/observer03/store -i ens192 -l INFO [admin@db01 ~]$ kill -9 1559 2141 [admin@db01 ~]$ ps -ef|grep observer |grep -v grep admin 727 1 99 02:52 ? 20:43:22 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/observer01/store -i ens192 -l INFO [admin@db01 ~]$复制
3.2 登陆数据库，发现系统已经不能正常提供服务
3.3 启动zone2
[admin@db01 ~]$ cd /data/observer02/ && /home/admin/oceanbase/bin/observer -r "192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881" -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO rs list: 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 optstr: __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 zone: zone2 mysql port: 3881 rpc port: 3882 cluster id: 1 data_dir: /data/observer02/store devname: ens192 log level: INFO [admin@db01 observer02]$ [admin@db01 observer02]$ ps -ef|grep obs|grep -v grep admin 727 1 99 02:52 ? 20:47:35 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone1 -p 2881 -P 2882 -c 1 -d /data/observer01/store -i ens192 -l INFO admin 11948 1 99 19:56 ? 00:00:16 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO复制
3.4 查询系统表，发现正常提供服务
[admin@db01 ~]$ sh c.sh Enter password: Welcome to the OceanBase. Commands end with ; or \g. Your MySQL connection id is 3221567989 Server version: 5.7.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [oceanbase]> select * from __all_server; ERROR 4012 (HY000): Timeout MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-07 15:52:48.009135 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:52:52.289833 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-07 16:24:29.110670 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ 3 rows in set (3.090 sec) MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ 3 rows in set (0.001 sec) MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.000 sec) MySQL [oceanbase]>[admin@db01 ~]$ sh c.sh Enter password: Welcome to the OceanBase. Commands end with ; or \g. Your MySQL connection id is 3221567989 Server version: 5.7.25 OceanBase 3.1.1 (r4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e) (Built Oct 21 2021 10:33:14) Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MySQL [oceanbase]> select * from __all_server; ERROR 4012 (HY000): Timeout MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-07 15:52:48.009135 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:52:52.289833 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-07 16:24:29.110670 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ 3 rows in set (3.090 sec) MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | blo ck_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | wit h_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+---- -------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+---- ------------+-------------------+ 3 rows in set (0.001 sec) MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 08:57:05.281970 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 08:57:11.281488 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638925030293827 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 08:57:05.278698 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863789811151 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.000 sec) MySQL [oceanbase]>复制
状态截图：
4 故障模拟：kill zone2
对外不能提供服务，只有zone1,zone2正常才能正常提供服务
4.1检查租户副本信息
sys租户副本分布在：locality: FULL{1}@zone1, FULL{1}@zone2
MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G;
*************************** 1. row ***************************gmt_create: 2021-12-07 16:42:12.523525gmt_modified: 2021-12-07 16:42:12.523525tenant_id: 1tenant_name: sysreplica_num: -1zone_list: zone1;zone2primary_zone: zone1;zone2locked: 0collation_type: 0info: system tenantread_only: 0rewrite_merge_version: 0locality: FULL{1}@zone1, FULL{1}@zone2logonly_replica_num: 0previous_locality:
storage_format_version: 0
storage_format_work_version: 0default_tablegroup_id: -1compatibility_mode: 0drop_tenant_time: -1status: TENANT_STATUS_NORMALin_recyclebin: 0
1 row in set (0.001 sec)
ERROR: No query specified
复制4.2模拟zone2故障
4.3故障现在，不能正常对外提供服务
4.4启动zone2
[admin@db01 observer03]$ [admin@db01 observer03]$ cd /data/observer02/ && /home/admin/oceanbase/bin/observer -r "192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881" -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO rs list: 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 optstr: __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 zone: zone2 mysql port: 3881 rpc port: 3882 cluster id: 1 data_dir: /data/observer02/store devname: ens192 log level: INFO [admin@db01 observer02]$ ps -ef|grep 3882 admin 15281 1 99 20:25 ? 00:07:57 /home/admin/oceanbase/bin/observer -r 192.168.5.200:2882:2881;192.168.5.200:3882:3881;192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone2 -p 3881 -P 3882 -c 1 -d /data/observer02/store -i ens192 -l INFO admin 16343 14059 0 20:32 pts/3 00:00:00 grep --color=auto 3882复制
4.5查看服务状态
zone2 的observer2启动之后，可以正常查询。
MySQL [oceanbase]> select * from __all_server; ERROR 4012 (HY000): Timeout MySQL [oceanbase]> select * from __all_server; ERROR 4012 (25000): Statement is timeout MySQL [oceanbase]> select * from __all_tenant where tenant_id=1\G; ERROR 4012 (25000): Statement is timeout ERROR: No query specified MySQL [oceanbase]> select * from __all_server; ERROR 4012 (HY000): Timeout MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 09:26:07.204402 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 09:26:07.207814 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 09:26:07.208028 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926371893289 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.001 sec)复制
5 故障模拟：kill zone3
对外正常提供服务。
5.1检查server状态
MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 09:26:07.204402 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 09:26:15.207936 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926774220189 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 09:26:07.208028 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926371893289 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+--------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.001 sec)复制
5.2 模拟zone3 故障
[admin@db01 observer02]$ [admin@db01 observer02]$ ps -ef|grep '\-P 4882'|grep -v grep admin 14080 1 90 20:17 ? 00:19:29 /home/admin/oceanbase/bin/observer -r 192.168.5.200:4882:4881 -o __min_full_resource_pool_memory=268435456,memory_limit=8G,system_memory=4G,stack_size=512K,cpu_count=16,cache_wash_threshold=1G,workers_per_cpu_quota=10,schema_history_expire_time=1d,net_thread_count=4,sys_bkgd_migration_retry_num=3,minor_freeze_times=10,enable_separate_sys_clog=0,enable_merge_by_turn=False,datafile_size=50G,enable_syslog_recycle=True,max_syslog_file_count=10 -z zone3 -p 4881 -P 4882 -c 1 -d /data/observer03/store -i ens192 -l INFO [admin@db01 observer02]$ kill -9 14080 [admin@db01 observer02]$ ps -ef|grep '\-P 4882'|grep -v grep复制
5.3 查询系统表
数据库状态正常，对外正常提供服务
MySQL [oceanbase]> select * from __all_server; +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | gmt_create | gmt_modified | svr_ip | svr_port | id | zone | inner_port | with_rootserver | status | block_migrate_in_time | build_version | stop_time | start_service_time | first_sessid | with_partition | last_offline_time | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ | 2021-12-07 15:52:36.583969 | 2021-12-08 09:26:07.204402 | 192.168.5.200 | 2882 | 1 | zone1 | 2881 | 1 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638863566011754 | 0 | 1 | 0 | | 2021-12-07 15:56:07.681043 | 2021-12-08 09:26:15.207936 | 192.168.5.200 | 3882 | 2 | zone2 | 3881 | 0 | active | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 1638926774220189 | 0 | 1 | 0 | | 2021-12-07 15:56:11.908905 | 2021-12-08 09:39:00.647693 | 192.168.5.200 | 4882 | 3 | zone3 | 4881 | 0 | inactive | 0 | 3.1.1_4-8c615943cbd25a6f7b8bdfd8677a13a21709a05e(Oct 21 2021 10:33:14) | 0 | 0 | 0 | 1 | 0 | +----------------------------+----------------------------+---------------+----------+----+-------+------------+-----------------+----------+-----------------------+------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+ 3 rows in set (0.001 sec)复制
「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文章的来源（墨天轮），文章链接，文章作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
评论
相关阅读
【MySQL 30周年庆】MySQL 8.0 OCP考试限时免费！教你免费领考券
墨天轮小教习
3409次阅读
2025-04-25 18:53:11
2025年4月国产数据库大事记：4个千万级中标项目诞生！2024年达梦净利3.6亿、金仓净利8006.6万……
墨天轮编辑部
2311次阅读
2025-04-30 17:39:54
2025年5月中国数据库流行度排行榜：OB一枝独秀破 800，金仓奋起直追跻四强
墨天轮编辑部
1188次阅读
2025-05-13 10:55:54
MySQL 30 周年庆！MySQL 8.4 认证免费考！这次是认真的。。。
严少安
922次阅读
2025-04-25 15:30:58
2025年4月国产数据库中标情况一览：4个千万元级项目，GaussDB与OceanBase大放异彩！
通讯员
818次阅读
2025-04-30 15:24:06
【活动】分享你的压箱底干货文档，三篇解锁进阶奖励！
墨天轮编辑部
544次阅读
2025-04-17 17:02:24
MySQL 8.0 OCP 1Z0-908 考试解析指南(二)
JiekeXu
373次阅读
2025-04-30 17:37:37
MySQL 30 周年庆！MySQL 8.4 认证免费考！这次是认真的。。。
数据库运维之道
309次阅读
2025-04-28 11:01:25
MogDB 发布更新，解决 openGauss 数据库在长事务情况下Ustore表膨胀问题
MogDB
309次阅读
2025-04-17 10:41:41
SQL优化 - explain查看SQL执行计划（下）
金同学
303次阅读
2025-05-06 14:40:00
热门文章
OceanBase 社区版入门教程第九期 如何快速拿下 OBCA & OBCP 认证
2022-09-27 6162浏览
实战教程第四章4.6：如何使用 DataX 加载 CSV 数据文件到 OceanBase
2022-09-28 2534浏览
从Oracle、MySQL到OceanBase的入门介绍，看这篇就够了！
2022-10-17 2491浏览
产品模块原理系列 | OceanBase总控服务到底是啥？一文详解RootService总控服务
2022-10-19 2344浏览
OceanBase备份恢复
2022-10-12 2282浏览
最新文章
TuGraph（OceanBase图数据库）小课堂010：TuGraph-DataX-Tool 如何使用？
2023-05-15 674浏览
TuGraph（OceanBase图数据库）小课堂009：TuGraph Browser 用户如何使用？
2023-05-15 817浏览
TuGraph（OceanBase图数据库）小课堂009：tugraph_cypher 如何使用？
2023-05-15 505浏览
TuGraph（OceanBase图数据库）小课堂008：TuGraph服务如何运维？
2023-05-15 527浏览
TuGraph（OceanBase图数据库）小课堂007：TuGraph如何配置服务？
2023-05-15 467浏览
