暂无图片
暂无图片
1
暂无图片
暂无图片
1
暂无图片

OceanBase 2.2集群实战第十一篇——备份恢复之异机恢复

原创 gelyon 2021-01-27
1678

说明:

本篇所提到的异机是指从一个集群的备份,恢复其中一个租户到另一个集群中。也就是说,两个OB集群挂载相同的nfs,两个集群都做备份后,将其中一个集群备份中的某个租户,恢复到另一个集群中。

经过实际测试,执行命令恢复时没报错,也没有任何恢复任务,但查看rs事件,提示恢复event是restore_failed失败的。。查看历史恢复表__all_restore_history,info信息提示不支持 ROOTSERVICE : OB_NOT_SUPPORTED(-4007)。

毕竟本人也是初学,可能方法不对,还是不支持,暂且先不管结果如何,这类需求应该还算是实际正常的需求,本文先记录于此,后续继续跟踪寻求解决方法。

参考上一篇备份恢复黑屏命令行实操:OceanBase 2.2集群实战第十篇——OceanBase的备份与恢复

之所以命令行操作备份恢复,对于dba们来说,与ocp流程化的操作相比,还是切实的键盘操作体验更要踏实亲切些。。。不知道广大dba朋友们有没有这种感觉。。。(>_<)


本系列总的专栏:一步一步学习OceanBase系列


(补充)问题解决:

异机恢复问题解决!
测试发现从测试区的物理备份,恢复到生产区,验证是成功的。。而生产区的物理备份无法恢复到测试区。
最终分析找到问题原因是因为生产区和测试区两个OB集群的LDC不一致导致。
生产区环境是同城三中心部署,测试区环境当时测试LDC时,修改成了两地三中心部署模式,导致生产区的备份恢复到测试区时就不支持,而测试区的备份可以恢复到生产区。

本篇实操之前,先对上一篇备份恢复作一下补充。 1、关于备份清理策略 • auto_delete_expired_backup 参数控制是否自动清理备份文件,默认为False不自动清理。 • backup_recovery_window 参数控制备份文件保留周期(以天为单位),默认为0。 如: alter system set auto_delete_expired_backup=true; alter system set backup_recovery_window=7; 2、关于 log archivelog status 日志归档状态问题 经过我几次测试发现,开启日志archivelog归档后,并且执行了一次合并,合并也是完成后,在执行备份的时候,还是会遇到报错,日志归档状态是INTERRUPTED或者BEGINNING obclient> alter system backup database; ERROR 9040 (HY000): backup can not start, because log archive is not doing. log archive status : INTERRUPTED. obclient> obclient> alter system backup database; ERROR 9040 (HY000): backup can not start, because log archive is not doing. log archive status : BEGINNING. obclient> 就想知道怎么查看当前日志归档状态信息? 关于日志归档状态,主要是查看内部表cdb_ob_backup_archivelog_summary中的status,当状态变成DOING时,才能进行后续的备份操作。 obclient> select * from cdb_ob_backup_archivelog_summary; +-------------+-------------------+-----------+--------+----------------------------+----------------------------+-------------+--------------+-------------------+---------------------+----------------------+ | INCARNATION | LOG_ARCHIVE_ROUND | TENANT_ID | STATUS | MIN_FIRST_TIME | MAX_NEXT_TIME | INPUT_BYTES | OUTPUT_BYTES | COMPRESSION_RATIO | INPUT_BYTES_DISPLAY | OUTPUT_BYTES_DISPLAY | +-------------+-------------------+-----------+--------+----------------------------+----------------------------+-------------+--------------+-------------------+---------------------+----------------------+ | 1 | 1 | 1 | STOP | 2021-01-26 17:36:56.981339 | 1970-01-01 08:00:00.000000 | 0 | 0 | NULL | 0.00MB | 0.00MB | | 1 | 2 | 1 | STOP | 2021-01-26 17:46:53.051170 | 2021-01-26 18:08:57.072835 | 0 | 0 | NULL | 0.00MB | 0.00MB | | 1 | 1 | 1001 | STOP | 2021-01-26 17:36:56.981339 | 1970-01-01 08:00:00.000000 | 0 | 0 | NULL | 0.00MB | 0.00MB | | 1 | 2 | 1001 | STOP | 2021-01-26 17:46:53.051170 | 2021-01-26 18:09:01.620397 | 0 | 0 | NULL | 0.00MB | 0.00MB | | 1 | 1 | 1002 | STOP | 2021-01-26 17:36:56.981339 | 1970-01-01 08:00:00.000000 | 0 | 0 | NULL | 0.00MB | 0.00MB | | 1 | 2 | 1002 | STOP | 2021-01-26 17:46:53.051170 | 2021-01-26 18:08:57.072835 | 0 | 0 | NULL | 0.00MB | 0.00MB | | 1 | 3 | 1 | DOING | 2021-01-26 18:49:24.340965 | 2021-01-27 09:28:44.928343 | 0 | 0 | NULL | 0.00MB | 0.00MB | | 1 | 3 | 1001 | DOING | 2021-01-26 18:49:24.340965 | 2021-01-27 09:28:44.928343 | 0 | 0 | NULL | 0.00MB | 0.00MB | | 1 | 3 | 1002 | DOING | 2021-01-26 18:49:24.340965 | 2021-01-27 09:29:36.882344 | 0 | 0 | NULL | 0.00MB | 0.00MB | | 1 | 3 | 1004 | DOING | 2021-01-26 18:49:24.340965 | 2021-01-27 09:29:36.882344 | 0 | 0 | NULL | 0.00MB | 0.00MB | +-------------+-------------------+-----------+--------+----------------------------+----------------------------+-------------+--------------+-------------------+---------------------+----------------------+ 10 rows in set (0.02 sec) obclient> log_archive_status 日志归档状态的种类,主要有: • STOP: 没有日志归档备份 • BEGINNING: 正在启动日志归档备份 • DOING: 正在日志归档备份 • STOPPING: 停止日志归档备份 • INTERRUPTED: 日志归档备份断流了 关于log_archive_status状态中,BEGINGING 一般是正常的,持续时间和租户数量有关,一般单租户 60s,两个租户 120s。 关于 INTERRUPTED状态:可以查rs事件信息,从内部表__all_rootservice_event_history可以看到哪个rs设置的INTERRUPTED,需要到对应的server上看rs的日志。 obclient> select * from __all_rootservice_event_history where module like '%archive%' order by gmt_create desc limit 30; +----------------------------+-------------+--------------------------+------------+-------------+-------+--------+--------+--------+-------+--------+-------+--------+-------+--------+------------+--------------+-------------+ | gmt_create | module | event | name1 | value1 | name2 | value2 | name3 | value3 | name4 | value4 | name5 | value5 | name6 | value6 | extra_info | rs_svr_ip | rs_svr_port | +----------------------------+-------------+--------------------------+------------+-------------+-------+--------+--------+--------+-------+--------+-------+--------+-------+--------+------------+--------------+-------------+ | 2021-01-26 18:52:47.454655 | log_archive | change_status | new_status | DOING | round | 3 | | | | | | | | | | 10.10.10.185 | 2882 | | 2021-01-26 18:49:24.387421 | log_archive | handle_start_log_archive | new_status | BEGINNING | round | 3 | result | 0 | | | | | | | | 10.10.10.185 | 2882 | | 2021-01-26 18:10:18.694953 | log_archive | change_status | new_status | STOP | round | 2 | | | | | | | | | | 10.10.10.185 | 2882 | | 2021-01-26 18:10:08.594152 | log_archive | handle_stop_log_archive | status | STOPPING | round | 2 | result | 0 | | | | | | | | 10.10.10.185 | 2882 | | 2021-01-26 17:49:45.024487 | log_archive | change_status | new_status | DOING | round | 2 | | | | | | | | | | 10.10.10.185 | 2882 | | 2021-01-26 17:46:53.114636 | log_archive | handle_start_log_archive | new_status | BEGINNING | round | 2 | result | 0 | | | | | | | | 10.10.10.185 | 2882 | | 2021-01-26 17:46:44.754471 | log_archive | change_status | new_status | STOP | round | 1 | | | | | | | | | | 10.10.10.185 | 2882 | | 2021-01-26 17:46:44.669171 | log_archive | handle_stop_log_archive | status | STOPPING | round | 1 | result | 0 | | | | | | | | 10.10.10.185 | 2882 | | 2021-01-26 17:37:07.399362 | log_archive | change_status | new_status | INTERRUPTED | round | 1 | | | | | | | | | | 10.10.10.185 | 2882 | | 2021-01-26 17:36:57.044151 | log_archive | handle_start_log_archive | new_status | BEGINNING | round | 1 | result | 0 | | | | | | | | 10.10.10.185 | 2882 | +----------------------------+-------------+--------------------------+------------+-------------+-------+--------+--------+--------+-------+--------+-------+--------+-------+--------+------------+--------------+-------------+ 10 rows in set (0.40 sec) obclient> 另外,如果当日志归档开启后,没有执行过合并,执行备份时也会报错,因为日志开启时间大于上一次合并冻结时间,需要执行一次合并。 obclient> alter system backup database; ERROR 9040 (HY000): backup can not start, because log archive start timestamp is bigger than frozen timestamp, need major freeze first. start timestamp : 1612147273821930, frozen timestamp : 1612116010844801 . obclient> 接下来开始本篇的异机恢复操作:(这里不详细介绍备份操作细节,具体怎么备份上一篇文章有介绍) 我这里两套OB集群: --生产区OB集群: cluster_id=2,cluster=obdemo , 有两个业务租户gjj_mysql_tent和gjj_ora_tent obclient> show parameters like 'cluster%'; +-------+----------+--------------+----------+------------+-----------+--------+---------------------+----------+---------+---------+-------------------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | +-------+----------+--------------+----------+------------+-----------+--------+---------------------+----------+---------+---------+-------------------+ | zone3 | observer | 10.10.10.65 | 2882 | cluster_id | NULL | 2 | ID of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone3 | observer | 10.10.10.65 | 2882 | cluster | NULL | obdemo | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone2 | observer | 10.10.10.196 | 2882 | cluster_id | NULL | 2 | ID of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone2 | observer | 10.10.10.196 | 2882 | cluster | NULL | obdemo | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone1 | observer | 10.10.10.185 | 2882 | cluster_id | NULL | 2 | ID of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone1 | observer | 10.10.10.185 | 2882 | cluster | NULL | obdemo | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | +-------+----------+--------------+----------+------------+-----------+--------+---------------------+----------+---------+---------+-------------------+ 6 rows in set (0.01 sec) obclient> source tent.sql +-----------+----------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ | tenant_id | tenant_name | compatibility_mode | zone_list | locality | primary_zone | gmt_modified | +-----------+----------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ | 1 | sys | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2,zone3 | 2020-12-21 15:57:13.995853 | | 1001 | gjj_mysql_tent | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1,zone2,zone3 | 2020-12-23 10:39:29.044498 | | 1002 | gjj_ora_tent | 1 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1,zone2,zone3 | 2020-12-23 14:52:09.341057 | +-----------+----------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ 3 rows in set (0.00 sec) obclient> --生产区obdemo集群的备份信息,可以看到做了一次全备 obclient> select * from cdb_ob_backup_set_details; +-------------+-----------+--------+---------+-------------+-----------------+----------------------------+----------------------------+------------------+------+------------+-------------+------------+--------------+-------------------+-------------------+----------------------+---------------------------+--------------------+-----------+ | INCARNATION | TENANT_ID | BS_KEY | COPY_ID | BACKUP_TYPE | ENCRYPTION_MODE | START_TIME | COMPLETION_TIME | ELAPSED_SECONDES | KEEP | KEEP_UNTIL | DEVICE_TYPE | COMPRESSED | OUTPUT_BYTES | OUTPUT_RATE_BYTES | COMPRESSION_RATIO | OUTPUT_BYTES_DISPLAY | OUTPUT_RATE_BYTES_DISPLAY | TIME_TAKEN_DISPLAY | STATUS | +-------------+-----------+--------+---------+-------------+-----------------+----------------------------+----------------------------+------------------+------+------------+-------------+------------+--------------+-------------------+-------------------+----------------------+---------------------------+--------------------+-----------+ | 1 | 1 | 1 | 0 | D | NONE | 2021-01-26 17:53:49.673935 | 2021-01-26 17:54:45.213432 | 56 | NO | | FILE | NO | 0 | 0.0000 | NULL | 0.00MB | 0.00MB/S | 00:00:55.539497 | COMPLETED | | 1 | 1001 | 1 | 0 | D | NONE | 2021-01-26 17:53:49.673935 | 2021-01-26 17:54:01.109643 | 11 | NO | | FILE | NO | 9172678 | 802108.4484 | 0.02 | 8.75MB | 0.76MB/S | 00:00:11.435708 | COMPLETED | | 1 | 1002 | 1 | 0 | D | NONE | 2021-01-26 17:53:49.673935 | 2021-01-26 17:54:44.751065 | 55 | NO | | FILE | NO | 972533740 | 17657669.1632 | 0.27 | 927.48MB | 16.84MB/S | 00:00:55.077130 | COMPLETED | +-------------+-----------+--------+---------+-------------+-----------------+----------------------------+----------------------------+------------------+------+------------+-------------+------------+--------------+-------------------+-------------------+----------------------+---------------------------+--------------------+-----------+ 3 rows in set (0.03 sec) obclient> --测试区OB集群: cluster_id=1,cluster=ob2273 obclient> show parameters like 'cluster%'; +-------+----------+--------------+----------+------------+-----------+--------+---------------------+----------+---------+---------+-------------------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | +-------+----------+--------------+----------+------------+-----------+--------+---------------------+----------+---------+---------+-------------------+ | zone1 | observer | 10.10.10.91 | 2882 | cluster_id | NULL | 1 | ID of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone1 | observer | 10.10.10.91 | 2882 | cluster | NULL | ob2273 | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone2 | observer | 10.10.10.202 | 2882 | cluster_id | NULL | 1 | ID of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone2 | observer | 10.10.10.202 | 2882 | cluster | NULL | ob2273 | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone3 | observer | 10.10.10.131 | 2882 | cluster_id | NULL | 1 | ID of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone3 | observer | 10.10.10.131 | 2882 | cluster | NULL | ob2273 | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | +-------+----------+--------------+----------+------------+-----------+--------+---------------------+----------+---------+---------+-------------------+ 6 rows in set (0.01 sec) obclient> source tent.sql +-----------+------------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ | tenant_id | tenant_name | compatibility_mode | zone_list | locality | primary_zone | gmt_modified | +-----------+------------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ | 1 | sys | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2,zone3 | 2021-01-06 19:20:14.239456 | | 1002 | mysql_tent_restore | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2 | 2021-01-25 18:09:40.325184 | | 1003 | mysql_tent_restore_inc | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2 | 2021-01-25 18:21:04.674520 | | 1005 | mysql_test_tent | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2 | 2021-01-26 10:33:36.554032 | +-----------+------------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ 4 rows in set (0.00 sec) obclient> --测试区ob2273集群的备份信息 obclient> select * from cdb_ob_backup_set_details; +-------------+-----------+--------+---------+-------------+-----------------+----------------------------+----------------------------+------------------+------+------------+-------------+------------+--------------+-------------------+-------------------+----------------------+---------------------------+--------------------+-----------+ | INCARNATION | TENANT_ID | BS_KEY | COPY_ID | BACKUP_TYPE | ENCRYPTION_MODE | START_TIME | COMPLETION_TIME | ELAPSED_SECONDES | KEEP | KEEP_UNTIL | DEVICE_TYPE | COMPRESSED | OUTPUT_BYTES | OUTPUT_RATE_BYTES | COMPRESSION_RATIO | OUTPUT_BYTES_DISPLAY | OUTPUT_RATE_BYTES_DISPLAY | TIME_TAKEN_DISPLAY | STATUS | +-------------+-----------+--------+---------+-------------+-----------------+----------------------------+----------------------------+------------------+------+------------+-------------+------------+--------------+-------------------+-------------------+----------------------+---------------------------+--------------------+-----------+ | 1 | 1 | 1 | 0 | D | NONE | 2021-01-25 17:44:03.774420 | 2021-01-25 17:44:07.530663 | 4 | NO | | FILE | NO | 0 | 0.0000 | NULL | 0.00MB | 0.00MB/S | 00:00:03.756243 | COMPLETED | | 1 | 1 | 2 | 0 | I | NONE | 2021-01-25 17:49:42.781455 | 2021-01-25 17:49:48.266004 | 5 | NO | | FILE | NO | 0 | 0.0000 | NULL | 0.00MB | 0.00MB/S | 00:00:05.484549 | COMPLETED | | 1 | 1 | 3 | 0 | D | NONE | 2021-01-26 09:45:12.217643 | 2021-01-26 09:45:19.192917 | 7 | NO | | FILE | NO | 0 | 0.0000 | NULL | 0.00MB | 0.00MB/S | 00:00:06.975274 | COMPLETED | | 1 | 1 | 4 | 0 | I | NONE | 2021-01-26 09:46:55.798867 | 2021-01-26 09:47:08.656009 | 13 | NO | | FILE | NO | 0 | 0.0000 | NULL | 0.00MB | 0.00MB/S | 00:00:12.857142 | COMPLETED | | 1 | 1001 | 1 | 0 | D | NONE | 2021-01-25 17:44:03.774420 | 2021-01-25 17:44:06.913808 | 3 | NO | | FILE | NO | 1242592 | 395807.0809 | 0.02 | 1.19MB | 0.38MB/S | 00:00:03.139388 | COMPLETED | | 1 | 1001 | 2 | 0 | I | NONE | 2021-01-25 17:49:42.781455 | 2021-01-25 17:49:47.739042 | 5 | NO | | FILE | NO | 882144 | 177938.1784 | 5.03 | 0.84MB | 0.17MB/S | 00:00:04.957587 | COMPLETED | | 1 | 1001 | 3 | 0 | D | NONE | 2021-01-26 09:45:12.217643 | 2021-01-26 09:45:15.843197 | 4 | NO | | FILE | NO | 1249325 | 344588.7166 | 0.02 | 1.19MB | 0.33MB/S | 00:00:03.625554 | COMPLETED | | 1 | 1001 | 4 | 0 | I | NONE | 2021-01-26 09:46:55.798867 | 2021-01-26 09:47:01.067561 | 5 | NO | | FILE | NO | 888877 | 168709.1716 | 4.99 | 0.85MB | 0.16MB/S | 00:00:05.268694 | COMPLETED | | 1 | 1002 | 3 | 0 | D | NONE | 2021-01-26 09:45:12.217643 | 2021-01-26 09:45:16.959876 | 5 | NO | | FILE | NO | 1234467 | 260313.4431 | 0.02 | 1.18MB | 0.25MB/S | 00:00:04.742233 | COMPLETED | | 1 | 1002 | 4 | 0 | I | NONE | 2021-01-26 09:46:55.798867 | 2021-01-26 09:47:04.490146 | 9 | NO | | FILE | NO | 882211 | 101505.3135 | 5.03 | 0.84MB | 0.10MB/S | 00:00:08.691279 | COMPLETED | | 1 | 1003 | 3 | 0 | D | NONE | 2021-01-26 09:45:12.217643 | 2021-01-26 09:45:18.469056 | 6 | NO | | FILE | NO | 1247146 | 199498.2574 | 0.02 | 1.19MB | 0.19MB/S | 00:00:06.251413 | COMPLETED | | 1 | 1003 | 4 | 0 | I | NONE | 2021-01-26 09:46:55.798867 | 2021-01-26 09:47:07.947229 | 12 | NO | | FILE | NO | 886698 | 72989.0993 | 5.04 | 0.85MB | 0.07MB/S | 00:00:12.148362 | COMPLETED | +-------------+-----------+--------+---------+-------------+-----------------+----------------------------+----------------------------+------------------+------+------------+-------------+------------+--------------+-------------------+-------------------+----------------------+---------------------------+--------------------+-----------+ 12 rows in set (0.02 sec) obclient> 由于生产区和测试区的两个OB集群,都挂载的同一个nfs,并且有权限读取nfs目录的所有文件权限。 [root@obnfs201 ~]# cd /obnfs/ [root@obnfs201 obnfs]# ll total 0 drwx------ 3 500 500 15 Jan 25 17:00 ob2273 drwx------ 3 500 500 15 Jan 26 17:36 obdemo [root@obnfs201 obnfs]# du -sh * 794M ob2273 3.1G obdemo [root@obnfs201 obnfs]# 现在要将生产区obdemo集群的租户恢复到测试区ob2273集群中。 登录测试区ob2273集群 [admin@ocp201 my]$ obclient -h10.10.10.33 -P2883 -uroot@sys#ob2273 -pYH_admin123.com -c -A oceanbase obclient: [Warning] Using a password on the command line interface can be insecure. Welcome to the OceanBase monitor. Commands end with ; or \g. Your OceanBase connection id is 2782033 Server version: 5.6.25 OceanBase 2.2.75 (r20210107182621-81357ec10e1342ef9f9e993ea38ef1cdd8778cf6) (Built Jan 7 2021 18:54:53) Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. obclient> show parameters like 'cluster%'; +-------+----------+--------------+----------+------------+-----------+--------+---------------------+----------+---------+---------+-------------------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | +-------+----------+--------------+----------+------------+-----------+--------+---------------------+----------+---------+---------+-------------------+ | zone2 | observer | 10.10.10.202 | 2882 | cluster_id | NULL | 1 | ID of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone2 | observer | 10.10.10.202 | 2882 | cluster | NULL | ob2273 | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone3 | observer | 10.10.10.131 | 2882 | cluster_id | NULL | 1 | ID of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone3 | observer | 10.10.10.131 | 2882 | cluster | NULL | ob2273 | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone1 | observer | 10.10.10.91 | 2882 | cluster_id | NULL | 1 | ID of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | | zone1 | observer | 10.10.10.91 | 2882 | cluster | NULL | ob2273 | Name of the cluster | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE | +-------+----------+--------------+----------+------------+-----------+--------+---------------------+----------+---------+---------+-------------------+ 6 rows in set (0.00 sec) obclient> source tent.sql +-----------+------------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ | tenant_id | tenant_name | compatibility_mode | zone_list | locality | primary_zone | gmt_modified | +-----------+------------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ | 1 | sys | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2,zone3 | 2021-01-06 19:20:14.239456 | | 1002 | mysql_tent_restore | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2 | 2021-01-25 18:09:40.325184 | | 1003 | mysql_tent_restore_inc | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2 | 2021-01-25 18:21:04.674520 | | 1005 | mysql_test_tent | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2 | 2021-01-26 10:33:36.554032 | +-----------+------------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ 4 rows in set (0.01 sec) obclient> source unit_config.sql +----------------+-----------------+---------+---------+------------+------------+------------------+ | unit_config_id | name | max_cpu | min_cpu | max_mem_gb | min_mem_gb | max_disk_size_gb | +----------------+-----------------+---------+---------+------------+------------+------------------+ | 1 | sys_unit_config | 5 | 2.5 | 16 | 12 | 380 | | 1001 | my_unit_1c5g | 1 | 1 | 5 | 5 | 20 | +----------------+-----------------+---------+---------+------------+------------+------------------+ 2 rows in set (0.00 sec) obclient> obclient> source pool.sql +--------------------------+------------------+---------+---------+------------+------------+---------+-------+-------------------+-----------+------------------------+ | resource_pool_name | unit_config_name | max_cpu | min_cpu | max_mem_gb | min_mem_gb | unit_id | zone | observer | tenant_id | tenant_name | +--------------------------+------------------+---------+---------+------------+------------+---------+-------+-------------------+-----------+------------------------+ | sys_pool | sys_unit_config | 5 | 2.5 | 16 | 12 | 1 | zone1 | 10.10.10.91:2882 | 1 | sys | | sys_pool | sys_unit_config | 5 | 2.5 | 16 | 12 | 2 | zone2 | 10.10.10.202:2882 | 1 | sys | | sys_pool | sys_unit_config | 5 | 2.5 | 16 | 12 | 3 | zone3 | 10.10.10.131:2882 | 1 | sys | | mysql_pool_test | my_unit_1c5g | 1 | 1 | 5 | 5 | 1001 | zone1 | 10.10.10.91:2882 | 1005 | mysql_test_tent | | mysql_pool_test | my_unit_1c5g | 1 | 1 | 5 | 5 | 1002 | zone2 | 10.10.10.202:2882 | 1005 | mysql_test_tent | | mysql_pool_test | my_unit_1c5g | 1 | 1 | 5 | 5 | 1003 | zone3 | 10.10.10.131:2882 | 1005 | mysql_test_tent | | mysql_pool_test_restore | my_unit_1c5g | 1 | 1 | 5 | 5 | 1004 | zone1 | 10.10.10.91:2882 | 1002 | mysql_tent_restore | | mysql_pool_test_restore | my_unit_1c5g | 1 | 1 | 5 | 5 | 1005 | zone2 | 10.10.10.202:2882 | 1002 | mysql_tent_restore | | mysql_pool_test_restore | my_unit_1c5g | 1 | 1 | 5 | 5 | 1006 | zone3 | 10.10.10.131:2882 | 1002 | mysql_tent_restore | | mysql_pool_test_restore2 | my_unit_1c5g | 1 | 1 | 5 | 5 | 1007 | zone1 | 10.10.10.91:2882 | 1003 | mysql_tent_restore_inc | | mysql_pool_test_restore2 | my_unit_1c5g | 1 | 1 | 5 | 5 | 1008 | zone2 | 10.10.10.202:2882 | 1003 | mysql_tent_restore_inc | | mysql_pool_test_restore2 | my_unit_1c5g | 1 | 1 | 5 | 5 | 1009 | zone3 | 10.10.10.131:2882 | 1003 | mysql_tent_restore_inc | +--------------------------+------------------+---------+---------+------------+------------+---------+-------+-------------------+-----------+------------------------+ 12 rows in set (0.00 sec) obclient> create resource unit my_unit_2c16g max_cpu=2, min_cpu=2, max_memory='16g', min_memory='16g', max_iops=10000, min_iops=1000, max_session_num=1000000, max_disk_size='100g'; Query OK, 0 rows affected (0.01 sec) obclient> source unit_config.sql +----------------+-----------------+---------+---------+------------+------------+------------------+ | unit_config_id | name | max_cpu | min_cpu | max_mem_gb | min_mem_gb | max_disk_size_gb | +----------------+-----------------+---------+---------+------------+------------+------------------+ | 1 | sys_unit_config | 5 | 2.5 | 16 | 12 | 380 | | 1001 | my_unit_1c5g | 1 | 1 | 5 | 5 | 20 | | 1002 | my_unit_2c16g | 2 | 2 | 16 | 16 | 100 | +----------------+-----------------+---------+---------+------------+------------+------------------+ 3 rows in set (0.00 sec) obclient> create resource pool gjj_pool_restore unit = 'my_unit_2c16g', unit_num = 1; Query OK, 0 rows affected (0.06 sec) obclient> source pool.sql +--------------------------+------------------+---------+---------+------------+------------+---------+-------+-------------------+-----------+------------------------+ | resource_pool_name | unit_config_name | max_cpu | min_cpu | max_mem_gb | min_mem_gb | unit_id | zone | observer | tenant_id | tenant_name | +--------------------------+------------------+---------+---------+------------+------------+---------+-------+-------------------+-----------+------------------------+ | sys_pool | sys_unit_config | 5 | 2.5 | 16 | 12 | 1 | zone1 | 10.10.10.91:2882 | 1 | sys | | sys_pool | sys_unit_config | 5 | 2.5 | 16 | 12 | 2 | zone2 | 10.10.10.202:2882 | 1 | sys | | sys_pool | sys_unit_config | 5 | 2.5 | 16 | 12 | 3 | zone3 | 10.10.10.131:2882 | 1 | sys | | mysql_pool_test | my_unit_1c5g | 1 | 1 | 5 | 5 | 1001 | zone1 | 10.10.10.91:2882 | 1005 | mysql_test_tent | | mysql_pool_test | my_unit_1c5g | 1 | 1 | 5 | 5 | 1002 | zone2 | 10.10.10.202:2882 | 1005 | mysql_test_tent | | mysql_pool_test | my_unit_1c5g | 1 | 1 | 5 | 5 | 1003 | zone3 | 10.10.10.131:2882 | 1005 | mysql_test_tent | | mysql_pool_test_restore | my_unit_1c5g | 1 | 1 | 5 | 5 | 1004 | zone1 | 10.10.10.91:2882 | 1002 | mysql_tent_restore | | mysql_pool_test_restore | my_unit_1c5g | 1 | 1 | 5 | 5 | 1005 | zone2 | 10.10.10.202:2882 | 1002 | mysql_tent_restore | | mysql_pool_test_restore | my_unit_1c5g | 1 | 1 | 5 | 5 | 1006 | zone3 | 10.10.10.131:2882 | 1002 | mysql_tent_restore | | mysql_pool_test_restore2 | my_unit_1c5g | 1 | 1 | 5 | 5 | 1007 | zone1 | 10.10.10.91:2882 | 1003 | mysql_tent_restore_inc | | mysql_pool_test_restore2 | my_unit_1c5g | 1 | 1 | 5 | 5 | 1008 | zone2 | 10.10.10.202:2882 | 1003 | mysql_tent_restore_inc | | mysql_pool_test_restore2 | my_unit_1c5g | 1 | 1 | 5 | 5 | 1009 | zone3 | 10.10.10.131:2882 | 1003 | mysql_tent_restore_inc | | gjj_pool_restore | my_unit_2c16g | 2 | 2 | 16 | 16 | 1013 | zone1 | 10.10.10.91:2882 | NULL | NULL | | gjj_pool_restore | my_unit_2c16g | 2 | 2 | 16 | 16 | 1014 | zone2 | 10.10.10.202:2882 | NULL | NULL | | gjj_pool_restore | my_unit_2c16g | 2 | 2 | 16 | 16 | 1015 | zone3 | 10.10.10.131:2882 | NULL | NULL | +--------------------------+------------------+---------+---------+------------+------------+---------+-------+-------------------+-----------+------------------------+ 15 rows in set (0.00 sec) obclient> obclient> 开始异机恢复,将生产区obdemo集群中的gjj_mysql_tent租户,恢复到测试区ob2273集群中。 obclient> obclient> alter system restore gjj_mysql_tent from gjj_mysql_tent -> at 'file:///obbackup' -> until '2021-01-26 17:54:01.109643' -> with 'backup_cluster_name=obdemo&backup_cluster_id=2&restore_cluster_name=ob2273&restore_cluster_id=1&pool_list=gjj_pool_restore&locality=F@zone1,F@zone2,F@zone3'; Query OK, 0 rows affected (0.02 sec) obclient> obclient> source tent.sql +-----------+------------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ | tenant_id | tenant_name | compatibility_mode | zone_list | locality | primary_zone | gmt_modified | +-----------+------------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ | 1 | sys | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2,zone3 | 2021-01-06 19:20:14.239456 | | 1002 | mysql_tent_restore | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2 | 2021-01-25 18:09:40.325184 | | 1003 | mysql_tent_restore_inc | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2 | 2021-01-25 18:21:04.674520 | | 1005 | mysql_test_tent | 0 | zone1;zone2;zone3 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 | zone1;zone2 | 2021-01-26 10:33:36.554032 | +-----------+------------------------+--------------------+-------------------+---------------------------------------------+-------------------+----------------------------+ 4 rows in set (0.00 sec) obclient> select * from __all_restore_info; Empty set (0.00 sec) obclient> 以上可以看到,执行restore恢复命令后,没有报错,但是也没有恢复任务在执行。。 查看rs事件,提示恢复event是restore_failed失败的。。 obclient> select * from __all_rootservice_event_history where module like '%restore%' order by gmt_create desc limit 30; +----------------------------+------------------+-----------------------+--------+-----------------+-------------+-----------------+--------+------------------------+-------+--------+-------+--------+-------+--------+------------+-------------+-------------+ | gmt_create | module | event | name1 | value1 | name2 | value2 | name3 | value3 | name4 | value4 | name5 | value5 | name6 | value6 | extra_info | rs_svr_ip | rs_svr_port | +----------------------------+------------------+-----------------------+--------+-----------------+-------------+-----------------+--------+------------------------+-------+--------+-------+--------+-------+--------+------------+-------------+-------------+ | 2021-01-26 18:04:56.362187 | physical_restore | restore_failed | tenant | gjj_mysql_tent | | | | | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 18:04:56.328814 | physical_restore | change_restore_status | job_id | 10 | tenant | gjj_mysql_tent | status | RESTORE_FAIL | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 18:04:56.212048 | physical_restore | change_restore_status | job_id | 10 | tenant_name | gjj_mysql_tent | status | CREATE_TENANT | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 18:04:56.211978 | physical_restore | restore_start | ret | 0 | tenant_name | gjj_mysql_tent | | | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:33:36.638068 | physical_restore | restore_success | tenant | mysql_test_tent | | | | | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:33:36.591701 | physical_restore | change_restore_status | job_id | 9 | tenant | mysql_test_tent | status | RESTORE_SUCCESS | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:33:36.551392 | physical_restore | change_restore_status | job_id | 9 | tenant | mysql_test_tent | status | POST_CHECK | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:33:36.510388 | physical_restore | change_restore_status | job_id | 9 | tenant | mysql_test_tent | status | REBUILD_INDEX | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:30:14.838115 | physical_restore | change_restore_status | job_id | 9 | tenant | mysql_test_tent | status | RESTORE_USER_REPLICA | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:30:14.003104 | physical_restore | change_restore_status | job_id | 9 | tenant | mysql_test_tent | status | CREATE_USER_PARTITIONS | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:30:08.857739 | physical_restore | change_restore_status | job_id | 9 | tenant | mysql_test_tent | status | MODIFY_SCHEMA | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:30:08.833693 | physical_restore | change_restore_status | job_id | 9 | tenant | mysql_test_tent | status | UPGRADE_POST | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:30:08.650592 | physical_restore | change_restore_status | job_id | 9 | tenant | mysql_test_tent | status | UPGRADE_PRE | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:26:47.026405 | physical_restore | change_restore_status | job_id | 9 | tenant | mysql_test_tent | status | RESTORE_SYS_REPLICA | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:26:46.550141 | physical_restore | change_restore_status | job_id | 9 | tenant_name | mysql_test_tent | status | CREATE_TENANT | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:26:46.550115 | physical_restore | restore_start | ret | 0 | tenant_name | mysql_test_tent | | | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:09:21.531708 | physical_restore | restore_success | tenant | mysql_test_tent | | | | | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:09:21.482865 | physical_restore | change_restore_status | job_id | 8 | tenant | mysql_test_tent | status | RESTORE_SUCCESS | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:09:21.433395 | physical_restore | change_restore_status | job_id | 8 | tenant | mysql_test_tent | status | POST_CHECK | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:09:21.416419 | physical_restore | change_restore_status | job_id | 8 | tenant | mysql_test_tent | status | REBUILD_INDEX | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:05:59.424783 | physical_restore | change_restore_status | job_id | 8 | tenant | mysql_test_tent | status | RESTORE_USER_REPLICA | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:05:58.869420 | physical_restore | change_restore_status | job_id | 8 | tenant | mysql_test_tent | status | CREATE_USER_PARTITIONS | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:05:53.638764 | physical_restore | change_restore_status | job_id | 8 | tenant | mysql_test_tent | status | MODIFY_SCHEMA | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:05:53.626525 | physical_restore | change_restore_status | job_id | 8 | tenant | mysql_test_tent | status | UPGRADE_POST | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:05:53.246260 | physical_restore | change_restore_status | job_id | 8 | tenant | mysql_test_tent | status | UPGRADE_PRE | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:02:31.456649 | physical_restore | change_restore_status | job_id | 8 | tenant | mysql_test_tent | status | RESTORE_SYS_REPLICA | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:02:30.965806 | physical_restore | change_restore_status | job_id | 8 | tenant_name | mysql_test_tent | status | CREATE_TENANT | | | | | | | | 10.10.10.91 | 2882 | | 2021-01-26 10:02:30.965768 | physical_restore | restore_start | ret | 0 | tenant_name | mysql_test_tent | | | | | | | | | | 10.10.10.91 | 2882 | +----------------------------+------------------+-----------------------+--------+-----------------+-------------+-----------------+--------+------------------------+-------+--------+-------+--------+-------+--------+------------+-------------+-------------+ 28 rows in set (0.00 sec) obclient> 查看历史恢复表__all_restore_history,info信息提示不支持 ROOTSERVICE : OB_NOT_SUPPORTED(-4007) obclient> select * from __all_restore_history order by gmt_create; +----------------------------+----------------------------+--------+-----------------+-----------+------------------------+-----------------+----------------------------+----------------------------+----------+-----------------+-----------------+------------------------+-------------------+--------------------------+----------------------------+----------------------------+---------------------------+----------------------+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+-------------------+---------------------+------------------+--------------------+ | gmt_create | gmt_modified | job_id | external_job_id | tenant_id | tenant_name | status | start_time | completion_time | pg_count | finish_pg_count | partition_count | finish_partition_count | macro_block_count | finish_macro_block_count | restore_start_timestamp | restore_finish_timestamp | restore_current_timestamp | restore_data_version | backup_dest | restore_option | info | backup_cluster_id | backup_cluster_name | backup_tenant_id | backup_tenant_name | +----------------------------+----------------------------+--------+-----------------+-----------+------------------------+-----------------+----------------------------+----------------------------+----------+-----------------+-----------------+------------------------+-------------------+--------------------------+----------------------------+----------------------------+---------------------------+----------------------+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+-------------------+---------------------+------------------+--------------------+ | 2021-01-26 10:09:21.514640 | 2021-01-26 10:09:21.514640 | 8 | -1 | 1004 | mysql_test_tent | RESTORE_SUCCESS | 2021-01-26 10:02:30.949717 | 2021-01-26 10:09:21.514640 | 18 | 18 | 18 | 18 | -1 | -1 | 2021-01-26 09:39:55.676198 | 2021-01-26 09:45:15.843197 | NULL | 26 | file:///obbackup | backup_cluster_name=ob2273&backup_cluster_id=1&pool_list=mysql_pool_test&locality=F@zone1,F@zone2,F@zone3 | | 1 | ob2273 | 1001 | mysql_test_tent | | 2021-01-26 10:33:36.616247 | 2021-01-26 10:33:36.616247 | 9 | -1 | 1005 | mysql_test_tent | RESTORE_SUCCESS | 2021-01-26 10:26:46.542645 | 2021-01-26 10:33:36.616247 | 18 | 18 | 18 | 18 | -1 | -1 | 2021-01-26 09:39:55.676198 | 2021-01-26 09:47:01.067561 | NULL | 26 | file:///obbackup | backup_cluster_name=ob2273&backup_cluster_id=1&pool_list=mysql_pool_test&locality=F@zone1,F@zone2,F@zone3 | | 1 | ob2273 | 1001 | mysql_test_tent | | 2021-01-26 18:04:56.349765 | 2021-01-26 18:04:56.349765 | 10 | -1 | 0 | gjj_mysql_tent | RESTORE_FAIL | 2021-01-26 18:04:56.173193 | 2021-01-26 18:04:56.349765 | -1 | -1 | -1 | -1 | -1 | -1 | NULL | 2021-01-26 17:54:01.109643 | NULL | 0 | file:///obbackup | backup_cluster_name=obdemo&backup_cluster_id=2&pool_list=gjj_mysql_pool&locality=F@zone1,F@zone2,F@zone3 | ROOTSERVICE : OB_NOT_SUPPORTED(-4007) | 2 | obdemo | 0 | gjj_mysql_tent | +----------------------------+----------------------------+--------+-----------------+-----------+------------------------+-----------------+----------------------------+----------------------------+----------+-----------------+-----------------+------------------------+-------------------+--------------------------+----------------------------+----------------------------+---------------------------+----------------------+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------+-------------------+---------------------+------------------+--------------------+ 3 rows in set (0.00 sec) obclient> 到此,OceanBase的异机恢复体验到此结束。 一步一步学习OceanBase系列
复制
最后修改时间:2021-02-01 14:04:53
「喜欢这篇文章,您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容,转载时必须标注文章的来源(墨天轮),文章链接,文章作者等基本信息,否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容,欢迎发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

文章被以下合辑收录

评论

gelyon
暂无图片
4年前
评论
暂无图片 1
异机恢复问题解决!测试发现从测试区的备份,恢复到生产区,验证是成功的。。最终找到问题原因LDC不一致导致。生产区是同城三中心部署,测试区当时测试LDC时,修改成了两地三中心部署模式,生产区的备份恢复到测试区时就不支持,而测试区的备份可以恢复到生产区。
4年前
暂无图片 1
评论