OceanBase OMS安装失败，创建oms 元数据租户时报ERROR 4179 (HY000) at line 1: Operation not allowed now

本系列总的专栏：一步一步学习OceanBase系列

本篇记录下在oms1.4.3安装失败，记录下整个分析过程。

安装oms时，在创建oms 元数据oms_tenant租户时报错，ERROR 4179 (HY000) at line 1: Operation not allowed now。
导致oms安装失败，通过ocp界面和命令行，在ocp 元数据所在的obcluster集群下，创建同样的租户也是报这个错误。

版本：ocp2.5.0，obcluster版本2.7.73，oms1.4.3

问题定位是ocp meta元数据集群出问题导致，但查看元数据集群状态是正常的，原因未找到，最后重装ocp解决。

oms安装报错：

[root@ocp201 t-oceanbase-antman]# ./install.sh -i 9
run install.sh with DEBUG=FALSE, INSTALL_STEPS=9 CLEAR_STEPS= CONFIG_FILE=/root/t-oceanbase-antman/obcluster.conf
[2021-02-23 18:46:09.112685] INFO [start antman API service]
LB_MODE=none
Tue Feb 23 18:46:09 CST 2021
[2021-02-23 18:46:09.638290] INFO [step9: check whether OMS port 8088 is in use or not on 10.10.10.145]
[2021-02-23 18:46:09.774685] INFO [step9: OMS port 8088 is idle on 10.10.10.145]
[2021-02-23 18:46:11.286924] INFO [step9: installing oms, logfile: /root/t-oceanbase-antman/logs/install_oms.log]
[2021-02-23 18:46:11.295632] INFO [installing oms on 10.10.10.145, pid=23240 log: /root/t-oceanbase-antman/logs/install_oms_10.10.10.145.log]
[2021-02-23 18:46:30.673533] INFO [create OMS meta tenant]
ERROR 4179 (HY000) at line 1: Operation not allowed now
ERROR 4179 (HY000) at line 1: Operation not allowed now
ERROR 1045 (42000): Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: NO)
ERROR 1045 (42000): Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: YES)
ERROR 1045 (42000): Access denied for user 'root'@'xxx.xxx.xxx.xxx' (using password: YES)
[2021-02-23 18:46:32.746152] INFO [replace params in /root/t-oceanbase-antman/config/oms_conf.yaml]
[2021-02-23 18:46:32.876418] INFO [load docker image: docker load -i oms_oms.feature_1.4.3.202012041419.tar.gz]
Loaded image: acs-reg.alipay.com/oceanbase/oms-all-in-one:feature_1.4.3
[2021-02-23 18:49:05.678194] INFO [start container: docker run -d -it     --name oms     --net=host     --cpu-period 100000     --cpu-quota 1200000     --memory=24G     -v /etc/oms_conf.yaml:/home/admin/conf/co
nfig.yaml     --restart on-failure:5     --privileged=true     acs-reg.alipay.com/oceanbase/oms-all-in-one:feature_1.4.3]a7b7db36486932030155d4b46f6174b60fa76ced676841f88b6cb47131e4f9cb
[2021-02-23 18:49:13.664445] INFO [docker init: docker exec oms /bin/bash -c "bash /root/docker_init.sh"]
.....
.....
[2021-02-23 18:49:14.507136] ERROR [oms init function return 1]
[2021-02-23 18:49:14.534831] WARN [error ERROR exists in /root/t-oceanbase-antman/logs/install_oms_10.10.10.145.log]
[2021-02-23 18:49:14.538937] ERROR [ANTMAN-311: error occurs in /root/t-oceanbase-antman/logs/install_oms_10.10.10.145.log]
[2021-02-23 18:49:14.549977] ERROR [ANTMAN-314: ERROR occurred in install_oms, install.sh exit]
[root@ocp201 t-oceanbase-antman]#

复制

手动命令行创建oms_tenant租户报错一样：

[root@ocp201 t-oceanbase-antman]# su - admin
Last login: Tue Feb 23 19:27:34 CST 2021 on pts/2
[admin@ocp201 ~]$ cd my
[admin@ocp201 my]$ obclient -h10.10.10.33 -P2883 -uroot@sys#obcluster -p8I%JWGpMi2 -c -A oceanbase
obclient: [Warning] Using a password on the command line interface can be insecure.
Welcome to the OceanBase monitor.  Commands end with ; or \g.
Your OceanBase connection id is 1929201
Server version: 5.6.25 OceanBase 2.2.73 (r20201211175814-5abff097aadf740bcae07de8f84321815907e062) (Built Dec 11 2020 18:26:06)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

obclient> source servers.sql
+----------------+-------------+----------+------------+-----------------+--------+----------------------------+----------------------------+-----------------------+
| zone           | svr_ip      | svr_port | inner_port | with_rootserver | status | gmt_create                 | start_service_time         | build_version         |
+----------------+-------------+----------+------------+-----------------+--------+----------------------------+----------------------------+-----------------------+
| META_OB_ZONE_1 | 10.10.10.33 |     2882 |       2881 |               1 | active | 2021-01-06 18:40:45.274610 | 2021-01-18 09:21:16.201942 | 2.2.73_20201211175814 |
+----------------+-------------+----------+------------+-----------------+--------+----------------------------+----------------------------+-----------------------+
1 row in set (0.00 sec)

obclient> source tent.sql
+-----------+-------------+--------------------+----------------+------------------------+----------------+----------------------------+
| tenant_id | tenant_name | compatibility_mode | zone_list      | locality               | primary_zone   | gmt_modified               |
+-----------+-------------+--------------------+----------------+------------------------+----------------+----------------------------+
|         1 | sys         |                  0 | META_OB_ZONE_1 | FULL{1}@META_OB_ZONE_1 | META_OB_ZONE_1 | 2021-01-06 18:40:56.670509 |
|      1001 | ocp_meta    |                  0 | META_OB_ZONE_1 | FULL{1}@META_OB_ZONE_1 | META_OB_ZONE_1 | 2021-01-06 18:43:42.430089 |
|      1002 | ocp_monitor |                  0 | META_OB_ZONE_1 | FULL{1}@META_OB_ZONE_1 | META_OB_ZONE_1 | 2021-01-06 18:43:44.368676 |
+-----------+-------------+--------------------+----------------+------------------------+----------------+----------------------------+
3 rows in set (0.01 sec)

obclient> create resource unit oms_unit_2c5g max_cpu=2, min_cpu=2, max_memory='5g', min_memory='5g', max_iops=10000, min_iops=1000, max_session_num=1000000, max_disk_size='100g';
Query OK, 0 rows affected (0.01 sec)

obclient> create resource pool oms_pool unit = 'oms_unit_2c5g', unit_num = 1;
Query OK, 0 rows affected (0.01 sec)

obclient> create tenant oms_tenant resource_pool_list=('oms_pool'), primary_zone='META_OB_ZONE_1',comment 'mysql tenant/instance', charset='utf8' set ob_tcp_invited_nodes='%', ob_compatibility_mode='mysql';
ERROR 4179 (HY000): Operation not allowed now
obclient> 

复制

排除obproxy问题，进入ocp meta docker容器，直连ocp元数据OB集群，创建租户一样报错：

[root@ocp201 ~]# docker ps
CONTAINER ID        IMAGE                                                               COMMAND                  CREATED             STATUS              PORTS               NAMES
5607cc1dc1c3        reg.docker.alibaba-inc.com/oceanbase/ocp-all-in-one:2.5.0-1915642   "/usr/bin/supervisor…"   6 weeks ago         Up 46 hours                             ocp
f67455f12bfe        reg.docker.alibaba-inc.com/antman/obproxy:OBP173_20200603_1923      "sh start_obproxy.sh"    6 weeks ago         Up 6 weeks                              obproxy
3a66a493b572        reg.docker.alibaba-inc.com/antman/ob-docker:OB2273_x86_20201214     "/usr/bin/supervisor…"   6 weeks ago         Up 12 hours                             META_OB_ZONE_1
[root@ocp201 ~]# 
[root@ocp201 ~]# docker exec -it META_OB_ZONE_1 bash
[root@ocp201 admin]# 
[root@ocp201 admin]# su - admin
Last login: Wed Feb 24 08:48:42 CST 2021 on pts/2
[admin@ocp201 ~]$ mysql -h127.1 -P2881 -uroot@sys -p8I%JWGpMi2 -c -A oceanbase
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 3221711814
Server version: 5.7.25 OceanBase 2.2.73 (r20201211175814-5abff097aadf740bcae07de8f84321815907e062) (Built Dec 11 2020 18:26:06)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

,gmt_create,base]> select zone,svr_ip,svr_port,inner_port,with_rootserver,status 
    -> usec_to_time(start_service_time) start_service_time,
    -> substr(build_version,1,instr(build_version,'-')-1) build_version
    -> from __all_server 
    -> order by zone, svr_ip;
+----------------+-------------+----------+------------+-----------------+--------+----------------------------+----------------------------+-----------------------+
| zone           | svr_ip      | svr_port | inner_port | with_rootserver | status | gmt_create                 | start_service_time         | build_version         |
+----------------+-------------+----------+------------+-----------------+--------+----------------------------+----------------------------+-----------------------+
| META_OB_ZONE_1 | 10.10.10.33 |     2882 |       2881 |               1 | active | 2021-01-06 18:40:45.274610 | 2021-02-23 20:33:20.902062 | 2.2.73_20201211175814 |
+----------------+-------------+----------+------------+-----------------+--------+----------------------------+----------------------------+-----------------------+
1 row in set (0.00 sec)

MySQL [oceanbase]> select tenant_id, tenant_name, compatibility_mode,zone_list, locality ,primary_zone ,gmt_modified from __all_tenant; 
+-----------+-------------+--------------------+----------------+------------------------+----------------+----------------------------+
| tenant_id | tenant_name | compatibility_mode | zone_list      | locality               | primary_zone   | gmt_modified               |
+-----------+-------------+--------------------+----------------+------------------------+----------------+----------------------------+
|         1 | sys         |                  0 | META_OB_ZONE_1 | FULL{1}@META_OB_ZONE_1 | META_OB_ZONE_1 | 2021-01-06 18:40:56.670509 |
|      1001 | ocp_meta    |                  0 | META_OB_ZONE_1 | FULL{1}@META_OB_ZONE_1 | META_OB_ZONE_1 | 2021-01-06 18:43:42.430089 |
|      1002 | ocp_monitor |                  0 | META_OB_ZONE_1 | FULL{1}@META_OB_ZONE_1 | META_OB_ZONE_1 | 2021-01-06 18:43:44.368676 |
+-----------+-------------+--------------------+----------------+------------------------+----------------+----------------------------+
3 rows in set (0.00 sec)

MySQL [oceanbase]> 
MySQL [oceanbase]> create tenant oms_tenant resource_pool_list=('oms_pool'), 
    -> primary_zone='META_OB_ZONE_1',comment 'mysql tenant/instance', 
    -> charset='utf8' set ob_tcp_invited_nodes='%', 
    -> ob_compatibility_mode='mysql';
ERROR 4179 (HY000): Operation not allowed now
MySQL [oceanbase]> 
MySQL [oceanbase]> select * from __all_zone where name='status';
+----------------------------+----------------------------+----------------+--------+-------+--------+
| gmt_create                 | gmt_modified               | zone           | name   | value | info   |
+----------------------------+----------------------------+----------------+--------+-------+--------+
| 2021-01-06 18:40:56.840934 | 2021-01-06 18:40:56.840934 | META_OB_ZONE_1 | status |     2 | ACTIVE |
+----------------------------+----------------------------+----------------+--------+-------+--------+
1 row in set (0.00 sec)

MySQL [oceanbase]> select * from __all_zone where name like 'merge%';
+----------------------------+----------------------------+----------------+------------------+------------------+------+
| gmt_create                 | gmt_modified               | zone           | name             | value            | info |
+----------------------------+----------------------------+----------------+------------------+------------------+------+
| 2021-01-06 18:40:56.840319 | 2021-02-23 02:02:13.537591 |                | merge_status     |                0 | IDLE |
| 2021-01-06 18:40:56.841407 | 2021-02-23 02:00:20.679118 | META_OB_ZONE_1 | merge_start_time | 1614016820678242 |      |
| 2021-01-06 18:40:56.841643 | 2021-02-23 02:02:03.430065 | META_OB_ZONE_1 | merge_status     |                0 | IDLE |
+----------------------------+----------------------------+----------------+------------------+------------------+------+
3 rows in set (0.01 sec)

MySQL [oceanbase]> 
MySQL [oceanbase]> 

复制

从docker中拷贝observer日志分析：

[root@ocp201 ~]# docker cp 3a66a493b572:/home/admin/oceanbase/log/observer.log /tmp/observer.log
[root@ocp201 ~]# docker cp 3a66a493b572:/home/admin/oceanbase/log/observer.log.wf /tmp/observer.log.wf
复制

observer.log 报错日志信息:

[2021-02-23 21:01:10.029215] INFO  [SHARE.SCHEMA] ob_schema_struct.cpp:1875 [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=27] [dc=1] serialize schema(tenant_id=18446744073709551615, schema_version=1, tenant_name=oms_tenant, replica_num=0, primary_zone=META_OB_ZONE_1, locked=false, comment=mysql tenant/instance, charset_type=2, collation_type=45, name_case_mode=-1, rewrite_merge_version=0, locality_str=, logonly_replica_num=0, primary_zone_array=[], storage_format_version=0, storage_format_work_version=0, default_tablegroup_id=18446744073709551615, default_tablegroup_name=, drop_tenant_time=-1, in_recyclebin=false, ret=0)
[2021-02-23 21:01:10.029635] WARN  [RPC.OBRPC] rpc_call (ob_rpc_proxy.ipp:369) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=33] [dc=0] execute rpc fail(ret=-4179, dst="10.10.10.33:2882")
[2021-02-23 21:01:10.029655] WARN  log_user_error_and_warn (ob_rpc_proxy.cpp:304) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=19] [dc=0] 
[2021-02-23 21:01:10.029665] WARN  [SQL.ENG] execute (ob_tenant_executor.cpp:73) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=4] [dc=0] rpc proxy create tenant failed(ret=-4179)
[2021-02-23 21:01:10.029735] INFO  [SHARE] ob_event_history_table_operator.h:242 [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=4] [dc=0] event table add task(ret=0, event_table_name="__all_server_event_history", sql=INSERT INTO __all_server_event_history (gmt_create, module, event, name1, value1, name2, value2, name3, value3, svr_ip, svr_port) VALUES (usec_to_time(1614085270029671), 'sql', 'execute_cmd', 'cmd_type', 8, 'sql_text', X'6372656174652074656E616E74206F6D735F74656E616E74207265736F757263655F706F6F6C5F6C6973743D28276F6D735F706F6F6C27292C207072696D6172795F7A6F6E653D274D4554415F4F425F5A4F4E455F31272C636F6D6D656E7420276D7973716C2074656E616E742F696E7374616E6365272C20636861727365743D27757466382720736574206F625F7463705F696E76697465645F6E6F6465733D2725272C206F625F636F6D7061746962696C6974795F6D6F64653D276D7973716C27', 'return_code', -4179, '10.10.10.33', 2882))
[2021-02-23 21:01:10.029751] WARN  [SQL] open_cmd (ob_result_set.cpp:83) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=9] [dc=0] execute cmd failed(ret=-4179)
[2021-02-23 21:01:10.029755] WARN  [SQL.SESSION] sync_open (ob_result_set.cpp:149) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=4] [dc=0] fail to exec execute()(ret=-4179)
[2021-02-23 21:01:10.029765] WARN  [SERVER] response_result (ob_sync_cmd_driver.cpp:61) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=3] [dc=0] close result set fail(cret=-4179)
[2021-02-23 21:01:10.029778] WARN  [SERVER] test_and_save_retry_state (ob_query_retry_ctrl.cpp:151) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=11] [dc=0] ddl, and errno is not OB_EAGAIN or OB_SNAPSHOT_DISCARDED or OB_ERR_PARALLEL_DDL_CONFLICT, do not need retry(client_ret=-4179, err=-4179, retry_type_=0)
[2021-02-23 21:01:10.029782] INFO  [SERVER] ob_query_retry_ctrl.cpp:555 [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=4] [dc=0] check if need retry(client_ret=-4179, err=-4179, retry_type_=0, retry_times=1, multi_stmt_item={is_part_of_multi_stmt:true, seq_num:0, sql:"create tenant oms_tenant resource_pool_list=('oms_pool'), primary_zone='META_OB_ZONE_1',comment 'mysql tenant/instance', charset='utf8' set ob_tcp_invited_nodes='%', ob_compatibility_mode='mysql'"})
[2021-02-23 21:01:10.029800] WARN  [SERVER] response_result (ob_sync_cmd_driver.cpp:67) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=7] [dc=0] result set open failed, check if need retry(ret=-4179, cli_ret=-4179, retry_ctrl_.need_retry()=0)
[2021-02-23 21:01:10.029842] INFO  [SERVER] obmp_base.cpp:1213 [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=5] [dc=0] sending error packet(err=-4179, bt="0x9419b9a 0x22d36f2 0x6fb2e9b 0x6fa71ec 0x6fe0331 0x94d6182 0x48545b5 0x484e850 0x92b7997 0x92b5fa3 0x92b547f", extra_err_info=NULL)
[2021-02-23 21:01:10.029879] WARN  [SERVER] do_process (obmp_query.cpp:703) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=9] [dc=0] execute query fail(ret=-4179, timeout_timestamp=1614086270028744)
[2021-02-23 21:01:10.029933] WARN  [SERVER] process (obmp_query.cpp:304) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=15] [dc=0] fail execute sql(sql_id="", sql=create tenant oms_tenant resource_pool_list=('oms_pool'), primary_zone='META_OB_ZONE_1',comment 'mysql tenant/instance', charset='utf8' set ob_tcp_invited_nodes='%', ob_compatibility_mode='mysql', sessid=3221487981, ret=-4179, ret="OB_OP_NOT_ALLOW", need_disconnect=false)
复制

observer.log.wf 报错日志信息：

[2021-02-23 21:01:10.029933] WARN  [SERVER] process (obmp_query.cpp:304) [1208][1240][YB420A0A0A21-0005BC001F9DC0C1] [lt=15] [dc=0] fail execute sql(sql_id="", sql=create tenant oms_tenant resource_pool_list=('oms_pool'), primary_zone='META_OB_ZONE_1',comment 'mysql tenant/instance', charset='utf8' set ob_tcp_invited_nodes='%', ob_compatibility_mode='mysql', sessid=3221487981, ret=-4179, ret="OB_OP_NOT_ALLOW", need_disconnect=false)
[2021-02-23 21:01:10.240463] WARN  [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:66) [1829][1956][Y0-0000000000000000] [lt=4] [dc=0] tenant not exist when generate min weak read version, use default max stale time instead(tenant_id=1, base_version_when_no_valid_partition=1614085265740412, lbt()="0x9419b9a 0xb1c7da 0x41f0137 0x6eb567f 0x6f79d43 0x6f7d7a4 0x3c586da 0x92b7997 0x92b5fa3 0x92b547f")
[2021-02-23 21:01:11.241565] WARN  [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:66) [1829][1956][Y0-0000000000000000] [lt=4] [dc=0] tenant not exist when generate min weak read version, use default max stale time instead(tenant_id=1, base_version_when_no_valid_partition=1614085266741532, lbt()="0x9419b9a 0xb1c7da 0x41f0137 0x6eb567f 0x6f79d43 0x6f7d7a4 0x3c586da 0x92b7997 0x92b5fa3 0x92b547f")
[2021-02-23 21:01:11.625382] WARN  [RPC.OBRPC] rpc_call (ob_rpc_proxy.ipp:369) [881][592][YB420A0A0A21-0005BC001C5DBE8C] [lt=18] [dc=0] execute rpc fail(ret=-4179, dst="10.10.10.33:2882")
[2021-02-23 21:01:11.625403] WARN  log_user_error_and_warn (ob_rpc_proxy.cpp:304) [881][592][YB420A0A0A21-0005BC001C5DBE8C] [lt=19] [dc=0] 
[2021-02-23 21:01:11.625420] WARN  [SQL.ENG] create_user (ob_user_cmd_executor.cpp:210) [881][592][YB420A0A0A21-0005BC001C5DBE8C] [lt=3] [dc=0] Create user error(ret=-4179)
[2021-02-23 21:01:11.625424] WARN  [SQL.ENG] execute (ob_user_cmd_executor.cpp:193) [881][592][YB420A0A0A21-0005BC001C5DBE8C] [lt=4] [dc=0] Create user rpc failed(ret=-4179)
[2021-02-23 21:01:11.625490] WARN  [SQL] open_cmd (ob_result_set.cpp:83) [881][592][YB420A0A0A21-0005BC001C5DBE8C] [lt=10] [dc=0] execute cmd failed(ret=-4179)
[2021-02-23 21:01:11.625497] WARN  [SQL.SESSION] sync_open (ob_result_set.cpp:149) [881][592][YB420A0A0A21-0005BC001C5DBE8C] [lt=7] [dc=0] fail to exec execute()(ret=-4179)
[2021-02-23 21:01:11.625500] WARN  [SERVER] open (ob_inner_sql_result.cpp:95) [881][592][YB420A0A0A21-0005BC001C5DBE8C] [lt=2] [dc=0] open result set failed(ret=-4179)
复制

试了 oms2.1.0 版本，还是不行，还是在ocp元数据集群中创建oms租户失败，报错一样。

准备重装ocp，但卸载ocp时也报同样的错：

[root@ocp201 t-oceanbase-antman]# ./install.sh -c 1-
run install.sh with DEBUG=FALSE, INSTALL_STEPS= CLEAR_STEPS=8 7 6 5 4 3 2 1 CONFIG_FILE=/root/t-oceanbase-antman/obcluster.conf
[2021-02-24 09:57:43.537599] INFO [start antman API service]
LB_MODE=none
[2021-02-24 09:57:43.901346] INFO [clear_step8: no need to clear for step network post-check]
[2021-02-24 09:57:43.904847] INFO [clear_step7: uninstalling ocp and remove docker, logfile: /root/t-oceanbase-antman/logs/uninstall_ocp.log]
[2021-02-24 09:57:44.606097] INFO [remove ocp docker on host:10.10.10.33]
[2021-02-24 09:57:44.683331] INFO [docker rm -f 5607cc1dc1c3]
5607cc1dc1c3
[2021-02-24 09:57:51.632582] INFO [remove ocp docker on host:10.10.10.33 done]
[2021-02-24 09:57:51.992983] INFO [uninstall.sh ocp  finished and reg.docker.alibaba-inc.com/oceanbase/ocp-all-in-one:2.5.0-1915642 removed on 10.10.10.33]
[2021-02-24 09:57:51.996852] INFO [ocp docker on 10.10.10.33 is removed]
[2021-02-24 09:57:52.007919] INFO [clear_step7: uninstallation of ocp done]
[2021-02-24 09:57:52.011646] INFO [clear_step6: uninstalling obproxy and remove docker, logfile: /root/t-oceanbase-antman/logs/uninstall_obproxy.log]
[2021-02-24 09:57:52.704401] INFO [remove obproxy docker on host:10.10.10.33]
[2021-02-24 09:57:52.814588] INFO [docker rm -f f67455f12bfe]
f67455f12bfe
[2021-02-24 09:57:55.247156] INFO [remove obproxy docker on host:10.10.10.33 done!]
[2021-02-24 09:57:55.610945] INFO [uninstall.sh obproxy  finished and reg.docker.alibaba-inc.com/antman/obproxy:OBP173_20200603_1923 removed on 10.10.10.33]
[2021-02-24 09:57:55.614558] INFO [obproxy docker on 10.10.10.33 is removed]
[2021-02-24 09:57:55.625474] INFO [clear_step6: uninstallation of obproxy done]
[2021-02-24 09:57:55.629075] INFO [clear_step5: uninstalling ocp and remove docker, logfile: /root/t-oceanbase-antman/logs/uninstall_tmp_ocp.log]
[2021-02-24 09:57:56.316979] INFO [remove ocp docker on host:10.10.10.33]
[2021-02-24 09:57:56.385885] INFO [remove ocp docker on host:10.10.10.33 done]
[2021-02-24 09:57:56.741161] INFO [uninstall.sh ocp  finished and reg.docker.alibaba-inc.com/oceanbase/ocp-all-in-one:2.5.0-1915642 removed on 10.10.10.33]
[2021-02-24 09:57:56.744644] INFO [tmp_ocp docker on 10.10.10.33 is removed]
ERROR 4179 (HY000) at line 1: Operation not allowed now
ERROR 4179 (HY000) at line 1: Operation not allowed now
[2021-02-24 09:57:56.797013] ERROR [ANTMAN-314: ERROR occurred in uninstall_tmp_ocp, install.sh exit]
[root@ocp201 t-oceanbase-antman]# 
复制

查看docker容器，发现ocp meta 容器未卸载，只能手动停掉后，强制删除容器

[root@ocp201 t-oceanbase-antman]# docker ps
CONTAINER ID        IMAGE                                                             COMMAND                  CREATED             STATUS              PORTS               NAMES
3a66a493b572        reg.docker.alibaba-inc.com/antman/ob-docker:OB2273_x86_20201214   "/usr/bin/supervisor…"   6 weeks ago         Up 13 hours                             META_OB_ZONE_1
[root@ocp201 t-oceanbase-antman]# 
[root@ocp201 t-oceanbase-antman]# docker stop META_OB_ZONE_1
META_OB_ZONE_1
[root@ocp201 t-oceanbase-antman]#
[root@ocp201 t-oceanbase-antman]# docker ps -a
CONTAINER ID        IMAGE                                                             COMMAND                  CREATED             STATUS                       PORTS               NAMES
3a66a493b572        reg.docker.alibaba-inc.com/antman/ob-docker:OB2273_x86_20201214   "/usr/bin/supervisor…"   6 weeks ago         Exited (137) 5 minutes ago                       META_OB_ZONE_1
[root@ocp201 t-oceanbase-antman]# 
[root@ocp201 t-oceanbase-antman]# docker rm 3a66a493b572
3a66a493b572
[root@ocp201 t-oceanbase-antman]# 
[root@ocp201 t-oceanbase-antman]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@ocp201 t-oceanbase-antman]#
复制

再手动清理残留的文件

[root@ocp201 t-oceanbase-antman]# rm -rf /home/admin/oceanbase/*
[root@ocp201 t-oceanbase-antman]# rm -rf /data/1/*
[root@ocp201 t-oceanbase-antman]# rm -rf /data/log1/*
[root@ocp201 t-oceanbase-antman]# 
复制

开始重装ocp2.5.0…

重装ocp后，oms安装成功。

[root@ocp201 t-oceanbase-antman]# ./install.sh -i 9
run install.sh with DEBUG=FALSE, INSTALL_STEPS=9 CLEAR_STEPS= CONFIG_FILE=/root/t-oceanbase-antman/obcluster.conf
[2021-02-24 10:59:15.143067] INFO [start antman API service]
LB_MODE=none
Wed Feb 24 10:59:15 CST 2021
[2021-02-24 10:59:15.639319] INFO [step9: check whether OMS port 8088 is in use or not on 10.10.10.145]
[2021-02-24 10:59:15.779273] INFO [step9: OMS port 8088 is idle on 10.10.10.145]
[2021-02-24 10:59:17.324039] INFO [step9: installing oms, logfile: /root/t-oceanbase-antman/logs/install_oms.log]
[2021-02-24 10:59:17.332360] INFO [installing oms on 10.10.10.145, pid=4207 log: /root/t-oceanbase-antman/logs/install_oms_10.10.10.145.log]
[2021-02-24 10:59:38.093717] INFO [create OMS meta tenant]
[2021-02-24 10:59:43.777075] INFO [replace params in /root/t-oceanbase-antman/config/oms_conf.yaml]
[2021-02-24 10:59:43.953287] INFO [start container: docker run -d -it     --name oms     --net=host     --cpu-period 100000     --cpu-quota 1200000     --memory=24G     -v /etc/oms_conf.yaml:/home/admin/conf/co
nfig.yaml     --restart on-failure:5     --privileged=true     acs-reg.alipay.com/oceanbase/oms-all-in-one:feature_1.4.3]a97982661aa15f80a5150bdd2a150a21ed79454bba55f0db33153ee84ebbc1b2
[2021-02-24 10:59:49.757523] INFO [docker init: docker exec oms /bin/bash -c "bash /root/docker_init.sh"]
。。。。。
。。。。。
。。。。。
。。。。。
[2021-02-24 11:06:17.249586] INFO [install_oms.sh finished and acs-reg.alipay.com/oceanbase/oms-all-in-one:feature_1.4.3 started on 10.10.10.145]
[2021-02-24 11:06:17.253838] INFO [oms docker on 10.10.10.145 is started]
[2021-02-24 11:06:17.265374] INFO [oms installation on all hosts done]
[2021-02-24 11:06:17.270298] INFO [step9: installation of oms done]
[root@ocp201 t-oceanbase-antman]# 
复制

一步一步学习OceanBase系列