在写这篇文章的时候，刚刚加班结束，顺手写了这篇文章。
前言
某大型全国性行业核心系统数据库需要A、B两个物理隔离的双网卡架构方案，已成为行业标准。而最新发布的MogDB 5.0.6的CM新增支持流复制双网段部署，用于网卡级高可用容灾(PTK 1.4及更高版本)。支持双网段后，某个节点在发生单网段故障后仍然能保证正常工作，提供网段级故障的高可用管理能力，保驾了数据库底座的稳定运行。
CM简介
CM(Cluster Manager)是一款集群资源管理软件。支持自定义资源 监控，提供了数据库主备的状态监控、网络通信故障监控、文件系统 故障监控、故障自动主备切换等能力。
MogDB数据库和CM是解耦的，可以分开安装。CM为数据库 集群提供管理服务，功能十分强大。
CM功能特性
- 数据库实例主备角色仲裁
- CM自身高可用仲裁
- 数据库实例运行状态监控
- 数据库实例所在节点资源检测
- 数据库集群、节点、实例的启动、停止
- 数据库集群状态查询、状态更新
- 数据库集群数据库实例switchover、failover
- 支持双网虚拟ip(VIP)
- CM支持两节点部署
- CM事件触发器
- CM支持双网段
生产落地方案
如图为完整架构方案，MogDB支持主备双机房的方案，A和B网络物理隔离，分别联通独立的A和B交换机，两者不互通，并且在A和B上挂载两个vip，对应用服务器供服务。
流复制网络方案
C和D网卡做Bond0，再通过A和B两个交换机中单独虚拟出一个vlan，通过VLAN Trunking技术做堆叠，提供流复制网络的交换机高可用，为流复制网络提供通讯。
本次落地的方案
此次方案为一主一备的架构。同样A和B网络提供业务网络。
C和D网卡做Bond0+VLAN Trunking技术做堆叠，为流复制提供冗余。
安装部署
| 主机 | 业务A网 | 业务B网 | 流复制网 |
|---|---|---|---|
| mogdb1 | 10.130.0.5 | 10.130.4.5 | 192.168.100 |
| mogdb2 | 10.130.0.7 | 10.130.4.7 | 192.168.101 |
配置config
PTK对CM实现了适配，安装时在config.yaml文件中配置参数ha_ips即可实现双网段部署。需要下载PTK 1.4及更高版本。
vi config.2024-04-01T17_37_01.yaml global: cluster_name: ats_cs user: omm506 group: omm506 db_password: pTk6YmQwZmY1ODU8QDw9PUM/Q085ZDFjV2g5dGZFQVRpSlJfQ2tGeG1GTzB3WjRfa3lDSmpsdjVTcVdVLUE= db_port: 28000 base_dir: /opt/mogdb506 db_conf: log_min_messages: 'DEBUG5' ssh_option: port: 22 user: root key_file: "/root/.ssh/id_rsa" cm_option: cm_server_port: 15300 cm_server_conf: ddb_type: 1 enable_ssl: on third_party_gateway_ip: 10.130.7.254,10.130.3.254 cms_enable_failover_on2nodes: 'true' cms_enable_db_crash_recovery: 'true' log_min_messages: 'DEBUG5' cm_agent_conf: enable_ssl: on log_min_messages: 'DEBUG5' db_servers: - host: 10.130.4.5 role: primary az_name: AZ1 az_priority: 1 ha_ips: [10.130.0.5] ssh_option: password: pTk6MzkzYTk5ZTg8QDw9PUM/REpvRHhwYk1LWGRjS3dER0I5RC1NNFVnMFNjalU1NUpQSGVCWXFHTU5LLVU= - host: 10.130.4.7 role: standby az_name: AZ1 az_priority: 1 ha_ips: [10.130.0.7] ssh_option: password: pTk6MzkzYTk5ZTg8QDw9PUM/REpvRHhwYk1LWGRjS3dER0I5RC1NNFVnMFNjalU1NUpQSGVCWXFHTU5LLVU=复制
查看网卡的网关
两节点部署（一主一备），需要在yaml配置文件中配置third_party_gateway_ip参数作为仲裁ip（单网段的三方网关）为双网段参数，以逗号分隔。这里配置如下：
third_party_gateway_ip
：10.130.7.254、10.130.3.254
[root@mogdb1 ~]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.130.7.254 0.0.0.0 UG 102 0 0 em1 0.0.0.0 10.130.3.254 0.0.0.0 UG 103 0 0 em2 10.130.0.0 0.0.0.0 255.255.252.0 U 103 0 0 em2 10.130.4.0 0.0.0.0 255.255.252.0 U 102 0 0 em1 192.168.1.0 0.0.0.0 255.255.255.0 U 300 0 0 bond0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0复制
安装数据库
加上参数 --install-cm表示同数据库一起安装cm，也可以单独部署。
ptk install -f config.2024-04-01T17_37_01.yaml --install-cm -p MogDB-5.0.6-Kylin-x86_64-all.tar.gz INFO[2024-04-22T16:21:59.149] PTK 版本: 1.4.1 release INFO[2024-04-22T16:21:59.149] 从 config.2024-04-01T17_37_01.yaml 加载配置 如果您选择继续安装软件, 就代表您接受该软件的许可协议。 [Y]: 接受并继续 [C]: 显示许可协议内容 [N]: 中止安装并退出 ✔ 请输入 (默认: Y): y INFO[2024-04-22T16:22:00.884] CM 启用, 但 cm_servers 为空，默认使用数据库的服务器作为 CM 服务器 集群将包含 2 个 CM 节点，所以请确认以下 cms 的配置: - (Optional) db_service_vip="" - (Required) third_party_gateway_ip="10.130.7.254,10.130.3.254" - (Optional) cms_enable_failover_on2nodes="true" - (Optional) cms_enable_db_crash_recovery="true" ✔ 请输入虚拟 IP (仅支持 IPv4): 10.130.4.6,10.130.0.6 现在这些配置的值是: - db_service_vip="10.130.4.6,10.130.0.6" - third_party_gateway_ip="10.130.7.254,10.130.3.254" - cms_enable_failover_on2nodes="true" - cms_enable_db_crash_recovery="true" ✔ 您想修改它们吗 (默认 n) [y/n]: n 集群名："ats_cs" +--------------+------------+----------------+----------+------------+-----------------------+----------+ | az(priority) | ip | user(group) | port | role | data dir | upstream | +--------------+------------+----------------+----------+------------+-----------------------+----------+ | AZ1(1) | 10.130.4.7 | omm506(omm506) | db:28000 | db:standby | db:/opt/mogdb506/data | - | | | | | cm:15300 | | cm:/opt/mogdb506/cm | | | | 10.130.4.5 | omm506(omm506) | db:28000 | db:primary | db:/opt/mogdb506/data | - |复制
MogDB和CM安装完毕。
添加VIP
建议添加参数–log-level debug方便查看和分析报错。ptk会自动判断把vip挂载同网段的地址上，简单易用。
ptk cluster -n ats_cs load-cm-vip --vip 10.130.4.6 --action install --log-level debug ptk cluster -n ats_cs load-cm-vip --vip 10.130.0.6 --action install --log-level debug复制
添加完后，会在 cd /opt/mogdb506/cm/cm_agent的下新增cm_resource.json的配置文件,“float_ip”: “10.130.4.6”，“float_ip”: "10.130.0.6"已经添加。
{ "resources": [{ "name": "VIP_az996037", "resources_type": "VIP", "instances": [{ "node_id": 1, "res_instance_id": 6001, "inst_attr": "base_ip=10.130.4.5" }, { "node_id": 2, "res_instance_id": 6002, "inst_attr": "base_ip=10.130.4.7" }], "float_ip": "10.130.4.6" }, { "name": "VIP_az40655", "resources_type": "VIP", "instances": [{ "node_id": 1, "res_instance_id": 6001, "inst_attr": "base_ip=10.130.0.5" }, { "node_id": 2, "res_instance_id": 6002, "inst_attr": "base_ip=10.130.0.7" }], "float_ip": "10.130.0.6"复制
cm_ctl show命令查看vip信息
vip：10.130.4.6挂载到10.130.4.5
，vip：10.130.0.5挂载到10.130.0.6上面。
[omm506@mogdb1 cm_agent]$ cm_ctl show [ Network Connect State ] Network timeout: 6s Current CMServer time: 2024-04-22 18:29:06 Network stat('Y' means connected, otherwise 'N'): | \ | Y | | Y | \ | [ Node Disk HB State ] Node disk hb timeout: 200s Current CMServer time: 2024-04-22 18:29:07 Node disk hb stat('Y' means connected, otherwise 'N'): | N | N | [ FloatIp Network State ] node instance base_ip float_ip_name float_ip ------------------------------------------------------ 1 mogdb1 6001 10.130.4.5 VIP_az996037 10.130.4.6 1 mogdb1 6001 10.130.0.5 VIP_az40655 10.130.0.6复制
修改流复制ip
两台机器的postgresql.con都修改replconninfo[1-2]参数，如下。
replconninfo1 = 'localhost=192.168.1.100 localport=28001 localheartbeatport=26005 localservice=28004 remotehost=192.168.1.101 remoteport=28001 remoteheartbeatport=28005 remoteservice=28004' replconninfo2 = 'localhost=192.168.1.101 localport=28001 localheartbeatport=28005 localservice=28004 remotehost=192.168.1.100 remoteport=28001 remoteheartbeatport=28005 remoteservice=28004'复制
修改pg_hba
添加互信，不然会出现Forbid remote connection with initia1 user的报错，添加#add MogDB那部分内容。
# "local" is for Unix domain socket connections only local all all trust host all omm506 10.130.0.5/32 trust host all omm506 10.130.4.5/32 trust host all omm506 10.130.0.7/32 trust host all omm506 10.130.4.7/32 trust # add MogDB host all omm506 192.168.1.101/32 trust host all omm506 192.168.1.100/32 trust # IPv4 local connections: host all all 127.0.0.1/32 trust host all all 10.130.0.5/32 sha256 host all all 10.130.4.5/32 sha256 host all all 10.130.0.7/32 sha256 host all all 10.130.4.7/32 sha256 # add MogDB host all all 192.168.1.101/32 sha256 host all all 192.168.1.100/32 sha256 host all all 10.130.4.6/32 sha256 host all all 10.130.0.6/32 sha256 # IPv6 local connections: host all all ::1/128 trust # Allow replication connections from localhost, by a user with the # replication privilege. #local replication omm506 trust #host replication omm506 127.0.0.1/32 trust #host replication omm506 ::1/128 trust # add MogDB host all all 0.0.0.0/0 md5复制
修改local_bind_address参数
设置为*。
vi /opt/mogdb506/data/postgresql.conf local_bind_address = '*'复制
重启生效
这里能看到流复制通道为：192.168.1.100:28001–>192.168.1.101:56732，cm接管的为业务ip：10.130.4.5,10.130.0.5，10.130.4.7,10.130.0.7，大功告成。
[root@mogdb1 ~]# ptk cluster -n ats_cs status --detail [ Cluster State ] cluster_name : ats_cs cluster_state : Normal database_version : MogDB 5.0.6 (build 8b0a6ca8) active_vip : 10.130.4.6,10.130.0.6 [ CMServer State ] id | ip | port | hostname | role -----+-----------------------+-------+----------+---------- 1 | 10.130.4.5,10.130.0.5 | 15300 | mogdb1 | primary 2 | 10.130.4.7,10.130.0.7 | 15300 | mogdb2 | standby [ Datanode State ] cluster_name | id | ip | port | user | nodename | db_role | state | uptime | upstream ---------------+------+------------+-------+--------+----------+---------+--------+----------+----------- ats_cs | 6001 | 10.130.4.5 | 28000 | omm506 | dn_6001 | primary | Normal | 00:02:16 | - | | 10.130.0.5 | | | | | | | | 6002 | 10.130.4.7 | 28000 | omm506 | dn_6002 | standby | Normal | 00:02:17 | - | | 10.130.0.7 | | | | | | | [ DataNode Detail ] --------------- 10.130.4.5:28000(dn_6001) --------------- role : primary data_dir : /opt/mogdb506/data az_name : AZ1 [Senders Info]: sender_pid : 3158066 local_role : Primary peer_role : Standby peer_state : Normal state : Streaming sender_sent_location : 0/5E95998 sender_write_location : 0/5E95998 sender_flush_location : 0/5E95998 sender_replay_location : 0/5E95998 receiver_received_location : 0/5E95998 receiver_write_location : 0/5E95998 receiver_flush_location : 0/5E95998 receiver_replay_location : 0/5E95998 sync_percent : 100% sync_state : Sync sync_priority : 1 sync_most_available : On channel : 192.168.1.100:28001-->192.168.1.101:56732 --------------- 10.130.4.7:28000(dn_6002) --------------- role : standby data_dir : /opt/mogdb506/data az_name : AZ1 [Receiver Info]: receiver_pid : 3121372 local_role : Standby peer_role : Primary peer_state : Normal state : Normal sender_sent_location : 0/5E95AB8 sender_write_location : 0/5E95AB8 sender_flush_location : 0/5E95AB8 sender_replay_location : 0/5E95AB8 receiver_received_location : 0/5E95AB8 receiver_write_location : 0/5E95AB8 receiver_flush_location : 0/5E95AB8 receiver_replay_location : 0/5E95AB8 sync_percent,channel : 100%复制
CM双网高可用测试
测试场景：
-
- 备库单网卡故障
-
- 主库A网卡故障
-
- 主库B网卡故障
预期：
-
- 无影响，不影响主备同步，不影响业务
-
- 主库A网故障，无影响，不影响主备同步，不影响业务
-
- 主库B网故障，进行主备切换，备库升主（switchover）
备库业务单网故障
[root@mogdb2 dn_6002]# ifdown em2 WARN : [ifdown] 您正在使用由 ‘network-scripts’ 所提供的 ‘ifdown’，这一命令现在不推荐使用。 WARN : [ifdown] ‘network-scripts’ 将在不久的将来的发行版中被移除。 WARN : [ifdown] 建议切换到 ‘NetworkManager’ 作为代替，它也提供了 ‘ifup/ifdown’ 脚本。 [root@mogdb1 ~]# ptk cluster -n ats_cs status --detail [ Cluster State ] cluster_name : ats_cs cluster_state : Normal database_version : MogDB 5.0.6 (build 8b0a6ca8) active_vip : 10.130.4.6,10.130.0.6 [ CMServer State ] id | ip | port | hostname | role -----+-----------------------+-------+----------+---------- 1 | 10.130.4.5,10.130.0.5 | 15300 | mogdb1 | primary 2 | 10.130.4.7,10.130.0.7 | 15300 | mogdb2 | standby [ Datanode State ] cluster_name | id | ip | port | user | nodename | db_role | state | uptime | upstream ---------------+------+------------+-------+--------+----------+---------+--------+----------+----------- ats_cs | 6001 | 10.130.4.5 | 28000 | omm506 | dn_6001 | primary | Normal | 00:03:16 | - | | 10.130.0.5 | | | | | | | | 6002 | 10.130.4.7 | 28000 | omm506 | dn_6002 | standby | Normal | 00:03:17 | - | | 10.130.0.7 | | | | | | | [ DataNode Detail ] --------------- 10.130.4.5:28000(dn_6001) --------------- role : primary data_dir : /opt/mogdb506/data az_name : AZ1 [Senders Info]: sender_pid : 3158066 local_role : Primary peer_role : Standby peer_state : Normal state : Streaming sender_sent_location : 0/5E95998 sender_write_location : 0/5E95998 sender_flush_location : 0/5E95998 sender_replay_location : 0/5E95998 receiver_received_location : 0/5E95998 receiver_write_location : 0/5E95998 receiver_flush_location : 0/5E95998 receiver_replay_location : 0/5E95998 sync_percent : 100% sync_state : Sync sync_priority : 1 sync_most_available : On channel : 192.168.1.100:28001-->192.168.1.101:56732 --------------- 10.130.4.7:28000(dn_6002) --------------- role : standby data_dir : /opt/mogdb506/data az_name : AZ1 [Receiver Info]: receiver_pid : 3121372 local_role : Standby peer_role : Primary peer_state : Normal state : Normal sender_sent_location : 0/5E95AB8 sender_write_location : 0/5E95AB8 sender_flush_location : 0/5E95AB8 sender_replay_location : 0/5E95AB8 receiver_received_location : 0/5E95AB8 receiver_write_location : 0/5E95AB8 receiver_flush_location : 0/5E95AB8 receiver_replay_location : 0/5E95AB8 sync_percent,channel : 100%复制
无影响，不影响主备同步，不影响业务,集群一切正常。
主库A网故障
down掉A网。
[root@mogdb1 ~]# ifdown em1 WARN : [ifdown] 您正在使用由 ‘network-scripts’ 所提供的 ‘ifdown’，这一命令现在不推荐使用。 WARN : [ifdown] ‘network-scripts’ 将在不久的将来的发行版中被移除。 WARN : [ifdown] 建议切换到 ‘NetworkManager’ 作为代替，它也提供了 ‘ifup/ifdown’ 脚本。复制
数据库同步正常
[2024-04-23 19:44:12.469][944266][][gs_ctl]: gs_ctl query ,datadir is /opt/mogdb506/data HA state: local_role : Primary static_connections : 2 db_state : Normal detail_information : Normal Senders info: sender_pid : 226949 local_role : Primary peer_role : Standby peer_state : Normal state : Streaming sender_sent_location : 0/A1575E8 sender_write_location : 0/A1575E8 sender_flush_location : 0/A1575E8 sender_replay_location : 0/A1575E8 receiver_received_location : 0/A1575E8 receiver_write_location : 0/A1575E8 receiver_flush_location : 0/A1575E8 receiver_replay_location : 0/A1575E8 sync_percent : 100% sync_state : Sync sync_priority : 1 sync_most_available : On channel : 192.168.1.100:28001-->192.168.1.101:41842 Receiver info: No information复制
主库B网故障
down掉B网。
[root@mogdb1 ~]# ifdown em2 WARN : [ifdown] 您正在使用由 ‘network-scripts’ 所提供的 ‘ifdown’，这一命令现在不推荐使用。 WARN : [ifdown] ‘network-scripts’ 将在不久的将来的发行版中被移除。 WARN : [ifdown] 建议切换到 ‘NetworkManager’ 作为代替，它也提供了 ‘ifup/ifdown’ 脚本。 成功断开设备 "em2"。复制
主库数据库被cm关闭。
[omm506@mogdb1 ~]$ cm_ctl query -Cvi [ CMServer State ] node node_ip instance state ---------------------------------------------------------- cm_ctl: [DoConnCmserver] ip 10.130.4.5 is not reachable. cm_ctl: [DoConnCmserver] ip 10.130.0.5 is not reachable. cm_ctl: [DoConnCmserver] ip 10.130.4.7 is not reachable. cm_ctl: [DoConnCmserver] ip 10.130.0.7 is not reachable. 1 mogdb1 10.130.4.5,10.130.0.5 1 Down 2 mogdb2 10.130.4.7,10.130.0.7 2 Down cm_ctl: can't connect to cm_server. Maybe cm_server is not running, or timeout expired. Please try again.复制
主库数据库被cm关闭。
[omm506@mogdb1 ~]$ ps -ef |grep mogdb pcp 2823 1 0 19:27 ? 00:00:00 /usr/bin/pmie -b -h local: -l /var/log/pcp/pmie/mogdb1/pmie.log -c config.default omm506 8941 1 0 19:28 ? 00:00:05 /opt/mogdb506/app/bin/om_monitor -L /opt/mogdb506/log/cm/om_monitor omm506 8945 8941 5 19:28 ? 00:01:18 /opt/mogdb506/app/bin/cm_agent omm506 8988 1 0 19:28 ? 00:00:00 mogdb fenced UDF master process omm506 1115099 1114674 0 19:51 pts/1 00:00:00 grep mogdb [omm506@mogdb1 ~]$ gsql -r failed to connect /opt/mogdb506/tmp:28000.复制
cm_ctl query -Cvi查看集群状态,可以看出备库已经是Primary Normal，已经升主。
[ CMServer State ] node node_ip instance state ---------------------------------------------------------- cm_ctl: [DoConnCmserver] ip 10.130.4.5 is not reachable. cm_ctl: [DoConnCmserver] ip 10.130.0.5 is not reachable. 1 mogdb1 10.130.4.5,10.130.0.5 1 Down 2 mogdb2 10.130.4.7,10.130.0.7 2 Primary [ Cluster State ] cluster_state : Degraded redistributing : No balanced : No current_az : AZ_ALL [ Datanode State ] node node_ip instance state | node node_ip instance state -------------------------------------------------------------------------------------------------------------------------------------------- 1 mogdb1 10.130.4.5,10.130.0.5 6001 P Down Unknown | 2 mogdb2 10.130.4.7,10.130.0.7 6002 S Primary Normal复制
cm_ctl show查看vip状态，全部漂移到mogdb2的A和B网上，对业务透明无感知。
[omm506@mogdb2 ~]$ cm_ctl show cm_ctl: [DoConnCmserver] ip 10.130.4.5 is not reachable. cm_ctl: [DoConnCmserver] ip 10.130.0.5 is not reachable. [ Network Connect State ] Network timeout: 6s Current CMServer time: 2024-04-23 19:52:21 Network stat('Y' means connected, otherwise 'N'): | \ | N | | N | \ | [ Node Disk HB State ] Node disk hb timeout: 200s Current CMServer time: 2024-04-23 19:52:22 Node disk hb stat('Y' means connected, otherwise 'N'): | N | N | [ FloatIp Network State ] node instance base_ip float_ip_name float_ip ------------------------------------------------------ 2 mogdb2 6002 10.130.4.7 VIP_az996037 10.130.4.6复制
总结
MogDB 5.0.6特性CM支持双网段，在某个节点在发生单网段故障后仍然能保证正常工作，提供网段级故障的高可用管理能力。Mogdb首先实现了对Oracle传统双网方案的兼容，并且为保证双网方案具有较好的业务切换体验，还实现了IT环境解耦，持续稳定支撑核心系统。
最后修改时间：2024-04-25 09:59:43
「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文章的来源（墨天轮），文章链接，文章作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
