multipath多链路冗余环境(资源池化)搭建指南

本次multipath多链路冗余环境搭建设备信息详情：

0. 需要安装的软件

所需安装的软件及功能概述：

(1) 安装软件

yum install -y open-iscsi
yum install -y libvirt-daemon-driver-storage-iscsi
yum install -y device-mapper-multipath

(2) 使用rpm -qa|grep app检查软件是否正确安装，若未正确安装该命令的输出为空

[root@centos131 ~]# rpm -qa|grep iscsi
iscsi-initiator-utils-6.2.0.874-22.el7_9.x86_64
libiscsi-1.9.0-7.el7.x86_64
libvirt-daemon-driver-storage-iscsi-4.5.0-36.el7_9.5.x86_64
iscsi-initiator-utils-iscsiuio-6.2.0.874-22.el7_9.x86_64

[root@centos131 ~]# rpm -qa|grep libvirt-daemon-driver-storage-iscsi
libvirt-daemon-driver-storage-iscsi-4.5.0-36.el7_9.5.x86_64

[root@centos131 ~]# rpm -qa|grep multipath
device-mapper-multipath-0.4.9-136.el7_9.x86_64
device-mapper-multipath-libs-0.4.9-136.el7_9.x86_64

1. 检查网卡与磁阵的连通性

(1) 使用ping命令测试四张网卡与磁阵的连通性

ping 30.0.0.229 -I NIC // NIC表示待测网卡

131机器上的网卡 enp3s0f0,enp3s0f1

若网卡不通，则使用命令关闭反向路由检查，这里提供两种方法

a. 法一：重启会失效

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/enp3s0f0/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/enp3s0f1/rp_filter

b. 法二：写入文件，长期有效，之后使用命令sysctl -p让配置信息生效

vim /etc/sysctl.conf

net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.tcp_tw_recycle = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.enp3s0f0.rp_filter = 0
net.ipv4.conf.enp3s0f1.rp_filter = 0

133机器上的网卡 enp2s0f0，enp129s0f0

使用命令关闭反向路由检查

重启失效

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/enp2s0f0/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/enp129s0f0/rp_filter

vim /etc/sysctl.conf

net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.tcp_tw_recycle = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.enp129s0f0.rp_filter = 0
net.ipv4.conf.enp2s0f0.rp_filter = 0

备注：若关闭反向路由检查后，网卡依旧ping不通磁阵，可联系物料管理员协助处理

2. 发现并连接磁阵

131

(1) 使用iscsiadm命令执行discovery发现目标磁阵，并用指定网卡登录

iscsiadm -m discovery -t st -p 30.0.0.229 -l enp3s0f0
iscsiadm -m discovery -t st -p 30.0.0.229 -l enp3s0f1

[root@centos131 ~]# iscsiadm -m discovery -t st -p 30.0.0.229 -l enp3s0f0

30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229
30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229
Logging in to [iface: iscsi_enp3s0f0, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260] (multiple)
Logging in to [iface: iscsi_enp3s0f1, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260] (multiple)
Login to [iface: iscsi_enp3s0f0, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260] successful.
Login to [iface: iscsi_enp3s0f1, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260] successful.

[root@centos131 ~]# iscsiadm -m discovery -t st -p 30.0.0.229 -l enp3s0f1

30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229
30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229

133上操作与131相同

[root@opengauss133 ~]# iscsiadm -m discovery -t st -p 30.0.0.229 -l enp2s0f0
30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229

[root@opengauss133 ~]# iscsiadm -m discovery -t st -p 30.0.0.229 -l enp129s0f0
30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229

从上面的命令输出可以看到，133少了2条链路，这里介绍冗余链路的配置过程

**********************************
******  冗余链路的配置过程  ********
**********************************

(1) 建立iscsi端口

iscsiadm -m iface -I iscsi_enp2s0f0 --op=new
iscsiadm -m iface -I iscsi_enp129s0f0 --op=new

(2) 绑定iscsi端口，建立网卡与iscsi端口之间的映射关系

iscsiadm -m iface -I iscsi_enp2s0f0 --op=update -n iface.net_ifacename -v enp2s0f0
iscsiadm -m iface -I iscsi_enp129s0f0 --op=update -n iface.net_ifacename -v enp129s0f0

(3) 确认iface配置

iscsiadm -m iface

(4) 重启iscsi服务

systemctl restart iscsi.service

(5) 若重启导致网络不通，执行关闭反向路由

echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/enp2s0f0/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/enp129s0f0/rp_filter

(6) 再次发射存储映射信息

iscsiadm -m discovery -t st -p 30.0.0.229

(7) 查看端口服务信息

netstat -anp|grep 3260

**********************************
******  冗余链路的配置过程  ********
**********************************

3. 在Dorado的device manager平台进行主机和启动器绑定映射，并生成4块盘

(1) 创建主机cenos131，centos133

(2) 创建主机组centos_hg

(3) 将主机加入主机组

(4) 创建lun—为了生成4块盘

[15:0:0:1]   disk    HUAWEI   XSG1             6000  /dev/sdl   36382028100c0772b338f5c8900000011   214GB
[15:0:0:2]   disk    HUAWEI   XSG1             6000  /dev/sdm   36382028100c0772b338f5c8900000012   214GB
[15:0:0:3]   disk    HUAWEI   XSG1             6000  /dev/sdn   36382028100c0772b338f5c8900000015   214GB
[15:0:0:4]   disk    HUAWEI   XSG1             6000  /dev/sdo   36382028100c0772b338f5c8900000016   214GB

4. 启动multipath

systemctl enable multipath
systemctl start multipath

使用multipath -ll查看盘的映射情况，当其输出为空时，表示配置有误，修复方法如下：

(1) 检查multipath的相关模块是否安装成功

[root@centos131 ~]# lsmod |grep dm_multipath

dm_multipath           27792  2 dm_round_robin,dm_service_time
dm_mod                128595  12 dm_multipath,dm_log,dm_mirror

(2) 将dm-multipath内核模块加载到Linux系统内核中,该模块是多路径功能的核心组成部分，主要用于实现存储设备的多路径访问

modprobe dm-multipath

(3) 将dm-round-robin内核模块加载到Linux系统内核中，该模块是Linux系统中设备映射器的一部分，主要功能是实现一种轮询的I/O调度策略

modprobe dm-round-robin

(4) 开启multipath服务

service multipathd start

Redirecting to /bin/systemctl start multipathd.service

(5) 使用multipath -v2命令验证多路径功能是否正常启用

multipath -v2

Nov 15 16:34:16 | sdg: alua not supported
Nov 15 16:34:16 | sdi: alua not supported
Nov 15 16:34:16 | sdk: alua not supported
Nov 15 16:34:16 | sdm: alua not supported
Nov 15 16:34:16 | sdf: alua not supported
Nov 15 16:34:16 | sdh: alua not supported
Nov 15 16:34:16 | sdj: alua not supported
Nov 15 16:34:16 | sdl: alua not supported

[root@centos131 ~]# multipath -ll

[root@centos131 ~]# iscsiadm -m session
tcp: [38] 30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229 (non-flash)
tcp: [39] 30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229 (non-flash)

(6) 退出之前建立的会话

[root@centos131 ~]# iscsiadm -m session -u
Logging out of session [sid: 38, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260]
Logging out of session [sid: 39, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260]

(7) 重新发现磁阵并连接

[root@centos131 ~]# iscsiadm -m discovery -t st -p 30.0.0.229 -l enp3s0f0
30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229
30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229
Logging in to [iface: iscsi_enp3s0f0, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260] (multiple)
Logging in to [iface: iscsi_enp3s0f1, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260] (multiple)
Login to [iface: iscsi_enp3s0f0, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260] successful.
Login to [iface: iscsi_enp3s0f1, target: iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229, portal: 30.0.0.229,3260] successful.

[root@centos131 ~]# iscsiadm -m discovery -t st -p 30.0.0.229 -l enp3s0f1
30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229
30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229

(8) 查看建立的会话

[root@centos131 ~]# iscsiadm -m session
tcp: [40] 30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229 (non-flash)
tcp: [41] 30.0.0.229:3260,36354 iqn.2006-08.com.huawei:oceanstor:2100382028c0772b::28e01:30.0.0.229 (non-flash)

(9) 使用命令multipath -ll查看盘的映射情况

[root@centos131 ~]# multipath -ll
Nov 15 16:35:38 | sdk: alua not supported
Nov 15 16:35:38 | sdm: alua not supported
mpathe (36382028100c0772b338f5c8900000016) dm-6 HUAWEI  ,XSG1
size=200G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=-1 status=active
  |- 50:0:0:4 sdk 8:160 active ready running
  `- 51:0:0:4 sdm 8:192 active ready running
Nov 15 16:35:38 | sdi: alua not supported
Nov 15 16:35:38 | sdl: alua not supported
mpathd (36382028100c0772b338f5c8900000015) dm-5 HUAWEI  ,XSG1
size=200G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=-1 status=active
  |- 50:0:0:3 sdi 8:128 active ready running
  `- 51:0:0:3 sdl 8:176 active ready running
Nov 15 16:35:38 | sdg: alua not supported
Nov 15 16:35:38 | sdj: alua not supported
mpathc (36382028100c0772b338f5c8900000012) dm-4 HUAWEI  ,XSG1
size=200G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=-1 status=active
  |- 50:0:0:2 sdg 8:96  active ready running
  `- 51:0:0:2 sdj 8:144 active ready running
Nov 15 16:35:38 | sdf: alua not supported
Nov 15 16:35:38 | sdh: alua not supported
mpathb (36382028100c0772b338f5c8900000011) dm-3 HUAWEI  ,XSG1
size=200G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=-1 status=active
  |- 50:0:0:1 sdf 8:80  active ready running
  `- 51:0:0:1 sdh 8:112 active ready running

经过该步骤之后，可观察到第三步生成的4块盘所对应的盘符，每个盘都有两条链路且状态都是active ready running

5. 使用命令ll /dev/dm-*依次查看四块盘生成的盘符

[root@centos131 script]# ll /dev/dm-3
brw-rw---- 1 root disk 253, 3 Nov 15 16:35 /dev/dm-3
[root@centos131 script]# ll /dev/dm-4
brw-rw---- 1 root disk 253, 4 Nov 15 16:35 /dev/dm-4
[root@centos131 script]# ll /dev/dm-5
brw-rw---- 1 root disk 253, 5 Nov 15 16:35 /dev/dm-5
[root@centos131 script]# ll /dev/dm-6
brw-rw---- 1 root disk 253, 6 Nov 15 16:35 /dev/dm-6

6. 资源池化环境搭建

在资源池化集群的xml文件中，将四个盘的盘符修改为相应的编号/dev/dm-*即可，后续流程与资源池化环境搭建相同

7. 可靠性&&可用性测试

后续测试，可在主备同时跑TPCC，期间可使用命令ip link set NIC down或者ifdown NIC对网卡进行DOWN操作，可持续几分钟，或者使用脚本在一段时间内反复执行DOWN和UP操作且中间可夹杂SLEEP操作，观察集群状态和业务执行情况，以验证multipath多链路冗余环境给集群带来的可用性及可靠性