Redhat7下multipath配置及udev绑定

原创清风～ 2020-01-05

5923

Openfiler安装配置multipath多路径作为Oracle ASM磁盘

一、软件准备

VMware® Workstation 15 Pro
openfileresa-2.99.2-x86_64-disc1.iso
rhel-server-7.7-x86_64-dvd.iso

二、Openfiler配置

在VMworkstation上Openfiler系统已经安装完成并且配置了两块块物理网卡用于实现存储多路径，相应的IP地址已经配置规划；

安装参考文档：https://www.modb.pro/doc/656

在虚拟机上添加虚拟硬盘

 ocr_voting 5120M

 redo   5120M

 data   5120M

 arch   5120M
复制

在openfiler主机上扫盘

 echo '- - -' >/sys/class/scsi_host/host0/scan

 echo '- - -' >/sys/class/scsi_host/host1/scan

 echo '- - -' >/sys/class/scsi_host/host2/scan

 root@localhost host0]# lsblk
复制

创建硬盘分区，配置raid，本测试由于本地空间不足，不配置raid；

Volumes-Block Devices,创建gpt类型磁盘分区

创建卷组

Volumes-Volume Groups

 rac_arch_vg	4.75 GB	

 rac_data_vg	4.75 GB
 
 rac_redo_vg	4.75 GB	

 rac_ocr+voting_vg	4.75 GB	
复制

创建逻辑卷

Volumes-Add Volumes
启动iSCSI目标服务器

点击 iSCSI Target 后面的 Enable 和 Start ，把 iSCSI 打开，并设置自启动
创建iSCSI Targets，并配置访问控制列表

Volumes-iSCSI Targets

Add new iSCSI Target

LUNs mapped to target: iqn.2006-01.com.openfiler:tsn.813052860478
Network Access Configuration

在下方的网络访问配置里把两块网卡的地址段都配置进去，要配置地址段，不要配置地址Name 可以根据实际需要填写即可

三、客户端配置

两个客户端主机分别配置IP

 nmcli conn show
 
 nmcli dev st
 
 nmcli  conn add type ethernet ifname ens38 con-name ens38

 nmcli conn mod ens38 ipv4.address 192.168.37.204 gw4 192.168.37.1 ipv4.method manual autoconnect yes
复制

配置本地YUM源

 --挂载系统镜像

 mount -t iso9660 -o loop /dev/sr0 /media/

 --设置开机启动，修改/etc/rc.d/rc.local文件

 chmod +x /etc/rc.d/rc.local 

 echo "mount -t iso9660 -o loop /dev/sr0 /media/" >> /etc/rc.d/rc.local

 --配置yum配置文件

 touch /etc/yum.repos.d/yum77.repo

 echo "[dvd]" >>/etc/yum.repos.d/yum77.repo 
 
 echo "name=install iso" >>/etc/yum.repos.d/yum77.repo 
 
 echo "baseurl=file:///media/">>/etc/yum.repos.d/yum77.repo 
 
 echo "enabled=1">>/etc/yum.repos.d/yum77.repo 
 
 echo "gpgcheck=0">>/etc/yum.repos.d/yum77.repo

 --清理软件源

 yum clean all
 
 --生成元数据缓

 yum makecache	
复制

安装iscsi-initiator-utils.x86_64.0.6.2.0.874-11.el7

 yum install iscsi-initiator*

 rpm -qa | grep iscsi-initiator
复制

用iscsiadm命令检查网络存储服务器上的所有可用目标
```
 iscsiadm -m discovery -t sendtargets -p 192.168.0.202
复制
```

对扫描到的ISCSI识别存储进行登陆

 iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.813052860478 -p 192.168.0.202:3260 -l

 iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.813052860478 -p 192.168.37.202:3260 -l
复制

修改iscsi软件开机扫描信息

 vi /etc/iscsi/initiatorname.iscsi

 InitiatorName=iqn.2006-01.com.openfiler:tsn.813052860478
复制

查看ISCSI当前登陆会话是否正常
```
 iscsiadm -m session
复制
```

安装mulpiath多路径软件

 --查看是否安装多路径软件包
 
 rpm -qa | grep device-mapper-multipath

 --如未安装，则安装device-mapper-multipath软件包

 yum install device-mapper-multipath
复制

设置开机自启动

 systemctl enable multipathd.service

 --确认是否已设置开机启动

 systemctl list-unit-files |grep multipath
复制

检查multipath模块是否加载成功，如未加载则进行初始化

 --检查内核模块加载情况

 lsmod|grep dm_multipath

 --加载内核模块

 modprobe dm-multipath

 modprobe dm-round-robin
复制

生成多路径配置文件,并启动multipathd服务

 mpathconf --enable
 
 systemctl start multipathd.service
复制

查看并配置multipath配置文件

multipathd show config

–查看wwid

cat /etc/multipath/bindings

mpatha 14f504e46494c4552526674326c752d494e6e732d424c4c74

mpathb 14f504e46494c45527141734c56542d743130322d42374f50

mpathc 14f504e46494c45526c49643830372d726154482d54414247

mpathd 14f504e46494c45524467383979392d6a4630322d6e465052

 --配置文件

 vi /etc/multipath.conf   添加如下：
 
 multipaths{
 multipath{
 	wwid	14f504e46494c4552526674326c752d494e6e732d424c4c74
 	alias	asm_ocr1
 	path_grouping_policy	multibus
 	}
 	multipath{
 	wwid	14f504e46494c45527141734c56542d743130322d42374f50
 	alias   asm_redo1
 	path_grouping_policy	multibus
 	}
 	multipath{
 	wwid	14f504e46494c45526c49643830372d726154482d54414247
 	alias   asm_data1
 	path_grouping_policy	multibus
 	}
 	multipath{
 	wwid	14f504e46494c45524467383979392d6a4630322d6e465052
 	alias   asm_arch1
 	path_grouping_policy	multibus
 	}		
 }
复制

重启multipathd服务，并查看

 systemctl restart multipathd.service

 multipath -ll

 [root@localhost etc]# multipathd show paths
 hcil dev dev_t pri dm_st  chk_st dev_st  next_check 
 2:0:0:0  sda 8:0   -1  undef  undef  unknown orphan 
 33:0:0:0 sdc 8:32  1   active ready  running .......... 1/20
 33:0:0:1 sde 8:64  1   active ready  running .......... 1/20
 33:0:0:2 sdg 8:96  1   active ready  running .......... 1/20
 33:0:0:3 sdi 8:128 1   active ready  running .......... 1/20
 34:0:0:0 sdb 8:16  1   active ready  running .......... 1/20
 34:0:0:1 sdd 8:48  1   active ready  running .......... 1/20
 34:0:0:2 sdf 8:80  1   active ready  running .......... 1/20
 34:0:0:3 sdh 8:112 1   active ready  running .......... 1/20
 [root@localhost etc]# multipathd show status
 path checker states:
 up  9
 
 paths: 8
 busy: False
 
 ls -lrt /dev/mapper/asm*
 lrwxrwxrwx. 1 root root 7 1月   6 02:41 /dev/mapper/asm_redo1 -> ../dm-3
 lrwxrwxrwx. 1 root root 7 1月   6 02:41 /dev/mapper/asm_ocr1 -> ../dm-2
 lrwxrwxrwx. 1 root root 7 1月   6 02:41 /dev/mapper/asm_data1 -> ../dm-4
 lrwxrwxrwx. 1 root root 7 1月   6 02:41 /dev/mapper/asm_arch1 -> ../dm-5
复制

将multipath.conf文件复制到第二个客户端节点，重启服务

 scp multipath.conf 192.168.0.204:/etc
 
 systemctl restart multipathd.service
复制

对共享磁盘分区，采用 GPT 格式的分区以进行分区对齐

 	[root@localhost mapper]# parted asm_ocr1 
 GNU Parted 3.1
 使用 /dev/dm-2
 Welcome to GNU Parted! Type 'help' to view a list of commands.
 (parted) h
 align-check TYPE Ncheck partition N for TYPE(min|opt) alignment
 help [COMMAND]   print general help, or help on COMMAND
 mklabel,mktable LABEL-TYPE   create a new disklabel (partition table)
 mkpart PART-TYPE [FS-TYPE] START END make a partition
 name NUMBER NAME name partition NUMBER as NAME
 print [devices|free|list,all|NUMBER] display the partition table, available devices, free space, all found partitions, or a particular partition
 quit exit program
 rescue START END rescue a lost partition near START and END
 
 resizepart NUMBER ENDresize partition NUMBER
 rm NUMBERdelete partition NUMBER
 select DEVICEchoose the device to edit
 disk_set FLAG STATE  change the FLAG on selected device
 disk_toggle [FLAG]   toggle the state of FLAG on selected device
 set NUMBER FLAG STATEchange the FLAG on partition NUMBER
 toggle [NUMBER [FLAG]]   toggle the state of FLAG on partition NUMBER
 unit UNITset the default unit to UNIT
 version  display the version number and copyright information of GNU Parted
 (parted) mklabel  
 新的磁盘标签类型？ gpt
 (parted) mkpart primary 8192s -1  
 (parted) p
 Model: Linux device-mapper (multipath) (dm)
 Disk /dev/dm-2: 5100MB
 Sector size (logical/physical): 512B/512B
 Partition Table: gpt
 Disk Flags: 
 
 Number  Start   End     Size    File system  Name     标志
 1      4194kB  5099MB  5095MB               primary
复制

在第二节点更新分区表

 kpartx -a /dev/mapper/asm_ocr1 

 ls -lrt /dev/mapper/asm*
复制

使用udev方式配置ASM磁盘

查询磁盘的dm_uuid

 --查询DM_UUID	
 udevadm info --query=all --name=/dev/mapper/asm_ocr1p1 |grep -i DM_UUID

 --批量获得DM_UUID

 for i in asm_ocr1p1 asm_redo1p1 asm_data1p1 asm_arch1p1; 
 do
 printf "%s %s\n" "$i" "$(udevadm info --query=all --name=/dev/mapper/$i |grep -i dm_uuid)"; 
 done
复制

创建新的udev规则文件

 touch /etc/udev/rules.d/99-oracle-asmdevices.rules

 --根据查到的dm_uuid配置规则如下：

 KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c4552526674326c752d494e6e732d424c4c74",OWNER="grid",GROUP="asmadmin",MODE="0660"
 KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c45527141734c56542d743130322d42374f50",OWNER="grid",GROUP="asmadmin",MODE="0660"
 KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c45526c49643830372d726154482d54414247",OWNER="grid",GROUP="asmadmin",MODE="0660"
 KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-14f504e46494c45524467383979392d6a4630322d6e465052",OWNER="grid",GROUP="asmadmin",MODE="0660"
复制

保存99-oracle-asmdevices.rules文件，重新加载udev生效

 udevadm control --reload-rules

 udevadm trigger --type=devices --action=change
复制

查询dm-device对应的磁盘分区

 for i in asm_ocr1p1 asm_redo1p1 asm_data1p1 asm_arch1p1; 
 do
 printf "%s %s\n" "$i" "$(ls -ll /dev/mapper/$i)";
 done
复制

测试规则文件是否生效，使用udevadm test
```
 udevadm test /sys/block/dm-6
复制
```

确认分区对应的dm设备权限是否正确

 [root@localhost dev]# ls -lrt dm-6
 brw-rw----. 1 grid asmadmin 253, 6 1月   6 07:04 dm-6
复制

oracle

最后修改时间：2020-01-06 21:20:56

「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」

关注作者

Uncopyrightable

3年前

咨询下，(parted) mkpart primary 8192s -1 这条命令中8192s有什么具体含义嘛？

3年前

Uncopyrightable

3年前

Here’s a quick step-by-step guide to aligning partitions properly. It’s just an abstraction of the HP post, but hopefully easier to follow. This will work for most arrays (in fact it works for all the arrays that I’ve seen); there are more options in HP’s post, but I’ve included the most common configuration here. 1. Get the alignment parameters for your array (remember to replace sdb with the name of your device as seen by the kernel). # cat /sys/block/sdb/queue/optimal_io_size 1048576 # cat /sys/block/sdb/queue/minimum_io_size 262144 # cat /sys/block/sdb/alignment_offset 0 # cat /sys/block/sdb/queue/physical_block_size 512 2. Add optimal_io_size to alignment_offset and divide the result by physical_block_size. In my case this was (1048576 + 0) / 512 = 2048. 3. This number is the sector at which the partition should start. Your new parted command should look like mkpart primary 2048s 100% The trailing ‘s’ is important: it tells parted that you’re talking about sectors, not bytes or megabytes.

3年前

泥鳅也是鱼

4年前

为什么我的看不到WWID

4年前

Redhat7下multipath配置及udev绑定

Openfiler安装配置multipath多路径作为Oracle ASM磁盘

一、软件准备

二、Openfiler配置

三、客户端配置

使用udev方式配置ASM磁盘

评论

相关阅读