适用范围
Red Hat 7.2
zData 3.10
oracle RAC 11.2.0.4
该文档迁移OCR和VOTEDISK也可用于非zData一体机环境。
方案概述
在老的zData版本，底层经常使用hdd磁盘，当io压力大时asm disk经常掉盘，如果掉盘的是ocr磁盘组就可能导致集群crash重启。为了解决该问题，可以将OCR和VOTEDISK迁移到高速盘ssd磁盘上去。
目前OCR和VOTEDISK在SYS磁盘组上，SYS磁盘组是三块HDD磁盘narmol冗余。将OCR和VOTEDISK迁移到五块SSD磁盘high冗余的磁盘组上。
实施步骤
1、检查OCR和VOTEDISK状态
grid$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 0ba9ff8a4b264fbebf88f8dd35a0a498 (/dev/mapper/ZDATA_HDISK_CELL01_001) [SYS] 2. ONLINE 87b5b01eadff4fa9bf73edcd74b0e606 (/dev/mapper/ZDATA_HDISK_CELL02_001) [SYS] 3. ONLINE 324e58a5139c4ff7bfa85c9b0c2c074d (/dev/mapper/ZDATA_HDISK_CELL03_001) [SYS] Located 3 voting disk(s). grid$ ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3164 Available space (kbytes) : 258956 ID : 1813559706 Device/File Name : +SYS Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check bypassed due to non-privileged user复制
2、检查ocr磁盘组asm磁盘状态
SQL> select group_number g_nu,os_mb,header_status,mount_status,name,path from v$asm_disk where GROUP_NUMBER=1; G_NU OS_MB HEADER_STATU MOUNT_S NAME ---------- ---------- ------------ ------- -------------------- PATH ------------------------------------------------------------------------------------------------------------------------------------------------------ 1 5120 MEMBER CACHED DISK3 /dev/mapper/ZDATA_HDISK_CELL03_001 1 5120 MEMBER CACHED DISK2 /dev/mapper/ZDATA_HDISK_CELL02_001 1 5120 MEMBER CACHED DISK1 /dev/mapper/ZDATA_HDISK_CELL01_001复制
3、zdata存储节点分散分配5块5G的ssd磁盘
3.1、存储节点1 zcli> create volume FDISK_CELL01_008 size=5G celldisk=CDISK_CELL01_001 hostgroup=ODS zcli> create volume FDISK_CELL01_009 size=5G celldisk=CDISK_CELL01_003 hostgroup=ODS zcli> alter volume FDISK_CELL01_008 active zcli> alter volume FDISK_CELL01_009 active 3.2、存储节点2 zcli> create volume FDISK_CELL02_008 size=5G celldisk=CDISK_CELL02_001 hostgroup=ODS zcli> alter volume FDISK_CELL02_008 active 3.3、存储节点3 zcli> create volume FDISK_CELL03_008 size=5G celldisk=CDISK_CELL03_001 hostgroup=ODS zcli> create volume FDISK_CELL03_009 size=5G celldisk=CDISK_CELL03_003 hostgroup=ODS zcli> alter volume FDISK_CELL03_008 active zcli> alter volume FDISK_CELL03_009 active复制
4、计算节点配置新的5块磁盘，所有计算节点执行相同的操作
4.1、刷新磁盘链路
# rescan-scsi-bus.sh复制
4.2、添加多路径映射
# vi /etc/multipath.conf 添加： multipath { wwid SZDATA_FDISK_CELL01_008 alias ZDATA_FDISK_CELL01_008 path_selector "queue-length 0" path_grouping_policy multibus } multipath { wwid SZDATA_FDISK_CELL02_008 alias ZDATA_FDISK_CELL02_008 path_selector "queue-length 0" path_grouping_policy multibus } multipath { wwid SZDATA_FDISK_CELL03_008 alias ZDATA_FDISK_CELL03_008 path_selector "queue-length 0" path_grouping_policy multibus } multipath { wwid SZDATA_FDISK_CELL01_009 alias ZDATA_FDISK_CELL01_009 path_selector "queue-length 0" path_grouping_policy multibus } multipath { wwid SZDATA_FDISK_CELL03_009 alias ZDATA_FDISK_CELL03_009 path_selector "queue-length 0" path_grouping_policy multibus }复制
4.3、刷新多路径
# multipath -r复制
4.4、查看磁盘状态
# ls -l /dev/mapper/* | grep 008 lrwxrwxrwx 1 root root 8 Dec 2 13:35 /dev/mapper/ZDATA_FDISK_CELL01_008 -> ../dm-32 lrwxrwxrwx 1 root root 8 Dec 2 13:35 /dev/mapper/ZDATA_FDISK_CELL02_008 -> ../dm-31 lrwxrwxrwx 1 root root 8 Dec 2 13:35 /dev/mapper/ZDATA_FDISK_CELL03_008 -> ../dm-34 # ls -l /dev/mapper/* | grep 009 lrwxrwxrwx 1 root root 8 Dec 2 13:35 /dev/mapper/ZDATA_FDISK_CELL01_009 -> ../dm-33 lrwxrwxrwx 1 root root 8 Dec 2 13:35 /dev/mapper/ZDATA_FDISK_CELL03_009 -> ../dm-35 # ls -l /dev/dm-3[1-5] brw-rw---- 1 grid asmadmin 253, 31 Dec 2 13:35 /dev/dm-31 brw-rw---- 1 grid asmadmin 253, 32 Dec 2 13:35 /dev/dm-32 brw-rw---- 1 grid asmadmin 253, 33 Dec 2 13:35 /dev/dm-33 brw-rw---- 1 grid asmadmin 253, 34 Dec 2 13:35 /dev/dm-34 brw-rw---- 1 grid asmadmin 253, 35 Dec 2 13:35 /dev/dm-35复制
5、计算节点将新的5块磁盘创建新的磁盘组
5.1、检查新分配的5块磁盘状态
SQL> select group_number g_nu,os_mb,header_status,mount_status,name,path from v$asm_disk order by 1; G_NU OS_MB HEADER_STATU MOUNT_S NAME ---------- ---------- ------------ ------- ------------------------------ PATH -------------------------------------------------------------------------------- 0 5120 CANDIDATE CLOSED /dev/mapper/ZDATA_FDISK_CELL03_009 0 5120 CANDIDATE CLOSED /dev/mapper/ZDATA_FDISK_CELL03_008 0 5120 CANDIDATE CLOSED /dev/mapper/ZDATA_FDISK_CELL02_008 G_NU OS_MB HEADER_STATU MOUNT_S NAME ---------- ---------- ------------ ------- ------------------------------ PATH -------------------------------------------------------------------------------- 0 5120 CANDIDATE CLOSED /dev/mapper/ZDATA_FDISK_CELL01_009 0 5120 CANDIDATE CLOSED /dev/mapper/ZDATA_FDISK_CELL01_008复制
5.2、创建新的磁盘组
SQL> create diskgroup OCR_DG high redundancy FAILGROUP fg1 DISK '/dev/mapper/ZDATA_FDISK_CELL01_008' FAILGROUP fg2 DISK '/dev/mapper/ZDATA_FDISK_CELL02_008' FAILGROUP fg3 DISK '/dev/mapper/ZDATA_FDISK_CELL03_008' FAILGROUP fg4 DISK '/dev/mapper/ZDATA_FDISK_CELL01_009' FAILGROUP fg5 DISK '/dev/mapper/ZDATA_FDISK_CELL03_009' attribute 'COMPATIBLE.ASM' = '11.2'; SQL> alter diskgroup OCR_DG set attribute 'compatible.rdbms' = '11.2'; SQL> select v1.name,v2.failgroup,v2.failgroup_type,v2.total_mb,v2.free_mb from v$asm_diskgroup v1,v$asm_disk v2 where v1.group_number=v2.group_number and v1.name='OCR_DG'; NAME FAILGROUP FAILGRO TOTAL_MB FREE_MB -------------------- ---------- ------- ---------- ---------- OCR_DG FG3 REGULAR 5120 5060 OCR_DG FG2 REGULAR 5120 5061 OCR_DG FG5 REGULAR 5120 5065 OCR_DG FG1 REGULAR 5120 5062 OCR_DG FG4 REGULAR 5120 5063复制
5.3、在所有节点检查磁盘组状态
SQL> select GROUP_NUMBER,name,state,type,COMPATIBILITY,DATABASE_COMPATIBILITY from v$asm_diskgroup where name='OCR_DG'; GROUP_NUMBER NAME STATE TYPE COMPATIBILITY DATABASE_COMPATIBILITY ------------ -------------------- ----------- ------ ------------------------------ ------------------------------ 5 OCR_DG MOUNTED HIGH 11.2.0.0.0 11.2.0.0.0复制
6、迁移OCR
6.1、 检查OCR和集群状态
# crsctl status res -t 略...复制
# /data/grid/product/11.2.0/bin/ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3208 Available space (kbytes) : 258912 ID : 1813559706 Device/File Name : +SYS Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded复制
6.2 检查OCR备份
# /data/grid/product/11.2.0/bin/ocrconfig -showbackup ods01 2021/12/02 18:00:38 /data/grid/product/11.2.0/cdata/ods-cluster/backup00.ocr ods01 2021/12/02 14:00:38 /data/grid/product/11.2.0/cdata/ods-cluster/backup01.ocr ods01 2021/12/02 10:00:38 /data/grid/product/11.2.0/cdata/ods-cluster/backup02.ocr ods01 2021/12/01 02:00:33 /data/grid/product/11.2.0/cdata/ods-cluster/day.ocr ods01 2021/11/21 17:59:53 /data/grid/product/11.2.0/cdata/ods-cluster/week.ocr ods01 2021/12/02 14:20:32 /data/grid/product/11.2.0/cdata/ods-cluster/backup_20211202_142032.ocr复制
6.3 手动备份
# /data/grid/product/11.2.0/bin/ocrconfig -manualbackup ods01 2021/12/02 18:10:05 /data/grid/product/11.2.0/cdata/ods-cluster/backup_20211202_181005.ocr ods01 2021/12/02 14:20:32 /data/grid/product/11.2.0/cdata/ods-cluster/backup_20211202_142032.ocr复制
6.4 添加OCR到新的磁盘组
# /data/grid/product/11.2.0/bin/ocrconfig -add +OCR_DG复制
6.5 再次检查OCR，发现已有Device/File Name:+OCR_DG
# /data/grid/product/11.2.0/bin/ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3228 Available space (kbytes) : 258892 ID : 1813559706 Device/File Name : +SYS Device/File integrity check succeeded Device/File Name : +OCR_DG Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded复制
6.6 OCR信息从老磁盘组删除
# /data/grid/product/11.2.0/bin/ocrconfig -delete +SYS复制
7、迁移VOTEDISK
7.1 检查VOTEDISK
/data/grid/product/11.2.0/bin/crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 0ba9ff8a4b264fbebf88f8dd35a0a498 (/dev/mapper/ZDATA_HDISK_CELL01_001) [SYS] 2. ONLINE 87b5b01eadff4fa9bf73edcd74b0e606 (/dev/mapper/ZDATA_HDISK_CELL02_001) [SYS] 3. ONLINE 324e58a5139c4ff7bfa85c9b0c2c074d (/dev/mapper/ZDATA_HDISK_CELL03_001) [SYS] Located 3 voting disk(s).复制
7.2 迁移VOTEDISK到新的磁盘组
grid$ /data/grid/product/11.2.0/bin/crsctl replace votedisk +OCR_DG Successful addition of voting disk 50843c23687e4fa0bfb48d1078215741. Successful addition of voting disk 8c467bf6a4564ff2bf0e02235d888593. Successful addition of voting disk 8f6cba4e17854fc3bf533085757ae1c5. Successful addition of voting disk 10faa71035404fc5bf5e6f6dc3e08591. Successful addition of voting disk a850b284b91c4f2dbf4f589f3ed415a2. Successful deletion of voting disk 09cb63637c454f74bf0bae43d5851a52. Successful deletion of voting disk b1b8878b4b224f35bfb6636030e1af24. Successful deletion of voting disk 0ecba1f8e0544fa0bf756d26153d3df7. Successfully replaced voting disk group with +OCR_DG. CRS-4266: Voting file(s) successfully replaced复制
7.3 再次检查VOTEDISK，已迁移到新的磁盘组
grid$ /data/grid/product/11.2.0/bin/crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 50843c23687e4fa0bfb48d1078215741 (/dev/mapper/ZDATA_FDISK_CELL01_008) [OCR_DG] 2. ONLINE 8c467bf6a4564ff2bf0e02235d888593 (/dev/mapper/ZDATA_FDISK_CELL02_008) [OCR_DG] 3. ONLINE 8f6cba4e17854fc3bf533085757ae1c5 (/dev/mapper/ZDATA_FDISK_CELL03_008) [OCR_DG] 4. ONLINE 10faa71035404fc5bf5e6f6dc3e08591 (/dev/mapper/ZDATA_FDISK_CELL01_009) [OCR_DG] 5. ONLINE a850b284b91c4f2dbf4f589f3ed415a2 (/dev/mapper/ZDATA_FDISK_CELL03_009) [OCR_DG] Located 5 voting disk(s).复制
8、迁移asm实例spfile文件
8.1 检查spfile
SQL> show parameter spfile NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ spfile string +SYS/ods-cluster/asmparameterf ile/registry.253.1089164317 SQL>复制
8.2 在新的磁盘组创建spfile
SQL> create pfile='/tmp/asmpfile.ora' from spfile; File created. SQL> create spfile='+OCR_DG' from pfile='/tmp/asmpfile.ora'; File created. SQL>复制
8.3 核实gpnptool信息
grid$ gpnptool get 结果略...复制
8.4 重启集群
在所有节点检查磁盘组状态
SQL> select name, state, type from v$asm_diskgroup;复制
重启所有节点集群是其使用新的spfile
# crsctl stop crs # crsctl start crs复制
9、核对磁盘、磁盘组及集群运行状态
SQL> set line 1000 SQL> set pages 599 SQL> col path format a30 SQL> select name,path,group_number,header_status,total_mb,free_mb from v$asm_disk; SQL> select name,state,usable_file_mb,total_mb,free_mb,required_mirror_free_mb from v$asm_diskgroup; $ORACLE_HOME/bin/crsctl stat res -init -t $ORACLE_HOME/bin/crsctl check cluster -all $ORACLE_HOME/bin/crsctl stat res -t复制
「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」
关注作者
【版权声明】本文为墨天轮用户原创内容，转载时必须标注文章的来源（墨天轮），文章链接，文章作者等基本信息，否则作者和墨天轮有权追究责任。如果您发现墨天轮中有涉嫌抄袭或者侵权的内容，欢迎发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
