60分钟部署Oracle 21c RAC

原创王先生 2025-03-04

239

[TOC]

60分钟部署Oracle 21c RAC

官方发布，Oracle 21c RAC支持在Docker环境生产使用

https://container-registry.oracle.com/ords/ocr/ba/database/rac
Oracle RAC is supported for production use on Docker starting with Oracle Database 21c (21.3).

资源准备

宿主机1台:
操作系统:Oracle Linux 7.9
https://yum.oracle.com/ISOS/OracleLinux/OL7/u9/x86_64/OracleLinux-R7-U9-Server-x86_64-dvd.iso
cpu:12核
内存:48G
磁盘:200G+(swap分32G，/boot分1G，/分剩下的)

装好后查看

[root@ol79 ~]# cat /etc/oracle-release 
Oracle Linux Server release 7.9
[root@ol79 ~]# uname -r
5.4.17-2102.201.3.el7uek.x86_64
[root@ol79 ~]# df -Th
文件系统            类型      容量  已用  可用 已用% 挂载点
devtmpfs            devtmpfs   24G     0   24G    0% /dev
tmpfs               tmpfs      24G     0   24G    0% /dev/shm
tmpfs               tmpfs      24G  8.7M   24G    1% /run
tmpfs               tmpfs      24G     0   24G    0% /sys/fs/cgroup
/dev/mapper/ol-root xfs       167G  1.4G  166G    1% /
/dev/sda1           xfs      1014M  184M  831M   19% /boot
tmpfs               tmpfs     4.7G     0  4.7G    0% /run/user/0
复制

RAC网络规划

(仅在Docker容器内部通讯，外部通讯使用cman映射实现)

172.16.1.100    racnode1.example.com            racnode1
192.168.17.100  racnode1-priv.example.com       racnode1-priv
172.16.1.130    racnode1-vip.example.com        racnode1-vip

172.16.1.101    racnode2.example.com            racnode2
192.168.17.101  racnode2-priv.example.com       racnode2-priv
172.16.1.131    racnode2-vip.example.com        racnode2-vip

172.16.1.230    racnodec1-scan.example.com      racnodec1-scan
172.16.1.164    racnodec1-cman.example.com      racnodec1-cman
复制

宿主机参数设置

vi /etc/sysctl.conf
复制

添加

fs.aio-max-nr=1048576
fs.file-max=6815744
net.core.rmem_max=4194304
net.core.rmem_default=262144
net.core.wmem_max=1048576
net.core.wmem_default=262144
net.core.rmem_default=262144
复制

强制生效

sysctl -a
sysctl -p
复制

安装仓库源

Installing Software from Oracle Linux Yum Server
https://yum.oracle.com/getting-started.html#installing-software-from-oracle-linux-yum-server
yum localinstall -y oraclelinux-release-el7-1.0-17.el7.x86_64.rpm
https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/oraclelinux-release-el7-1.0-17.el7.x86_64.rpm

安装Docker

https://docs.oracle.com/en/operating-systems/oracle-linux/docker/

yum-config-manager --enable ol7_addons
yum clean all
yum makecache
yum list available | grep -i docker
yum install -y docker-engine
systemctl enable --now docker

[root@ol79 ~]# rpm -qa | grep docker
docker-engine-19.03.11.ol-13.el7.x86_64
docker-cli-19.03.11.ol-13.el7.x86_64
复制

修改Docker启动参数为实时模式

vi /usr/lib/systemd/system/docker.service
复制

修改这一行

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --cpu-rt-runtime=950000
复制

重启生效

systemctl daemon-reload
systemctl stop docker
systemctl start docker
复制

下载rac镜像

docker pull registry.cn-hangzhou.aliyuncs.com/hd2020/ka:rac21.3.0
docker pull registry.cn-hangzhou.aliyuncs.com/hd2020/ka:cman23.5.0.0
复制

或

docker pull container-registry.oracle.com/database/rac:21.3.0.0
docker pull container-registry.oracle.com/database/cman:23.5.0.0
复制

创建docker网桥

docker network create -d bridge --subnet=172.16.1.0/24 rac_pub1_nw
docker network create -d bridge --subnet=192.168.17.0/24 rac_priv1_nw
docker network ls
复制

创建rac集群共享host文件

mkdir -p /opt/containers/
touch /opt/containers/rac_host_file
复制

指定操作系统oracle用户、grid用户，数据库sys用户的密码

mkdir -p /opt/.secrets/
openssl rand -hex 64 -out /opt/.secrets/pwd.key
echo 'Qq00000000!' > /opt/.secrets/common_os_pwdfile
openssl enc -aes-256-cbc -salt -in /opt/.secrets/common_os_pwdfile -out /opt/.secrets/common_os_pwdfile.enc -pass file:/opt/.secrets/pwd.key
rm -f /opt/.secrets/common_os_pwdfile
复制

共享块设备没有，咱走NFS

Deploying Oracle RAC on Container With NFS Volume

yum -y install nfs-utils
mkdir -p /oradata
vi /etc/exports
复制

这里没做任何访问限制，感兴趣的可以ds

/oradata *(rw,no_root_squash)
复制

启动nfs服务

systemctl start nfs-server
systemctl enable nfs-server
exportfs -a
exportfs -rv
复制

创建NFS卷，使用本机对外ip

export NFS_SERVER_NAME=172.17.10.82
docker volume create \
  --driver local \
  --opt type=nfs \
  --opt o=addr=$NFS_SERVER_NAME,rw,bg,hard,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 \
  --opt device=:/oradata racstorage
docker volume ls
复制

在nfs上创建集群共享文件

5个文件，每个2G大小，可以自由修改

dd if=/dev/zero of=/oradata/asm_disk01.img bs=1024k count=2048 
dd if=/dev/zero of=/oradata/asm_disk02.img bs=1024k count=2048 
dd if=/dev/zero of=/oradata/asm_disk03.img bs=1024k count=2048 
dd if=/dev/zero of=/oradata/asm_disk04.img bs=1024k count=2048 
dd if=/dev/zero of=/oradata/asm_disk05.img bs=1024k count=2048 
复制

创建集群的第一个节点

docker create -t -i \
  --hostname racnode1 \
  --volume /boot:/boot:ro \
  --volume /dev/shm \
  --tmpfs /dev/shm:rw,exec,size=4G \
  --volume /opt/containers/rac_host_file:/etc/hosts  \
  --volume /opt/.secrets:/run/secrets:ro \
  --volume /etc/localtime:/etc/localtime:ro \
  --cpuset-cpus 0-3 \
  --memory 16G \
  --memory-swap 32G \
  --sysctl kernel.shmall=2097152  \
  --sysctl "kernel.sem=250 32000 100 128" \
  --sysctl kernel.shmmax=8589934592  \
  --sysctl kernel.shmmni=4096 \
  --dns-search=example.com \
  --privileged=false \
  --volume racstorage:/oradata \
  --cap-add=SYS_NICE \
  --cap-add=SYS_RESOURCE \
  --cap-add=NET_ADMIN \
  -e NODE_VIP=172.16.1.130  \
  -e VIP_HOSTNAME=racnode1-vip  \
  -e PRIV_IP=192.168.17.100  \
  -e PRIV_HOSTNAME=racnode1-priv \
  -e PUBLIC_IP=172.16.1.100 \
  -e PUBLIC_HOSTNAME=racnode1  \
  -e SCAN_NAME=racnodec1-scan \
  -e SCAN_IP=172.16.1.230  \
  -e OP_TYPE=INSTALL \
  -e DOMAIN=example.com \
  -e ASM_DISCOVERY_DIR=/oradata \
  -e ASM_DEVICE_LIST=/oradata/asm_disk01.img,/oradata/asm_disk02.img,/oradata/asm_disk03.img,/oradata/asm_disk04.img,/oradata/asm_disk05.img  \
  -e CMAN_HOSTNAME=racnodec1-cman \
  -e CMAN_IP=172.16.1.164 \
  -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc \
  -e PWD_KEY=pwd.key \
  --restart=always \
  --tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --cpu-rt-runtime=95000 \
  --ulimit rtprio=99  \
  --name racnode1 \
  registry.cn-hangzhou.aliyuncs.com/hd2020/ka:rac21.3.0

docker network disconnect bridge racnode1
docker network connect rac_pub1_nw --ip 172.16.1.100 racnode1
docker network connect rac_priv1_nw --ip 192.168.17.100 racnode1

docker start racnode1
docker logs -f racnode1
复制

创建集群的第一个节点至少需要40分钟或更长时间。

您应该在最后看到数据库创建成功的消息：

####################################
ORACLE RAC DATABASE IS READY TO USE!
####################################
复制

docker exec -i -t racnode1 /bin/bash

[root@ol79 ~]# ps -ef|grep pmon
54332     6264 17133  0 09:17 ?        00:00:00 asm_pmon_+ASM1
54321    18966 17133  0 09:40 ?        00:00:00 ora_pmon_ORCLCDB1
复制

添加集群的第二个节点

docker create -t -i \
  --hostname racnode2 \
  --volume /dev/shm \
  --tmpfs /dev/shm:rw,exec,size=4G \
  --volume /boot:/boot:ro \
  --dns-search=example.com \
  --volume /opt/containers/rac_host_file:/etc/hosts \
  --volume /opt/.secrets:/run/secrets:ro \
  --volume /etc/localtime:/etc/localtime:ro \
  --cpuset-cpus 4-7 \
  --memory 16G \
  --memory-swap 32G \
  --sysctl kernel.shmall=2097152 \
  --sysctl "kernel.sem=250 32000 100 128" \
  --sysctl kernel.shmmax=8589934592 \
  --sysctl kernel.shmmni=4096 \
  --privileged=false \
  --volume racstorage:/oradata \
  --cap-add=SYS_NICE \
  --cap-add=SYS_RESOURCE \
  --cap-add=NET_ADMIN \
  -e EXISTING_CLS_NODES=racnode1 \
  -e NODE_VIP=172.16.1.131 \
  -e VIP_HOSTNAME=racnode2-vip \
  -e PRIV_IP=192.168.17.101 \
  -e PRIV_HOSTNAME=racnode2-priv \
  -e PUBLIC_IP=172.16.1.101 \
  -e PUBLIC_HOSTNAME=racnode2 \
  -e DOMAIN=example.com \
  -e SCAN_NAME=racnodec1-scan \
  -e SCAN_IP=172.16.1.230 \
  -e ASM_DISCOVERY_DIR=/oradata \
  -e ASM_DEVICE_LIST=/oradata/asm_disk01.img,/oradata/asm_disk02.img,/oradata/asm_disk03.img,/oradata/asm_disk04.img,/oradata/asm_disk05.img \
  -e ORACLE_SID=ORCLCDB \
  -e OP_TYPE=ADDNODE \
  -e COMMON_OS_PWD_FILE=common_os_pwdfile.enc \
  -e PWD_KEY=pwd.key \
  --tmpfs=/run -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --cpu-rt-runtime=95000 \
  --ulimit rtprio=99  \
  --restart=always \
  --name racnode2 \
  registry.cn-hangzhou.aliyuncs.com/hd2020/ka:rac21.3.0

docker network disconnect bridge racnode2  
docker network connect rac_pub1_nw --ip 172.16.1.101 racnode2  
docker network connect rac_priv1_nw --ip 192.168.17.101 racnode2  

docker start racnode2  
docker logs -f racnode2  
复制

添加集群的第二个节点至少需要20分钟或更长时间。

您应该在最后看到数据库创建成功的消息：

####################################
ORACLE RAC DATABASE IS READY TO USE!
####################################
复制

docker exec -i -t racnode2 /bin/bash

[grid@racnode2 ~]$ ps -ef|grep pmon
grid     18981     1  0 10:08 ?        00:00:00 asm_pmon_+ASM2
oracle   23686     1  0 10:11 ?        00:00:00 ora_pmon_ORCLCDB2
复制

检查rac集群

crsctl query css votedisk
crsctl check cluster -all
crsctl stat res -t
复制

[grid@racnode2 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   60b6365c42184ff4bfe469834faf3af7 (/oradata/asm_disk01.img) [DATA]
Located 1 voting disk(s).
[grid@racnode2 ~]$ 
[grid@racnode2 ~]$ crsctl check cluster -all
**************************************************************
racnode1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
racnode2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[grid@racnode2 ~]$ 
[grid@racnode2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
ora.chad
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
ora.net1.network
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
ora.ons
               ONLINE  ONLINE       racnode1                 STABLE
               ONLINE  ONLINE       racnode2                 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 Started,STABLE
      2        ONLINE  ONLINE       racnode2                 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
ora.cdp1.cdp
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.cvu
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.orclcdb.db
      1        ONLINE  ONLINE       racnode1                 Open,HOME=/u01/app/o
                                                             racle/product/21.3.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       racnode2                 Open,HOME=/u01/app/o
                                                             racle/product/21.3.0
                                                             /dbhome_1,STABLE
ora.orclcdb.orclpdb.pdb
      1        ONLINE  ONLINE       racnode1                 STABLE
      2        ONLINE  ONLINE       racnode2                 STABLE
ora.qosmserver
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.racnode1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
ora.racnode2.vip
      1        ONLINE  ONLINE       racnode2                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       racnode1                 STABLE
--------------------------------------------------------------------------------
复制

切换到oracle用户

sudo su
su - oracle
export ORACLE_HOME=/u01/app/oracle/product/21.3.0/dbhome_1
export ORACLE_SID=ORCLCDB2
sqlplus / as sysdba
复制

查看数据库运行模式、ASM磁盘使用情况等

SQL> select open_mode,name from v$database;

OPEN_MODE            NAME
-------------------- -------------------------
READ WRITE           ORCLCDB

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 ORCLPDB                        READ WRITE NO
SQL> select group_number,name,total_mb,free_mb from v$ASM_DISKGROUP;

GROUP_NUMBER NAME                        TOTAL_MB    FREE_MB
------------ ------------------------- ---------- ----------
           1 DATA                           10240       4476
复制

部署cman

我们规划是在一台宿主机部署2个docker容器搭建RAC，
需要使用cman(Oracle Connection Manager)做连接代理服务器，在宿主机暴露1521端口；
https://github.com/oracle/docker-images/tree/main/OracleDatabase/OracleConnectionManager

docker create -t -i --hostname racnodec1-cman --dns-search=example.com \
-e DOMAIN=example.com -e PUBLIC_IP=172.16.1.164 \
-e PUBLIC_HOSTNAME=racnodec1-cman  --privileged=false \
-e DB_HOSTDETAILS="HOST=racnode1:IP=172.16.1.100:RULE_ACT=accept,HOST=racnode2:IP=172.16.1.101:RULE_ACT=accept" \
--publish=1521:1521 \
--restart=always \
--name racnodec1-cman \
registry.cn-hangzhou.aliyuncs.com/hd2020/ka:cman23.5.0.0

docker network disconnect bridge racnodec1-cman
docker network connect rac_pub1_nw --ip 172.16.1.164 racnodec1-cman

docker start racnodec1-cman
docker logs -f racnodec1-cman
复制

您应该在最后看到cman创建成功的消息：

####################################
CONNECTION MANAGER IS READY TO USE!
####################################
复制

docker exec -it racnodec1-cman bash
复制

修改cman配置文件
将racnode1改为*
将racnode2改为*

export cmanfile=/u01/app/oracle/product/23ai/client_1/network/admin/cman.ora
sed -i 's/racnode1/*/g' $cmanfile
sed -i 's/racnode2/*/g' $cmanfile
复制

进入cman依次操作

cmctl
administer
reload
show all
复制

在其他电脑连接到RAC实例

sqlplus /nolog
conn sys/Qq00000000!@172.17.10.82:1521/orclcdb as sysdba 
复制

SQL> select instance_name,status from gv$instance;

INSTANCE_NAME    STATUS
---------------- ----------
ORCLCDB1         OPEN
ORCLCDB2         OPEN
复制

Have fun!

「喜欢这篇文章，您的关注和赞赏是给作者最好的鼓励」

关注作者

60分钟部署Oracle 21c RAC

60分钟部署Oracle 21c RAC

官方发布，Oracle 21c RAC支持在Docker环境生产使用

资源准备

RAC网络规划

宿主机参数设置

安装仓库源

安装Docker

修改Docker启动参数为实时模式

下载rac镜像

创建docker网桥

创建rac集群共享host文件

指定操作系统oracle用户、grid用户，数据库sys用户的密码

共享块设备没有，咱走NFS

在nfs上创建集群共享文件

创建集群的第一个节点

添加集群的第二个节点

检查rac集群

部署cman

在其他电脑连接到RAC实例

评论

相关阅读