openGauss 集群 2.0.1 企业版 灰度升级至 2.1.0 操作过程

详细内容可参照：https://www.modb.pro/doc/80693

一、准备工作

1.1 下载2.1.0 软件包

-- 移动 /opt/software/openGauss 目录下除 clusterconfig.xml文件外 至 备份目录
-- 下载 2.1.0 软件包至节点一 /opt/software/openGauss目录
[root@opengauss-node1 openGauss]# ll
total 98284
drwxr-xr-x 5 root root       4096 Nov  4 10:24 2.0.1
drwxr-xr-x 5 root root       4096 Nov  3 10:37 bak
-rw------- 1 omm  dbgrp      2055 Nov  1 17:32 clusterconfig.xml
-rw-r--r-- 1 root root  100623501 Nov  2 11:23 openGauss-2.1.0-CentOS-64bit-all.tar.gz
复制

1.2 解压2.1.0 软件包

-- 节点一root用户执行
[root@opengauss-node1 openGauss]# tar -zxvf openGauss-2.1.0-CentOS-64bit-all.tar.gz 
[root@opengauss-node1 openGauss]# tar -zxvf openGauss-2.1.0-CentOS-64bit-om.tar.gz
复制

1.3 查询版本信息

-- omm 用户
[root@opengauss-node1 ~]# su - omm
Last login: Fri Nov  4 10:29:33 CST 2022 on pts/1
-- 查询所有节点版本信息
[omm@opengauss-node1 ~]$ gs_ssh -c "gsql -V"
Successfully execute command on all nodes.

Output:
[SUCCESS] opengauss-node1:
gsql (openGauss 2.0.1 build d97c0e8a) compiled at 2021-06-02 19:37:17 commit 0 last mr  
[SUCCESS] opengauss-node2:
gsql (openGauss 2.0.1 build d97c0e8a) compiled at 2021-06-02 19:37:17 commit 0 last mr 
复制

1.4 查询集群状态

-- omm 用户
[root@opengauss-node1 ~]# su - omm
Last login: Fri Nov  4 10:32:52 CST 2022 on pts/1
[omm@opengauss-node1 ~]$ gs_om -t status --detail --all
[   Cluster State   ]

cluster_state   : Normal   -- Normal 代表数据库状态正常
redistributing  : No
current_az      : AZ_ALL

[  Datanode State   ]

node               node_ip         instance                  state            | node               node_ip         instance                  state
--------------------------------------------------------------------------------------------------------
1  opengauss-node1 192.168.17.136  6001 /gaussdb/data/db1 P Primary Normal | 2  opengauss-node2 192.168.17.137  6002 /gaussdb/data/db1 S Standby Normal
复制

1.5 停止集群

-- 停集群，节点一 omm 用户
[root@opengauss-node1 ~]# su - omm
Last login: Fri Nov  4 10:33:09 CST 2022 on pts/1
[omm@opengauss-node1 ~]$ gs_om -t stop
Stopping cluster.
=========================================
Successfully stopped cluster.
=========================================
End stop cluster.
[omm@opengauss-node1 ~]$ gs_om -t status --detail --all
[   Cluster State   ]

cluster_state   : Unavailable  -- Unavailable 代表数据库状态不可用
redistributing  : No
current_az      : AZ_ALL

[  Datanode State   ]

node               node_ip         instance                  state            | node               node_ip         instance                  state
--------------------------------------------------------------------------------------------------------
1  opengauss-node1 192.168.17.136  6001 /gaussdb/data/db1 P Down    Manually stopped | 2  opengauss-node2 192.168.17.137  6002 /gaussdb/data/db1 S Down    Manually stopped
复制

1.6 备份目录及文件

-- 两节点都需操作
-- 升级前建议参照clusterconfig.xml文件对相应目录及文件进行备份，以防升级失败
-- 本次测试环境数据库相应目录如下
<PARAM name="gaussdbAppPath" value="/opt/gaussdb/app" />
<PARAM name="gaussdbLogPath" value="/opt/gaussdb/log" />
<PARAM name="tmpMppdbPath" value="/opt/gaussdb/tmp" />
<PARAM name="gaussdbToolPath" value="/opt/gaussdb/gausstools/om" />
<PARAM name="corePath" value="/opt/gaussdb/corefile" />
<PARAM name="dataNode1" value="/gaussdb/data/db1,opengauss-node2,/gaussdb/data/db1"/>

-- 备份目录
[root@opengauss-nodeX ~]# cd /opt
[root@opengauss-nodeX opt]# tar -czf gaussdb.tar ./gaussdb/
[root@opengauss-nodeX opt]# cd /
[root@opengauss-node1 /]# tar -czf gaussdb.tar.gz ./gaussdb/
复制

1.7 启动集群

-- omm 用户，节点一操作
[root@opengauss-node1 ~]# su - omm
Last login: Fri Nov  4 10:37:14 CST 2022 on pts/1
[omm@opengauss-node1 ~]$ gs_om -t start
Starting cluster.
=========================================
[SUCCESS] opengauss-node1
2022-11-04 10:50:25.162 63647df0.1 [unknown] 140210156164864 [unknown] 0 dn_6001_6002 01000  0 [BACKEND] WARNING:  Failed to initialize the memory protect for g_instance.attr.attr_storage.cstore_buffers (1024 Mbytes) or shared memory (4250 Mbytes) is larger.
[SUCCESS] opengauss-node2
2022-11-04 10:50:31.617 63647df7.1 [unknown] 140188793579264 [unknown] 0 dn_6001_6002 01000  0 [BACKEND] WARNING:  Failed to initialize the memory protect for g_instance.attr.attr_storage.cstore_buffers (1024 Mbytes) or shared memory (4250 Mbytes) is larger.
=========================================
Successfully started.
[omm@opengauss-node1 ~]$ gs_om -t status --detail --all
[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
current_az      : AZ_ALL

[  Datanode State   ]

node               node_ip         instance                  state            | node               node_ip         instance                  state
-------------------------------------------------------------------------------------------------------
1  opengauss-node1 192.168.17.136  6001 /gaussdb/data/db1 P Primary Normal | 2  opengauss-node2 192.168.17.137  6002 /gaussdb/data/db1 S Standby Normal
复制

二、集群升级

2.1 升级前预检查

-- root 用户，节点一操作
-- 设置lib库
[root@opengauss-node1 ~]# export LD_LIBRARY_PATH=/opt/software/openGauss/script/gspylib/clib:$LD_LIBRARY_PATH

-- 执行预检查
[root@opengauss-node1 ~]# python3 /opt/software/openGauss/script/gs_preinstall -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml

-- 以下是预检查执行过程
[root@opengauss-node1 ~]# cd /opt/software/openGauss/
[root@opengauss-node1 openGauss]# export LD_LIBRARY_PATH=/opt/software/openGauss/script/gspylib/clib:$LD_LIBRARY_PATH
[root@opengauss-node1 openGauss]# python3 /opt/software/openGauss/script/gs_preinstall -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml
Parsing the configuration file.
Successfully parsed the configuration file.
Installing the tools on the local node.
Successfully installed the tools on the local node.
Are you sure you want to create trust for root (yes/no)? yes  --输入root口令
Please enter password for root.
Password: 
Creating SSH trust for the root permission user.
Checking network information.
All nodes in the network are Normal.
Successfully checked network information.
Creating SSH trust.
Creating the local key file.
Successfully created the local key files.
Appending local ID to authorized_keys.
Successfully appended local ID to authorized_keys.
Updating the known_hosts file.
Successfully updated the known_hosts file.
Appending authorized_key on the remote node.
Successfully appended authorized_key on all remote node.
Checking common authentication file content.
Successfully checked common authentication content.
Distributing SSH trust file to all node.
Successfully distributed SSH trust file to all node.
Verifying SSH trust on all hosts.
Successfully verified SSH trust on all hosts.
Successfully created SSH trust.
Successfully created SSH trust for the root permission user.
Setting pssh path
Successfully set core path.
Distributing package.
Begin to distribute package to tool path.
Successfully distribute package to tool path.
Begin to distribute package to package path.
Successfully distribute package to package path.
Successfully distributed package.
Are you sure you want to create the user[omm] and create trust for it (yes/no)? yes -- 如已创建omm，可选no
Preparing SSH service.
Successfully prepared SSH service.
Installing the tools in the cluster.
Successfully installed the tools in the cluster.
Checking hostname mapping.
Successfully checked hostname mapping.
Creating SSH trust for [omm] user.
Please enter password for current user[omm].
Password: 
Checking network information.
All nodes in the network are Normal.
Successfully checked network information.
Creating SSH trust.
Creating the local key file.
Successfully created the local key files.
Appending local ID to authorized_keys.
Successfully appended local ID to authorized_keys.
Updating the known_hosts file.
Successfully updated the known_hosts file.
Appending authorized_key on the remote node.
Successfully appended authorized_key on all remote node.
Checking common authentication file content.
Successfully checked common authentication content.
Distributing SSH trust file to all node.
Successfully distributed SSH trust file to all node.
Verifying SSH trust on all hosts.
Successfully verified SSH trust on all hosts.
Successfully created SSH trust.
Successfully created SSH trust for [omm] user.
Checking OS software.
Successfully check os software.
Checking OS version.
Successfully checked OS version.
Creating cluster's path.
Successfully created cluster's path.
Set and check OS parameter.
Setting OS parameters.
Successfully set OS parameters.
Warning: Installation environment contains some warning messages.
Please get more details by "/opt/software/openGauss/script/gs_checkos -i A -h opengauss-node1,opengauss-node2 --detail".
Set and check OS parameter completed.
Preparing CRON service.
Successfully prepared CRON service.
Setting user environmental variables.
Successfully set user environmental variables.
Setting the dynamic link library.
Successfully set the dynamic link library.
Setting Core file
Successfully set core path.
Setting pssh path
Successfully set pssh path.
Setting Cgroup.
Successfully set Cgroup.
Set ARM Optimization.
No need to set ARM Optimization.
Fixing server package owner.
Setting finish flag.
Successfully set finish flag.
Preinstallation succeeded.

-- 查看预检查执行详细信息，可执行如下命令，另外也可以进入到/opt/gaussdb/log/omm/om目录下查看执行过程日志
[root@opengauss-node1 ~]# /opt/software/openGauss/script/gs_checkos -i A -h opengauss-node1,opengauss-node2 --detail

-- 执行结果如下
Checking items:
    A1. [ OS version status ]                                   : Normal     
        [opengauss-node2]
        centos_7.6.1810_64bit
        [opengauss-node1]
        centos_7.6.1810_64bit

    A2. [ Kernel version status ]                               : Normal     
        The names about all kernel versions are same. The value is "3.10.0-957.27.2.el7.x86_64".
    A3. [ Unicode status ]                                      : Normal     
        The values of all unicode are same. The value is "LANG=en_US.UTF-8".
    A4. [ Time zone status ]                                    : Normal     
        The informations about all timezones are same. The value is "+0800".
    A5. [ Swap memory status ]                                  : Normal     
        The value about swap memory is correct.            
    A6. [ System control parameters status ]                    : Normal     
        All values about system control  parameters are correct.
    A7. [ File system configuration status ]                    : Normal     
        Both soft nofile and hard nofile are correct.      
    A8. [ Disk configuration status ]                           : Normal     
        The value about XFS mount parameters is correct.   
    A9. [ Pre-read block size status ]                          : Normal     
        The value about Logical block size is correct.     
    A10.[ IO scheduler status ]                                 : Normal     
        The value of IO scheduler is correct.              
    A11.[ Network card configuration status ]                   : Warning    
        [opengauss-node2]
BondMode Null
        Warning reason: network 'ens33' 'mtu' RealValue '1500' ExpectedValue '8192' -- 忽略

        [opengauss-node1]
BondMode Null
        Warning reason: network 'ens33' 'mtu' RealValue '1500' ExpectedValue '8192'


    A12.[ Time consistency status ]                             : Warning    
        [opengauss-node2]
        The NTPD not detected on machine and local time is "2022-11-04 11:08:57".
        [opengauss-node1]
        The NTPD not detected on machine and local time is "2022-11-04 11:08:57".

    A13.[ Firewall service status ]                             : Normal     
        The firewall service is stopped.                   
    A14.[ THP service status ]                                  : Normal     
        The THP service is stopped.                        
Total numbers:14. Abnormal numbers:0. Warning numbers:2.
复制

-- omm用户
# 预检查执行后会发现数据库版本号信息已发生改变，显示为 2.1.0
[root@opengauss-node1 ~]# su - omm
Last login: Fri Nov  4 11:32:24 CST 2022 on pts/1
[omm@opengauss-node1 ~]$ gs_om -V
gs_om (openGauss OM 2.1.0 build 956ee0b1) compiled at 2021-09-30 14:40:54 commit 0 last mr

-- 此时集群两节点数据库版本还未发生改变
[omm@opengauss-node1 ~]$ gs_ssh -c "gsql -V"
Successfully execute command on all nodes.

Output:
[SUCCESS] opengauss-node1:
gsql (openGauss 2.0.1 build d97c0e8a) compiled at 2021-06-02 19:37:17 commit 0 last mr  
[SUCCESS] opengauss-node2:
gsql (openGauss 2.0.1 build d97c0e8a) compiled at 2021-06-02 19:37:17 commit 0 last mr 
复制

2.2 执行升级

[root@opengauss-node1 ~]# chmod -R 755 /opt/software/openGauss/script/
[root@opengauss-node1 ~]# chown -R omm:dbgrp /opt/software/openGauss/script/
[root@opengauss-node1 ~]# su - omm
Last login: Thu Nov  3 11:13:42 CST 2022 on pts/1

-- 修改version.cfg权限,两节点都需操作
[root@opengauss-nodeX ~]# chown omm:dbgrp /opt/software/openGauss/version.cfg
# 如未修改version.cfg权限，执行升级时会报如下错：
# [omm@opengauss-node1 ~]$ /opt/software/openGauss/script/gs_upgradectl -t auto-upgrade --grey -X /opt/software/openGauss/clusterconfig.xml
# Static configuration matched with old static configuration files.
# Successfully set upgrade_mode to 0.
# [GAUSS-52900] : Failed to upgrade strategy: [Errno 13] Permission denied: '/opt/software/openGauss/version.cfg'. Do nothing this time.
# [GAUSS-52900] : Failed to upgrade strategy: [Errno 13] Permission denied: '/opt/software/openGauss/version.cfg'. Do nothing this time.


-- 灰度升级
[omm@opengauss-node1 ~]$ /opt/software/openGauss/script/gs_upgradectl -t auto-upgrade --grey -X /opt/software/openGauss/clusterconfig.xml

-- 执行过程如下
[omm@opengauss-node1 ~]$ /opt/software/openGauss/script/gs_upgradectl -t auto-upgrade --grey -X /opt/software/openGauss/clusterconfig.xml
Static configuration matched with old static configuration files.
Successfully set upgrade_mode to 0.
Checking upgrade environment.
Successfully checked upgrade environment.
Start to do health check.
Successfully checked cluster status.
Upgrade all nodes.
Performing grey rollback.
No need to rollback.
The directory /opt/gaussdb/app_d97c0e8a will be deleted after commit-upgrade, please make sure there is no personal data.
Installing new binary.
Successfully backup hotpatch config file.
Sync cluster configuration.
Successfully synced cluster configuration.
Switch symbolic link to new binary directory.
Successfully switch symbolic link to new binary directory.
Switching all db processes.
Wait for the cluster status normal or degrade.
Successfully switch all process version
The nodes ['opengauss-node1', 'opengauss-node2'] have been successfully upgraded to new version. Then do health check.
Start to do health check.
Successfully checked cluster status.
Waiting for the cluster status to become normal.
.
The cluster status is normal.
Upgrade main process has been finished, user can do some check now.
Once the check done, please execute following command to commit upgrade:

    gs_upgradectl -t commit-upgrade -X /opt/software/openGauss/clusterconfig.xml   

Successfully upgrade nodes.

-- 升级提交
[omm@opengauss-node1 ~]$ gs_upgradectl -t commit-upgrade -X /opt/software/openGauss/clusterconfig.xml
-- 升级提交执行结果如下
Start to do health check.
Successfully checked cluster status.
Successfully cleaned old install path.
Commit upgrade succeeded.

# 升级过程可跟踪查看两节点 /opt/gaussdb/log/omm/bin/gs_ctl 目录下 gs_ctl-*.log 日志，查看升级过程
复制

2.3 信息核查

-- 查看版本信息
[root@opengauss-node1 ~]# su - omm
Last login: Fri Nov  4 11:43:21 CST 2022 on pts/1

-- 版本信息为 2.1.0
[omm@opengauss-node1 ~]$ gs_om -V
gs_om (openGauss OM 2.1.0 build 956ee0b1) compiled at 2021-09-30 14:40:54 commit 0 last mr

-- 查看两节点数据库版本信息，都已升级到2.1.0
[omm@opengauss-node1 ~]$ gs_ssh -c "gsql -V"
Successfully execute command on all nodes.

Output:
[SUCCESS] opengauss-node1:
gsql (openGauss 2.1.0 build 590b0f8e) compiled at 2021-09-30 14:29:04 commit 0 last mr  
[SUCCESS] opengauss-node2:
gsql (openGauss 2.1.0 build 590b0f8e) compiled at 2021-09-30 14:29:04 commit 0 last mr

-- 集群状态信息
[omm@opengauss-node1 ~]$ gs_om -t status --detail --all
[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
current_az      : AZ_ALL

[  Datanode State   ]

    node           node_ip         port      instance                  state
--------------------------------------------------------------------------------------------
1  opengauss-node1 192.168.17.136  26000      6001 /gaussdb/data/db1   P Primary Normal
2  opengauss-node2 192.168.17.137  26000      6002 /gaussdb/data/db1   S Standby Normal
复制