standby master的模拟故障与修复

本文档主要演示GreenPlum数据库的Master节点的Standby在异常后的数据同步及Master节点故障后，Standby的切换。在操作前通过gpstate命令查看Standby的状态是正常的，数据也在同步状态。同时将现有master（原有的standy）和原有的master再做切换，恢复原状。

1 检查集群gpstate -f

[gpadmin@mdw ~]$ gpstate -f
20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24'
20200420:00:23:47:010679 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-Standby master details
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-----------------------
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-   Standby address          = smdw
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-   Standby data directory   = /greenplum/gpdata/master/gpseg-1
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-   Standby port             = 5432
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-   Standby PID              = 7593
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:-   Standby status           = Standby host passive
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CADFDD8
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CADFDD8
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CADFDD8
20200420:00:23:48:010679 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
复制

2 模拟standby 故障

mdw 
cat .bash_profile 
export PATH
export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1
source /usr/local/greenplum-db/greenplum_path.sh
export PGPORT=5432
export PGDATABASE=archdata

sdw3 and smdw
vi .bash_profile

export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1
source /usr/local/greenplum-db/greenplum_path.sh
export PGPORT=5432
export PGDATABASE=archdata

vi .bashr
export MASTER_DATA_DIRECTORY=/greenplum/gpdata/master/gpseg-1
source /usr/local/greenplum-db/greenplum_path.sh
export PGPORT=5432
export PGDATABASE=archdata

smdw：
pg_ctl stop -D $MASTER_DATA_DIRECTORY
[gpadmin@sdw3 ~]$ pg_ctl stop -D $MASTER_DATA_DIRECTORY
waiting for server to shut down.... done
server stopped
[gpadmin@sdw3 ~]$
复制

3 mdw上检查集群状态

gpstate -f
复制

[gpadmin@mdw ~]$ gpstate -f
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24'
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-Standby master details
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-----------------------
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-   Standby address          = smdw
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-   Standby data directory   = /greenplum/gpdata/master/gpseg-1
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-   Standby port             = 5432
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[WARNING]:-Standby PID              = 0                                  <<<<<<<<
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[WARNING]:-Standby status           = Standby process not running        <<<<<<<<
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:-No entries found.
20200420:01:03:58:011908 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
复制

4 mdw上创建一个表，测试master的元数据变化

psql

psql -h 10.102.254.27 -U gpadmin -d archdata

[gpadmin@mdw ~]$ psql -h 10.102.254.27 -U gpadmin -d archdata
psql (8.3.23)
Type "help" for help.

archdata=# \c
You are now connected to database "archdata" as user "gpadmin".
archdata=# 
archdata=# \d
No relations found.
archdata=# create table test (id  int);
NOTICE:  Table doesn't have 'DISTRIBUTED BY' clause -- Using column named 'id' as the Greenplum Database data distribution key for this table.
HINT:  The 'DISTRIBUTED BY' clause determines the distribution of data. Make sure column(s) chosen are the optimal data distribution key to minimize skew.
CREATE TABLE
archdata=# \d
             List of relations
 Schema | Name | Type  |  Owner  | Storage 
--------+------+-------+---------+---------
 public | test | table | gpadmin | heap
(1 row)
复制

5 启动standby节点

[gpadmin@sdw3 ~]$ pg_ctl start -D $MASTER_DATA_DIRECTORY  
server starting
[gpadmin@sdw3 ~]$ 2020-04-19 17:08:36.469615 GMT,,,p29616,th-410839168,,,,0,,,seg-1,,,,,"FATAL","22023","dbid (from -b option) is not specified or is invalid.  This value must be >= 0, or >= -1 in utility mode.  The dbid value to pass can be determined from this server's entry in the segment configuration; it may be -1 if running in utility mode.",,,,,,,,"PostmasterMain","postmaster.c",1175,
复制

6 检查集群状态

gpstate -f 

[gpadmin@mdw ~]$ gpstate -f 
20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24'
20200420:01:10:47:012122 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-Standby master details
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-----------------------
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-   Standby address          = smdw
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-   Standby data directory   = /greenplum/gpdata/master/gpseg-1
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-   Standby port             = 5432
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[WARNING]:-Standby PID              = 0                                  <<<<<<<<
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[WARNING]:-Standby status           = Standby process not running        <<<<<<<<
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:-No entries found.
20200420:01:10:48:012122 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
[gpadmin@mdw ~]$ 
复制

7 尝试重启集群

gpstop -a

gpstart -a

[gpadmin@mdw ~]$ gpstop -a
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart'
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart
20200420:01:15:44:012279 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/greenplum/gpdata/master/gpseg-1
20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process
20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment /greenplum/gpdata/master/gpseg-1
20200420:01:15:45:012279 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast
20200420:01:15:46:012279 gpstop:mdw:gpadmin-[WARNING]:-Error occured while stopping the standby master: ExecutionError: 'non-zero rc: 1' occured.  Details: 'ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 smdw ". /usr/local/greenplum-db/./greenplum_path.sh; $GPHOME/bin/pg_ctl -D /greenplum/gpdata/master/gpseg-1 -m fast -w -t 120 stop"'  cmd had rc=1 completed=True halted=False
  stdout=''
  stderr='Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26'
Offending key for IP in /home/gpadmin/.ssh/known_hosts:1
Matching host key in /home/gpadmin/.ssh/known_hosts:9
pg_ctl: PID file "/greenplum/gpdata/master/gpseg-1/postmaster.pid" does not exist
Is server running?
'
20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw
20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Targeting dbid [2, 8, 3, 9, 4, 10, 5, 11, 6, 12, 7, 13] for shutdown
20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20200420:01:15:46:012279 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed
20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed
20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20200420:01:15:47:012279 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-----------------------------------------------------
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-   Segments stopped successfully      = 12
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-   Segments with errors during stop   = 0
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-----------------------------------------------------
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 12 of 12 segment instances 
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-No leftover gpmmon process found
20200420:01:15:49:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes
20200420:01:15:50:012279 gpstop:mdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20200420:01:15:50:012279 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory
[gpadmin@mdw ~]$ gpstart -a
20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -a
20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051'
20200420:01:16:06:012440 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Setting new master era
20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Master Started...
20200420:01:16:07:012440 gpstart:mdw:gpadmin-[INFO]:-Shutting down master
20200420:01:16:08:012440 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
.. 
20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Process results...
20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-   Successful segment starts                                            = 12
20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-   Failed segment starts                                                = 0
20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   = 0
20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Successfully started 12 of 12 segment instances 
20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20200420:01:16:10:012440 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /greenplum/gpdata/master/gpseg-1 
20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active
20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Starting standby master
20200420:01:16:11:012440 gpstart:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw  in directory: /greenplum/gpdata/master/gpseg-1
20200420:01:16:12:012440 gpstart:mdw:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26'
Offending key for IP in /home/gpadmin/.ssh/known_hosts:1
Matching host key in /home/gpadmin/.ssh/known_hosts:9
'
20200420:01:16:13:012440 gpstart:mdw:gpadmin-[INFO]:-Database successfully started
复制

8 再次确认状态

gpstat -f

[gpadmin@mdw ~]$ gpstate -f
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24'
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-Standby master details
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-----------------------
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-   Standby address          = smdw
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-   Standby data directory   = /greenplum/gpdata/master/gpseg-1
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-   Standby port             = 5432
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-   Standby PID              = 30408
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:-   Standby status           = Standby host passive
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CB21DB0
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CB21DB0
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CB21DB0
20200420:01:17:30:012591 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------

状态正常
复制

9 如果不能同步数据的处理

如果Standby数据不能自动同步，那么就需要重新初始化Standby的数据，也就是手动全量同步数据，因为Master只存数据库的元数据，通常都很小，这个过程很快即可完成，但是需要将Master数据库启动到Utility模式，否则会遇到下面的错误。

[gpadmin@mdw ~]$ gpinitstandby -n

20200420:01:20:23:012710 gpinitstandby:mdw:gpadmin-[INFO]:-Standy master is already up and running.

使用gpstart –m只将Master节点启动到Utility模式。


[gpadmin@mdw ~]$ gpstop -a
[gpadmin@mdw ~]$ gpstop -a
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Starting gpstop with args: -a
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-There are 0 connections to the database
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart'
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Master host=mdw
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing Master instance shutdown with mode=smart
20200420:01:21:46:012747 gpstop:mdw:gpadmin-[INFO]:-Master segment instance directory=/greenplum/gpdata/master/gpseg-1
20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Attempting forceful termination of any leftover master process
20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Terminating processes for segment /greenplum/gpdata/master/gpseg-1
20200420:01:21:47:012747 gpstop:mdw:gpadmin-[INFO]:-Stopping master standby host smdw mode=fast
20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown standby process on smdw
20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Targeting dbid [2, 8, 3, 9, 4, 10, 5, 11, 6, 12, 7, 13] for shutdown
20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel primary segment instance shutdown, please wait...
20200420:01:21:49:012747 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed
20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed
20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-Commencing parallel mirror segment instance shutdown, please wait...
20200420:01:21:50:012747 gpstop:mdw:gpadmin-[INFO]:-0.00% of jobs completed
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-100.00% of jobs completed
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-----------------------------------------------------
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-   Segments stopped successfully      = 12
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-   Segments with errors during stop   = 0
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-----------------------------------------------------
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Successfully shutdown 12 of 12 segment instances 
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Database successfully shutdown with no errors reported
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpmmon process
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-No leftover gpmmon process found
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover gpsmon processes
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-No leftover gpsmon processes on some hosts. not attempting forceful termination on these hosts
20200420:01:21:52:012747 gpstop:mdw:gpadmin-[INFO]:-Cleaning up leftover shared memory


[gpadmin@mdw ~]$ gpstart -m
[gpadmin@mdw ~]$ gpstart -m
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -m
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051'
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-****************************************************************************
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-Master-only start requested in a configuration with a standby master.
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-This is advisable only under the direct supervision of Greenplum support. 
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-This mode of operation is not supported in a production environment and 
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-may lead to a split-brain condition and possible unrecoverable data loss.
20200420:01:22:05:012908 gpstart:mdw:gpadmin-[WARNING]:-****************************************************************************

Continue with master-only startup Yy|Nn (default=N):

> y
> 20200420:01:22:07:012908 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
> 20200420:01:22:08:012908 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
> 20200420:01:22:08:012908 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
> 20200420:01:22:09:012908 gpstart:mdw:gpadmin-[INFO]:-Setting new master era
> 20200420:01:22:09:012908 gpstart:mdw:gpadmin-[INFO]:-Master Started...
> 此处需要输入Y确认。


将Master启动到Utility模式后，使用gpinitstandby来重新初始化Standby的数据。


[gpadmin@mdw ~]$ gpinitstandby -n

[gpadmin@mdw ~]$ gpinitstandby -n
20200420:01:22:35:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Starting standby master
20200420:01:22:35:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw  in directory: /greenplum/gpdata/master/gpseg-1
20200420:01:22:36:012938 gpinitstandby:mdw:gpadmin-[WARNING]:-Unable to cleanup previously started standby: 'Warning: the ECDSA host key for 'smdw' differs from the key for the IP address '10.102.254.26'
Offending key for IP in /home/gpadmin/.ssh/known_hosts:1
Matching host key in /home/gpadmin/.ssh/known_hosts:9
'
20200420:01:22:37:012938 gpinitstandby:mdw:gpadmin-[INFO]:-Successfully started standby master
重启数据库，Standby的状态就会正常。


[gpadmin@mdw ~]$ gpstop -a

[gpadmin@mdw ~]$ gpstart -a
[gpadmin@mdw ~]$ gpstate -f

'
20200420:01:23:39:013123 gpstart:mdw:gpadmin-[INFO]:-Database successfully started
[gpadmin@mdw ~]$ gpstate -f
20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -f
20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3.23 (Greenplum Database 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4) on x86_64-pc-linux-gnu, compiled by GCC gcc (GCC) 6.2.0, 64-bit compiled on Aug 10 2018 07:30:24'
20200420:01:23:41:013239 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-Standby master details
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-----------------------
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-   Standby address          = smdw
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-   Standby data directory   = /greenplum/gpdata/master/gpseg-1
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-   Standby port             = 5432
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-   Standby PID              = 31874
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:-   Standby status           = Standby host passive
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--pg_stat_replication
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--WAL Sender State: streaming
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Sync state: sync
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Sent Location: 0/CB22200
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Flush Location: 0/CB22200
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--Replay Location: 0/CB22010
20200420:01:23:42:013239 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
复制