GP Master首先会检测Primary状态，如果Primary不可连通，那么将会检测Mirror状态，Primary/Mirror状态总共有4种：
Primary活着，Mirror活着。GP Master探测Primary成功之后直接返回，进行下一个Segment检测；
Primary活着，Mirror挂了。GP Master探测Primary成功之后，通过Primary返回的状态得知Mirror挂掉了（Mirror挂掉之后，Primary将会探测到，将自己变成ChangeTracking模式），这时候更新Master元信息，进行下一个Segment检测；
Primary挂了，Mirror活着。GP Master探测Primary失败之后探测Mirror，发现Mirror是活着，这时候更新Master上面的元信息，同时使Mirror接管Primary（故障切换），进行下一个Segment检测；
Primary挂了，Mirror挂了。GP Master探测Primary失败之后探测Mirror，Mirror也是挂了，直到重试最大值，结束这个Segment的探测，也不更新Master元信息了，进行下一个Segment检测。
上面的2-4需要进行gprecoverseg进行segment恢复。
对失败的segment节点；启动时会直接跳过，忽略。
原文链接：https://blog.csdn.net/kjh2007abc/article/details/85001364以下内容为作者在启动数据库的时候遇到的故障情况
gpadmin@mdw ~]$ gpstart -a
20210414:22:49:05:005365 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -a
20210414:22:49:05:005365 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20210414:22:49:05:005365 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: ‘postgres (Greenplum Database) 6.14.0 build commit:62d24f4a455276cab4bf2ca4538e96dcf58db8ba Open Source‘
20210414:22:49:05:005365 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: ‘301908232‘
20210414:22:49:05:005365 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20210414:22:49:06:005365 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20210414:22:49:06:005365 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210414:22:49:06:005365 gpstart:mdw:gpadmin-[INFO]:-Setting new master era
20210414:22:49:06:005365 gpstart:mdw:gpadmin-[INFO]:-Master Started...
20210414:22:49:06:005365 gpstart:mdw:gpadmin-[INFO]:-Shutting down master
20210414:22:49:06:005365 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw3 directory /data01/gpdata/gpdatap1/gpseg4 <<<<<
20210414:22:49:06:005365 gpstart:mdw:gpadmin-[WARNING]:-Skipping startup of segment marked down in configuration: on sdw3 directory /data01/gpdata/gpdatap2/gpseg5 <<<<<
20210414:22:49:06:005365 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
..
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:-Process results...
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:- Successful segment starts = 10
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:- Failed segment starts = 0
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[WARNING]:-Skipped segment starts (segments are marked down in configuration) = 2 <<<<<<<<
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:-Successfully started 10 of 10 segment instances, skipped 2 other segments
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[WARNING]:-****************************************************************************
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[WARNING]:-There are 2 segment(s) marked down in the database
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[WARNING]:-To recover from this current state, review usage of the gprecoverseg
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[WARNING]:-management utility which will recover failed segment instance databases.
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[WARNING]:-****************************************************************************
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data01/gpdata/gpmaster/gpseg-1
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active
20210414:22:49:09:005365 gpstart:mdw:gpadmin-[INFO]:-Connecting to dbname=‘template1‘ connect_timeout=15
20210414:22:49:10:005365 gpstart:mdw:gpadmin-[INFO]:-Starting standby master
20210414:22:49:10:005365 gpstart:mdw:gpadmin-[INFO]:-Checking if standby master is running on host: smdw in directory: /data01/gpdata/gpmaster/gpseg-1
20210414:22:49:11:005365 gpstart:mdw:gpadmin-[WARNING]:-Number of segments not attempted to start: 2
20210414:22:49:11:005365 gpstart:mdw:gpadmin-[INFO]:-Check status of database with gpstate utility查看数据库的mirror的节点启动状态
[gpadmin@mdw ~]$ gpstate -m
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -m
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: ‘postgres (Greenplum Database) 6.14.0 build commit:62d24f4a455276cab4bf2ca4538e96dcf58db8ba Open Source‘
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: ‘PostgreSQL 9.4.24 (Greenplum Database 6.14.0 build commit:62d24f4a455276cab4bf2ca4538e96dcf58db8ba Open Source) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Feb 5 2021 18:58:52‘
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:--Current GPDB mirror list and status
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:--Type = Group
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data01/gpdata/gpdatam1/gpseg0 50000 Passive Not In Sync
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data01/gpdata/gpdatam2/gpseg1 50001 Passive Synchronized
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data01/gpdata/gpdatam1/gpseg2 50000 Passive Not In Sync
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data01/gpdata/gpdatam2/gpseg3 50001 Passive Not In Sync
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data01/gpdata/gpdatam1/gpseg4 50000 Acting as Primary Not In Sync
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data01/gpdata/gpdatam2/gpseg5 50001 Acting as Primary Not In Sync
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[WARNING]:-2 segment(s) configured as mirror(s) are acting as primaries
20210414:22:49:58:006300 gpstate:mdw:gpadmin-[WARNING]:-2 mirror segment(s) acting as primaries are not synchronized
如何恢复这个primary /mirror segment呢？
首先产生一个恢复的配置文件 ： gprecoverseg -o ./recov
[gpadmin@mdw ~]$ gprecoverseg -o ./recov
20210414:22:51:18:007797 gprecoverseg:mdw:gpadmin-[INFO]:-Starting gprecoverseg with args: -o ./recov
20210414:22:51:19:007797 gprecoverseg:mdw:gpadmin-[INFO]:-local Greenplum Version: ‘postgres (Greenplum Database) 6.14.0 build commit:62d24f4a455276cab4bf2ca4538e96dcf58db8ba Open Source‘
20210414:22:51:19:007797 gprecoverseg:mdw:gpadmin-[INFO]:-master Greenplum Version: ‘PostgreSQL 9.4.24 (Greenplum Database 6.14.0 build commit:62d24f4a455276cab4bf2ca4538e96dcf58db8ba Open Source) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Feb 5 2021 18:58:52‘
20210414:22:51:19:007797 gprecoverseg:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210414:22:51:20:007797 gprecoverseg:mdw:gpadmin-[INFO]:-Configuration file output to ./recov successfully.
可以知道哪些segment需要恢复
[gpadmin@mdw ~]$ cat recov
sdw3|40000|/data01/gpdata/gpdatap1/gpseg4
sdw3|40001|/data01/gpdata/gpdatap2/gpseg5
[gpadmin@mdw ~]$
[gpadmin@mdw ~]$ gprecoverseg -i ./recov
20210414:22:52:17:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Starting gprecoverseg with args: -i ./recov
20210414:22:52:18:008924 gprecoverseg:mdw:gpadmin-[INFO]:-local Greenplum Version: ‘postgres (Greenplum Database) 6.14.0 build commit:62d24f4a455276cab4bf2ca4538e96dcf58db8ba Open Source‘
20210414:22:52:18:008924 gprecoverseg:mdw:gpadmin-[INFO]:-master Greenplum Version: ‘PostgreSQL 9.4.24 (Greenplum Database 6.14.0 build commit:62d24f4a455276cab4bf2ca4538e96dcf58db8ba Open Source) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Feb 5 2021 18:58:52‘
20210414:22:52:18:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Heap checksum setting is consistent between master and the segments that are candidates for recoverseg
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Greenplum instance recovery parameters
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:----------------------------------------------------------
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Recovery from configuration -i option supplied
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:----------------------------------------------------------
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Recovery 1 of 2
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:----------------------------------------------------------
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Synchronization mode = Incremental
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance host = sdw3
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance address = sdw3
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance directory = /data01/gpdata/gpdatap1/gpseg4
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance port = 40000
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Source instance host = sdw1
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Source instance address = sdw1
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Source instance directory = /data01/gpdata/gpdatam1/gpseg4
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Source instance port = 50000
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Target = in-place
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:----------------------------------------------------------
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Recovery 2 of 2
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:----------------------------------------------------------
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Synchronization mode = Incremental
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance host = sdw3
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance address = sdw3
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance directory = /data01/gpdata/gpdatap2/gpseg5
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Failed instance port = 40001
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Source instance host = sdw1
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Source instance address = sdw1
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Source instance directory = /data01/gpdata/gpdatam2/gpseg5
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Source instance port = 50001
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:- Recovery Target = in-place
20210414:22:52:19:008924 gprecoverseg:mdw:gpadmin-[INFO]:----------------------------------------------------------
Continue with segment recovery procedure Yy|Nn (default=N):
> Y
20210414:22:52:24:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Starting to modify pg_hba.conf on primary segments to allow replication connections
20210414:22:52:30:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Successfully modified pg_hba.conf on primary segments to allow replication connections
20210414:22:52:30:008924 gprecoverseg:mdw:gpadmin-[INFO]:-2 segment(s) to recover
20210414:22:52:30:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Ensuring 2 failed segment(s) are stopped
20210414:22:52:31:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
20210414:22:52:31:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Updating configuration with new mirrors
20210414:22:52:31:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Updating mirrors
20210414:22:52:31:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Running pg_rewind on required mirrors
sdw3 (dbid 6): Done!
sdw3 (dbid 7): Done!
20210414:22:52:49:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Starting mirrors
20210414:22:52:49:008924 gprecoverseg:mdw:gpadmin-[INFO]:-era is 8d2c5a3f6db9998a_210414224906
20210414:22:52:49:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
.
20210414:22:52:51:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Process results...
20210414:22:52:52:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Triggering FTS probe
20210414:22:52:52:008924 gprecoverseg:mdw:gpadmin-[INFO]:-******************************************************************
20210414:22:52:52:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Updating segments for streaming is completed.
20210414:22:52:52:008924 gprecoverseg:mdw:gpadmin-[INFO]:-For segments updated successfully, streaming will continue in the background.
20210414:22:52:52:008924 gprecoverseg:mdw:gpadmin-[INFO]:-Use gpstate -s to check the streaming progress.
20210414:22:52:52:008924 gprecoverseg:mdw:gpadmin-[INFO]:-******************************************************************
[gpadmin@mdw ~]$ gpstate -m
20210414:22:53:00:009531 gpstate:mdw:gpadmin-[INFO]:-Starting gpstate with args: -m
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:-local Greenplum Version: ‘postgres (Greenplum Database) 6.14.0 build commit:62d24f4a455276cab4bf2ca4538e96dcf58db8ba Open Source‘
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:-master Greenplum Version: ‘PostgreSQL 9.4.24 (Greenplum Database 6.14.0 build commit:62d24f4a455276cab4bf2ca4538e96dcf58db8ba Open Source) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Feb 5 2021 18:58:52‘
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:--Current GPDB mirror list and status
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:--Type = Group
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data01/gpdata/gpdatam1/gpseg0 50000 Passive Synchronized
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:- sdw2 /data01/gpdata/gpdatam2/gpseg1 50001 Passive Synchronized
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data01/gpdata/gpdatam1/gpseg2 50000 Passive Synchronized
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:- sdw3 /data01/gpdata/gpdatam2/gpseg3 50001 Passive Synchronized
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data01/gpdata/gpdatam1/gpseg4 50000 Acting as Primary Synchronized
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:- sdw1 /data01/gpdata/gpdatam2/gpseg5 50001 Acting as Primary Synchronized
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[INFO]:--------------------------------------------------------------
20210414:22:53:01:009531 gpstate:mdw:gpadmin-[WARNING]:-2 segment(s) configured as mirror(s) are acting as primaries
用于修复Segment的是gprecoverseg。使用方式比较简单，几个主要参数如下：
-i ：主要参数，用于指定一个配置文件，该配置文件描述了需要修复的Segment和修复后的目的位置。
-F ：可选项，指定后，gprecoverseg会将”-i”中指定的或标记”d”的实例删除，并从活着的Mirror复制一个完整一份到目标位置。
-r ：当FTS发现有Primary宕机并进行主备切换，在gprecoverseg修复后，担当Primary的Mirror角色并不会立即切换回来，就会导致部分主机上活跃的Segment过多从而引起性能瓶颈。因此需要恢复Segment原先的角色，称为re-balance.
原文：https://blog.51cto.com/lishiyan/2706887