报错：
20200402:12:58:36:069637 gpstop:ares:gpadmin-[INFO]:- Segments stopped successfully = 60
20200402:12:58:36:069637 gpstop:ares:gpadmin-[INFO]:- Segments with errors during stop = 0
20200402:12:58:36:069637 gpstop:ares:gpadmin-[INFO]:-
20200402:12:58:36:069637 gpstop:ares:gpadmin-[WARNING]:-Segments that are currently marked down in configuration = 6 <<<<<<<<
并且启动成功的节点数少于原节点数量
文档来源
https://blog.csdn.net/dazuiba008/article/details/68925839
查找错误
[gpadmin@ares testdb]$ psql -d gp_sydb
psql (9.4.24)
Type "help" for help.
gp_sydb=# SELECT * FROM gp_segment_configuration WHERE status='d';
dbid | content | role | preferred_role | mode | status | port | hostname | address | datadir
------+---------+------+----------------+------+--------+------+--------------------+--------------------+--------------------------------------
8 | 6 | m | p | n | d | 6000 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data/primary/gpseg6
9 | 7 | m | p | n | d | 6001 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data/primary/gpseg7
10 | 8 | m | p | n | d | 6002 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data/primary/gpseg8
11 | 9 | m | p | n | d | 6003 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data2/primary/gpseg9
12 | 10 | m | p | n | d | 6004 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data2/primary/gpseg10
13 | 11 | m | p | n | d | 6005 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data2/primary/gpseg11
(6 rows)
psql -d postgres -c "SELECT * FROM gp_segment_configuration WHERE status='d';"
status字段 d=down u=up
mode = s, c, r s=synced, c=change logging, r=resyning。
已同步，说明节点现在是mirror状态，并且已经完成同步。
跟踪块变更，说明节点现在是primary状态，正在产生块变更日志。
同步中，说明节点现在是mirror状态，并且还没有同步完成。
关闭greenplum 报错
Continue with Greenplum instance shutdown Yy|Nn (default=N):
> y
20200402:15:06:05:089021 gpstop:ares:gpadmin-[INFO]:-Commencing Master instance shutdown with mode='smart'
20200402:15:06:05:089021 gpstop:ares:gpadmin-[INFO]:-Master segment instance directory=/opt/greenplum/data/master/gpseg-1
20200402:15:06:05:089021 gpstop:ares:gpadmin-[INFO]:-Stopping master segment and waiting for user connections to finish ...
server shutting down
20200402:15:08:06:089021 gpstop:ares:gpadmin-[WARNING]:-Smart mode failed to shutdown the master.
[gpadmin@ares testdb]$ gpstate -m
20200402:15:14:49:091070 gpstate:ares:gpadmin-[INFO]:-Starting gpstate with args: -m
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560'
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Nov 1 2019 22:06:07'
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:-Obtaining Segment details from master...
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:--------------------------------------------------------------
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:--Current GPDB mirror list and status
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:--Type = Group
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:--------------------------------------------------------------
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- Mirror Datadir Port Status Data Status
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- artemis.hadoop.com /opt/greenplum/data/mirror/gpseg0 7000 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- artemis.hadoop.com /opt/greenplum/data/mirror/gpseg1 7001 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- artemis.hadoop.com /opt/greenplum/data/mirror/gpseg2 7002 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- artemis.hadoop.com /opt/greenplum/data2/mirror/gpseg3 7003 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- artemis.hadoop.com /opt/greenplum/data2/mirror/gpseg4 7004 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- artemis.hadoop.com /opt/greenplum/data2/mirror/gpseg5 7005 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- bird.hadoop.com /opt/greenplum/data/mirror/gpseg6 7000 Acting as Primary Not In Sync
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- bird.hadoop.com /opt/greenplum/data/mirror/gpseg7 7001 Acting as Primary Not In Sync
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- bird.hadoop.com /opt/greenplum/data/mirror/gpseg8 7002 Acting as Primary Not In Sync
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- bird.hadoop.com /opt/greenplum/data2/mirror/gpseg9 7003 Acting as Primary Not In Sync
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- bird.hadoop.com /opt/greenplum/data2/mirror/gpseg10 7004 Acting as Primary Not In Sync
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- bird.hadoop.com /opt/greenplum/data2/mirror/gpseg11 7005 Acting as Primary Not In Sync
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- lover.hadoop.com /opt/greenplum/data/mirror/gpseg12 7000 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- lover.hadoop.com /opt/greenplum/data/mirror/gpseg13 7001 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- lover.hadoop.com /opt/greenplum/data/mirror/gpseg14 7002 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- lover.hadoop.com /opt/greenplum/data2/mirror/gpseg15 7003 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- lover.hadoop.com /opt/greenplum/data2/mirror/gpseg16 7004 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- lover.hadoop.com /opt/greenplum/data2/mirror/gpseg17 7005 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- uranus.hadoop.com /opt/greenplum/data/mirror/gpseg18 7000 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- uranus.hadoop.com /opt/greenplum/data/mirror/gpseg19 7001 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- uranus.hadoop.com /opt/greenplum/data/mirror/gpseg20 7002 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- uranus.hadoop.com /opt/greenplum/data2/mirror/gpseg21 7003 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- uranus.hadoop.com /opt/greenplum/data2/mirror/gpseg22 7004 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- uranus.hadoop.com /opt/greenplum/data2/mirror/gpseg23 7005 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- ares.hadoop.com /opt/greenplum/data/mirror/gpseg24 7000 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- ares.hadoop.com /opt/greenplum/data/mirror/gpseg25 7001 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- ares.hadoop.com /opt/greenplum/data/mirror/gpseg26 7002 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- ares.hadoop.com /opt/greenplum/data2/mirror/gpseg27 7003 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- ares.hadoop.com /opt/greenplum/data2/mirror/gpseg28 7004 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:- ares.hadoop.com /opt/greenplum/data2/mirror/gpseg29 7005 Passive Synchronized
20200402:15:14:50:091070 gpstate:ares:gpadmin-[INFO]:--------------------------------------------------------------
20200402:15:14:50:091070 gpstate:ares:gpadmin-[WARNING]:-6 segment(s) configured as mirror(s) are acting as primaries
20200402:15:14:50:091070 gpstate:ares:gpadmin-[WARNING]:-6 mirror segment(s) acting as primaries are not synchronized
You have new mail in /var/spool/mail/root
在每一个segment检查warning，error,fatal，panic日志级别的信息
gpssh -f /home/gpadmin/conf/seg_hosts -e 'source /usr/local/greenplum-db/greenplum_path.sh;gplogfilter -t /data01/gpadmin/gpdata/primary/*/pg_log/gpdb*.csv' > seglog.out
这里直接gprecoverseg = gprecoverseg -o ./recov(生成配置文件) + gprecoverseg -i ./recov(读取配置文件进行恢复)
gprecoverseg -o /home/gpadmin/recover_config_file
[gpadmin@ares bin]$ gprecoverseg -o /home/gpadmin/recover_config_file
20200402:15:24:49:092737 gprecoverseg:ares:gpadmin-[INFO]:-Starting gprecoverseg with args: -o /home/gpadmin/recover_config_file
20200402:15:24:49:092737 gprecoverseg:ares:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560'
20200402:15:24:49:092737 gprecoverseg:ares:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Nov 1 2019 22:06:07'
20200402:15:24:49:092737 gprecoverseg:ares:gpadmin-[INFO]:-Obtaining Segment details from master...
20200402:15:24:49:092737 gprecoverseg:ares:gpadmin-[INFO]:-Configuration file output to /home/gpadmin/recover_config_file successfully.
gprecoverseg -i /home/gpadmin/recover_config_file
[gpadmin@ares bin]$ gprecoverseg -i /home/gpadmin/recover_config_file
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Starting gprecoverseg with args: -i /home/gpadmin/recover_config_file
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560'
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Nov 1 2019 22:06:07'
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Obtaining Segment details from master...
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Heap checksum setting is consistent between master and the segments that are candidates for recoverseg
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Greenplum instance recovery parameters
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Recovery from configuration -i option supplied
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 1 of 6
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data/primary/gpseg6
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 6000
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data/mirror/gpseg6
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 7000
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 2 of 6
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data/primary/gpseg7
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 6001
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data/mirror/gpseg7
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 7001
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 3 of 6
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data/primary/gpseg8
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 6002
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data/mirror/gpseg8
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 7002
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 4 of 6
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data2/primary/gpseg9
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 6003
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data2/mirror/gpseg9
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 7003
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 5 of 6
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data2/primary/gpseg10
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 6004
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data2/mirror/gpseg10
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 7004
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 6 of 6
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = artemis.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data2/primary/gpseg11
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 6005
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = bird.hadoop.com
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data2/mirror/gpseg11
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 7005
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:25:01:092823 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
Continue with segment recovery procedure Yy|Nn (default=N):
> y
20200402:15:25:04:092823 gprecoverseg:ares:gpadmin-[INFO]:-6 segment(s) to recover
20200402:15:25:04:092823 gprecoverseg:ares:gpadmin-[INFO]:-Ensuring 6 failed segment(s) are stopped
20200402:15:25:06:092823 gprecoverseg:ares:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
20200402:15:25:06:092823 gprecoverseg:ares:gpadmin-[INFO]:-Updating configuration with new mirrors
20200402:15:25:06:092823 gprecoverseg:ares:gpadmin-[INFO]:-Updating mirrors
20200402:15:25:06:092823 gprecoverseg:ares:gpadmin-[INFO]:-Running pg_rewind on required mirrors
我的等待的时间有点长，大约30分钟？？？
[B20200402:15:47:11:092823 gprecoverseg:ares:gpadmin-[INFO]:-Starting mirrors
20200402:15:47:11:092823 gprecoverseg:ares:gpadmin-[INFO]:-era is a85e24bf5268647b_200402150816
20200402:15:47:11:092823 gprecoverseg:ares:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
.............................................
20200402:15:47:56:092823 gprecoverseg:ares:gpadmin-[INFO]:-Process results...
20200402:15:47:56:092823 gprecoverseg:ares:gpadmin-[INFO]:-Triggering FTS probe
20200402:15:47:56:092823 gprecoverseg:ares:gpadmin-[INFO]:-******************************************************************
20200402:15:47:56:092823 gprecoverseg:ares:gpadmin-[INFO]:-Updating segments for streaming is completed.
20200402:15:47:56:092823 gprecoverseg:ares:gpadmin-[INFO]:-For segments updated successfully, streaming will continue in the background.
20200402:15:47:56:092823 gprecoverseg:ares:gpadmin-[INFO]:-Use gpstate -s to check the streaming progress.
20200402:15:47:56:092823 gprecoverseg:ares:gpadmin-[INFO]:-******************************************************************
You have new mail in /var/spool/mail/root
观察节点gpstate -s | grep Sync，如果说还没有恢复完= Not In Sync，恢复完了就都是= Synchronized
[gpadmin@ares root]$ gpstate -s | grep Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Not In Sync
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
20200402:15:47:00:096303 gpstate:ares:gpadmin-[INFO]:- Mirror status = Synchronized
查看节点状态gpstate -a
[gpadmin@ares root]$ gpstate -a
20200402:15:48:33:096837 gpstate:ares:gpadmin-[INFO]:-Starting gpstate with args: -a
20200402:15:48:33:096837 gpstate:ares:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560'
20200402:15:48:33:096837 gpstate:ares:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Nov 1 2019 22:06:07'
20200402:15:48:33:096837 gpstate:ares:gpadmin-[INFO]:-Obtaining Segment details from master...
20200402:15:48:34:096837 gpstate:ares:gpadmin-[INFO]:-Gathering data from segments...
.
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:-Greenplum instance status summary
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:-----------------------------------------------------
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Master instance = Active
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Master standby = No master standby configured
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total segment instance count from metadata = 60
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:-----------------------------------------------------
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Primary Segment Status
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:-----------------------------------------------------
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total primary segments = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total primary segment valid (at master) = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total primary segment failures (at master) = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of postmaster.pid files found = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of /tmp lock files found = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number postmaster processes missing = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number postmaster processes found = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:-----------------------------------------------------
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Mirror Segment Status
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:-----------------------------------------------------
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total mirror segments = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total mirror segment valid (at master) = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total mirror segment failures (at master) = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of postmaster.pid files missing = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of postmaster.pid files found = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of postmaster.pid PIDs missing = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of postmaster.pid PIDs found = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of /tmp lock files missing = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number of /tmp lock files found = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number postmaster processes missing = 0
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number postmaster processes found = 30
20200402:15:48:35:096837 gpstate:ares:gpadmin-[WARNING]:-Total number mirror segments acting as primary segments = 6 <<<<<<<<
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:- Total number mirror segments acting as mirror segments = 24
20200402:15:48:35:096837 gpstate:ares:gpadmin-[INFO]:-----------------------------------------------------
查看节点，这里看到角色优先级互换了
select * from gp_segment_configuration;
gp_sydb=# select * from gp_segment_configuration;
dbid | content | role | preferred_role | mode | status | port | hostname | address | datadir
------+---------+------+----------------+------+--------+------+--------------------+--------------------+--------------------------------------
1 | -1 | p | p | n | u | 5432 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data/master/gpseg-1
31 | 29 | p | p | s | u | 6005 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data2/primary/gpseg29
61 | 29 | m | m | s | u | 7005 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data2/mirror/gpseg29
3 | 1 | p | p | s | u | 6001 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data/primary/gpseg1
33 | 1 | m | m | s | u | 7001 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data/mirror/gpseg1
16 | 14 | p | p | s | u | 6002 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data/primary/gpseg14
46 | 14 | m | m | s | u | 7002 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data/mirror/gpseg14
19 | 17 | p | p | s | u | 6005 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data2/primary/gpseg17
49 | 17 | m | m | s | u | 7005 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data2/mirror/gpseg17
23 | 21 | p | p | s | u | 6003 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data2/primary/gpseg21
53 | 21 | m | m | s | u | 7003 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data2/mirror/gpseg21
24 | 22 | p | p | s | u | 6004 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data2/primary/gpseg22
54 | 22 | m | m | s | u | 7004 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data2/mirror/gpseg22
27 | 25 | p | p | s | u | 6001 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data/primary/gpseg25
57 | 25 | m | m | s | u | 7001 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data/mirror/gpseg25
28 | 26 | p | p | s | u | 6002 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data/primary/gpseg26
58 | 26 | m | m | s | u | 7002 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data/mirror/gpseg26
29 | 27 | p | p | s | u | 6003 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data2/primary/gpseg27
59 | 27 | m | m | s | u | 7003 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data2/mirror/gpseg27
26 | 24 | p | p | s | u | 6000 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data/primary/gpseg24
56 | 24 | m | m | s | u | 7000 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data/mirror/gpseg24
30 | 28 | p | p | s | u | 6004 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data2/primary/gpseg28
60 | 28 | m | m | s | u | 7004 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data2/mirror/gpseg28
17 | 15 | p | p | s | u | 6003 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data2/primary/gpseg15
47 | 15 | m | m | s | u | 7003 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data2/mirror/gpseg15
14 | 12 | p | p | s | u | 6000 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data/primary/gpseg12
44 | 12 | m | m | s | u | 7000 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data/mirror/gpseg12
21 | 19 | p | p | s | u | 6001 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data/primary/gpseg19
51 | 19 | m | m | s | u | 7001 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data/mirror/gpseg19
20 | 18 | p | p | s | u | 6000 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data/primary/gpseg18
50 | 18 | m | m | s | u | 7000 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data/mirror/gpseg18
22 | 20 | p | p | s | u | 6002 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data/primary/gpseg20
52 | 20 | m | m | s | u | 7002 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data/mirror/gpseg20
25 | 23 | p | p | s | u | 6005 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data2/primary/gpseg23
55 | 23 | m | m | s | u | 7005 | uranus.hadoop.com | uranus.hadoop.com | /opt/greenplum/data2/mirror/gpseg23
15 | 13 | p | p | s | u | 6001 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data/primary/gpseg13
45 | 13 | m | m | s | u | 7001 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data/mirror/gpseg13
18 | 16 | p | p | s | u | 6004 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data2/primary/gpseg16
48 | 16 | m | m | s | u | 7004 | lover.hadoop.com | lover.hadoop.com | /opt/greenplum/data2/mirror/gpseg16
2 | 0 | p | p | s | u | 6000 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data/primary/gpseg0
32 | 0 | m | m | s | u | 7000 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data/mirror/gpseg0
7 | 5 | p | p | s | u | 6005 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data2/primary/gpseg5
37 | 5 | m | m | s | u | 7005 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data2/mirror/gpseg5
5 | 3 | p | p | s | u | 6003 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data2/primary/gpseg3
35 | 3 | m | m | s | u | 7003 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data2/mirror/gpseg3
6 | 4 | p | p | s | u | 6004 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data2/primary/gpseg4
36 | 4 | m | m | s | u | 7004 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data2/mirror/gpseg4
4 | 2 | p | p | s | u | 6002 | ares.hadoop.com | ares.hadoop.com | /opt/greenplum/data/primary/gpseg2
34 | 2 | m | m | s | u | 7002 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data/mirror/gpseg2
41 | 9 | p | m | s | u | 7003 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data2/mirror/gpseg9
11 | 9 | m | p | s | u | 6003 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data2/primary/gpseg9
38 | 6 | p | m | s | u | 7000 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data/mirror/gpseg6
8 | 6 | m | p | s | u | 6000 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data/primary/gpseg6
39 | 7 | p | m | s | u | 7001 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data/mirror/gpseg7
9 | 7 | m | p | s | u | 6001 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data/primary/gpseg7
40 | 8 | p | m | s | u | 7002 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data/mirror/gpseg8
10 | 8 | m | p | s | u | 6002 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data/primary/gpseg8
42 | 10 | p | m | s | u | 7004 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data2/mirror/gpseg10
12 | 10 | m | p | s | u | 6004 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data2/primary/gpseg10
43 | 11 | p | m | s | u | 7005 | bird.hadoop.com | bird.hadoop.com | /opt/greenplum/data2/mirror/gpseg11
13 | 11 | m | p | s | u | 6005 | artemis.hadoop.com | artemis.hadoop.com | /opt/greenplum/data2/primary/gpseg11
(61 rows)
进行角色调换：
gprecoverseg -r
[gpadmin@ares bin]$ gprecoverseg -r
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Starting gprecoverseg with args: -r
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560'
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Nov 1 2019 22:06:07'
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Obtaining Segment details from master...
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Greenplum instance recovery parameters
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Recovery type = Rebalance
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 1 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data/mirror/gpseg6
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 7000
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 2 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data/primary/gpseg6
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 6000
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 3 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data/mirror/gpseg7
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 7001
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 4 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data/primary/gpseg7
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 6001
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 5 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data/mirror/gpseg8
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 7002
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 6 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data/primary/gpseg8
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 6002
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 7 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data2/mirror/gpseg9
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 7003
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 8 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data2/primary/gpseg9
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 6003
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 9 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data2/mirror/gpseg10
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 7004
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 10 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data2/primary/gpseg10
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 6004
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 11 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = bird.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data2/mirror/gpseg11
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 7005
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:-Unbalanced segment 12 of 12
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance host = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance address = artemis.hadoop.com
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance directory = /opt/greenplum/data2/primary/gpseg11
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Unbalanced instance port = 6005
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Balanced role = Primary
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:- Current role = Mirror
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[WARNING]:-This operation will cancel queries that are currently executing.
20200402:15:55:46:098288 gprecoverseg:ares:gpadmin-[WARNING]:-Connections to the database however will not be interrupted.
Continue with segment rebalance procedure Yy|Nn (default=N):
> y
20200402:15:55:49:098288 gprecoverseg:ares:gpadmin-[INFO]:-Getting unbalanced segments
20200402:15:55:49:098288 gprecoverseg:ares:gpadmin-[INFO]:-Stopping unbalanced primary segments...
20200402:15:55:50:098288 gprecoverseg:ares:gpadmin-[INFO]:-Triggering segment reconfiguration
20200402:15:56:00:098288 gprecoverseg:ares:gpadmin-[INFO]:-Starting segment synchronization
20200402:15:56:00:098288 gprecoverseg:ares:gpadmin-[INFO]:-=============================START ANOTHER RECOVER=========================================
20200402:15:56:00:098288 gprecoverseg:ares:gpadmin-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560'
20200402:15:56:00:098288 gprecoverseg:ares:gpadmin-[INFO]:-master Greenplum Version: 'PostgreSQL 9.4.24 (Greenplum Database 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Nov 1 2019 22:06:07'
20200402:15:56:00:098288 gprecoverseg:ares:gpadmin-[INFO]:-Obtaining Segment details from master...
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Heap checksum setting is consistent between master and the segments that are candidates for recoverseg
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Greenplum instance recovery parameters
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Recovery type = Standard
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 1 of 6
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data/mirror/gpseg6
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 7000
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data/primary/gpseg6
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 6000
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 2 of 6
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data/mirror/gpseg7
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 7001
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data/primary/gpseg7
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 6001
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 3 of 6
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data/mirror/gpseg8
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 7002
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data/primary/gpseg8
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 6002
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 4 of 6
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data2/mirror/gpseg9
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 7003
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data2/primary/gpseg9
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 6003
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 5 of 6
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data2/mirror/gpseg10
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 7004
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data2/primary/gpseg10
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 6004
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Recovery 6 of 6
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Synchronization mode = Incremental
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance host = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance address = bird.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance directory = /opt/greenplum/data2/mirror/gpseg11
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Failed instance port = 7005
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance host = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance address = artemis.hadoop.com
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance directory = /opt/greenplum/data2/primary/gpseg11
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Source instance port = 6005
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:- Recovery Target = in-place
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:----------------------------------------------------------
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-6 segment(s) to recover
20200402:15:56:01:098288 gprecoverseg:ares:gpadmin-[INFO]:-Ensuring 6 failed segment(s) are stopped
20200402:15:56:03:098288 gprecoverseg:ares:gpadmin-[INFO]:-Ensuring that shared memory is cleaned up for stopped segments
20200402:15:56:04:098288 gprecoverseg:ares:gpadmin-[INFO]:-Updating configuration with new mirrors
20200402:15:56:04:098288 gprecoverseg:ares:gpadmin-[INFO]:-Updating mirrors
20200402:15:56:04:098288 gprecoverseg:ares:gpadmin-[INFO]:-Running pg_rewind on required mirrors
20200402:15:56:07:098288 gprecoverseg:ares:gpadmin-[INFO]:-Starting mirrors
20200402:15:56:07:098288 gprecoverseg:ares:gpadmin-[INFO]:-era is a85e24bf5268647b_200402150816
20200402:15:56:07:098288 gprecoverseg:ares:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
.
20200402:15:56:09:098288 gprecoverseg:ares:gpadmin-[INFO]:-Process results...
20200402:15:56:09:098288 gprecoverseg:ares:gpadmin-[INFO]:-Triggering FTS probe
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-******************************************************************
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-Updating segments for streaming is completed.
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-For segments updated successfully, streaming will continue in the background.
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-Use gpstate -s to check the streaming progress.
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-******************************************************************
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-==============================END ANOTHER RECOVER==========================================
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-******************************************************************
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-The rebalance operation has completed successfully.
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-There is a resynchronization running in the background to bring all
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-segments in sync.
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-Use gpstate -e to check the resynchronization progress.
20200402:15:56:10:098288 gprecoverseg:ares:gpadmin-[INFO]:-******************************************************************
重启节点
[gpadmin@ares bin]$ gpstart -a
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Starting gpstart with args: -a
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Gathering information and validating the environment...
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 6.1.0 build commit:6788ca8c13b2bd6e8976ccffea07313cbab30560'
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Greenplum Catalog Version: '301908232'
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Starting Master instance in admin mode
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Obtaining Segment details from master...
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Setting new master era
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Master Started...
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Shutting down master
20200402:15:59:28:099866 gpstart:ares:gpadmin-[INFO]:-Commencing parallel primary and mirror segment instance startup, please wait...
..
20200402:15:59:31:099866 gpstart:ares:gpadmin-[INFO]:-Process results...
20200402:15:59:31:099866 gpstart:ares:gpadmin-[INFO]:-----------------------------------------------------
20200402:15:59:31:099866 gpstart:ares:gpadmin-[INFO]:- Successful segment starts = 60
20200402:15:59:31:099866 gpstart:ares:gpadmin-[INFO]:- Failed segment starts = 0
20200402:15:59:31:099866 gpstart:ares:gpadmin-[INFO]:- Skipped segment starts (segments are marked down in configuration) = 0
20200402:15:59:31:099866 gpstart:ares:gpadmin-[INFO]:-----------------------------------------------------
20200402:15:59:31:099866 gpstart:ares:gpadmin-[INFO]:-Successfully started 60 of 60 segment instances
20200402:15:59:31:099866 gpstart:ares:gpadmin-[INFO]:-----------------------------------------------------
20200402:15:59:31:099866 gpstart:ares:gpadmin-[INFO]:-Starting Master instance ares.hadoop.com directory /opt/greenplum/data/master/gpseg-1
20200402:15:59:32:099866 gpstart:ares:gpadmin-[INFO]:-Command pg_ctl reports Master ares.hadoop.com instance active
20200402:15:59:32:099866 gpstart:ares:gpadmin-[INFO]:-Connecting to dbname='template1' connect_timeout=15
20200402:15:59:32:099866 gpstart:ares:gpadmin-[INFO]:-No standby master configured. skipping...
20200402:15:59:32:099866 gpstart:ares:gpadmin-[INFO]:-Database successfully started
