お元気そうで残念です

もんじゃ焼きが転がってるところにいます。

MySQL 5.7.23 で Group Replicationの復旧などを確認

前回はGroup Replicationを構築しました。
今回は、Primary Node死亡時の流れや復旧の流れを確認します。
my.cnfへの書き出しができていれば、mysqlプロセスを再起動するだけでONLINE状態まで戻ることが確認できました。

環境

環境は前回と同じ以下の環境です。

OS アプリ IP ホスト名
CentOS7 MySQL 5.7.23, MySQL Shell 8.0.16 192.168.10.101 node01
CentOS7 MySQL 5.7.23, MySQL Shell 8.0.16 192.168.10.102 node02
CentOS7 MySQL 5.7.23, MySQL Shell 8.0.16 192.168.10.103 node03

Primary Nodeの死

Primary Nodeが死亡したときの動作を確認します。

node2でmysqlshを実行してclusterの状態を見てみましょう。
どのnodeも正常に動作していて、node1がPrimary Nodeになっています。
あと、node2で実行してもclusterの状態はnode1から取得しているようです。

[node2] $ mysqlsh --log-level=DEBUG3  --uri repl@node2:3306
 MySQL  node2:3306  JS > var cluster = dba.getCluster()
 MySQL  node2:3306  JS > cluster.status()
{
    "clusterName": "myCluster", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "node1:3306", 
        "ssl": "DISABLED", 
        "status": "OK", 
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", 
        "topology": {
            "node1:3306": {
                "address": "node1:3306", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "node2:3306": {
                "address": "node2:3306", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "node3:3306": {
                "address": "node3:3306", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "node1:3306"
}

続いて、node1のmysqlを止めます。

[node1] $ sudo systemctl stop mysqld

node2で状態を確認します。
このとき、node2で実行しているmysqlshでcluster.status()を実行すると、node1のmysqlに紐付いているのでCluster.status: MySQL server has gone away (MySQL Error 2006) と出て情報が取れません。

再度getClusterして確認します。
node1の情報が取得できない状態になり、Primary Nodeがnode2になっています。

[node2] $ mysqlsh --log-level=DEBUG3  --uri repl@node2:3306
 MySQL  node2:3306  JS > var cluster = dba.getCluster()
 MySQL  node2:3306  JS > cluster.status()
{
    "clusterName": "myCluster", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "node2:3306", 
        "ssl": "DISABLED", 
        "status": "OK_NO_TOLERANCE", 
        "statusText": "Cluster is NOT tolerant to any failures. 1 member is not active", 
        "topology": {
            "node1:3306": {
                "address": "node1:3306", 
                "mode": "n/a", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)"
            }, 
            "node2:3306": {
                "address": "node2:3306", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "node3:3306": {
                "address": "node3:3306", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "node2:3306"
}

Nodeの復旧

node1を復旧してみましょう。

mysqlを再起動します。

[node1] $ sudo systemctl start mysqld

node2でcluster.status()を確認してみましょう。
node1がONLINEになり復旧できていることが確認できます。

[node2] $ mysqlsh --log-level=DEBUG3  --uri repl@node2:3306
 MySQL  node2:3306  JS > var cluster = dba.getCluster()
 MySQL  node2:3306  JS > cluster.status()
{
    "clusterName": "myCluster", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "node2:3306", 
        "ssl": "DISABLED", 
        "status": "OK", 
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", 
        "topology": {
            "node1:3306": {
                "address": "node1:3306", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "node2:3306": {
                "address": "node2:3306", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "node3:3306": {
                "address": "node3:3306", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "node2:3306"
}

続いて、node3を止めて復旧してみましょう。

[node3] $ sudo systemctl stop mysqld
[node3] $ sudo systemctl start mysqld

statusを確認してみると、node3はmissingのままになっています。

[node2] $ mysqlsh --log-level=DEBUG3  --uri repl@node2:3306
 MySQL  node2:3306  JS > var cluster = dba.getCluster()
 MySQL  node2:3306  JS > cluster.status()
{
    "clusterName": "myCluster", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "node2:3306", 
        "ssl": "DISABLED", 
        "status": "OK_NO_TOLERANCE", 
        "statusText": "Cluster is NOT tolerant to any failures. 1 member is not active", 
        "topology": {
            "node1:3306": {
                "address": "node1:3306", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "node2:3306": {
                "address": "node2:3306", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "node3:3306": {
                "address": "node3:3306", 
                "mode": "n/a", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "node2:3306"
}

journalを確認します。

[node3] $ sudo journalctl -xe
...
...[ERROR] Plugin group_replication reported: 'The group name '' is not a valid UUID'
...[ERROR] Plugin group_replication reported: 'Unable to start Group Replication on boot'
...

group nameが空っぽだから起動できないようですね?
addInstanceしたあとで、my.cnfに書き出したはずですが、my.cnfを確認してみるとgroup_replication_group_name =と空っぽになっていました。

もう一度設定してみましょう。
node3でgroup replicationが実行されたままで、addInstanceができないため、停止してからaddInstanceする必要があります。
追加したら、dba.configureLocalInstance()を実行して設定を永続化させましょう。

[node3] $ sudo mysql -u root
mysql > STOP GROUP_REPLICATION;

[node2] $ mysqlsh --log-level=DEBUG3  --uri repl@node2:3306
 MySQL  node2:3306  JS > var cluster = dba.getCluster()
 MySQL  node2:3306  JS > cluster.addInstance('repl@node3:3306')

[node3] $ mysqlsh --log-level=DEBUG3  --uri repl@node2:3306
 MySQL  node3:3306  JS > dba.configureLocalInstance()

無事追加できたのでstatusを確認してみましょう。
node3のstatusがRECOVERINGとなっており復帰できていないようですね。
復帰できていないくせに、clusterのstatusはOKになっているのが罠感あります。

[node2] $ mysqlsh --log-level=DEBUG3  --uri repl@node2:3306
 MySQL  node2:3306  JS > var cluster = dba.getCluster()
 MySQL  node2:3306  JS > cluster.status()
{
    "clusterName": "myCluster", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "node2:3306", 
        "ssl": "DISABLED", 
        "status": "OK", 
        "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", 
        "topology": {
            "node1:3306": {
                "address": "node1:3306", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "node2:3306": {
                "address": "node2:3306", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "ONLINE"
            }, 
            "node3:3306": {
                "address": "node3:3306", 
                "mode": "n/a", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "RECOVERING"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "node2:3306"
}

node3のjournalを確認してみましょう。
ERRORがでているようです。内容的にwarning感がありますが他にめぼしいログは無いですね。

[node3] $ sudo journalctl -xe
...
...[ERROR] Plugin group_replication reported: 'Group contains 3 members which is greater than group_replication_auto_increment_increment value of 1. This can lead to an higher rate of transactional aborts.'
...

エラーはgroup_replication_auto_increment_increment = が1なのにmemberは3やでと言っていますが、設定ファイルにはgroup_replication_auto_increment_increment = 7となっています。
mysqlで確認しても7になっていますね。

第49回 MySQLのAUTO_INCREMENTについて:MySQL道普請便り|gihyo.jp … 技術評論社
こちらのサイトの解説を見てもこれが原因で復帰できないとは思えません。

MySQL Group Replicationのモニタリング – variable.jp [データベース,パフォーマンス,運用]
こちらのサイトを参考に状況を調査してみましょう。
queueにtransactionが残っているようですね?

[node3] $ sudo mysql -u root
mysql >  select * from performance_schema.replication_group_member_stats \G;
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 15628341822720480:10
                         MEMBER_ID: d46d27c7-a24e-11e9-a4e6-fa163e4b9377
       COUNT_TRANSACTIONS_IN_QUEUE: 1
        COUNT_TRANSACTIONS_CHECKED: 0
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: 
    LAST_CONFLICT_FREE_TRANSACTION: 
1 row in set (0.00 sec)

他のnodeの状況も調査してみましょう。
どちらもqueueは0ですね。checkedの値が違うのが気になりますがそれはおいておきましょう。

このCOUNT_TRANSACTIONS_IN_QUEUEは、https://mysqlhighavailability.com/mysql-group-replication-monitoring/ このサイトによると、

Count_Transactions_in_queue – Number of transactions in queue pending certification and apply. For the simple run that we had this field shows value 0 but if the load is high we will have transactions in queue.

となっており、なんか処理されていないtransactionがあるようです。

[node1] $ sudo mysql -u root
mysql> select * from performance_schema.replication_group_member_stats \G
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 15628341822720480:10
                         MEMBER_ID: 440dd2c5-a22b-11e9-b851-fa163e47fb1b
       COUNT_TRANSACTIONS_IN_QUEUE: 0
        COUNT_TRANSACTIONS_CHECKED: 4
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: 10d5fb2f-a3b1-11e9-b146-fa163e47fb1b:1-15,
15f39657-a3b2-11e9-b146-fa163e47fb1b:1-15,
19a85aa3-a3b0-11e9-b146-fa163e47fb1b:1-15,
34f9b48b-a3ac-11e9-b146-fa163e47fb1b:1-15,
440dd2c5-a22b-11e9-b851-fa163e47fb1b:1-156,
7ea76381-a3ab-11e9-b146-fa163e47fb1b:1-3,
9c72025d-a3b5-11e9-b146-fa163e47fb1b:1-25,
a70e6e35-a3b0-11e9-b146-fa163e47fb1b:1-15,
b4c0063d-a3ad-11e9-b146-fa163e47fb1b:1-15,
ea69819e-a3ac-11e9-b146-fa163e47fb1b:1-15,
f1813b82-a242-11e9-bb2d-fa163e47fb1b:1-2,
f5a6d972-a3b6-11e9-b146-fa163e47fb1b:1-29,
f9d4b90c-a246-11e9-b146-fa163e47fb1b:1-268:1000193-1000256:2000233-2000244
    LAST_CONFLICT_FREE_TRANSACTION: f5a6d972-a3b6-11e9-b146-fa163e47fb1b:29
1 row in set (0.00 sec)

[node2] $ sudo mysql -u root
mysql> select * from performance_schema.replication_group_member_stats \G
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 15628341822720480:10
                         MEMBER_ID: 34ce3aed-a249-11e9-b3e8-fa163e4531c6
       COUNT_TRANSACTIONS_IN_QUEUE: 0
        COUNT_TRANSACTIONS_CHECKED: 8
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: 10d5fb2f-a3b1-11e9-b146-fa163e47fb1b:1-15,
15f39657-a3b2-11e9-b146-fa163e47fb1b:1-15,
19a85aa3-a3b0-11e9-b146-fa163e47fb1b:1-15,
34f9b48b-a3ac-11e9-b146-fa163e47fb1b:1-15,
440dd2c5-a22b-11e9-b851-fa163e47fb1b:1-156,
7ea76381-a3ab-11e9-b146-fa163e47fb1b:1-3,
9c72025d-a3b5-11e9-b146-fa163e47fb1b:1-25,
a70e6e35-a3b0-11e9-b146-fa163e47fb1b:1-15,
b4c0063d-a3ad-11e9-b146-fa163e47fb1b:1-15,
ea69819e-a3ac-11e9-b146-fa163e47fb1b:1-15,
f1813b82-a242-11e9-bb2d-fa163e47fb1b:1-2,
f5a6d972-a3b6-11e9-b146-fa163e47fb1b:1-29,
f9d4b90c-a246-11e9-b146-fa163e47fb1b:1-268:1000193-1000256:2000233-2000244
    LAST_CONFLICT_FREE_TRANSACTION: f5a6d972-a3b6-11e9-b146-fa163e47fb1b:29
1 row in set (0.00 sec)

処理を進めて貰う方法を調べます。

MySQL 5.7と8.0でロック状態を確認する(sys.innodb_lock_waitsビュー) - Qiita
こちらのサイトを参考に確認してみましょう。
何も情報が無いですね?

[node3] $ sudo mysql -u root
mysql> select * from sys.innodb_lock_waits \G
Empty set, 3 warnings (0.01 sec)

mysql> show warnings;
+---------+------+-----------------------------------------------------------------------------------------------+
| Level   | Code | Message                                                                                       |
+---------+------+-----------------------------------------------------------------------------------------------+
| Warning | 1681 | 'INFORMATION_SCHEMA.INNODB_LOCK_WAITS' is deprecated and will be removed in a future release. |
| Warning | 1681 | 'INFORMATION_SCHEMA.INNODB_LOCKS' is deprecated and will be removed in a future release.      |
| Warning | 1681 | 'INFORMATION_SCHEMA.INNODB_LOCKS' is deprecated and will be removed in a future release.      |
+---------+------+-----------------------------------------------------------------------------------------------+
3 rows in set (0.01 sec)

なぜあなたは SHOW ENGINE INNODB STATUS を読まないのか - そーだいなるらくがき帳
Inno DBのstatusを見てみましょう。
問題が起きてそうには見えないですねー。

[node3] $ sudo mysql -u root
mysql> show engine innodb status\G
*************************** 1. row ***************************
  Type: InnoDB
  Name: 
Status: 
=====================================
2019-07-11 19:57:31 0x7fb7dcfb7700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 5 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 4 srv_active, 0 srv_shutdown, 4770 srv_idle
srv_master_thread log flush and writes: 4774
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 5
OS WAIT ARRAY INFO: signal count 5
RW-shared spins 0, rounds 10, OS waits 5
RW-excl spins 0, rounds 0, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 10.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 6690
Purge done for trx's n:o < 6688 undo n:o < 0 state: running but idle
History list length 12
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421911789999952, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421911790001776, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421911790000864, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
 ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
406 OS file reads, 111 OS file writes, 40 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
 insert 0, delete mark 0, delete 0
discarded operations:
 insert 0, delete mark 0, delete 0
Hash table size 2365399, node heap has 0 buffer(s)
Hash table size 2365399, node heap has 0 buffer(s)
Hash table size 2365399, node heap has 0 buffer(s)
Hash table size 2365399, node heap has 0 buffer(s)
Hash table size 2365399, node heap has 0 buffer(s)
Hash table size 2365399, node heap has 0 buffer(s)
Hash table size 2365399, node heap has 0 buffer(s)
Hash table size 2365399, node heap has 0 buffer(s)
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 7554795
Log flushed up to   7554795
Pages flushed up to 7554795
Last checkpoint at  7554786
0 pending log flushes, 0 pending chkp writes
27 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 8795455488
Dictionary memory allocated 133496
Buffer pool size   524256
Free buffers       523847
Database pages     409
Old database pages 0
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 373, created 36, written 71
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 409, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
----------------------
INDIVIDUAL BUFFER POOL INFO
----------------------
---BUFFER POOL 0
Buffer pool size   65536
Free buffers       65499
Database pages     37
Old database pages 0
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 37, created 0, written 16
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 37, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 1
Buffer pool size   65528
Free buffers       65517
Database pages     11
Old database pages 0
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 11, created 0, written 0
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 11, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 2
Buffer pool size   65536
Free buffers       65521
Database pages     15
Old database pages 0
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 15, created 0, written 0
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 15, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 3
Buffer pool size   65528
Free buffers       65452
Database pages     76
Old database pages 0
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 76, created 0, written 0
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 76, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 4
Buffer pool size   65536
Free buffers       65465
Database pages     71
Old database pages 0
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 71, created 0, written 1
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 71, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 5
Buffer pool size   65528
Free buffers       65460
Database pages     68
Old database pages 0
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 68, created 0, written 3
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 68, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 6
Buffer pool size   65536
Free buffers       65458
Database pages     78
Old database pages 0
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 42, created 36, written 41
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 78, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
---BUFFER POOL 7
Buffer pool size   65528
Free buffers       65475
Database pages     53
Old database pages 0
Modified db pages  0
Pending reads      0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 53, created 0, written 10
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 53, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=15367, Main thread ID=140427713836800, state: sleeping
Number of rows inserted 90, updated 12, deleted 0, read 143
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================

1 row in set (0.01 sec)

各nodeでshow master status;をしてみると、node3だけずれています。
一旦RESET MASTER;してみましょう。
MySQL :: MySQL 5.6 リファレンスマニュアル :: 13.4.1.2 RESET MASTER 構文

[node3] $ sudo mysql -u root
mysql> show master status;
+---------------+----------+--------------+------------------+-------------------+
| File          | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+---------------+----------+--------------+------------------+-------------------+
| binlog.000001 |      150 |              |                  |                   |
+---------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
mysql> STOP GROUP_REPLICATION;
Query OK, 0 rows affected (10.26 sec)

mysql> RESET MASTER;
mysql> START GROUP_REPLICATION;
mysql> select * from performance_schema.replication_group_member_stats \G
*************************** 1. row ***************************
                      CHANNEL_NAME: group_replication_applier
                           VIEW_ID: 15628341822720480:12
                         MEMBER_ID: d46d27c7-a24e-11e9-a4e6-fa163e4b9377
       COUNT_TRANSACTIONS_IN_QUEUE: 0
        COUNT_TRANSACTIONS_CHECKED: 0
          COUNT_CONFLICTS_DETECTED: 0
COUNT_TRANSACTIONS_ROWS_VALIDATING: 0
TRANSACTIONS_COMMITTED_ALL_MEMBERS: 
    LAST_CONFLICT_FREE_TRANSACTION: 
1 row in set (0.00 sec)

COUNT_TRANSACTIONS_IN_QUEUEは0になりました。
node2でcluster.status()を見るとnode3がONLINEになっています。

おそらく原因としては、node3をaddInstanceしたタイミングでSTART GROUP_REPLICATION;が走り、group_nameが空っぽのclusterのPrimaryとして動作し始めたので、group_nameを修正して元のclusterに追加した際に前のclusterのログが残っていたことで不整合が発生していたのではないでしょうか。
詳しい原因を調べたかったですが、node3に消えては困るログが残っていたわけでも無いのでRESET MASTER;で解決しました。

とりあえず、Primary NodeとSecondary Nodeの復旧方法が確認できました。

おわりに

冒頭にも書きましたが、my.cnfへの書き出しができていれば、mysqlプロセスを再起動するだけでONLINE状態まで戻ることが確認できました。
MySQL 5.7系を使っていると、remoteからMySQL Shellで設定の書き出しができないため、nodeにsshしてMySQL Shellでdba.configureLocalInstance()を実行する必要があります。

データを流してみて同期の流れや不整合についてなども確認したかったのですが、ボリュームが増えちゃったので一旦区切ります。
次回はデータ周りを確認してみましょう。