redis cluster 强制kill某一个节点和shutdown某一个节点后修复过程

Redis cluster 命令行，执行以下命令需登录cluster，是集群所独有的

集群(cluster)

CLUSTER INFO 打印集群的信息

CLUSTER nodeS 列出集群当前已知的所有节点（node），以及这些节点的相关信息

节点(node)

CLUSTER MEET <ip> <port> 将 ip 和 port 所指定的节点添加到集群当中，让它成为集群的一份子

CLUSTER FORGET <node_id> 从集群中移除 node_id 指定的节点

CLUSTER REPLICATE <node_id> 将当前节点设置为 node_id 指定的节点的从节点

CLUSTER SAVECONFIG 将节点的配置文件保存到硬盘里面

槽(slot)

CLUSTER ADDSLOTS <slot> [slot ...] 将一个或多个槽（slot）指派（assign）给当前节点

CLUSTER DELSLOTS <slot> [slot ...] 移除一个或多个槽对当前节点的指派

CLUSTER FLUSHSLOTS 移除指派给当前节点的所有槽，让当前节点变成一个没有指派任何槽的节点

CLUSTER SETSLOT <slot> NODE <node_id> 将槽 slot 指派给 node_id 指定的节点，如果槽已经指派给另一个节点，那么先让另一个节点删除该槽>，然后再进行指派

CLUSTER SETSLOT <slot> MIGRATING <node_id> 将本节点的槽 slot 迁移到 node_id 指定的节点中

CLUSTER SETSLOT <slot> IMPORTING <node_id> 从 node_id 指定的节点中导入槽 slot 到本节点

CLUSTER SETSLOT <slot> STABLE 取消对槽 slot 的导入（import）或者迁移（migrate）

键 (key) 

CLUSTER KEYSLOT <key> 计算键 key 应该被放置在哪个槽上

CLUSTER COUNTKEYSINSLOT <slot> 返回槽 slot 目前包含的键值对数量

CLUSTER GETKEYSINSLOT <slot> <count> 返回 count 个 slot 槽中的键

验证将集群某一个节点强行kill后是否可以自动加入集群：

查看redis cluster关系：

[root@192-13-168-77 ~]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes

06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master - 0 1575439453000 3 connected 10923-16383

d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575439454000 1 connected 0-5460

532ef94c81188111827fef599ee73c0996a04e5e 192.13.168.77:7007@17007 slave 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 0 1575439455000 6 connected

46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575439455742 5 connected

ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575439454743 2 connected 5461-10922

b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575439454000 4 connected

查看redis cluster进程：

[root@192-13-168-77 ~]# ps -ef | grep redis

root      2039     1  0 14:00 ?        00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:2002 [cluster]              

root      2236     1  0 14:00 ?        00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:3003 [cluster]              

root      2335     1  0 14:00 ?        00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:4004 [cluster]              

root      2365     1  0 14:00 ?        00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:5005 [cluster]              

root      2391     1  0 14:00 ?        00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:6006 [cluster]              

root      2418     1  0 14:00 ?        00:00:01 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:7007 [cluster] 

[root@192-13-168-77 ~]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes

06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master - 0 1575439453000 3 connected 10923-16383

d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575439454000 1 connected 0-5460

532ef94c81188111827fef599ee73c0996a04e5e 192.13.168.77:7007@17007 slave 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 0 1575439455000 6 connected

46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575439455742 5 connected

ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575439454743 2 connected 5461-10922

b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575439454000 4 connected

通过redis cluster slots查看主从关系：

[root@192-13-168-77 ~]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster slots | xargs  -n8 | awk '{print $3":"$4"->"$6":"$7}' | sort -nk2 -t ':' | uniq

192.13.168.77:2002->192.13.168.77:5005

192.13.168.77:3003->192.13.168.77:6006

192.13.168.77:7007->192.13.168.77:4004

强行kill：

[root@192-13-168-77 ~]# kill -9 2418

[root@192-13-168-77 ~]# ps -ef | grep redis

root      2039     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:2002 [cluster]              

root      2236     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:3003 [cluster]              

root      2335     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:4004 [cluster]              

root      2365     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:5005 [cluster]              

root      2391     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:6006 [cluster]  

查看7007从4004的日志，4004升级为新master：

[root@192-13-168-77 log]# cat redis_4004.log 

23377:C 04 Dec 13:40:40.502 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

23377:C 04 Dec 13:40:40.502 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=23377, just started

23377:C 04 Dec 13:40:40.502 # Configuration loaded

23378:M 04 Dec 13:40:40.517 # Server initialized

23378:M 04 Dec 13:40:40.517 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

23378:M 04 Dec 13:40:40.517 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

23378:M 04 Dec 13:57:15.782 # User requested shutdown...

23378:M 04 Dec 13:57:15.783 # Redis is now ready to exit, bye bye...

2334:C 04 Dec 14:00:25.888 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

2334:C 04 Dec 14:00:25.888 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=2334, just started

2334:C 04 Dec 14:00:25.889 # Configuration loaded

2335:M 04 Dec 14:00:25.896 # Server initialized

2335:M 04 Dec 14:00:25.896 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

2335:M 04 Dec 14:00:25.896 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

2335:M 04 Dec 14:01:24.003 # configEpoch set to 3 via CLUSTER SET-CONFIG-EPOCH

2335:M 04 Dec 14:01:24.180 # IP address for this node updated to 192.13.168.77

2335:M 04 Dec 14:01:29.005 # Cluster state changed: ok

2335:M 04 Dec 14:06:56.069 # Manual failover requested by slave 532ef94c81188111827fef599ee73c0996a04e5e.

2335:M 04 Dec 14:06:56.258 # Failover auth granted to 532ef94c81188111827fef599ee73c0996a04e5e for epoch 7

2335:M 04 Dec 14:06:56.263 # Connection with slave 192.13.168.77:7007 lost.

2335:M 04 Dec 14:06:56.266 # Configuration change detected. Reconfiguring myself as a replica of 532ef94c81188111827fef599ee73c0996a04e5e

2335:S 04 Dec 14:06:56.538 # Master replication ID changed to 880dfbbc29acda08aa0f997ec0d3D9238f987cca

2335:S 04 Dec 14:46:13.970 # Connection with master lost.

2335:S 04 Dec 14:46:14.427 # Error condition on Socket for SYNC: Connection refused

2335:S 04 Dec 14:46:15.427 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:16.430 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:17.433 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:18.436 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:19.438 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:20.441 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:21.443 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:22.445 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:23.448 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:24.449 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:25.451 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:26.453 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:27.455 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:28.457 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:29.459 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:29.960 # Start of election delayed for 658 milliseconds (rank #0, offset 3738).

2335:S 04 Dec 14:46:30.461 # Error condition on socket for SYNC: Connection refused

2335:S 04 Dec 14:46:30.662 # Starting a failover election for epoch 8.

2335:S 04 Dec 14:46:30.667 # Failover election won: I'm the new master.

2335:S 04 Dec 14:46:30.667 # configEpoch set to 8 after successful failover

2335:M 04 Dec 14:46:30.667 # Setting secondary replication ID to 880dfbbc29acda08aa0f997ec0d3d9238f987cca, valid up to offset: 3739. New replication ID is 8ae5eb556fbfd6c13ec33b1123c87de1fbe4db05

到7007data目录下删除nodes-7007.conf：

[root@192-13-168-77 ~]# cd /u02/redis/7007/

[root@192-13-168-77 7007]# ls

conf  data  log  pid

[root@192-13-168-77 7007]# cd data/

[root@192-13-168-77 data]# ls

nodes-7007.conf  redis_7007_dump.rdb

[root@192-13-168-77 data]# rm -rf nodes-7007.conf 

[root@192-13-168-77 data]# ps -ef | grep redis

root      2039     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:2002 [cluster]              

root      2236     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:3003 [cluster]              

root      2335     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:4004 [cluster]              

root      2365     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:5005 [cluster]              

root      2391     1  0 14:00 ?        00:00:02 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:6006 [cluster]              

root     32397 30103  0 14:49 pts/1    00:00:00 grep redis

root     37840     1  0 Nov05 ?        00:25:35 /usr/sbin/glusterfs --volfile-server=10.66.5.10 --volfile-id=/volume-10-0-5-16-db01 /sharedisk/

启动7007进程：

[root@192-13-168-77 data]#  /usr/local/redis-4.0.9/bin/redis-server /u02/redis/7007/conf/redis_7007.conf 

因强制kill无法自动加入集群：

[root@192-13-168-77 data]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes

06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master - 0 1575442290000 8 connected 10923-16383

d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575442289000 1 connected 0-5460

532ef94c81188111827fef599ee73c0996a04e5e :0@0 master,fail,noaddr - 1575441974024 1575441972822 7 disconnected

46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575442290483 5 connected

ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575442291483 2 connected 5461-10922

b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575442289481 4 connected

在每个节点执行：

CLUSTER FORGET 06031e33797ef0aa6427bddb1ff958f7af0f1a4a

[root@192-13-168-77 data]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB"

192.13.168.77:2002> CLUSTER FORGET 532ef94c81188111827fef599ee73c0996a04e5e

OK

192.13.168.77:3003> CLUSTER FORGET 532ef94c81188111827fef599ee73c0996a04e5e

OK

192.13.168.77:5005> CLUSTER FORGET 532ef94c81188111827fef599ee73c0996a04e5e

OK

192.13.168.77:6006> CLUSTER FORGET 532ef94c81188111827fef599ee73c0996a04e5e

OK

重新启动7007 redis进程：

[root@192-13-168-77 data]# /usr/local/redis-4.0.9/bin/redis-server /u02/redis/7007/conf/redis_7007.conf 

[root@192-13-168-77 data]# ps -ef | grep redis

root      2039     1  0 14:00 ?        00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:2002 [cluster]              

root      2236     1  0 14:00 ?        00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:3003 [cluster]              

root      2335     1  0 14:00 ?        00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:4004 [cluster]              

root      2365     1  0 14:00 ?        00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:5005 [cluster]              

root      2391     1  0 14:00 ?        00:00:03 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:6006 [cluster]              

root      2931     1  0 15:07 ?        00:00:00 /usr/local/redis-4.0.9/bin/redis-server 0.0.0.0:7007 [cluster] 

将7007重新加入为4004的slave

[root@192-13-168-77 data]# redis-trib.rb add-node --slave --master-id 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:7007 192.13.168.77:2002

>>> Adding node 192.13.168.77:7007 to cluster 192.13.168.77:2002

>>> PerfORMing Cluster Check (using node 192.13.168.77:2002)

M: d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002

   slots:0-5460 (5461 slots) master

   1 additional replica(s)

M: 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004

   slots:10923-16383 (5461 slots) master

   0 additional replica(s)

S: 46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006

   slots: (0 slots) slave

   replicates ef76f232efb578249e8d0ec8fef8ec02b3524010

M: ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003

   slots:5461-10922 (5462 slots) master

   1 additional replica(s)

S: b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005

   slots: (0 slots) slave

   replicates d49ebf2a5f3605487ea4c8deee7e2aa2782667e6

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.

>>> Send CLUSTER MEET to node 192.13.168.77:7007 to make it join the cluster.

Waiting for the cluster to join.

>>> Configure node as replica of 192.13.168.77:4004.

[OK] New node added correctly.

检查redis cluster状态：

[root@192-13-168-77 data]# redis-trib.rb check 192.13.168.77:2002

>>> Performing Cluster Check (using node 192.13.168.77:2002)

M: d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002

   slots:0-5460 (5461 slots) master

   1 additional replica(s)

M: 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004

   slots:10923-16383 (5461 slots) master

   1 additional replica(s)

S: 46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006

   slots: (0 slots) slave

   replicates ef76f232efb578249e8d0ec8fef8ec02b3524010

M: ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003

   slots:5461-10922 (5462 slots) master

   1 additional replica(s)

S: 1dba8274e36bde79a215a77d1f241ae6fc81c03e 192.13.168.77:7007

   slots: (0 slots) slave

   replicates 06031e33797ef0aa6427bddb1ff958f7af0f1a4a

S: b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005

   slots: (0 slots) slave

   replicates d49ebf2a5f3605487ea4c8deee7e2aa2782667e6

[OK] All nodes agree about slots configuration.

>>> Check for open slots...

>>> Check slots coverage...

[OK] All 16384 slots covered.

重新查看集群状态：

[root@192-13-168-77 data]# redis-cli -h 192.13.168.77 -p 3003 -c -a "ysBhqkYHDifB" cluster nodes

b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575452066959 4 connected

1dba8274e36bde79a215a77d1f241ae6fc81c03e 192.13.168.77:7007@17007 slave 06031e33797ef0aa6427bddb1ff958f7af0f1a4a 0 1575452066000 8 connected

46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575452065958 5 connected

06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master - 0 1575452067960 8 connected 10923-16383

ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 myself,master - 0 1575452065000 2 connected 5461-10922

d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 master - 0 1575452067000 1 connected 0-5460

正常shtudown某一个节点后重新加入集群：

[root@192-13-168-77 data]# redis-cli -h 192.13.168.77 -p 4004 -c -a "ysBhqkYHDifB" shutdown

[root@192-13-168-77 log]# cat redis_7007.log

4540:S 04 Dec 18:03:32.106 # Connection with master lost.

4540:S 04 Dec 18:03:32.921 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:33.924 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:34.925 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:35.928 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:36.930 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:37.932 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:38.934 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:39.935 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:40.936 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:41.938 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:42.940 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:43.942 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:44.943 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:45.945 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:46.948 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:47.550 # Start of election delayed for 554 milliseconds (rank #0, offset 17724).

4540:S 04 Dec 18:03:47.950 # Error condition on socket for SYNC: Connection refused

4540:S 04 Dec 18:03:48.150 # Starting a failover election for epoch 9.

4540:S 04 Dec 18:03:48.155 # Failover election won: I'm the new master.

4540:S 04 Dec 18:03:48.155 # configEpoch set to 9 after successful failover

4540:M 04 Dec 18:03:48.155 # Setting secondary replication ID to 8ae5eb556fbfd6c13ec33b1123c87de1fbe4db05, valid up to offset: 17725. New replication ID is e44a29dcbf3dce8e1d29c06bd310ba2ba3d0c41b

查看集群状态：

redis-cli -h 192.13.168.77 -p 4004 -c -a "ysBhqkYHDifB" cluster nodes

Could not connect to Redis at 192.13.168.77:4004: Connection refused

[root@192-13-168-77 log]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes  

06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 master,fail - 1575453812190 1575453808482 8 disconnected

d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575453939000 1 connected 0-5460

46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575453940773 5 connected

ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575453939000 2 connected 5461-10922

1dba8274e36bde79a215a77d1f241ae6fc81c03e 192.13.168.77:7007@17007 master - 0 1575453939770 9 connected 10923-16383

b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575453940000 4 connected

重新启动4004进程：

[root@192-13-168-77 log]# /usr/local/redis-4.0.9/bin/redis-server /u02/redis/4004/conf/redis_4004.conf 

查看集群状态：（4004自动添加进来并变成slave）

[root@192.13.168.77 log]# redis-cli -h 192.13.168.77 -p 2002 -c -a "ysBhqkYHDifB" cluster nodes         

06031e33797ef0aa6427bddb1ff958f7af0f1a4a 192.13.168.77:4004@14004 slave 1dba8274e36bde79a215a77d1f241ae6fc81c03e 0 1575454019000 9 connected

d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 192.13.168.77:2002@12002 myself,master - 0 1575454018000 1 connected 0-5460

46233f6d8b508be0cedafc5f07aca04210f654ea 192.13.168.77:6006@16006 slave ef76f232efb578249e8d0ec8fef8ec02b3524010 0 1575454019942 5 connected

ef76f232efb578249e8d0ec8fef8ec02b3524010 192.13.168.77:3003@13003 master - 0 1575454020944 2 connected 5461-10922

1dba8274e36bde79a215a77d1f241ae6fc81c03e 192.13.168.77:7007@17007 master - 0 1575454018000 9 connected 10923-16383

b11fc826c15cdee6e026a59ed98f31c9fa490aaa 192.13.168.77:5005@15005 slave d49ebf2a5f3605487ea4c8deee7e2aa2782667e6 0 1575454020000 4 connected