Redis Cluster | 8lovelife's life

8lovelife's life

M&M

0%

Redis Cluster

发表于 更新于 分类于
本文字数： 4.1k 阅读时长 ≈ 15 分钟

Redis Cluster 是 Redis 的分布式解决方案，它使用哈希槽分片技术将数据分散到不同的 Redis 节点中，实现数据的水平扩展，从而提高系统性能与吞吐量。Redis Cluster 支持主从复制和故障转移，当主节点发生故障时，从节点可以继续为系统提供服务，保证系统的高可用性。Redis Cluster 支持数据自动迁移，可以在节点的新增和删除时，进行数据的自动迁移，并且操作十分简单。在本篇文章中，我们将搭建一个 3主3从 的 Redis Cluster，并演示集群中的基本操作

集群搭建

我们使用 Docker 来搭建 Redis 集群

创建 Redis 实例

创建6个 Redis 实例，并且要求容器之间可以通过服务名称进行网络通信

1
2
3
4
5
6
7
8
9
10
11
12
13
14

docker network create -d bridge redis-cluster-net

docker run -itd --name redis-node-1 --net=redis-cluster-net -v ~/data/redis/cluster/redis-node-1:/data redis --cluster-enabled yes --appendonly yes

docker run -itd --name redis-node-2 --net=redis-cluster-net -v ~/data/redis/cluster/redis-node-2:/data redis --cluster-enabled yes --appendonly yes

docker run -itd --name redis-node-3 --net=redis-cluster-net -v ~/data/redis/cluster/redis-node-3:/data redis --cluster-enabled yes --appendonly yes

docker run -itd --name redis-node-4 --net=redis-cluster-net -v ~/data/redis/cluster/redis-node-4:/data redis --cluster-enabled yes --appendonly yes

docker run -itd --name redis-node-5 --net=redis-cluster-net -v ~/data/redis/cluster/redis-node-5:/data redis --cluster-enabled yes --appendonly yes

docker run -itd --name redis-node-6 --net=redis-cluster-net -v ~/data/redis/cluster/redis-node-6:/data redis --cluster-enabled yes --appendonly yes

构建集群节点

开始构建集群，这里我们使用 redis-node-1 作为操作入口

  1. 进入 redis-node-1 容器内
1
docker exec -it redis-node-1 /bin/sh
  1. 创建集群节点
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
redis-cli --cluster create redis-node-1:6379 redis-node-2:6379 redis-node-3:6379 redis-node-4:6379 redis-node-5:6379 redis-node-6:6379 --cluster-replicas 1

>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica redis-node-5:6379 to redis-node-1:6379
Adding replica redis-node-6:6379 to redis-node-2:6379
Adding replica redis-node-4:6379 to redis-node-3:6379
M: 6d088a123c9054e680bb679a4a90dd096c5d6f13 redis-node-1:6379
   slots:[0-5460] (5461 slots) master
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 redis-node-2:6379
   slots:[5461-10922] (5462 slots) master
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 redis-node-3:6379
   slots:[10923-16383] (5461 slots) master
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d redis-node-4:6379
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
S: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   replicates 6d088a123c9054e680bb679a4a90dd096c5d6f13
S: 4e95919b1382d9576e11c93be686240d5d639e32 redis-node-6:6379
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3

   Can I set the above configuration? (type 'yes' to accept): yes # 这里确认配置
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node redis-node-1:6379)
M: 6d088a123c9054e680bb679a4a90dd096c5d6f13 redis-node-1:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
S: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea 172.18.0.5:6379
   slots: (0 slots) slave
   replicates 6d088a123c9054e680bb679a4a90dd096c5d6f13
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
  1. 查看集群状况
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
redis-cli -c #集群客服端进入 

127.0.0.1:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:479
cluster_stats_messages_pong_sent:524
cluster_stats_messages_sent:1003
cluster_stats_messages_ping_received:519
cluster_stats_messages_pong_received:479
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:1003
total_cluster_links_buffer_limit_exceeded:0


127.0.0.1:6379> cluster nodes
4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379@16379 slave dc2b88c5dd76c479639a428dbb2a57a34be919e3 0 1680019648000 2 connected
6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379@16379 slave 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 0 1680019647583 3 connected
6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379@16379 myself,master - 0 1680019649000 1 connected 0-5460
42afc0262d0ebdce755aa1a61e12d70b80ffa9ea 172.18.0.5:6379@16379 slave 6d088a123c9054e680bb679a4a90dd096c5d6f13 0 1680019648626 1 connected
dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379@16379 master - 0 1680019649670 2 connected 5461-10922
7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379@16379 master - 0 1680019648000 3 connected 10923-16383

通过集群信息，我们可以得出当前集群节点的关系分布图

imag

  1. 指令操作测试
1
2
3
4
5
6
7
8
9
10
11
12
13
redis-cli -c #集群客服端进入 

127.0.0.1:6379> set k1 v1
-> Redirected to slot [12706] located at 172.18.0.3:6379
OK
172.18.0.3:6379> 

172.18.0.3:6379> set k2 v2
-> Redirected to slot [449] located at 172.18.0.7:6379
OK
172.18.0.7:6379> set k3 v3
OK
172.18.0.7:6379>
  1. 检查 KEY 情况
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
redis-cli --cluster check redis-node-1:6379 # 集群检查

redis-node-1:6379 (6d088a12...) -> 2 keys | 5461 slots | 1 slaves.
172.18.0.2:6379 (dc2b88c5...) -> 0 keys | 5462 slots | 1 slaves.
172.18.0.3:6379 (7dceb962...) -> 1 keys | 5461 slots | 1 slaves.
[OK] 3 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node redis-node-1:6379)
M: 6d088a123c9054e680bb679a4a90dd096c5d6f13 redis-node-1:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
S: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea 172.18.0.5:6379
   slots: (0 slots) slave
   replicates 6d088a123c9054e680bb679a4a90dd096c5d6f13
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

故障转移

前面我们搭建了 3主3从 的 Redis 集群，我们来测试一下当集群中某一个 master 节点宕机后，集群中会有什么变化

  1. 停止 master 节点 redis-node-1
1
2
3
4
5
6
7
8
9
10
11
docker stop redis-node-1 # 停止 redis-node-1

docker exec -it redis-node-2 /bin/sh # 通过 redis-node-2 与集群交互

127.0.0.1:6379> cluster nodes
7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379@16379 master - 0 1680021042893 3 connected 10923-16383
dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379@16379 myself,master - 0 1680021038000 2 connected 5461-10922
42afc0262d0ebdce755aa1a61e12d70b80ffa9ea 172.18.0.5:6379@16379 master - 0 1680021042000 7 connected 0-5460
4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379@16379 slave dc2b88c5dd76c479639a428dbb2a57a34be919e3 0 1680021040000 2 connected
6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379@16379 master,fail - 1680020721097 1680020718000 1 connected
6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379@16379 slave 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 0 1680021041000 3 connected

可以发现当我们停止 redis-node-1 后，从节点 redis-node-5 会升级为 master 节点

imag

  1. 检查 KEY 情况
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
redis-cli --cluster check redis-node-2:6379

Could not connect to Redis at 172.18.0.7:6379: No route to host
redis-node-2:6379 (dc2b88c5...) -> 0 keys | 5462 slots | 1 slaves.
172.18.0.3:6379 (7dceb962...) -> 1 keys | 5461 slots | 1 slaves.
172.18.0.5:6379 (42afc026...) -> 2 keys | 5461 slots | 0 slaves.    # KEY 数量正确
[OK] 3 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node redis-node-2:6379)
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 redis-node-2:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea 172.18.0.5:6379
   slots:[0-5460] (5461 slots) master
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

当我们再次启动 redis-node-1 后，redis-node-1 会变为 redis-node-5 的从节点

imag

  • 如果集群中的主从节点同时宕机，那么在恢复的时候需要先启动 master，然后再启动 slave

扩容

假设我们的服务遇到了流量高峰，现在需要将 Redis Cluster 扩容至 4主4从，Redis Cluster 的扩容非常简单，我们来试试看该如何扩容

  1. 新建2个 Redis 实例
1
2
3
docker run -itd --name redis-node-7 --net=redis-cluster-net -v ~/data/redis/cluster/redis-node-7:/data redis --cluster-enabled yes --appendonly yes

docker run -itd --name redis-node-8 --net=redis-cluster-net -v ~/data/redis/cluster/redis-node-8:/data redis --cluster-enabled yes --appendonly yes
  1. 新 redis-node-7 以 master 节点加入集群
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
docker exec -it redis-node-7 /bin/sh

redis-cli --cluster add-node redis-node-7:6379 redis-node-5:6379 # 通过 redis-node-5 与集群交互

>>> Adding node redis-node-7:6379 to cluster redis-node-5:6379
>>> Performing Cluster Check (using node redis-node-5:6379)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Getting functions from cluster
>>> Send FUNCTION LIST to redis-node-7:6379 to verify there is no functions in it
>>> Send FUNCTION RESTORE to redis-node-7:6379
>>> Send CLUSTER MEET to node redis-node-7:6379 to make it join the cluster.
[OK] New node added correctly.
  1. 集群检查
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
redis-cli --cluster check redis-node-5:6379 # 检查集群情况

redis-node-5:6379 (42afc026...) -> 2 keys | 5461 slots | 1 slaves.
172.18.0.3:6379 (7dceb962...) -> 1 keys | 5461 slots | 1 slaves.
172.18.0.8:6379 (a26bb6fb...) -> 0 keys | 0 slots | 0 slaves.     # 新加入集群的 master 节点，此时为空槽
172.18.0.2:6379 (dc2b88c5...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 3 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node redis-node-5:6379)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: a26bb6fbc64644a4807c13d692247510e10815ff 172.18.0.8:6379
   slots: (0 slots) master
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
  1. 为新加入的 master 节点分配槽位
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
redis-cli --cluster reshard redis-node-5:6379 # 通过 redis-node-5 与集群交互

>>> Performing Cluster Check (using node redis-node-5:6379)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: a26bb6fbc64644a4807c13d692247510e10815ff 172.18.0.8:6379 # 新加入的 master 节点
   slots: (0 slots) master
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 

How many slots do you want to move (from 1 to 16384)? 4096 # 这里我们选择平均分配槽号（16384/4 = 4096）
What is the receiving node ID? a26bb6fbc64644a4807c13d692247510e10815ff # 新加入的 master 节点 ID
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: all
  1. 查看新 master 节点集群状况
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
redis-cli --cluster check redis-node-5:6379 # 通过 redis-node-5 与集群交互

redis-node-5:6379 (42afc026...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.3:6379 (7dceb962...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.8:6379 (a26bb6fb...) -> 1 keys | 4096 slots | 0 slaves.  # 可以看到 redis-node-5 其中的一个 KEY 分配给了新加入的 master 节点
172.18.0.2:6379 (dc2b88c5...) -> 0 keys | 4096 slots | 1 slaves.
[OK] 3 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node redis-node-5:6379)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
M: a26bb6fbc64644a4807c13d692247510e10815ff 172.18.0.8:6379
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master   # 新加入的 master 节点，槽位分配完成
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

新 master 节点 redis-node-8 加入集群后的槽位分布

imag

  1. 从节点 redis-node-8 加入集群
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
redis-cli --cluster add-node redis-node-8:6379 redis-node-7:6379 --cluster-slave --cluster-master-id a26bb6fbc64644a4807c13d692247510e10815ff

>>> Adding node redis-node-8:6379 to cluster redis-node-7:6379
>>> Performing Cluster Check (using node redis-node-7:6379)
M: a26bb6fbc64644a4807c13d692247510e10815ff redis-node-7:6379
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea 172.18.0.5:6379
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node redis-node-8:6379 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of redis-node-7:6379.
[OK] New node added correctly.
  1. 查看新 slave 节点集群状况
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
redis-cli --cluster check redis-node-5:6379

redis-node-5:6379 (42afc026...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.3:6379 (7dceb962...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.8:6379 (a26bb6fb...) -> 1 keys | 4096 slots | 1 slaves.  # redis-node-7 的从节点加入成功
172.18.0.2:6379 (dc2b88c5...) -> 0 keys | 4096 slots | 1 slaves.
[OK] 3 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node redis-node-5:6379)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
S: bd3887b976a2d52d3dc70a1ad4d5f6045f08a021 172.18.0.9:6379 # 新加入的从节点
   slots: (0 slots) slave
   replicates a26bb6fbc64644a4807c13d692247510e10815ff
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
M: a26bb6fbc64644a4807c13d692247510e10815ff 172.18.0.8:6379
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
   1 additional replica(s)
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

扩容后的集群节点分布情况

imag

缩容

假设我们的服务过了流量高峰期，现在需要将 Redis Cluster 缩容至 3主3从，Redis Cluster 的缩容也非常简单，我们来试试看该如何缩容

  1. 先缩容 slave 节点 redis-node-8
1
2
3
4
5
redis-cli --cluster del-node redis-node-8:6379 bd3887b976a2d52d3dc70a1ad4d5f6045f08a021

>>> Removing node bd3887b976a2d52d3dc70a1ad4d5f6045f08a021 from cluster redis-node-8:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
  1. 集群检查
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
redis-cli --cluster check redis-node-5:6379

redis-node-5:6379 (42afc026...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.3:6379 (7dceb962...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.8:6379 (a26bb6fb...) -> 1 keys | 4096 slots | 0 slaves.  # 从节点 redis-node-8 不见了
172.18.0.2:6379 (dc2b88c5...) -> 0 keys | 4096 slots | 1 slaves.
[OK] 3 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node redis-node-5:6379)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
M: a26bb6fbc64644a4807c13d692247510e10815ff 172.18.0.8:6379
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

imag

  1. 将 master 节点 redis-node-7 槽清空
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
redis-cli --cluster reshard redis-node-5:6379

>>> Performing Cluster Check (using node redis-node-5:6379)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
M: a26bb6fbc64644a4807c13d692247510e10815ff 172.18.0.8:6379
   slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[6827-10922] (4096 slots) master
   1 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 

How many slots do you want to move (from 1 to 16384)? 4096  # 清空 redis-node-7 中的 4096 个槽位
What is the receiving node ID? dc2b88c5dd76c479639a428dbb2a57a34be919e3 # 指定哪个节点接收 redis-node-7 中槽位，这里选择 redis-node-2
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1: a26bb6fbc64644a4807c13d692247510e10815ff # redis-node-7 的集群 ID
Source node #2: done
  1. 集群中 KEY 情况检查
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
redis-cli --cluster check redis-node-5:6379

redis-node-5:6379 (42afc026...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.3:6379 (7dceb962...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.2:6379 (dc2b88c5...) -> 1 keys | 8192 slots | 2 slaves.  # redis-node-7 中的 KEY 转移到 redis-node-2 中了
[OK] 3 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node redis-node-5:6379)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
S: a26bb6fbc64644a4807c13d692247510e10815ff 172.18.0.8:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[0-1364],[5461-12287] (8192 slots) master
   2 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

imag

  1. 删除 master 节点 redis-node-7
1
2
3
4
5
redis-cli --cluster del-node redis-node-7:6379 a26bb6fbc64644a4807c13d692247510e10815ff

>>> Removing node a26bb6fbc64644a4807c13d692247510e10815ff from cluster redis-node-7:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
  1. 集群检查
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
redis-cli --cluster check redis-node-5:6379

redis-node-5:6379 (42afc026...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.3:6379 (7dceb962...) -> 1 keys | 4096 slots | 1 slaves.
172.18.0.2:6379 (dc2b88c5...) -> 1 keys | 8192 slots | 1 slaves.
[OK] 3 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node redis-node-5:6379)
M: 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea redis-node-5:6379
   slots:[1365-5460] (4096 slots) master
   1 additional replica(s)
S: 4e95919b1382d9576e11c93be686240d5d639e32 172.18.0.6:6379
   slots: (0 slots) slave
   replicates dc2b88c5dd76c479639a428dbb2a57a34be919e3
M: 7dceb96259a1e2eeece5a15e33250ffcde71aaf0 172.18.0.3:6379
   slots:[12288-16383] (4096 slots) master
   1 additional replica(s)
S: 6d088a123c9054e680bb679a4a90dd096c5d6f13 172.18.0.7:6379
   slots: (0 slots) slave
   replicates 42afc0262d0ebdce755aa1a61e12d70b80ffa9ea
M: dc2b88c5dd76c479639a428dbb2a57a34be919e3 172.18.0.2:6379
   slots:[0-1364],[5461-12287] (8192 slots) master
   1 additional replica(s)
S: 6aaae0c14c2a7bf2517659ff6a2530400b40968d 172.18.0.4:6379
   slots: (0 slots) slave
   replicates 7dceb96259a1e2eeece5a15e33250ffcde71aaf0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

缩容后的最新槽位分布情况

imag

MULTIPLE KEYS

一个命令同时操作多个 KEY 的情况

1
2
3
4
5
6
7
8
9
10
11
12
172.18.0.5:6379> mget k1 k2
(error) CROSSSLOT Keys in request don't hash to the same slot

# 执行错误，因为 k1 与 k2 分布在不同的槽位上

172.18.0.5:6379> get k1
-> Redirected to slot [12706] located at 172.18.0.3:6379
"v1"

172.18.0.3:6379> get k2
-> Redirected to slot [449] located at 172.18.0.2:6379
"v2"

我们可以通过 hashtag 将一组相关联的数据存储到同一个槽位中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
172.18.0.2:6379> del k1
-> Redirected to slot [12706] located at 172.18.0.3:6379
(integer) 1
172.18.0.3:6379> del k2
-> Redirected to slot [449] located at 172.18.0.2:6379
(integer) 1

172.18.0.2:6379> set {user.100}.k1 v1
-> Redirected to slot [13578] located at 172.18.0.3:6379
OK
172.18.0.3:6379> set {user.100}.k2 v2
OK

172.18.0.3:6379> mget {user.100}.k1 {user.100}.k2
1) "v1"
2) "v2"

哈希槽

Redis Cluster 通过对 KEY 的哈希计算（ CRC16(k1) mod 16384 ）得出槽的范围，从而找到真实的 Redis 节点，实现数据的水平扩展和负载均衡。哈希槽的引入使得 Redis Cluster 能够很灵活的解决数据倾斜问题，通过哈希槽的再分配能够快速的调整集群中数据的负载，并同时完成数据的自动迁移