Redis三主三从集群搭建

ZFIREH 2020年11月10日 17次浏览

Redis简介

Redis 是完全开源的,遵守 BSD 协议,是一个高性能的 key-value 数据库。

Redis 与其他 key - value 缓存产品有以下三个特点:

  • Redis支持数据的持久化,可以将内存中的数据保存在磁盘中,重启的时候可以再次加载进行使用。
  • Redis不仅仅支持简单的key-value类型的数据,同时还提供list,set,zset,hash等数据结构的存储。
  • Redis支持数据的备份,即master-slave模式的数据备份。

Redis3.0开始支持集群,在每个Master上存放的数据可各不相同,即分布式存储的思想。集群中的每个节点都需要知道集群中自己之外的其它节点,这些需要在每个Redis节点的nodes-xxxx.conf中体现,该文件是集群建立后自动设置的,不需要手动修改。

环境信息

系统版本及内核版本

[zhouhuo@zhouhuo1 ~]$ cat /etc/redhat-release 
CentOS Linux release 7.8.2003 (Core)
[zhouhuo@zhouhuo1 ~]$ uname -r
3.10.0-1127.19.1.el7.x86_64

gcc版本

[zhouhuo@zhouhuo1 ~]$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu --enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC)

三主三从架构说明

三台服务器,启动6个实例,形成三主三从,其中存储相同数据的主从节点不能落在同一台机器上,目的是防止部署redis的虚拟机宕机从而造成主从节点全部失效。本文机器的地址与端口如下:

HostIPPort集群规划
zhouhuo1192.168.8.67000master
zhouhuo1192.168.8.67001slave
zhouhuo2192.168.8.77000master
zhouhuo2192.168.8.77001slave
zhouhuo3192.168.8.87000master
zhouhuo3192.168.8.87001slave

Redis 版本

5.0.0

下载Redis安装包

[zhouhuo@zhouhuo1 ~]$ wget https://download.redis.io/releases/redis-5.0.0.tar.gz

安装

创建相关目录

依次在三台服务器上创建相关目录

# 存放redis脚本
[zhouhuo@zhouhuo1 ~]$ mkdir -p redis/bin
[zhouhuo@zhouhuo1 ~]$ mkdir -p redis/cluster/7000
[zhouhuo@zhouhuo1 ~]$ mkdir -p redis/cluster/7001

解压安装包

依次在三台服务器上解压安装包

[zhouhuo@zhouhuo1 ~]$ tar -zxvf redis-5.0.0.tar.gz 

编译安装包并安装到指定目录

依次在三台服务器上编译安装

[zhouhuo@zhouhuo1 ~]$ cd redis-5.0.0
[zhouhuo@zhouhuo1 redis-5.0.0]$ make PREFIX=/home/zhouhuo/redis install

安装结果如下:

CC rax.o
CC t_stream.o
CC listpack.o
CC localtime.o
CC lolwut.o
CC lolwut5.o
LINK redis-server
INSTALL redis-sentinel
CC redis-cli.o
LINK redis-cli
CC redis-benchmark.o
LINK redis-benchmark
INSTALL redis-check-rdb
INSTALL redis-check-aof

Hint: It's a good idea to run 'make test' ;)

INSTALL install
INSTALL install
INSTALL install
INSTALL install
INSTALL install

make[1]: Leaving directory `/home/zhouhuo/redis-5.0.0/src'

复制配置文件

依次在三台服务器上复制配置文件

[zhouhuo@zhouhuo1 redis-5.0.0]$ cp redis.conf /home/zhouhuo/redis/cluster/7000
[zhouhuo@zhouhuo1 redis-5.0.0]$ cp redis.conf /home/zhouhuo/redis/cluster/7001

修改配置文件

依次修改三台服务器上的6个节点的配置文件(redis.conf),修改项如下:

# 必须配置,绑定当前机器IP
bind 192.168.8.6
# 必须配置,配置redis作为守护进程运行
daemonize yes
# 必须配置,开启集群模式
cluster-enabled yes
# 节点端口,依次修改
port 7000
# 配置数据持久化
appendonly yes
# pid文件路径,依次修改
pidfile /home/zhouhuo/redis/cluster/7000/redis_7000.pid
#cluster配置文件(启动自动生成)  
cluster-config-file nodes-7000.conf
# 指定数据文件存放的位置,必须指定为不同目录,不然会丢失数据,深坑误入
dir /home/zhouhuo/redis/cluster/7000

检查配置

检查每个配置文件(redis.conf)

检查每个配置文件(redis.conf)

检查每个配置文件(redis.conf)

重要的事说三遍!!!

在bin目录下编写启停脚本

在一台服务器上编写好了后,依次复制到另外两台服务器

# 启动脚本
[zhouhuo@zhouhuo1 bin]$ vi start-redis-cluster.sh
#!/bin/bash

/home/zhouhuo/redis/bin/redis-server /home/zhouhuo/redis/cluster/7000/redis.conf
/home/zhouhuo/redis/bin/redis-server /home/zhouhuo/redis/cluster/7001/redis.conf

# 停止脚本
[zhouhuo@zhouhuo1 bin]$ vi stop-redis-cluster.sh
#!/bin/bash

ps -ef |grep redis |grep -v grep |awk '{print $2}' |xargs kill

# 赋予执行权限
[zhouhuo@zhouhuo1 bin]$ chmod +x *-redis-cluster.sh

防火墙策略

每台服务器均需要关闭防火墙或者开放指定指定端口(7000/7001/17000/17001)

每台服务器均需要关闭防火墙或者开放指定指定端口(7000/7001/17000/17001)

每台服务器均需要关闭防火墙或者开放指定指定端口(7000/7001/17000/17001)

重要的事说三遍!!!

关闭防火墙

# 查看防火墙状态
[root@zhouhuo1 ~]# $ systemctl disabled firewalld
# 关闭防火墙
[root@zhouhuo1 ~]# $ systemctl stop firewalld
# 关闭开机自启
[root@zhouhuo1 ~]# $ systemctl disabled firewalld

开放端口

# 开放7000-7001端口
[root@zhouhuo1 ~]# firewall-cmd --zone=public --add-port=7000-7001/tcp --permanent
# 开放17000-17001端口
[root@zhouhuo1 ~]# firewall-cmd --zone=public --add-port=17000-17001/tcp --permanent
# 重载防火墙策略
[root@zhouhuo1 ~]# firewall-cmd --reload

启动redis

在三台服务器上依次启动redis

[zhouhuo@zhouhuo1 redis]$ bin/start-redis-cluster.sh 

创建集群

在其中一台服务器上创建redis集群,为了使主从节点不落在同一台机器上,使用IP+Port交叉的方式(暂时没有找到命令指定主从节点的关系的方法)

5.0.0以下版本

[zhouhuo@zhouhuo1 redis]$ bin/redis-trib.rb create --replicas 1 192.168.8.6:7000 192.168.8.7:7000 192.168.8.8:7000 192.168.8.6:7001 192.168.8.7:7001 192.168.8.8:7001

5.0.0及以上版本

[zhouhuo@zhouhuo1 redis]$ bin/redis-cli --cluster create 192.168.8.6:7000 192.168.8.7:7000 192.168.8.8:7000 192.168.8.6:7001 192.168.8.7:7001 192.168.8.8:7001 --cluster-replicas 1

创建集群过程如下

[zhouhuo@zhouhuo1 redis]$ bin/redis-cli --cluster create 192.168.8.6:7000 192.168.8.7:7000 192.168.8.8:7000 192.168.8.6:7001 192.168.8.7:7001 192.168.8.8:7001 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.8.7:7001 to 192.168.8.6:7000
Adding replica 192.168.8.6:7001 to 192.168.8.7:7000
Adding replica 192.168.8.8:7001 to 192.168.8.8:7000
>>> Trying to optimize slaves allocation for anti-affinity
[OK] Perfect anti-affinity obtained!
M: 31ee6430c4d9f6c6fccd388336bef9434bb539a8 192.168.8.6:7000
   slots:[0-5460] (5461 slots) master
M: 075323c7e5c260a8150c2666c22f0bc0fcad7489 192.168.8.7:7000
   slots:[5461-10922] (5462 slots) master
M: 6ed0866aa43c3e85b6da73ccb8d2774d80118c51 192.168.8.8:7000
   slots:[10923-16383] (5461 slots) master
S: e04854306f7aea1c92baa4e656b4bcbeb5f9d576 192.168.8.6:7001
   replicates 6ed0866aa43c3e85b6da73ccb8d2774d80118c51
S: 0fd4238d42bc3adfa1538e69c69b0d901374a322 192.168.8.7:7001
   replicates 31ee6430c4d9f6c6fccd388336bef9434bb539a8
S: d3838fac3780ca761dc58f38b3bf3624e5137a18 192.168.8.8:7001
   replicates 075323c7e5c260a8150c2666c22f0bc0fcad7489
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...
>>> Performing Cluster Check (using node 192.168.8.6:7000)
M: 31ee6430c4d9f6c6fccd388336bef9434bb539a8 192.168.8.6:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: e04854306f7aea1c92baa4e656b4bcbeb5f9d576 192.168.8.6:7001
   slots: (0 slots) slave
   replicates 6ed0866aa43c3e85b6da73ccb8d2774d80118c51
S: 0fd4238d42bc3adfa1538e69c69b0d901374a322 192.168.8.7:7001
   slots: (0 slots) slave
   replicates 31ee6430c4d9f6c6fccd388336bef9434bb539a8
S: d3838fac3780ca761dc58f38b3bf3624e5137a18 192.168.8.8:7001
   slots: (0 slots) slave
   replicates 075323c7e5c260a8150c2666c22f0bc0fcad7489
M: 6ed0866aa43c3e85b6da73ccb8d2774d80118c51 192.168.8.8:7000
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
M: 075323c7e5c260a8150c2666c22f0bc0fcad7489 192.168.8.7:7000
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

查看集群状态

[zhouhuo@zhouhuo1 redis]$ bin/redis-cli -c -h 192.168.8.6 -p 7000
192.168.8.6:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:40
cluster_stats_messages_pong_sent:45
cluster_stats_messages_sent:85
cluster_stats_messages_ping_received:40
cluster_stats_messages_pong_received:40
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:85
192.168.8.6:7000>

测试set/get键值对

192.168.8.6:7000> set zhouhuo hello
-> Redirected to slot [16207] located at 192.168.8.8:7000
OK
192.168.8.8:7000> get zhouhuo
"hello"
192.168.8.8:7000> 

故障转移测试

先断掉集群中的一台master节点,可以看到集群依旧是正常,且会自动将这故障master对应的slave节点顶替上,变为master

[zhouhuo@zhouhuo1 redis]$ bin/redis-cli -c -h 192.168.8.6 -p 7000
192.168.8.6:7000> cluster nodes
e04854306f7aea1c92baa4e656b4bcbeb5f9d576 192.168.8.6:7001@17001 slave 6ed0866aa43c3e85b6da73ccb8d2774d80118c51 0 1604939536000 4 connected
0fd4238d42bc3adfa1538e69c69b0d901374a322 192.168.8.7:7001@17001 slave 31ee6430c4d9f6c6fccd388336bef9434bb539a8 0 1604939538000 5 connected
d3838fac3780ca761dc58f38b3bf3624e5137a18 192.168.8.8:7001@17001 slave 075323c7e5c260a8150c2666c22f0bc0fcad7489 0 1604939538774 6 connected
6ed0866aa43c3e85b6da73ccb8d2774d80118c51 192.168.8.8:7000@17000 master - 0 1604939537758 3 connected 10923-16383
31ee6430c4d9f6c6fccd388336bef9434bb539a8 192.168.8.6:7000@17000 myself,master - 0 1604939537000 1 connected 0-5460
075323c7e5c260a8150c2666c22f0bc0fcad7489 192.168.8.7:7000@17000 master - 0 1604939539793 2 connected 5461-10922
192.168.8.6:7000> get zhouhuo
-> Redirected to slot [16207] located at 192.168.8.8:7000
"hello"
192.168.8.8:7000> 

# 此时我们kill掉192.168.8.8上的7000端口的redis节点
[zhouhuo@zhouhuo3 redis]$ ps -ef | grep redis
zhouhuo    9659      1  0 00:35 ?        00:00:00 /home/zhouhuo/redis/bin/redis-server 192.168.8.8:7000 [cluster]
zhouhuo    9664      1  0 00:35 ?        00:00:00 /home/zhouhuo/redis/bin/redis-server 192.168.8.8:7001 [cluster]
zhouhuo    9672   9497  0 00:37 pts/0    00:00:00 grep --color=auto redis
[zhouhuo@zhouhuo3 redis]$ kill 9659

# 再次查看节点信息
[zhouhuo@zhouhuo1 redis]$ bin/redis-cli -c -h 192.168.8.6 -p 7000
192.168.8.6:7000> cluster nodes
2a8fb21e5f4f71db89efb645ccf5741e6472fdec 192.168.8.6:7000@17000 myself,master - 0 1604939894000 1 connected 0-5460
360e4d608cf686c98393c316d5bc43b9e710dc75 192.168.8.7:7001@17001 master - 0 1604939895865 7 connected 10923-16383
0f6bd0691ebee836223cef6976e7775674ea743f 192.168.8.7:7000@17000 master - 0 1604939893820 2 connected 5461-10922
a1d765a40b3ce47a95a9ff68b5827a1b19751381 192.168.8.6:7001@17001 slave 0f6bd0691ebee836223cef6976e7775674ea743f 0 1604939891000 4 connected
90950917a8210d1325db34c3088315c033bc5590 192.168.8.8:7000@17000 master,fail - 1604939858507 1604939857175 3 disconnected
3e04e4eb9acff8f9dfefd15eabeadb553b7827cf 192.168.8.8:7001@17001 slave 2a8fb21e5f4f71db89efb645ccf5741e6472fdec 0 1604939894847 6 connected
192.168.8.6:7000> get zhouhuo
-> Redirected to slot [16207] located at 192.168.8.7:7001
"hello"
192.168.8.7:7001> 

测试一台机器上面断掉某个master节点后,在此节点上的数据有无丢失,从下面可以看出master节点断掉后,从集群中读取该节点下的数据会自动切换到该master节点的slave节点下,而slave节点已经替代原master节点成为新的master节点

重启集群

逐个执行每台服务器的stop-redis-cluster.sh脚本,停止完毕后,再逐个执行start-redis-cluster.sh脚本

重新创建集群

创建集群之后,再使用相同的集群创建命令会报错

解决:进入每个节点目录删掉节点信息相关文件,主要是和node相关的文件,
rdb和aof文件,如需保留数据可以不用删除。

[zhouhuo@zhouhuo1 7000]$ rm -rf appendonly.aof dump.rdb nodes-7000.conf nodes.conf
[zhouhuo@zhouhuo1 7001]$ rm -rf appendonly.aof dump.rdb nodes-7001.conf nodes.conf