Kubernetes: redis-cluster on GKE (Google Kubernetes Engine)

My goal was to create a redis-cluster with one command. I was able to accomplish this by using a kubernetes job. It wasn’t the ideal solution due to all the additional Terminated: Error pods it leaves behind, but it is however a solution.

There are two things you need to pay attention to.

  1. namespace: itsmetommy
  2. storageClassName: fast

namespace

You can update the namespace to whatever you’d like or delete the namespace line completely to set it to default.

storageClassName

I created a storageClass named fast that uses SSD. This is why you will see  storageClassName: fast within the volumeClaimTemplates section. You can simply remove this line to set it back to your default settings.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd

Clone git repository

https://github.com/itsmetommy/kubernetes-redis-cluster

Create

ONE COMMAND TO RULE THEM ALL!

kubectl create -f .

Example

kubectl create -f .
configmap "redis-cluster-conf" created
job.batch "redis-join" created
service "redis-service" created
statefulset.apps "redis-cluster" created

Check pods

kubectl get pods -n itsmetommy -l app=redis

Example

kubectl get pods -n itsmetommy -l app=redis
NAME READY STATUS RESTARTS AGE
redis-cluster-0 1/1 Running 0 19m
redis-cluster-1 1/1 Running 0 19m
redis-cluster-2 1/1 Running 0 19m
redis-cluster-3 1/1 Running 0 18m
redis-cluster-4 1/1 Running 0 18m
redis-cluster-5 1/1 Running 0 17m

Logs

kubectl logs redis-cluster-0 -n itsmetommy

Example

kubectl logs redis-cluster-0 -n itsmetommy
1:C 09 Jun 20:08:58.268 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 09 Jun 20:08:58.268 # Redis version=4.0.2, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 09 Jun 20:08:58.268 # Configuration loaded
1:M 09 Jun 20:08:58.269 * No cluster configuration found, I'm face3a65719530427ad09a188a70fea150d22ea1
1:M 09 Jun 20:08:58.272 * Running mode=cluster, port=6379.
1:M 09 Jun 20:08:58.272 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 09 Jun 20:08:58.272 # Server initialized
1:M 09 Jun 20:08:58.272 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 09 Jun 20:08:58.272 * Ready to accept connections
1:M 09 Jun 20:10:53.031 # configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH
1:M 09 Jun 20:10:53.043 # IP address for this node updated to 10.43.12.235
1:M 09 Jun 20:10:57.392 * Slave 10.43.11.13:6379 asks for synchronization
1:M 09 Jun 20:10:57.392 * Partial resynchronization not accepted: Replication ID mismatch (Slave asked for '2e3eac549dfc8815a6eae482e5884caec12837f9', my replication IDs are '6749228b457a1208ba105fa3130f8820860ba731' and '0000000000000000000000000000000000000000')
1:M 09 Jun 20:10:57.392 * Starting BGSAVE for SYNC with target: disk
1:M 09 Jun 20:10:57.392 * Background saving started by pid 10
10:C 09 Jun 20:10:57.395 * DB saved on disk
10:C 09 Jun 20:10:57.395 * RDB: 0 MB of memory used by copy-on-write
1:M 09 Jun 20:10:57.463 * Background saving terminated with success
1:M 09 Jun 20:10:57.463 * Synchronization with slave 10.43.11.13:6379 succeeded
1:M 09 Jun 20:10:57.964 # Cluster state changed: ok

Connect

Note: -c Enable cluster mode (follow -ASK and -MOVED redirections)

kubectl exec redis-cluster-0 -n itsmetommy -it -- redis-cli -c

Create keys

127.0.0.1:6379> set mykey1 1
OK
127.0.0.1:6379> set mykey2 2
-> Redirected to slot [14119] located at 10.43.13.244:6379
OK
10.43.13.244:6379> set mykey3 3
-> Redirected to slot [9990] located at 10.43.1.53:6379
OK
10.43.1.53:6379>

View Nodes

> cluster nodes

Example

10.43.1.53:6379> cluster nodes
5f4b8d887e5ab846fd2cd441ce444132190e60e2 10.43.11.13:6379@16379 slave face3a65719530427ad09a188a70fea150d22ea1 0 1528576197000 5 connected
834d17ef54a792cf561591c32687cfc316ac877a 10.43.10.149:6379@16379 slave e8cbe17e5e6024536c49729ddf1bd4860d966d73 0 1528576198514 4 connected
fcd20223a3c6e02993c5aad23f84d8222825d6a8 10.43.1.53:6379@16379 myself,master - 0 1528576197000 2 connected 5461-10922
face3a65719530427ad09a188a70fea150d22ea1 10.43.12.235:6379@16379 master - 0 1528576197000 1 connected 0-5460
e8cbe17e5e6024536c49729ddf1bd4860d966d73 10.43.13.244:6379@16379 master - 0 1528576198000 3 connected 10923-16383
958a6d195c24670f4d73a94972b49b29d417dc74 10.43.13.252:6379@16379 slave fcd20223a3c6e02993c5aad23f84d8222825d6a8 0 1528576198114 6 connected
10.43.1.53:6379>

View slots

> cluster slots

Example

10.43.1.53:6379> cluster slots
1) 1) (integer) 5461
   2) (integer) 10922
   3) 1) "10.43.1.53"
      2) (integer) 6379
      3) "fcd20223a3c6e02993c5aad23f84d8222825d6a8"
   4) 1) "10.43.13.252"
      2) (integer) 6379
      3) "958a6d195c24670f4d73a94972b49b29d417dc74"
2) 1) (integer) 0
   2) (integer) 5460
   3) 1) "10.43.12.235"
      2) (integer) 6379
      3) "face3a65719530427ad09a188a70fea150d22ea1"
   4) 1) "10.43.11.13"
      2) (integer) 6379
      3) "5f4b8d887e5ab846fd2cd441ce444132190e60e2"
3) 1) (integer) 10923
   2) (integer) 16383
   3) 1) "10.43.13.244"
      2) (integer) 6379
      3) "e8cbe17e5e6024536c49729ddf1bd4860d966d73"
   4) 1) "10.43.10.149"
      2) (integer) 6379
      3) "834d17ef54a792cf561591c32687cfc316ac877a"
10.43.1.53:6379>

Clean up

kubectl delete -f .

Example

kubectl delete -f .
configmap "redis-cluster-conf" deleted
job.batch "redis-join" deleted
service "redis-service" deleted
statefulset.apps "redis-cluster" deleted

Delete Persistent Volumes

kubectl delete pvc -l app=redis -n itsmetommy

Example

kubectl delete pvc -l app=redis -n itsmetommy
persistentvolumeclaim "redis-data-redis-cluster-0" deleted
persistentvolumeclaim "redis-data-redis-cluster-1" deleted
persistentvolumeclaim "redis-data-redis-cluster-2" deleted
persistentvolumeclaim "redis-data-redis-cluster-3" deleted
persistentvolumeclaim "redis-data-redis-cluster-4" deleted
persistentvolumeclaim "redis-data-redis-cluster-5" deleted