Kubernetes: Cluster Backup with Heptio Velero on GCP

Heptio Ark is now Velero! As I’ve already written HERE, I thought I’d give an update to the slightly different installation of Heptio Velero.

This is going to be more of a cliff notes blog, as the previous blog has a lot of information I’d rather not repeat in this one. So here we go.

Install Client

Option 1

I got an error, but I assume it will work in the future.

brew install velero

Error

brew install velero
Error: No available formula with the name "velero"
==> Searching for a previously deleted formula (in the last month)…
Warning: homebrew/core is shallow clone. To get complete history run:
git -C "$(brew --repo homebrew/core)" fetch --unshallow
Error: No previously deleted formula found.
==> Searching for similarly named formulae…
Error: No similarly named formulae found.
==> Searching taps…
==> Searching taps on GitHub…
Error: No formulae found in taps.

Option 2

View latest release https://github.com/heptio/velero/releases.

The following commands downloads the Velero binary and copies it to /usr/local/bin.

{
VERSION=v0.11.0
OS=$(uname | tr '[:upper:]' '[:lower:]')
ARCH=amd64
curl -L https://github.com/heptio/velero/releases/download/$VERSION/velero-$VERSION-$OS-$ARCH.tar.gz \
-o velero-$VERSION-$OS-$ARCH.tar.gz
tar -zxvf velero-$VERSION-$OS-$ARCH.tar.gz velero
chmod +x ./velero
sudo mv ./velero /usr/local/bin/velero
rm -rf velero-$VERSION-$OS-$ARCH.tar.gz
}

Install Server

Download the config directory, which holds all the files required to install velero onto your cluster.

The following commands creates a directory called heptio-velero and downloads the config directory into it (keeps things more organized).

{
VERSION=v0.11.0
OS=$(uname | tr '[:upper:]' '[:lower:]')
ARCH=amd64
mkdir heptio-velero && cd heptio-velero
curl -L https://github.com/heptio/velero/releases/download/$VERSION/velero-$VERSION-$OS-$ARCH.tar.gz \
-o velero-$VERSION-$OS-$ARCH.tar.gz
tar -zxvf velero-$VERSION-$OS-$ARCH.tar.gz config
rm -rf velero-$VERSION-$OS-$ARCH.tar.gz
}

Run on GCP

Create GCS bucket

BUCKET=[YOUR_BUCKET]

Create service account

Store the project value from the results in the environment variable $PROJECT_ID.

PROJECT_ID=$(gcloud config get-value project)

Create service account.

gcloud iam service-accounts create velero \
--display-name "Velero service account"

Set the $SERVICE_ACCOUNT_EMAIL variable to match its email value.

SERVICE_ACCOUNT_EMAIL=velero@ace-element-175818.iam.gserviceaccount.com

Attach policies to give Velero the necessary permissions to function:

A

ROLE_PERMISSIONS=(
compute.disks.get
compute.disks.create
compute.disks.createSnapshot
compute.snapshots.get
compute.snapshots.create
compute.snapshots.useReadOnly
compute.snapshots.delete
compute.projects.get
)

B

gcloud iam roles create velero.server \
--project $PROJECT_ID \
--title "Velero Server" \
--permissions "$(IFS=","; echo "${ROLE_PERMISSIONS[*]}")"

C

gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
--role projects/$PROJECT_ID/roles/velero.server

D

gsutil iam ch serviceAccount:$SERVICE_ACCOUNT_EMAIL:objectAdmin gs://${BUCKET}

Create a service account key, specifying an output file (credentials-velero) in your local directory:

Note: This will download the file credentials-velero to your localhost.

gcloud iam service-accounts keys create credentials-velero \
--iam-account $SERVICE_ACCOUNT_EMAIL

Credentials and configuration

Prereqs

kubectl apply -f config/common/00-prereqs.yaml

Create a Secret

kubectl create secret generic cloud-credentials \
--namespace velero \
--from-file cloud=credentials-velero

Start the server

{
kubectl apply -f config/gcp/05-backupstoragelocation.yaml
kubectl apply -f config/gcp/06-volumesnapshotlocation.yaml
kubectl apply -f config/gcp/10-deployment.yaml
}

Testing

I’m using namespace itsmetommy-test to test backup situations.

# create namespace
kubectl create ns itsmetommy-test

# create a deployment
kubectl create deployment nginx --image=nginx -n itsmetommy-test

# confirm deployment
kubectl get deploy -n itsmetommy-test

velero backup create itsmetommy-test-ns --include-namespaces itsmetommy-test --snapshot-volumes

# confirm backup
velero get backups

# simulate disaster
kubectl delete ns itsmetommy-test

# confirm namespace is gone
k get ns itsmetommy-test

# restore
velero restore create --from-backup itsmetommy-test-ns

# confirm restore
velero get restores

# confirm deployment restored
kubectl get deploy -n itsmetommy-test

# clean up
{
kubectl delete ns itsmetommy-test-ns
sleep 60
velero backup delete itsmetommy-test-ns
}