Categories
Kubernetes

Kubernetes: cert-manager on GKE using Let’s Encrypt

The cert-manager project Automatically provisions and renews TLS certificates in Kubernetes. It supports using your own certificate authority, self signed certificates, certificates managed by the Hashicorp Vault PKI, and of course the free certificates issued by Let’s Encrypt.

If you followed my last post, I automated DNS using external-dns. Now it’s time to automate SSL Certificates with cert-manager using Let’s Encrypt.

Install

I’m using Helm to install cert-manager.

https://hub.helm.sh/charts/jetstack/cert-manager

# mac
brew install kubernetes-helm

# This will install Tiller to your running Kubernetes cluster
# It will also set up any necessary local configuration
helm init

# Grant the tiller service account cluster admin privileges
{
  kubectl create serviceaccount \
    --namespace kube-system tiller
  kubectl create clusterrolebinding tiller \
    --clusterrole=cluster-admin \
    --serviceaccount=kube-system:tiller
  kubectl patch deploy \
    --namespace kube-system tiller-deploy \
    -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
}

# Install the CustomResourceDefinition resources separately
# get latest from https://hub.helm.sh/charts/jetstack/cert-manager
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.10/deploy/manifests/00-crds.yaml

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install \
  --name cert-manager \
  --namespace cert-manager \
  jetstack/cert-manager

Verify

kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-5d669ffbd8-zhzm8 1/1 Running 0 2m18s
cert-manager-cainjector-79b7fc64f-rlcgx 1/1 Running 0 2m19s
cert-manager-webhook-6484955794-nmh84 1/1 Running 0 2m19s

Test Installation

# Create a ClusterIssuer to test the webhook works okay
cat <<EOF > test-resources.yaml
apiVersion: v1
kind: Namespace
metadata:
name: cert-manager-test
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: test-selfsigned
namespace: cert-manager-test
spec:
selfSigned: {}
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: selfsigned-cert
namespace: cert-manager-test
spec:
commonName: example.com
secretName: selfsigned-cert-tls
issuerRef:
name: test-selfsigned
EOF

# Create the test resources
kubectl apply -f test-resources.yaml

# Check the status of the newly created certificate
# You may need to wait a few seconds before cert-manager processes the
# certificate request
kubectl describe certificate -n cert-manager-test
...
Spec:
Common Name: example.com
Issuer Ref:
Name: test-selfsigned
Secret Name: selfsigned-cert-tls
Status:
Conditions:
Last Transition Time: 2019-01-29T17:34:30Z
Message: Certificate is up to date and has not expired
Reason: Ready
Status: True
Type: Ready
Not After: 2019-04-29T17:34:29Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CertIssued 4s cert-manager Certificate issued successfully

# Clean up the test resources
kubectl delete -f test-resources.yaml

Create Zone

I created a public zone within Google Cloud DNS.

https://cloud.google.com/dns/docs/quickstart

Create GCP service account and download key

Create a GCP service account to enable an account to edit Cloud DNS.

export PROJECT_NAME=[YOUR_PROJECT_NAME]

# create service account
gcloud iam service-accounts create k8s-cert-manager \
  --display-name="Service Account to manage SSL Certificates." \
  --project=$PROJECT_NAME

# create and download service account key
gcloud iam service-accounts keys create ./credentials.json \
  --iam-account=k8s-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com \
  --project=$PROJECT_NAME

# give dns admin permissions
gcloud projects add-iam-policy-binding $PROJECT_NAME \
  --member=serviceAccount:k8s-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com \
  --role=roles/dns.admin

Create secret from GCP service account

kubectl create secret generic cert-manager \
--from-file=./credentials.json \
--namespace=cert-manager

ClusterIssuer

I am using a ClusterIssuer instead of an Issuer.

“ClusterIssuers are a resource type similar to Issuers. They are specified in exactly the same way, but they do not belong to a single namespace and can be referenced by Certificate resources from multiple different namespaces.”

https://docs.cert-manager.io/en/latest/reference/clusterissuers.html
https://docs.cert-manager.io/en/latest/tutorials/acme/dns-validation.html

I will also be using a DNS-01 challenge mechanism (versus HTTP-01). More about this later.

There is a Let’s Encrypt staging and production API. I suggest you use the staging until you’ve worked out any bugs. The Let’s Encrypt production API will rate limit you.

“We highly recommend testing against our staging environment before using our production environment. This will allow you to get things right before issuing trusted certificates and reduce the chance of your running up against rate limits.”

https://letsencrypt.org/docs/staging-environment/

Create ClusterIssuer — Staging

cat <<EOF > clusterissuer-staging.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: username@gmail.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    # ACME DNS-01 provider configurations
    - dns01:
        # Google Cloud DNS
        clouddns:
          # Secret from the google service account key
          serviceAccountSecretRef:
            name: cert-manager
            key: credentials.json
          # The project in which to update the DNS zone
          project: $PROJECT_NAME
EOF
kubectl apply -f clusterissuer-staging.yaml
kubectl describe clusterissuer letsencrypt-staging

Create ClusterIssuer — Production

cat <<EOF > clusterissuer-production.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: username@gmail.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    # ACME DNS-01 provider configurations
    - dns01:
        # Google Cloud DNS
        clouddns:
          # Secret from the google service account key
          serviceAccountSecretRef:
            name: cert-manager
            key: credentials.json
          # The project in which to update the DNS zone
          project: $PROJECT_NAME
EOF
kubectl apply -f clusterissuer-production.yaml
kubectl describe clusterissuer letsencrypt

Certificate

Once we have created the above ClusterIssuer, we can use it to obtain a certificate.

To request a certificate from Let’s Encrypt (or any Certificate Authority), you need to provide some kind of proof that you are entitled to receive the certificate for a given domain. Let’s Encrypt support two methods of validation to prove control of your domain, http-01 (validation over HTTP) and dns-01 (validation via DNS). Wildcard domain certificates (those covering *.yourdomain.com) can only be requested using DNS validation.

cert-manager will periodically check its validity and attempt to renew it if it gets close to expiry. cert-manager considers certificates to be close to expiry when the ‘Not After’ field on the certificate is less than the current time plus 30 days.

Note: The certificate and ingress have to be in the same namespace.

Create Certificate: Staging – Wildcard

cat <<EOF > certificate-itsmetommy-io-tls-staging.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: itsmetommy-io-tls-staging
  namespace: itsmetommy
spec:
  secretName: itsmetommy-io-tls-staging
  commonName: '*.itsmetommy.io'
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
EOF
kubectl create -f certificate-itsmetommy-io-tls-staging.yaml
kubectl describe secret itsmetommy-io-tls-staging
kubectl describe cert itsmetommy-io-tls-staging

Create Certificate: Staging – Single domain

I won’t be using this certificate, but I wanted to show you how to create one.

cat <<EOF > certificate-tommy-itsmetommy-io-tls-staging.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: tommy-itsmetommy-io-tls-staging
  namespace: itsmetommy
spec:
  secretName: tommy-itsmetommy-io-tls-staging
  commonName: tommy.itsmetommy.io
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
EOF
kubectl create -f certificate-tommy-itsmetommy-io-tls-staging.yaml
kubectl describe secret tommy-itsmetommy-io-tls-staging
kubectl describe cert tommy-itsmetommy-io-tls-staging

Create Certificate: Staging – Sub-domain

cat <<EOF > certificate-dev-itsmetommy-io-tls-staging.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: dev-itsmetommy-io-tls-staging
  namespace: itsmetommy
spec:
  secretName: dev-itsmetommy-io-tls-staging
  commonName: '*.dev.itsmetommy.io'
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
EOF
kubectl create -f certificate-dev-itsmetommy-io-tls-staging.yaml
kubectl describe secret dev-itsmetommy-io-tls-staging
kubectl describe cert dev-itsmetommy-io-tls-staging

Create Certificate: Production – Wildcard

Note: I included my root domain itsmetommy.io.

cat <<EOF > certificate-itsmetommy-io-tls.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: itsmetommy-io-tls
  namespace: itsmetommy
spec:
  secretName: itsmetommy-io-tls
  commonName: '*.itsmetommy.io'
  dnsNames:
    - itsmetommy.io 
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
EOF
kubectl create -f certificate-itsmetommy-io-tls.yaml
kubectl describe secret itsmetommy-io-tls
kubectl describe cert itsmetommy-io-tls

Create Certificate: Production – Single domain

I won’t be using this certificate, but I wanted to show you how to create one.

cat <<EOF > certificate-tommy-itsmetommy-io-tls.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: tommy-itsmetommy-io-tls
  namespace: itsmetommy
spec:
  secretName: tommy-itsmetommy-io-tls
  commonName: tommy.itsmetommy.io
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
EOF
kubectl create -f certificate-tommy-itsmetommy-io-tls.yaml
kubectl describe secret tommy-itsmetommy-io-tls
kubectl describe cert tommy-itsmetommy-io-tls

Create Certificate: Production – Sub-domain

cat <<EOF > certificate-dev-itsmetommy-io-tls.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: dev-itsmetommy-io-tls
  namespace: itsmetommy
spec:
  secretName: dev-itsmetommy-io-tls
  commonName: '*.dev.itsmetommy.io'
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
EOF
kubectl create -f certificate-dev-itsmetommy-io-tls.yaml
kubectl describe secret dev-itsmetommy-io-tls
kubectl describe cert dev-itsmetommy-io-tls

Create Deployment

kubectl create deployment nginx --image=nginx -n itsmetommy

Create Service

kubectl expose deployment nginx --port=80 --target-port=80 --type=NodePort -n itsmetommy

Ingress

In this blog, I will show you two ways of utilizing Certificates within an Ingress.

  1. Ingress with an existing certificate
  2. Ingress with an autogenerated certificate

The examples with existing certificates use the already created certificates that we created above.

Note: The ingress and certificate have to be in the same namespace.

Ingress w/ existing Certificate: Staging – Multiple-domains

Uses existing certificate itsmetommy-io-tls-staging, which includes *.itsmetommy.io.

Note: There is no need to include the annotations certmanager.k8s.io section if you’ve already created a certificate.

cat <<EOF > ingress-itsmetommy-io-staging.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: itsmetommy-io-staging
  namespace: itsmetommy
spec:
  tls:
  - secretName: itsmetommy-io-tls-staging
    hosts:
    - staging-1.itsmetommy.io
    - staging-2.itsmetommy.io
  rules:
  - host: staging-1.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: staging-2.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  backend:
    serviceName: nginx
    servicePort: 80
EOF
kubectl create -f ingress-itsmetommy-io-staging.yaml
kubectl describe secret itsmetommy-io-tls-staging
kubectl describe ingress itsmetommy-io-staging
kubectl get ingress itsmetommy-io-staging

Ingress w/ existing Certificate: Staging – Sub-domains

Uses existing certificate dev-itsmetommy-io-tls-staging, which includes *.dev.itsmetommy.io.

Note: There is no need to include the annotations certmanager.k8s.io section if you’ve already created a certificate.

cat <<EOF > ingress-dev-itsmetommy-io-staging.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dev-itsmetommy-io-staging
  namespace: itsmetommy
spec:
  tls:
  - secretName: dev-itsmetommy-io-tls-staging
    hosts:
    - staging-1.dev.itsmetommy.io
    - staging-2.dev.itsmetommy.io
  rules:
  - host: staging-1.dev.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: staging-2.dev.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  backend:
    serviceName: nginx
    servicePort: 80
EOF
kubectl create -f ingress-dev-itsmetommy-io-staging.yaml
kubectl describe secret dev-itsmetommy-io-tls-staging
kubectl describe ingress dev-itsmetommy-io-staging
kubectl get ingress dev-itsmetommy-io-staging

Ingress w/ existing Certificate: Production – Multiple-domains

Uses existing certificate itsmetommy-io-tls, which includes *.itsmetommy.io.

I included itsmetommy.io in this one.

Note: There is no need to include the annotations certmanager.k8s.io section if you’ve already created a certificate.

cat <<EOF > ingress-itsmetommy-io.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: itsmetommy-io
  namespace: itsmetommy
spec:
  tls:
  - secretName: itsmetommy-io-tls
    hosts:
    - www.itsmetommy.io
    - itsmetommy.io
    - prod-1.itsmetommy.io
    - prod-2.itsmetommy.io
  rules:
  - host: www.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: prod-1.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: prod-2.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  backend:
    serviceName: nginx
    servicePort: 80
EOF
kubectl create -f ingress-itsmetommy-io.yaml
kubectl describe secret itsmetommy-io-tls
kubectl describe ingress itsmetommy-io
kubectl get ingress itsmetommy-io

Ingress w/ existing Certificate: Production – Sub-domains

Uses existing certificate dev-itsmetommy-io-tls, which includes *.dev.itsmetommy.io.

Note: There is no need to include the annotations certmanager.k8s.io section if you’ve already created a certificate.

cat <<EOF > ingress-dev-itsmetommy-io.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dev-itsmetommy-io
  namespace: itsmetommy
spec:
  tls:
  - secretName: dev-itsmetommy-io-tls
    hosts:
    - prod-1.dev.itsmetommy.io
    - prod-2.dev.itsmetommy.io
  rules:
  - host: prod-1.dev.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: prod-2.dev.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  backend:
    serviceName: nginx
    servicePort: 80
EOF
kubectl create -f ingress-dev-itsmetommy-io.yaml
kubectl describe secret dev-itsmetommy-io-tls
kubectl describe ingress dev-itsmetommy-io
kubectl get ingress dev-itsmetommy-io

Ingress w/ autogenerated Certificate: Staging – Multiple-domains

This Ingress will automatically create a certificate for you because the name of the secret does not exist, which hold the actual certificate. The most important part of having auto generated certificates is the secretName. Make sure you do not use the same name as a certificate that you already created before. For example, we already created a certificate called itsmetommy-io-tls-staging, which created a secret called itsmetommy-io-tls-staging with the certificate in it. In this case, we do not want to use the same name. This will cause changes that will break the certificate.

You will also notice that I included certmanager.k8s.io in the annotations section. This tells tells which issuer to use to generate the certificate.

cat <<EOF > ingress-itsmetommy-io-auto-staging.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: itsmetommy-io-auto-staging
  namespace: itsmetommy
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging
spec:
  tls:
  - secretName: itsmetommy-io-auto-tls-staging
    hosts:
    - staging-1-auto.itsmetommy.io
    - staging-2-auto.itsmetommy.io
  rules:
  - host: staging-1-auto.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: staging-2-auto.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  backend:
    serviceName: nginx
    servicePort: 80
EOF
kubectl create -f ingress-itsmetommy-io-auto-staging.yaml
kubectl describe secret itsmetommy-io-auto-tls-staging
kubectl describe ingress itsmetommy-io-auto-staging
kubectl get ingress itsmetommy-io-staging

Ingress w/ autogenerated Certificate: Production – Multiple-domains

cat <<EOF > ingress-itsmetommy-io-auto.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: itsmetommy-io-auto
  namespace: itsmetommy
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt
spec:
  tls:
  - secretName: itsmetommy-io-auto-tls
    hosts:
    - prod-1-auto.itsmetommy.io
    - prod-2-auto.itsmetommy.io
  rules:
  - host: prod-1-auto.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: prod-2-auto.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  backend:
    serviceName: nginx
    servicePort: 80
EOF
kubectl create -f ingress-itsmetommy-io-auto.yaml
kubectl describe secret itsmetommy-io-auto-tls
kubectl describe ingress itsmetommy-io-auto
kubectl get ingress itsmetommy-io

Check Certificate

If you are using a staging ssl certificate, your site will not be secure. It will however show that it has been signed, which means everything is working as it should and is ready for production.

Make sure the domain resolves.

while [ 1 ];do host itsmetommy.io;sleep 5;done

Check the connection.

echo | openssl s_client -connect itsmetommy.io:443

Errors

helm install \
--name cert-manager \
--namespace cert-manager \
--version v0.8.0 \
jetstack/cert-manager
Error: release cert-manager failed: namespaces "cert-manager" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "cert-manager"

Fix.

{
kubectl create serviceaccount \
--namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule \
--clusterrole=cluster-admin \
--serviceaccount=kube-system:tiller
kubectl patch deploy \
--namespace kube-system tiller-deploy \
-p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
}