Kubernetes: cert-manager on GKE using Let’s Encrypt

Updated: 2020-06-18

The cert-manager project Automatically provisions and renews TLS certificates in Kubernetes. It supports using your own certificate authority, self signed certificates, certificates managed by the Hashicorp Vault PKI, and of course the free certificates issued by Let’s Encrypt.

If you followed my last post, I automated DNS using external-dns. Now it’s time to automate SSL Certificates with cert-manager using Let’s Encrypt.

Install

Get latest version from https://hub.helm.sh/charts/jetstack/cert-manager.

# Install the CustomResourceDefinition
# Kubernetes 1.15+
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager.crds.yaml

# Kubernetes <1.15
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.1/cert-manager-legacy.crds.yaml

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update # create namespace kubectl create ns cert-manager

# Install the cert-manager Helm chart
helm install cert-manager \
-n cert-manager \
jetstack/cert-manager

Verify

kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-5d669ffbd8-zhzm8 1/1 Running 0 2m18s
cert-manager-cainjector-79b7fc64f-rlcgx 1/1 Running 0 2m19s
cert-manager-webhook-6484955794-nmh84 1/1 Running 0 2m19s

Test Installation

Create a ClusterIssuer to test the webhook works okay.

cat <<EOF > test-resources.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-test
---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: test-selfsigned
  namespace: cert-manager-test
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: selfsigned-cert
  namespace: cert-manager-test
spec:
  commonName: example.com
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned
EOF

Create the test resources.

kubectl apply -f test-resources.yaml

Check the status of the newly created certificate. You may need to wait a few seconds before cert-manager processes the certificate request.

kubectl describe certificate.cert-manager.io -n cert-manager-test
Name:         selfsigned-cert
Namespace:    cert-manager-test
Labels:       <none>
Annotations:  API Version:  cert-manager.io/v1alpha3
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-06-13T04:27:41Z
  Generation:          1
  Resource Version:    9390932
  Self Link:           /apis/cert-manager.io/v1alpha3/namespaces/cert-manager-test/certificates/selfsigned-cert
  UID:                 2a513598-80c2-465d-aa9f-6b318bf9c7a9
Spec:
  Common Name:  example.com
  Issuer Ref:
    Name:       test-selfsigned
  Secret Name:  selfsigned-cert-tls
Status:
  Conditions:
    Last Transition Time:  2020-06-13T04:27:41Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2020-09-11T04:27:41Z
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  GeneratedKey  80s   cert-manager  Generated a new private key
  Normal  Requested     80s   cert-manager  Created new CertificateRequest resource "selfsigned-cert-2334779822"
  Normal  Issued        80s   cert-manager  Certificate issued successfully

Clean up the test resources.

kubectl delete -f test-resources.yaml

Create Zone

I created a public zone within Google Cloud DNS.

https://cloud.google.com/dns/docs/quickstart

Create GCP service account and download key

Create a GCP service account with DNS admin access. The ClusterIssuer will use it to access DNS from the K8s cluster.

export PROJECT_NAME=[YOUR_PROJECT_NAME]

# create service account
gcloud iam service-accounts create k8s-cert-manager \
  --display-name="Service Account to manage SSL Certificates." \
  --project=$PROJECT_NAME

# create and download service account key
gcloud iam service-accounts keys create ./credentials.json \
  --iam-account=k8s-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com \
  --project=$PROJECT_NAME

# give dns admin permissions
gcloud projects add-iam-policy-binding $PROJECT_NAME \
  --member=serviceAccount:k8s-cert-manager@$PROJECT_NAME.iam.gserviceaccount.com \
  --role=roles/dns.admin

Create secret from GCP service account

kubectl create secret generic cert-manager \
  --from-file=./credentials.json \
  --namespace=cert-manager

ClusterIssuer

I am using a ClusterIssuer instead of an Issuer.

“ClusterIssuers are a resource type similar to Issuers. They are specified in exactly the same way, but they do not belong to a single namespace and can be referenced by Certificate resources from multiple different namespaces.”

https://docs.cert-manager.io/en/latest/reference/clusterissuers.html
https://docs.cert-manager.io/en/latest/tutorials/acme/dns-validation.html

I will also be using a DNS-01 challenge mechanism (versus HTTP-01). More about this later.

There is a Let’s Encrypt staging and production API. I suggest you use the staging until you’ve worked out any bugs. The Let’s Encrypt production API will rate limit you.

“We highly recommend testing against our staging environment before using our production environment. This will allow you to get things right before issuing trusted certificates and reduce the chance of your running up against rate limits.”

https://letsencrypt.org/docs/staging-environment/

Create ClusterIssuer Staging

cat <<EOF > clusterissuer-staging.yaml
apiVersion: certmanager.k8s.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: username@domain.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    # ACME DNS-01 provider configurations
    - dns01:
        # Google Cloud DNS
        clouddns:
          # Secret from the google service account key
          serviceAccountSecretRef:
            name: cert-manager
            key: credentials.json
          # The project in which to update the DNS zone
          project: $PROJECT_NAME
EOF

Apply.

kubectl apply -f clusterissuer-staging.yaml

Describe.

kubectl describe clusterissuer letsencrypt-staging

Create ClusterIssuer Production

cat <<EOF > clusterissuer-production.yaml
apiVersion: certmanager.k8s.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: username@domain.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    # ACME DNS-01 provider configurations
    - dns01:
        # Google Cloud DNS
        clouddns:
          # Secret from the google service account key
          serviceAccountSecretRef:
            name: cert-manager
            key: credentials.json
          # The project in which to update the DNS zone
          project: $PROJECT_NAME
EOF

Apply, describe.

kubectl apply -f clusterissuer-production.yaml
kubectl describe clusterissuer letsencrypt

Certificate

Once we have created the above ClusterIssuer, we can use it to obtain a certificate.

To request a certificate from Let’s Encrypt (or any Certificate Authority), you need to provide some kind of proof that you are entitled to receive the certificate for a given domain. Let’s Encrypt support two methods of validation to prove control of your domain, http-01 (validation over HTTP) and dns-01 (validation via DNS). Wildcard domain certificates (those covering *.yourdomain.com) can only be requested using DNS validation.

cert-manager will periodically check its validity and attempt to renew it if it gets close to expiry. cert-manager considers certificates to be close to expiry when the ‘Not After’ field on the certificate is less than the current time plus 30 days.

Create Certificate: Staging Wildcard

cat <<EOF > certificate-itsmetommy-io-tls-staging.yaml
apiVersion: cert-manager.io/v1alpha3
kind: Certificate
metadata:
  name: itsmetommy-io-tls-staging
  namespace: itsmetommy
spec:
  secretName: itsmetommy-io-tls-staging
  dnsNames:
  - '*.itsmetommy.io'
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
EOF

Create, describe.

kubectl create -f certificate-itsmetommy-io-tls-staging.yaml
kubectl describe secret itsmetommy-io-tls-staging
kubectl describe cert itsmetommy-io-tls-staging

Create Certificate: Production Wildcard

cat <<EOF > certificate-itsmetommy-io-tls.yaml
apiVersion: cert-manager.io/v1alpha3
kind: Certificate
metadata:
  name: itsmetommy-io-tls
  namespace: itsmetommy
spec:
  secretName: itsmetommy-io-tls
  dnsNames:
    - '*.itsmetommy.io'
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer
EOF

Create, describe.

kubectl create -f certificate-itsmetommy-io-tls.yaml
kubectl describe secret itsmetommy-io-tls
kubectl describe cert itsmetommy-io-tls

Create Deployment

kubectl create deployment nginx --image=nginx -n itsmetommy

Create Service

kubectl expose deployment nginx --port=80 --target-port=80 --type=NodePort -n itsmetommy

Ingress

In this blog, I will show you two ways of utilizing Certificates within an Ingress.

Ingress with an existing certificate
Ingress with an autogenerated certificate

Note: The ingress and certificate have to be in the same namespace.

Ingress w/ existing Certificate: Staging

Uses existing certificate itsmetommy-io-tls-staging, which includes *.itsmetommy.io.

Note: There is no need to include the annotations cert-manager.io section if you’ve already created a certificate.

cat <<EOF > ingress-itsmetommy-io-staging.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: itsmetommy-io-staging
  namespace: itsmetommy
annotations:
  # use ingress-nginx controller
  kubernetes.io/ingress.class: nginx
spec:
  tls:
  # use an already existing certificate within the same namespace
  - secretName: itsmetommy-io-tls-staging
    hosts:
    - staging-1.itsmetommy.io
    - staging-2.itsmetommy.io
  rules:
  - host: staging-1.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: staging-2.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
EOF

Create, get, describe.

kubectl create -f ingress-itsmetommy-io-staging.yaml
kubectl get ingress itsmetommy-io-staging
kubectl describe secret itsmetommy-io-tls-staging
kubectl describe ingress itsmetommy-io-staging

Ingress w/ existing Certificate: Production

Note: There is no need to include the annotations cert-manager.io section if you’ve already created a certificate.

cat <<EOF > ingress-itsmetommy-io.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: itsmetommy-io
  namespace: itsmetommy
annotations:
  # use ingress-nginx controller
  kubernetes.io/ingress.class: nginx
spec:
  tls:
  # use an already existing certificate within the same namespace
  - secretName: itsmetommy-io-tls
    hosts:
    - prod-1.itsmetommy.io
    - prod-2.itsmetommy.io
  rules:
  - host: prod-1.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: prod-2.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
EOF

Create, get, describe.

kubectl create -f ingress-itsmetommy-io.yaml
kubectl get ingress itsmetommy-io
kubectl describe secret itsmetommy-io-tls
kubectl describe ingress itsmetommy-io

Ingress w/ autogenerated Certificate: Staging

This Ingress will automatically create a certificate for you because the name of the secret does not exist, which hold the actual certificate. The most important part of having auto generated certificates is the secretName. Make sure you do not use the same name as a certificate that you already created before. For example, we already created a certificate called itsmetommy-io-tls-staging, which created a secret called itsmetommy-io-tls-staging with the certificate in it. In this case, we do not want to use the same name. This will cause changes that will break the certificate.

You will also notice that I included cert-manager.io in the annotations section. This tells tells which issuer to use to generate the certificate.

cat <<EOF > ingress-itsmetommy-io-auto-staging.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: itsmetommy-io-auto-staging
  namespace: itsmetommy
  annotations:
    # use ingress-nginx controller
    kubernetes.io/ingress.class: nginx
    # automatically create ssl certificate using cert-manager
    cert-manager.io/cluster-issuer: letsencrypt-staging
spec:
  tls:
  - secretName: itsmetommy-io-auto-tls-staging
    hosts:
    - staging-1-auto.itsmetommy.io
    - staging-2-auto.itsmetommy.io
  rules:
  - host: staging-1-auto.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: staging-2-auto.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
EOF

Create, get, describe.

kubectl create -f ingress-itsmetommy-io-auto-staging.yaml
kubectl get ingress itsmetommy-io-staging
kubectl describe secret itsmetommy-io-auto-tls-staging
kubectl describe ingress itsmetommy-io-auto-staging

Ingress w/ autogenerated Certificate: Production

cat <<EOF > ingress-itsmetommy-io-auto.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: itsmetommy-io-auto
  namespace: itsmetommy
  annotations:
    # use ingress-nginx controller
    kubernetes.io/ingress.class: nginx
    # automatically create ssl certificate using cert-manager
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
  - secretName: itsmetommy-io-auto-tls
    hosts:
    - prod-1-auto.itsmetommy.io
    - prod-2-auto.itsmetommy.io
  rules:
  - host: prod-1-auto.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
  - host: prod-2-auto.itsmetommy.io
    http:
      paths:
      - path:
        backend:
          serviceName: nginx
          servicePort: 80
EOF

Create, get, describe.

kubectl create -f ingress-itsmetommy-io-auto.yaml
kubectl get ingress itsmetommy-io
kubectl describe secret itsmetommy-io-auto-tls
kubectl describe ingress itsmetommy-io-auto

2 responses to “Kubernetes: cert-manager on GKE using Let’s Encrypt”

Kubernetes: Google-managed SSL Certificates using ManagedCertificate on GKE – It's Me, Tommy says:
May 31, 2022 at 12:15 AM
[…] previously went over how to create SSL Certificates using cert-manager, but Google also has a GKE specific way of somewhat doing the same thing by using a custom resource […]
Kubernetes: Istio + Cert-Manager + GKE – It's Me, Tommy says:
May 31, 2022 at 12:21 AM
[…] https://itsmetommy.com/2019/06/23/kubernetes-cert-manager-on-gke-using-lets-encrypt/ […]