k8s cert manager sync

background
In containerized environments, we use cert-manager to automatically provision, manage, and renew TLS certificates for our applications.

These certificates are managed entirely through code using git ops, and developers / operators never need to touch / see the actual plain-text certificate as it is automatically provisioned and attached to gateway.

However for applications that sit behind the Incapsula WAF, or have components in both EKS and CloudFront, there was not a seamless and secure process to attach certificates without operators manually passing DNS01 challenge records back and forth or worse, passing TLS certs back and forth.

In addition to the security risk this poses, it also introduces a level of human error and manual tracking of expiry / renewals.

To address this, I created a small operator that runs in the cluster alongside cert-manager, called cert-manager-sync.

This operator fully automates this process, so that developers must only annotate their cert-manager kube tls secrets with a flag indicating they want the certificate synced and to where they want it synced.

When the certificate is provisioned by cert-manager, the cert-manager-sync operator will sync the certificate to the upstream certificate provider(s) defined in the TLS secret annotations.

authentication
aws acm
Create an IRSA role with acm:* access, and attach the IAM Role to the k8s ServiceAccount in devops/k8s/sa.yaml.

incapsula
Create an Incapsula API Key and create a kube secret in the namespace in which the operator runs.

kubectl -n cert-manager \
	create secret generic example-incapsula-api-secret \
	--from-literal api_id=XXXXX --from-literal api_key=XXXXX


You will then annotate your k8s TLS secret with this secret name to tell the operator to retrieve the Incapsula API secret from this location.

configuration

The operator uses Kubernetes annotations to define the sync locations and configurations.

The following example contains all supported annotations.

---
apiVersion: v1
type: kubernetes.io/tls
kind: Secret
metadata:
  name: example
  namespace: cert-manager
  annotations:
    cert-manager-sync.lestak.sh/sync-enabled: "true" # enable sync on tls secret
    cert-manager-sync.lestak.sh/acm-enabled: "true" # sync certificate to ACM
    cert-manager-sync.lestak.sh/acm-certificate-arn: "" # will be auto-filled by operator for in-place renewals
    cert-manager-sync.lestak.sh/incapsula-site-id: "12345" # incapsula site to attach cert
    cert-manager-sync.lestak.sh/incapsula-secret-name: "cert-manager-sync-poc" # secret in same namespace which contains incapsula api key
data:
  ca.crt: ""
  tls.crt: ""
  tls.key: ""


last updated 2021-06-30T19:52:27-0700