kubernetes enterprise ca distribution

Within kubernetes clusters, cert-manager is the de facto standard for managing certificates. Coupled with a service mesh such as Istio, it is possible to automatically distribute certificates and trust chains to all pods in a cluster, and the service mesh will handle all intra-mesh communication, TLS termination, and certificate validation.

However an issue arises when workloads inside of Kubernetes must communicate out to services outside of the cluster/mesh which are secured with an internally signed certificate.

When distributing an internal root CA, one may use GPOs, Ansible, Chef, Jamf, etc. to distribute the root CA to all machines in the enterprise. This is a well understood problem, and there are many solutions to it.

However in a containerized environment, by default every container manages its own trust store, usually inherited from the base image used to build the container. Development teams rarely touch the trust store outside of possibly doing the equivalent of update-ca-certificates in the container's Dockerfile. This is all done at container build time, which means any changes to the trust store require a rebuild and redeploy of the container.

From an immutable artifact perspective this is ideal, but from an enterprise operational perspective this significantly limits the ability to centrally manage trust stores and certificates.

We either need to bake in the current root CA in every single microservice container image, or we need to update all of our deployment manifests to mount a ConfigMap containing the root CA into every single pod. This is a lot of work, and is not scalable.

In order to solve this problem, we need a way to centrally manage trust stores and certificates, and distribute them to containers at runtime, without requiring a rebuild and redeploy of the container, and without requiring a change to the deployment manifest.

This is where cert-manager's trust-manager comes in.

trust-manager is a relatively new project, but aims to address this very issue. It enables a cluster administrator to centrally define a Trust Bundle which is then distributed out as a ConfigMap in all namespaces in the cluster. When the trust bundle is updated, the ConfigMap is updated as well, without the use of an additional tool such as kubernetes-replicator.

Let's look at an example Trust Bundle:

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
    name: example-bundle
spec:
    sources:
    - useDefaultCAs: true
    - secret:
        name: "trust-manager-example-ca-secret"
        key: "tls.crt"
    target:
    configMap:
        key: "trust-bundle.pem"
    additionalFormats:
        jks:
            key: "bundle.jks"

This uses the default public CAs, and joins them with the CA provided in the trust-manager-example-ca-secret, and distributes them as an example-bundle ConfigMap in all namespaces in the cluster.

This is great - it even includes the Java keystore! - however we aren't done yet. We still need to get the trust bundle into the containers in the pods.

If you are familiar with the cert-manager ecosystem, you've probably seen a deployment called cert-manager-cainjector. One may think this would solve our problem, but in fact this is solely responsible for managing CA certs for apiserver and in-cluster webhook services. It does not manage CA certs for pods.

cainjector closes this loop. cainjector is a Mutating Admission Webhook which injects the trust bundle into all pods in the cluster. It does this by mutating the pod spec and adding a volume mount for the trust bundle ConfigMap, and a volume mount for the trust bundle secret.

Additionally, it exports a SSL_CERT_FILE environment variable which points to the trust bundle secret, which is used by many applications to specify the location of the trust store.

This is all done with zero change to the deployment manifest, and zero change to the container image.

cainjector is configured with a small yaml ConfigMap:

---
apiVersion: v1
kind: ConfigMap
metadata:
    name: cainjector-config
    namespace: cert-manager
data:
    config.yaml: |
    configMap: my-org.com
    mountPath: /cacerts
    certFile: ca.crt
    excludeContainers:
    - istio-proxy
    excludeNamespaces:
    - kube-system
    - cert-manager
    - istio-system

excludeNamespaces enables an implicit inject and explicit exclude model, while includeNamespaces enables an explicit inject and implicit exclude model. excludeContainers is always respected.

Additionally, pods can use annotations to exclude or configure the trust bundle, overriding the defaults set in the ConfigMap:

---
apiVersion: v1
kind: Pod
metadata:
    name: my-pod
    annotations:
        cainjector.lestak.sh/configMap: "myCustomCA"
        cainjector.lestak.sh/mountPath: "/etc/ssl/certs"
        cainjector.lestak.sh/certFile: "my-ca.crt"
        cainjector.lestak.sh/sslCertFile: "/etc/ssl/certs/my-ca.crt"
        cainjector.lestak.sh/setEnvVar: "false"

The combination of cert-manager, trust-manager, and cainjector enables cluster administrators to centrally manage and distribute trust stores to all pods in the cluster, regardless of the container image used, CI/CD pipeline, or service mesh adoption.

last updated 2024-10-03