kubernetes enterprise ca distribution
Within kubernetes clusters,
cert-manager is the de facto standard for managing certificates.
Coupled with a service mesh such as
Istio, it is possible to automatically distribute certificates and trust chains to all pods in a cluster, and the service mesh will handle all intra-mesh communication, TLS termination, and certificate validation.
However an issue arises when workloads inside of Kubernetes must communicate out to services outside of the cluster/mesh which are secured with an internally signed certificate.
When distributing an internal root CA, one may use GPOs, Ansible, Chef, Jamf, etc. to distribute the root CA to all machines in the enterprise. This is a well understood problem, and there are many solutions to it.
However in a containerized environment, by default every container manages its own trust store, usually inherited from the base image used to build the container. Development teams rarely touch the trust store outside of possibly doing the equivalent of
update-ca-certificates
in the container's Dockerfile. This is all done at container build time, which means any changes to the trust store require a rebuild and redeploy of the container.
From an immutable artifact perspective this is ideal, but from an enterprise operational perspective this significantly limits the ability to centrally manage trust stores and certificates.
We either need to bake in the current root CA in every single microservice container image, or we need to update all of our deployment manifests to mount a ConfigMap containing the root CA into every single pod. This is a lot of work, and is not scalable.
In order to solve this problem, we need a way to centrally manage trust stores and certificates, and distribute them to containers at runtime, without requiring a rebuild and redeploy of the container, and without requiring a change to the deployment manifest.
This is where
cert-manager's trust-manager comes in.
trust-manager is a relatively new project, but aims to address this very issue. It enables a cluster administrator to centrally define a
Trust Bundle
which is then distributed out as a
ConfigMap
in all namespaces in the cluster. When the trust bundle is updated, the ConfigMap is updated as well, without the use of an additional tool such as
kubernetes-replicator.
Let's look at an example
Trust Bundle
:
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
name: example-bundle
spec:
sources:
- useDefaultCAs: true
- secret:
name: "trust-manager-example-ca-secret"
key: "tls.crt"
target:
configMap:
key: "trust-bundle.pem"
additionalFormats:
jks:
key: "bundle.jks"
This uses the default public CAs, and joins them with the CA provided in the
trust-manager-example-ca-secret
, and distributes them as an
example-bundle
ConfigMap in all namespaces in the cluster.
This is great - it even includes the Java keystore! - however we aren't done yet. We still need to get the trust bundle into the containers in the pods.
If you are familiar with the
cert-manager
ecosystem, you've probably seen a deployment called
cert-manager-cainjector
. One may think this would solve our problem, but in fact this is solely responsible for managing CA certs for apiserver and in-cluster webhook services. It does not manage CA certs for pods.
cainjector closes this loop.
cainjector
is a Mutating Admission Webhook which injects the trust bundle into all pods in the cluster. It does this by mutating the pod spec and adding a volume mount for the trust bundle ConfigMap, and a volume mount for the trust bundle secret.
Additionally, it exports a
SSL_CERT_FILE
environment variable which points to the trust bundle secret, which is used by many applications to specify the location of the trust store.
This is all done with zero change to the deployment manifest, and zero change to the container image.
cainjector
is configured with a small yaml ConfigMap:
---
apiVersion: v1
kind: ConfigMap
metadata:
name: cainjector-config
namespace: cert-manager
data:
config.yaml: |
configMap: my-org.com
mountPath: /cacerts
certFile: ca.crt
excludeContainers:
- istio-proxy
excludeNamespaces:
- kube-system
- cert-manager
- istio-system
excludeNamespaces
enables an implicit inject and explicit exclude model, while
includeNamespaces
enables an explicit inject and implicit exclude model.
excludeContainers
is always respected.
Additionally, pods can use annotations to exclude or configure the trust bundle, overriding the defaults set in the ConfigMap:
---
apiVersion: v1
kind: Pod
metadata:
name: my-pod
annotations:
cainjector.lestak.sh/configMap: "myCustomCA"
cainjector.lestak.sh/mountPath: "/etc/ssl/certs"
cainjector.lestak.sh/certFile: "my-ca.crt"
cainjector.lestak.sh/sslCertFile: "/etc/ssl/certs/my-ca.crt"
cainjector.lestak.sh/setEnvVar: "false"
The combination of
cert-manager
,
trust-manager
, and
cainjector
enables cluster administrators to centrally manage and distribute trust stores to all pods in the cluster, regardless of the container image used, CI/CD pipeline, or service mesh adoption.
last updated 2024-03-18