Kubernetes External Secrets
Teams at GoDaddy use the AWS managed Kubernetes offering, EKS, to deploy their services. We also use AWS Secrets Manager for storing secrets, like private keys and database passwords. EKS, however, does not provide much support for accessing Secrets Manager. Therefore, teams develop custom solutions for accessing secret data from their EKS clusters. One approach, for example, is to fetch secrets from Secret Manager when the application starts. The result is overlapping efforts for developing these custom solutions and because security is critical in this context, the individual efforts can require substantial engineering time.
This blog post describes a generalized approach and implementation for supporting secret management systems, like AWS Secrets Manager, in Kubernetes. We call this system Kubernetes External Secrets and we have open sourced our initial implementation. Our current solution focuses on making it easy for developers to manage secrets with AWS and deploy them to their EKS clusters, but we think our approach is general enough for other types of Kubernetes clusters and secret management systems.
The rest of this blog post describes the motivation for and design of Kubernetes External Secrets. The README.md has instructions for adding Kubernetes External Secrets to your cluster if you are looking to get started immediately.
Overview
Kubernetes has a built-in object for managing secrets called a
Secret. The
Secret object is convenient to use: it provides a declarative API
that makes it easy for application
Pods
to access secret data without any special code. One downside of
Secret objects is that they do not support storing or retrieving
secret data from external secret management systems. It’s often
beneficial, however, to use Kubernetes with an external service that
handles secret management and includes useful features. For
example, Secrets Manager integrates with other AWS services, like
Lambda functions, includes encryption at rest, and has a useful
mechanism for codifying rotation policies.
Kubernetes External Secrets aims to provide the same ease of use as
native Secret objects and provide access to secrets stored
externally. It does this by adding an ExternalSecret object to the
Kubernetes API that allows developers to inject external secrets into
a Pod using a declarative API similar to the native Secret
one.
Instead of inlining base64 encoded secret data into a Secret object,
developers define an ExternalSecret object that specifies a secret
management system and properties to load from that system.
For example, instead of using a Secret object to store a database
password:
apiVersion: v1
kind: Secret
metadata:
name: cats-and-dogs
type: Opaque
data:
password: d29vZgo=
a developer can use an ExternalSecret, specifying the secret
management systems as backendType and the properties to access in the
data array:
apiVersion: 'kubernetes-client.io/v1'
kind: ExtrenalSecret
metadata:
name: cats-and-dogs
secretDescriptor:
backendType: secretsManager
data:
- key: cats-and-dogs/mysql-password
name: password
and access that password from a Pod in the same way they would
if they were using a Secret object:
apiVersion: v1
kind: Pod
metadata:
name: cats-and-dogs
spec:
containers:
- name: cats-and-dogs
image: cats-and-dogs
env:
- name: SECRET_PASSWORD
valueFrom:
secretKeyRef:
name: cats-and-dogs
key: password
Notice that the ExternalSecret does not contain secret data. It’s
safe to store in plain text along with your other Kubernetes manifest
files for your service.
Design
Kubernetes External Secrets adds the ExternalSecret object to
Kubernetes using a
CustomResourceDefinition,
and adds an ExternalSecret controller we wrote in Node.js that implements
the behavior of the object type itself.
The ExternalSecret controller follows a familiar pattern seen in
other Kubernetes objects, like
Deployments:
users declare a desired state in an ExternalSecret object and the
controller creates or updates a complementary Secret object to reach
that state. The controller monitors ExternalSecret objects, fetches
secret data from the specified external secret management system, and automatically
creates native Secret objects that hold the secret data. The
architecture diagram below illustrates this process.
Using
You can add Kubernetes External Secrets to your cluster with a single command:
kubectl apply -f https://raw.githubusercontent.com/godaddy/kubernetes-external-secrets/master/external-secrets.yml
and begin manipulating ExternalSecret objects like you would any
other Kubernetes object type:
kubectl -n cats-and-dogs get es
If you’re interested in converting existing Secret objects to
ExternalSecret objects, one of the authors of Kubernetes External
Secrets wrote a Kubernetes External Secrets
CLI that
makes migration easy by converting a Secret to a series of AWS
CLI commands and a kubectl command that loads your secret data
into AWS Secrets Manager and creates a complementary ExternalSecret
object.
Other approaches
We experimented with and drew inspiration from other projects that help manage access to secret data from Kubernetes.
cmattoon/aws-ssm uses annotations on
Secret objects to identify properties in AWS Parameter Store and
populate the Secret object with those properties’ values.
kubesec makes it easy to encrypt
data before storing it in a Secret object or manifest. At run-time,
applications are responsible for decrypting the data before using it.
The Kubernetes Vault
Integration project
provides Vault auth tokens to Pod objects. Application code running
in a Pod can use that token to authenticate with Vault and retrieve
secret data.
We explored configuring nodes in our CICD pipeline to inject
Secret objects during testing and deployment. One benefit of this
approach is that Kubernetes clusters would not need to access the
external secret management system directly, which might help mitigate
data leaks during an attack. On the other
hand, it required the Kubernetes manifest files for a single
application to be stored in several locations and complicated
application debugging and deployment because developers would need to
use a combination of kubectl and specialized CICD tools.
Upcoming improvements
Kubernetes External Secrets supports AWS Secrets Manager and AWS Systems Manager, however, we believe the approach is general enough to support other external secret management systems and would be excited about working with community members to add them.
The Kubernetes External Secrets controller does not have a broad attack surface since it is accessible only via the Kubernetes API. We have worked, however, to tighten security by leveraging RBAC to restrict what the controller can access, using eslint-plugin-security to identify potentially vulnerable code patterns, and doing design reviews at GoDaddy. Nevertheless, we think it is important to continually improve security and take advantage of new features and patterns as they emerge. We plan on continuing to improve security by:
- restricting network access to staunch potential attacks that might try to leak secrets;
- facilitating rotation by triggering restarts when there is an update to an
ExternalSecret; - working through a threat model analysis; and
- adopting upcoming EKS features, like IAM roles for
Pods, that will help harden our implementation.
Conclusion
We hope that Kubernetes External Secrets can provide teams outside of
GoDaddy with a common way to access secret data in secret management
systems. Using Kubernetes External Secrets does not require any
application modifications and existing Pod specifications that
leverage Secret objects will work with ExternalSecret objects
without any changes.
If you want to get started with Kubernetes External Secrets or contribute please visit https://github.com/godaddy/kubernetes-external-secrets.
Acknowledgements
Celia Waggoner and Jarrett Cruger provided feedback on the Kubernetes External Secrets design and contributed to early versions of the implementation.
