Kubernetes Secret Management at Scale (II): Moving Beyond SealedSecrets with Azure Key Vault

In the previous chapter we explained how we stopped storing secrets in plain text in our GitHub repositories and started using SealedSecrets. That change solved the most urgent problem we had: the real values were not visible anymore for anyone with read access to the code. For some time, this was enough.

But when a solution is used every day, by different teams and in different clusters, the small problems start to appear. In our case, all of them appeared at the same moment: when a secret had to be created, renewed or rotated. The kubeseal client did not work well on Windows and had conflicts with our security controls. Access to the clusters was restricted to the company's internal network, so only a few people could seal a new value, and some of them had to use ad-hoc tools running inside the cluster to do it. And the code review bots kept flagging the sealed values as exposed secrets, even though they were encrypted. None of these problems was a security issue by itself, but together they turned a simple password change into a small project.

At the end of the first chapter we wrote down what we expected from the next solution. We wanted to manage secrets from a central place, with tools that did not depend on a specific operating system. Only people with higher privileges should be able to manage them. The creation and rotation of secrets in production should be automated as much as possible. And the repositories should not contain secrets anymore, not even in encrypted form.

This article is about the discussions we had after that. It is not yet about tools. It is about how we decided to approach the problem, and about one detail that we underestimated at the beginning: the credentials needed to read the secrets.

Changing the question

With SealedSecrets, the question we had answered was "how can we store secrets in Git in a safe way?". When we started to analyse the new requirements, we realised that the last one changed everything. If the repositories should not contain secrets, even encrypted, then the question was a different one: how can we keep secrets outside Git and still have them in the clusters when the applications need them?

This was not a small change for us. Our DevOps team works following GitOps, and ArgoCD uses the information in the repositories as the single source of truth for what is deployed. As we explained in the first chapter, this was one of the reasons why we could not just create the secrets manually in the clusters. Anything created outside Git could be seen by ArgoCD as out of sync, and could be lost.

The solution was to separate two things that until then had always been together: the definition of a secret and its value. Git can still describe which secret an application needs, in which namespace it must exist and which component is going to use it. That is configuration, and it is perfectly fine to have it in a repository, reviewed and versioned like the rest of the code. What Git does not need to know is the password itself.

So we would have two sources of truth instead of one. Git for the desired state of the infrastructure and the applications, and an external secret store for the sensitive values. It may sound like a contradiction with GitOps, but in practice it is not. ArgoCD continues to deploy exactly what is defined in the repository, and the repository continues to describe the full deployment. It only points to the value instead of containing it.

There was another advantage that we discussed a lot in the team. With this model, the person who deploys an application does not need to know the database password, and the person who rotates that password does not need to touch a Helm chart. Every time a secret is copied, pasted into a file or typed in a terminal, there is a new chance of making a mistake or exposing it by accident. Reducing those moments was, for us, as important as the encryption itself.

A central vault: Azure Key Vault

Once we agreed that the values should live outside Git, the next question was where. The answer was quite easy. Our platform runs on Azure, all our applications are deployed in AKS clusters, and Azure Key Vault was already available inside the company. After some discussions in the team, we decided to use it as the central store for all our secrets.

Key Vault gave us what SealedSecrets could not. Access is controlled from Azure, with the same identity and permission model that we already use for the rest of the infrastructure, so we can decide exactly who can read or change each secret. It does not need any client installed on the user's laptop, so the problems with Windows disappear. And rotation becomes much simpler, at least in theory: the value is changed in one place, instead of generating a new sealed value for each cluster and committing it to several repositories.

That "in theory" is important. Having the secrets in Key Vault solves where they are stored. It does not solve how they reach the pods.

How do the values reach the clusters?

This was the longest part of our discussions. Before looking at specific tools, we tried to understand which questions the solution had to answer, because we had already learnt with SealedSecrets that a tool that looks good at the beginning can create a lot of work later.

The first question was who brings the value into the cluster. One option is that the value is injected from outside, during the deployment, by the same process that applies the manifests. The other option is that something running inside the cluster reads the value from Key Vault and creates it locally. The first one is simple and fits well with GitOps, but it makes secret management depend on the deployment tool. The second one is more independent, but it means that each cluster must be able to talk to Key Vault by itself.

The second question was how applications consume the value. Most of our applications, and especially third-party products like Redis, MongoDB, PostgreSQL or Superset, are deployed with Helm charts that expect a standard Kubernetes Secret. A solution that delivers secrets in a different way, for example as files mounted in a volume, could force us to modify those charts, and that is something we wanted to avoid because it makes future upgrades harder.

The third question was what happens after a rotation. Updating a value in Key Vault is easy. Updating the Kubernetes Secret is also possible. But many applications read their configuration only when the container starts, so a pod that is already running continues using the old password. If the solution did not take this into account, every rotation would still need a manual restart.

The last question was scale. We are not talking about one application in one cluster. We have several AKS clusters, each one with many namespaces and many applications. Any step that has to be repeated by hand for each namespace or cluster is a step that will be forgotten at some point.

These four questions became our criteria to evaluate any tool.

The secret behind the secrets

There was one more question, and it took us some time to see how important it was.

To read a secret from Azure Key Vault, any component needs to authenticate first. In our case this means credentials, for example a Service Principal with permissions on the vault. And those credentials are, of course, a secret too. So before the cluster can retrieve the secrets, it needs a secret to retrieve the secrets. We started calling this the bootstrap problem.

The first secret cannot be obtained with the mechanism that depends on it. If a tool synchronises secrets from Key Vault, it cannot create its own credentials to access Key Vault, because it needs them before it can do anything. Something else has to put them there.

In a single cluster this looks like a minor detail. An administrator creates the credentials manually once, and the synchronisation takes over from that moment. But Kubernetes Secrets are namespace-scoped: a secret created in one namespace cannot be referenced from another one. So if the credentials are needed in ten namespaces, they have to exist ten times. With several clusters, the number multiplies again.

And then the credentials expire, or must be rotated because of security policies. Every copy must be updated, in every namespace, in every cluster, by hand. This is exactly the type of work we wanted to eliminate when we left SealedSecrets, only now hidden in a place where it is easy to forget. A centralised secret store should reduce duplication, not create a new set of copies that nobody manages.

So the bootstrap problem is not only about starting the system for the first time. It is about the full lifecycle of the credentials that make the whole system possible. We also accepted that it cannot be eliminated completely. At some point there must be a first trust relationship, created by a privileged person. Our objective was to reduce that manual step to the minimum, ideally to one single moment, and to manage everything after it automatically.

What we were looking for

At the end of this analysis we had a much clearer picture of the architecture we wanted, even before choosing any tool.

Git would define the applications and which secrets each one needs, but never the values. Azure Key Vault would store all the sensitive information, including the credentials needed to access it. A bootstrap mechanism would place those credentials in the clusters and keep them updated. A synchronisation mechanism would read the application secrets from Key Vault and create standard Kubernetes Secrets in the right namespaces. And something would make sure that running applications start using a new value after a rotation, without anybody restarting them by hand.

What we did not have yet was the tools. And in a company like ours, the choice is not completely free. Security policies, support and the experience of other teams have a strong influence, and there was some urgency to have a solution soon, so we did not want to build anything from scratch. Inside the organization there were three options already approved and working in other projects: ArgoCD Vault Plugin, the Secrets Store CSI Driver and External Secrets Operator.

In the next chapter we will explain how each of them fitted, or did not fit, in this picture, and the combination we finally put in production.