Blog
Kubernetes Secret Management at Scale (I): From Hardcoded Secrets to Sealed Secrets
1. The Problem: Hardcoded Secrets
The problem described here is very common in companies that have been using code repositories to track their code for a long time: exists a moment when they need to handle sensitive data, such as passwords, access tokens, private keys, etc. The real problem is where to store this information in a secured way.
In our case, we found that the early stages of the project had taken place at a time when there were no corporate requirements regarding how to handle this type of data, coupled with the fact that we were using a version of GitHub Enterprise Server hosted on our own servers. This had led to each team storing these secrets directly in their GitHub repositories, where they could be read by anyone with read access to the repository.
As the number of repositories grew, along with the number of people with access to them, it became a genuine security issue, as we were unable to control who could view such sensitive information. And let’s not forget that, on top of everything else, this is a research-focused area, meaning that the exposed information posed a risk from a corporate perspective.
Added to this is the ‘drawback’ that the DevOps engineering team adopted the GITOPS philosophy, which makes the information in repositories and external tools the sole source of truth regarding deployed applications. As our application deployment platform consists of Azure AKS clusters, this meant that creating secrets directly within the clusters was not an ideal approach whenever an alternative existed, as it could result in data loss if tools such as ArgoCD interpreted manually created components as being out of sync with the single source of truth.
We were faced with two problems to resolve:
- We wanted our secrets to be stored somewhere outside the clusters, and for their values to be synchronised in some way within the production environment.
- The secret values could not be accessible to the general public, who did have read access to the code repositories; viewing and modifying them had to be restricted to a select group of people with higher-level permissions.
2. First Option: Sealed Secrets
The selected option was to use SealedSecrets Operator. SealedSecrets is a tool designed to obfuscate secrets in such a way that they can only be decrypted at the destination where they were sealed. As an example, the tool is installed in a Kubernetes cluster. Using a client (kubeseal), we can generate the obfuscated value for a given secret specifically for that cluster, and even for a specific namespace.
From that point on, this sealed value can be stored as-is in any repository, since the value itself does not provide any meaningful information to a third party. Only when that SealedSecret is deployed into the target Kubernetes cluster will the tool decrypt it again and create a regular Secret that can be consumed by an application running in a Pod.
3. Our First Improvement
The method for creating a secret in the cluster is now as follows:
- A user with access privileges to the cluster uses `kubeseal` from any device on which this client can run to encrypt a secret value. This encrypted secret can only be decrypted using the tool installed on the AKS cluster.
- The encrypted value is stored in a file within the GitHub repository. This value—either directly in a manifest or via a deployment tool such as Helm—is used to create a SealedSecret object in the AKS cluster.
- The tool’s controller detects the SealedSecret in the cluster and, using its own installed certificates, decrypts it to create a Kubernetes Secret object.
- The value is now available for use by the application ‘pods’, which will utilise it as required for the configurations that need it
Now we have fixed the two problems that we had before:
- We can keep our configuration in Git without exposing the actual secret values, and information is synchronized to the cluster with an automatic process
- Only people with privileges can use kubeseal connected to the AKS cluster to seal values that will be decrypted once deployed.
4. But Now We Have Other Problems
What at the beginning was a big help ended up leading to other issues down the line, all of them mainly related to secret renewal and rotation:
- The kubeseal client does not work well on Windows and conflicted heavily with our security controls. Combined with the need to have direct access to the cluster, this created a clear bottleneck when it came to rotating or renewing secrets, due to that only high-privileged people could do the changes.
- When access to the clusters was further restricted to be only available from within Company’s internal network, the number of people who could update secrets was reduced even more. In some cases, ad-hoc tools were even created to generate the sealed values from pods mounted inside the cluster itself, but access remained costly and highly restricted.
- Code review bots frequently flagged SealedSecrets as exposed secrets in GitHub repositories, which conflicted with existing security policies.
5. What's Next?
Now that we have resolved the actual problem – the security lack produced by having secrets stored in plain text directly in the repository – we need to rethink the way in which we are synchronising our secrets. The issues that came with the new approach, while not affecting security, do add a significant degree of complexity when it comes to maintaining and rotating secrets.
Therefore, now that there is no longer any urgency, we are re-evaluating our approach and need to find a solution that:
- Enables the most centralised management of secrets possible, using tools that do not require a specific operating system.
- Secret management must be carried out only by staff with a higher level of privileges.
- Enables the widest possible automation of secret creation and rotation processes in the production environment.
- Does not expose secrets, even in encrypted form, in the repositories to facilitate the validation of published code.
To be continued…
Written by
Manuel Gallego
DevOps Engineer
DevOps Engineer
Our Ideas
Explore More Blogs
Embedding Requirement Agents Across Jira, Confluence and Enterprise Agile...
Requirement Agents deliver value far beyond generating superior requirements. When deeply integrated with Jira, Confluence, and Azure DevOps,...
Manoj Sharma
Contact