Our experience with resources on secure deployment
Have you ever searched for resources on “Secure Software Deployment”? Most of the results revolve around the pentesting or putting security tools in your CI/CD pipeline. It would be the same as researching how to improve your cake baking skills, but end up with manuals of kitchen appliances. We want to address this gap: in this blog, we want to give you key pointers for a secure deployment.
So, what should you think of? We’ll start with a few aspects that we believe are important to think of when you work on a secure deployment. After that, we will touch upon the areas that you need to work on to actually achieve it. Finally, we’ll advise where to go from here.
Components of a secure deployment
Deployment of your software normally focuses on the migration of your artifacts to a given target environment. Such a deployment often requires instructions, which are present in code. Often the build process is still part of the deployment, as some artifacts require additional build steps in the target environment.
This leaves us with four main components that are relevant for a deployment:
- the instructions for the artifact and its deployment as code;
- building the artifact;
- deploying the build artifact to the target environment;
- the target environment.
Here, we’ll focus on the deployment itself – secure build and secured artifacts are building blocks that the deployment depends on.
The build tools you need, the way you want to code, and how you integrate code changes, are topics worthy of their own articles. Let’s focus on the deployment step instead. So what would you need?
Requirements, requirements, requirements…
Let’s start with an important note: doing secure deployment over a secure CI/CD pipeline is hard. It takes continuous effort, involving improvement of the security of the components we just mentioned. To get you started, we compiled a list of pointers based on our experience and the resources found at the bottom of this blog.
We’ll move on to what you should take care of, covering process, technology, and people:
- Formalization & repeatability: first make sure that you understand what you want to achieve. Either document this in a formal process or in code -or both, together-, to be able to automate the building and the deployment process.
- Reproducibility: every deployment should be reproducible. This becomes clear when you want to replay an incident in a different environment. You want to find out what went wrong, and why. This means that you’ll have to store some of the produced artifacts for a longer period of time. Alternatively, an artifact deployment process should have guarantees to end up in the same state with the exact same properties as when you deployed said artifacts previously.
- Possibility to rollback: this is often overlooked, but if you cannot rollback from a compromising deployment… You are in trouble.
- Provenance & traceability: in order to make sure that you know who or what started a task, you must be sure that you can trace the origin – whether it was a commit, an approval of a pull/merge-request, or a deployment approval. So, make sure that this applies to your audit logging and build or deployment pipeline.
- Access management: a DevOps engineer should not be allowed to just deploy anything anywhere. There should be segregation of duties in place. This holds for the technology in use as well: you should make sure that your means to deploy (e.g. agent/runner) should only be allowed to deploy to the designated target environment. For instance: when your runner deploys a container to data-science test-environment, that same runner should not be authorized to deploy to your production environment at the same time.
- Integrity verification: there are a few places where integrity matters:
- the integrity of the artifact when retrieved/made ready for deployment
- the integrity of the artifact after deployment to the target environment
- In case you have a mutable artifact, the integrity of the configuration instructions as well as the target result are key.
- Automated security checks: you can have various automated security checks to detect “low hanging fruit”, often spread over the build, deployment, and post-deployment phase. Think of dependency checks, SAST, DAST, docker container checks, host-audits, and/or cloud provider configuration security checks. Where you put which check depends on the check, the nature of the artifact, and the target environment.
- Security of the build & deployment pipeline: there are many things to say about the security of your CI/CD pipeline. It is your “production environment” which should be taken care of just as good as your actual production environment. You can find more pointers in this presentation.
- How you manage your secrets: of course, you should be careful with what you make public. Whether it is your intellectual property in source code, or the Personally Identifiable Information of your users. The actual secrets that require additional protection in this phase should be your authentication, encryption, and signing means. Think of deployment credentials, other passwords, keys, and such. Verify you deploy them securely, use temporary credentials where possible or otherwise rotate the secrets on a timely basis.
- Manual verification (or “4 eyes”): tools do not find everything, certainly not when starting out. Make sure you have manual verification in place. Where this happens depends on your process. Often, this is at the pull/merge request level or there’s a deployment approval. Make sure that the author of the code is not the one who does the manual verification. Otherwise they can approve the deployment of their own malware.
- Training: train your engineers to understand the goals you want to achieve with a secure deployment and enable them to use the tooling well. In fact, it might be a good idea to train them to build the pipeline themselves for best results.
- Recognize and report: your engineers should be able to recognize and report on bypassing the security controls. Reward them for the reporting!
Of course, there are many more opinionated aspects of a secure deployment. For instance: should your artifact be immutable at the target environment? What should be your artifact promotion strategy? (E.g. which environments do you have outside of production? and how do you propagate through?). And then we need to talk about the amount of testing of each phase of your build- and deployment needs to take place.
Where to go from here
These were a lot of pointers to start with! What should you do now?
- Start risk based. Make a threat model for your SDLC process and take care of the pointers on a scored risk basis.
- CI/CD security is complex and a continuous team effort. Make sure you embark on this journey together.
- Don’t get trapped in analysis paralysis. Start small, but more importantly: start!
If you have any questions, remarks, or additions, don’t hesitate to reach out!