Blog

How To Run A Post-Deployment Script On Kubernetes

18 Jan, 2022
Xebia Background Header Wave

Sometimes you need to run a post-deployment script to upgrade a database or configure your applications. For imperative deployments you just add a new deployment script task. For declarative deployments, such as Kubernetes deployments, you’ll have to wait for the deployment to complete. This blog shows you how to run a post-deployment script on Kubernetes.

Don’t Wait. Run A New Container

This solution runs the script in a new container instance. Because it’s a new container, you don’t have to wait for the deployment to complete. This is my prefered solution, because it allows you to run any type of container and assign script specific resources.

kubectl run the-update-task --attach --restart=Never --rm 
    --image ubuntu:latest --command -- bash -c "echo "Call your script here..""

if [ $? -ne 0 ]; then
    # The update script failed, do something.
fi

Note that this command uses a regular Pod. We would like to use a Job instead. Jobs, however, are not cleaned up by default. So you’ll need to write the scripting to poll and remove the job.

While this is incredibly easy, you run the containers imperatively (from the command line). Therefore, you likely specify the same container configuration as in your declarative (yaml-file) deployment files. Let’s explore other solutions to run your script that reuse your declarative deployment.

Check For Rollout And Pod Status

This solution runs the script on an existing workload, immediately after a deployment. It relies on kubectl rollout status to wait for the deployment to complete. Sadly, this command doesn’t wait for all containers to terminate. To prevent you from exec’ing into a terminating container, it polls the pod state before running the script.

kubectl apply -f my-deployment.yaml

# Wait for rollout to complete..
kubectl rollout status deploy/my-deployment-name --watch=true

# Wait for shutdown of terminating pods..
SELECTOR=$(kubectl get deploy/my-deployment-name -o wide --no-headers | awk '{print $NF}')

while :
do
    POD_STATES=$(kubectl get pods --selector ${SELECTOR} --no-headers | awk '{print $3}' | uniq)
    if [[ "$POD_STATES" == "Running" ]]; then
        break
    fi

    sleep 5
done

# Run the update script..
kubectl exec deploy/my-deployment-name -- echo "Call your script here.."

Reuse Terraform Provider Behavior

If you deploy using the Terraform Kubernetes provider, you are in luck. The kubernetes_deployment-resource exposes the wait_for_rollout-attribute for you. As a result, you don’t have to write or maintain any scripts.

terraform apply
kubectl exec deploy/my-deployment-name -- echo "Call your script here.."

Note that the Kubernetes provider contains a race condition that affects the wait_for_rollout behavior. I assume that this is fixed in the near future, as a fix is already proposed for the issue.

Conclusion

There are many ways to run a post-deployment script on Kubernetes. I prefer to run a new container because of it’s ease of use. Find your preferred way by balancing your tools, maintenance effort and ease of use.

Photo by Birger Strahl on Unsplash

Laurens Knoll
As a cloud consultant I enjoy taking software engineering practices to the cloud. Continuously improving the customers systems, tools and processes by focusing on integration and quality.
Questions?

Get in touch with us to learn more about the subject and related solutions

Explore related posts