Sometimes you need to run a post-deployment script to upgrade a database or configure your applications. For imperative deployments you just add a new deployment script task. For declarative deployments, such as Kubernetes deployments, you’ll have to wait for the deployment to complete. This blog shows you how to run a post-deployment script on Kubernetes.
Don’t Wait. Run A New Container
This solution runs the script in a new container instance. Because it’s a new container, you don’t have to wait for the deployment to complete. This is my prefered solution, because it allows you to run any type of container and assign script specific resources.
kubectl run the-update-task --attach --restart=Never --rm
--image ubuntu:latest --command -- bash -c "echo "Call your script here..""
if [ $? -ne 0 ]; then
# The update script failed, do something.
fi
Note that this command uses a regular
Pod
. We would like to use aJob
instead. Jobs, however, are not cleaned up by default. So you’ll need to write the scripting to poll and remove the job.
While this is incredibly easy, you run the containers imperatively (from the command line). Therefore, you likely specify the same container configuration as in your declarative (yaml-file) deployment files. Let’s explore other solutions to run your script that reuse your declarative deployment.
Check For Rollout And Pod Status
This solution runs the script on an existing workload, immediately after a deployment. It relies on kubectl rollout status to wait for the deployment to complete. Sadly, this command doesn’t wait for all containers to terminate. To prevent you from exec’ing into a terminating container, it polls the pod state before running the script.
kubectl apply -f my-deployment.yaml
# Wait for rollout to complete..
kubectl rollout status deploy/my-deployment-name --watch=true
# Wait for shutdown of terminating pods..
SELECTOR=$(kubectl get deploy/my-deployment-name -o wide --no-headers | awk '{print $NF}')
while :
do
POD_STATES=$(kubectl get pods --selector ${SELECTOR} --no-headers | awk '{print $3}' | uniq)
if [[ "$POD_STATES" == "Running" ]]; then
break
fi
sleep 5
done
# Run the update script..
kubectl exec deploy/my-deployment-name -- echo "Call your script here.."
Reuse Terraform Provider Behavior
If you deploy using the Terraform Kubernetes provider, you are in luck. The kubernetes_deployment
-resource exposes the wait_for_rollout-attribute for you. As a result, you don’t have to write or maintain any scripts.
terraform apply
kubectl exec deploy/my-deployment-name -- echo "Call your script here.."
Note that the Kubernetes provider contains a race condition that affects the
wait_for_rollout
behavior. I assume that this is fixed in the near future, as a fix is already proposed for the issue.
Conclusion
There are many ways to run a post-deployment script on Kubernetes. I prefer to run a new container because of it’s ease of use. Find your preferred way by balancing your tools, maintenance effort and ease of use.
Photo by Birger Strahl on Unsplash