How to define AWS ECS task image definitions to play nice with ECS new version consistency behaviour

In this blog I will provide two ways to work with the new ECS image version consistency behaviour: Either enable the ECR tag immutability or add the image digest to all your task definition images.

ecs version consistency

Since June 25, 2024, Amazon Elastic Container Service (Amazon ECS) enforces software version consistency for containerised applications. ECS no longer pulls images based on the tag that is defined in the task definition, but on the digest of the container image used for the first task. This ensures that all tasks of the same task definition, are launched with the exact same image. Moving the tag to a new image, will NOT effect the existing task definition.

This behaviour prevents the use of floating tags for images and may cause an outage if the tag is moved and the previous image it pointed to is deleted. If that is the case, you will see errors like:

level=warning msg="reference for unknown type: " digest="sha256:52f6876d1f340f2b5991105854dd541e0b1aed76eb9cfe963fc95bec90265a86" remote="<ACCOUNT>.dkr.ecr.<REGION>.amazonaws.com/<REPOSITORY>@sha256:52f6876d1f340f2b5991105854dd541e0b1aed76eb9cfe963fc95bec90265a86"
level=info msg="Attempting next endpoint for pull after error: manifest unknown: Requested image not found"

To avoid this problem there are two possible solutions:

• Enable tag immutability on your ECR image repository.
• Add image digests to the image references in the task definition.

Tag immutability

Tag immutability would be cool as it prevents tags from being moved, but unfortunately, it is an all-or-nothing switch. However, it is customary for container images to have symbolic tags to point to the latest image, the latest major version or the last minor version. The latest tag is for instance used for integration tests, where as the major and minor version tags allow for security updates to be applied. For instance, at the moment of writing, the following tags:

alpine:latest
alpine:3
alpine:3.20

all point to the version 3.20.1 of the alpine image. Tag immutability does not provide enough fine grained control to allow for this. So, to work with the new behaviour, we recommend to update your image references in the task definitions with the image digest.

Update image references with digest

The docker container image reference allows you to specify both the tag and the digest. The digest takes precedence, so the tag is just there as meta information. For example, the image reference:

mvanholsteijn/paas-monitor:3.1.0

can be written as:

mvanholsteijn/paas-monitor:3.1.0@sha256:c0717cab9....

An image definition with a digest ensures that the task definition matches the new ECS software version consistency behaviour.

add digest to container image reference

Back in 2020, Wietse Venema and I created a utility to update container references to include the digest while preserving the tag as a human readable name. For example, given the following task definition:

{
    "taskDefinitionArn": "arn:aws:ecs:eu-central-1:123456789012:task-definition/paas-monitor:2",
    "containerDefinitions": [
        {
            "name": "paas-monitor",
            "image": "mvanholsteijn/paas-monitor:3.1.0"

You can update and add the digest to the container image reference, by typing:

$ cru update --image-reference mvanholsteijn/paas-monitor:3.4.0 --resolve-digest .

2024/07/29 07:56:29 resolving repository mvanholsteijn/paas-monitor Tag 3.4.0 to Digest sha256:fdcfbed7e0a7beb8738e00fe8961c8e33e17bdeee94eab52cb8b85de1d04d024

2024/07/29 07:56:29 INFO: updating reference mvanholsteijn/paas-monitor:3.1.0 to mvanholsteijn/paas-monitor:3.4.0@sha256:fdcfbed7e0a7beb8738e00fe8961c8e33e17bdeee94eab52cb8b85de1d04d024 in paas-monitor.json

As you can see, cru updates the image reference and adds the digest:

{
    "taskDefinitionArn": "arn:aws:ecs:eu-central-1:123456789012:task-definition/paas-monitor:2",
    "containerDefinitions": [
        {
            "name": "paas-monitor",
            "image": "mmvanholsteijn/paas-monitor:3.4.0@sha256:fdcfbed7e0a7beb8738e00fe8961c8e33e17bdeee94eab52cb8b85de1d04d024"

Now the image reference includes both the tag and the digest, ECS will ignore the tag and use the digest, while for humans the semantic relevance of the tag is still in place.

Conclusion

Amazon ECS version consistency ensures that all ECS tasks of a task definition use the exact same container image when deployed to a cluster. Unfortunately this change was unannounced and broke quite a number of workflows out there in the real world. It would have been nice if AWS made this behaviour available as a feature toggle on the ECS task definition or ECS Cluster. Another improvement would be if the tag immutability could be selective based on the tag.

To avoid surprise enable tag immutability on your ECR repositories, or include the digest in the container image references of your ECS task definition.

Image by Monica Volpin from Pixabay