How to overcome Docker Hub rate limiting using AWS ECR and AWS CloudFormation

In this blog post, we will explore how ECR and AWS CloudFormation can be used to address the rate limiting imposed by Docker Hub and provide full control over your base images.

The popular registry Docker Hub is home to thousands of useful container images, which are used by many software delivery processes. Unfortunately, the registry enforces a rate limit for anonymous and free-tier users. Whenever you try to pull an image from a AWS CodeBuild project, you will immediately run it this problem. AWS offers many Docker Hub images directly from their public AWS ECR registry https://public.ecr.aws, but not for all of them. So, when you want to use a public image not on offer, you have to find another way.

Overcome Docker Hub rate limiting

Our Custom CloudFormation Container Image Provider offers an effective workaround by allowing you to clone public images into a private Amazon Elastic Container Registry repository. By leveraging the custom provider, you can avoid the rate limit imposed by Docker Hub. Once in, you can pull the image as often as you want from your own ECR repository. This ensures a smooth and uninterrupted development process.

Full control over updating base images

Another advantage of using the Custom CloudFormation Container Image Provider is that you gain complete control over the base images. You can enable container image scanning and see which vulnerabilities live inside the public image. By using a CloudFormation template, you specify the exact image version you want.

Our container reference update utility – cru can be used to updates image references in the CloudFormation template and trigger the provisioning of the latest version to your ECR repository.

This effectively gives you a well defined provisioning process for container images.

Example usage

To demonstrate the usage of the Custom CloudFormation Container Image Provider, let’s consider the following CloudFormation template:

Resources:
  Repository:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: python

  Python37:
    Type: 'Custom::ContainerImage'
    Properties:
      ImageReference: docker.io/library/python:3.7
      RepositoryArn: !GetAtt Repository.Arn
      ServiceToken: !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:cfn-container-image-provider'

In this example, we clone the current repository from the public image ‘python:3.7’ into our ‘python’ repository in ECR. The ‘Repository’ resource creates the ECR repository, and the ‘Python37’ resource uses the custom resource ‘Custom::ContainerImage’ to clone the image.

Updating the image reference

To pin the image to a specific version, you can use the container reference update utility – cruas follows:

$ cru update \
        --resolve-digest --all \
        --matching-tag \
         demo.yaml

023/10/07 16:20:56 INFO: 1 image references found
2023/10/07 16:20:57 resolving repository docker.io/library/python Tag 3.7 to Digest sha256:eedf63967cdb57d8214db38ce21f105003ed4e4d0358f02bedc057341bcf92a0
2023/10/07 16:20:57 INFO: updated a total of 1 files
2023/10/07 16:20:57 INFO: no commit message, skipping commit and push

Now the container image reference will have the associated digest of the image, so you now exactly which image is used.

Python37:
    Type: 'Custom::ContainerImage'
    Properties:
      ImageReference: 'docker.io/library/python:3.7@sha256:eedf63967cdb57d8214db38ce21f105003ed4e4d0358f02bedc057341bcf92a0'

Installing the provider

To install this custom resource provider, type:

aws cloudformation create-stack \
       --capabilities CAPABILITY_IAM \
       --stack-name cfn-container-image-provider \
       --template-url s3://binxio-public-eu-central-1/lambdas/cfn-container-image-provider-0.2.3.yaml

aws cloudformation wait stack-create-complete \
       --stack-name cfn-container-image-provider

or use We recommend to install the provider on the private subnets in your VPC, to ensure that your NAT’s ip addresses are used to pull images from docker hub. Just pass in the VPC id, subnet ids and appropriate security groups.

Conclusion

The Custom CloudFormation Container Image Provider addresses two important challenges that developers and organisations face when working with container images. By cloning public images into your ECR repository, you can overcome the rate limit imposed by Docker Hub, and ensure uninterrupted access to the images you need. Additionally, you gain full control over which images are used in your organisation.

Photo by Zé Maria on Unsplash