Fixing oversized artifacts AWS CDK Pipelines

I built a workload using AWS CDK, and the CodePipeline stopped working at the worst moment. It was right at the end of the sprint; we had done multiple deployments before. But at this moment, the moment you might recognize. When this last PR is released, we will have met our deadline! Kaboom, the pipeline stops working, and the reason for the failure is not related to the changes you made.

Some background

When you use CDK, you can write infrastructure as code using your favorite programming language. CDK comes with certain constructs that make your life easier. For example, you can easily create a CodePipeline that will fetch the code from your repository. It will then synthesize the CDK code into CloudFormation. The CodePipeline will then upload all assets to an S3 bucket, and the CloudFormation templates will be deployed to the target accounts. This CloudFormation template will contain references to the assets on S3. When the template is deployed, the assets are fetched from the S3 Bucket.

What has happened?

I ran into the problem that CodePipeline will pass an artifact from step to step. This artifact is a single zipped file, which makes it easy to pass it along the pipeline.

When CodePipeline wants to deploy the CloudFormation templates. It uploads the zipped file, and it knows where in the zipped file the template is located. However, all other assets are also in the zipfile; this could be the code that you want to run in Lambda functions or even Lambda Layers. You can imagine that when your workload starts to become more mature, there will be more assets in this zipfile.

Why is that a problem?

Well, the artifact has exceeded the maximum size! With CDK projects, this is something that you will run into sooner or later. Usually, you only add stuff to your projects and rarely remove things. And, as you already learned, this always happens at a moment that you don’t expect at all. And in my case, at the worst possible time, right before my deadline.

How can we fix this?

The answer is quite simple. As I already explained, the problem is with the size of the artifact being passed to the CloudFormation deploy action. We just have to reduce the size after all the assets in the artifact are uploaded to S3.

private stripAssets(pipeline: pipelines.CodePipeline) {
 const policies = [
   new iam.PolicyStatement({
     effect: iam.Effect.ALLOW,
     resources: [<code>arn:aws:s3:::${this.artifactBucket.bucketName}/*],
     actions: ['s3:PutObject'],
   }),
 ];

 if (this.artifactBucket.encryptionKey) {
   policies.push(
     new iam.PolicyStatement({
       effect: iam.Effect.ALLOW,
       resources: [this.artifactBucket.encryptionKey.keyArn],
       actions: ['kms:GenerateDataKey'],
     }),
   );
 }
 pipeline.addWave('BeforeStageDeploy', {
   pre: [
     new pipelines.CodeBuildStep('StripAssetsFromAssembly', {
       input: pipeline.cloudAssemblyFileSet,
       commands: [
         'S3_PATH=${CODEBUILD_SOURCE_VERSION#"arn:aws:s3:::"}',
         'ZIP_ARCHIVE=$(basename $S3_PATH)',
         'echo $S3_PATH',
         'echo $ZIP_ARCHIVE',
         'ls',
         'rm -rfv asset.*',
         'zip -r -q -A $ZIP_ARCHIVE *',
         'ls',
         'aws s3 cp $ZIP_ARCHIVE s3://$S3_PATH',
       ],
       rolePolicyStatements: policies,
     }),
   ],
 });
}

What happens in the code sample?

• We make sure that we have the s3:PutObject rights on the artifact bucket. (We need to overwrite the existing artifact.)
• We need the kms:GenerateDataKey rights. (Optional, I am using KMS encryption on the bucket.)
• We add a CodeBuildStep that will consume the artifact.
• The artifact is downloaded and unzipped by CodeBuild.
• We figure out the origin of the artifact.
• We remove all assets.
• We will zip the artifact.
• We will upload the artifact over the original.

Because we placed the CodeBuildStep in the pre-section it will be placed right before the deployment of the CloudFormation deploy action.

Conclusion

Hitting CodePipeline’s artifact limit can stall your sprint at the worst moment. By inserting a pre-deploy CodeBuildStep that strips the bulky assets after they’re safely uploaded to S3, you shrink the artifact without changing your workflow. This quick fix keeps your CDK pipeline flowing, your CloudFormation stacks deploying, and your release deadlines intact.

Photo by Ellie Burgin