Reduce AWS Costs with S3 Bucket Keys

Encrypting objects in S3 buckets is one of the best security practices to date. However, encrypting can lead to additional costs when AWS Key Management Service (KMS) is used with S3 objects. And it may also impact S3 performance if your application communicates too often with KMS. What if you could reduce traffic between S3 and KMS and still get the benefits of KMS for your security compliance?

On December 1st 2020, AWS announced a new S3 bucket key feature for bucket encryption. By enabling it we can reduce the request costs of Amazon S3 server-side encryption (SSE) with AWS Key Management Service (KMS) by up to 99% by decreasing the request traffic from S3 to KMS without making changes to client applications. 

Various applications, websites and mobile apps access millions of objects encrypted with SSE-KMS and this understandably generates large volumes of requests from S3 to AWS KMS. When we use SSE-KMS to protect data without an S3 Bucket Key, Amazon S3 uses an individual AWS KMS data key for every object. It makes a call to AWS KMS every time a request is made against a KMS-encrypted object.

By configuring an S3 Bucket Key for SSE-KMS on new objects, AWS KMS generates a bucket-level key that is used to create unique data keys for objects in the bucket. This key is used for a time-limited period within Amazon S3, reducing the need for Amazon S3 to make requests to AWS KMS to complete encryption operations. This reduces traffic from S3 to AWS KMS, allowing you to access AWS KMS-encrypted objects in S3 at a fraction of the previous cost.

We can enable the keys with no additional cost and it is available in all regions, allowing a cost saving potential for all customers.

Before you enable an S3 Bucket Key

When you enable an S3 bucket key, S3 uses the bucket Amazon Resource Name (ARN) as the encryption context instead of the object ARN. If you have IAM policies or AWS KMS key policies, use your object Amazon Resource Name (ARN) as the encryption context to refine or limit access to your AWS KMS CMKs, as these policies won’t work with an S3 Bucket Key. Again, these keys use the bucket ARN as encryption context. Before you enable an S3 Bucket Key, update your IAM policies or AWS KMS key policies to use your bucket ARN as encryption context.

AWS KMS CloudTrail events log your bucket ARN instead of your object ARN. Additionally, you will see fewer KMS CloudTrail events for SSE-KMS objects in your logs. Because key material is time-limited in Amazon S3, fewer requests are made to AWS KMS.

Enabling S3 Bucket Keys

We can configure buckets to use an S3 key for SSE-KMS on new objects using five different methods: AWS Console, CloudFormation, SDK, REST API and CLI.

1. AWS Console

We can use the AWS S3 Console to enable a bucket key for a new or existing bucket. In order to do so, go to your bucket and select Properties → Default Encryption.

2. CloudFormation

With CloudFormation, we can use the property BucketKeyEnabled, when using encryption for the resource AWS::S3::Bucket.

AWSTemplateFormatVersion: '2010-09-09'
Description: S3 bucket with default encryption
Resources:
  EncryptedS3Bucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      BucketName:
        'Fn::Sub': 'testaccount-${AWS::Region}-${AWS::AccountId}'
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: 'aws:kms'
              KMSMasterKeyID: <KMS-KEY-ARN>
            BucketKeyEnabled: true
    DeletionPolicy: Delete

For PutBucketEncryption and GetBucketEncryption: ServerSideEncryptionRule accepts the BucketKeyEnabled parameter for enabling and disabling an S3 Bucket Key, and it also returns a setting for encryption.

PutObject, CopyObject, CreateMutlipartUpload, and PostObject:

Use x-amz-server-side-encryption-bucket-key-enabled request header to enable or disable an S3 Bucket Key at the object level.

HeadObject, GetObject, UploadPartCopy, UploadPart, and CompleteMultipartUpload:

x-amz-server-side-encryption-bucket-key-enabled response header indicates if an S3 Bucket Key is enabled or disabled for an object.

4. CLI

The example below enables a default bucket encryption with SSE-KMS and an S3 Bucket Key using the CLI.

aws s3api put-bucket-encryption --bucket <bucket-name> --server-side-encryption-configuration '{
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms",
                    "KMSMasterKeyID": "<KMS-Key-ARN>"
                },
                "BucketKeyEnabled": true
            }
        ]
    }'

5. SDK

We can enable this parameter at object level also, e.g with the Python example below:

import boto3

s3 = boto3.resource('s3')

data = open('test.jpg', 'rb')

s3.Bucket('my-bucket').put_object(Key='test.jpg', Body=data).bucket_key_enabled(True)

Final Thoughts

Overall, without additional costs, no changes to the client-side application and with an easy configuration change, this feature is a quick win to reduce the request costs of Amazon S3 server-side encryption (SSE) with KMS.

For references and further reading:

Amazon S3 Bucket Keys reduce the costs of Server-Side Encryption with AWS Key Management Service (SSE-KMS)

Reducing the cost of SSE-KMS with Amazon S3 Bucket Keys – Amazon Simple Storage Service

About Xebia

Xebia focuses on Amazon Web Services (AWS) Consultancy, managed services and training. Our mission is to guide our customers to become self-sufficient and innovative on a successful and effective cloud journey. This is done by architecting, planning, migrating and implementing large scale Cloud Computing solutions and helping enterprises to enter the cloud and by training their staff. We are an AWS Premier Consultancy and Channel Partner.