Cross Region Replication with S3

Sometimes you need objects from one region into another region. A common use case is centralizing logs from multiple regions for data analytics. Amazon Simple Storage Storage (S3) is a service that stores objects by means of a key and a value. The data associated with a key can easily be accessed by means of a web service API. S3 is a core service of AWS and is perfectly suited for storing log data. S3 stores data locally in a region. Data is never copied to another region without explicit configuration. Lets take a look what is involved setting up a cross region copy between two S3 buckets.

Cross Region Replication (CRR)

Cross Region Replication (CRR) is a service that automatically copies objects from one region to another. The service is part of S3 and can be enabled by means of configuration. Data is then securely copied from one region to another and stored in another bucket.

Architecture

The CRR example consists of two buckets, one in us-east-1 and one is eu-west-1. The bucket in us-east-1 contains content that must be replicated to eu-west-1.

Example

The example project consists of two CloudFormation templates, that configures buckets in separate regions. For us-east-1 a bucket needs to be created that has VersioningConfiguration and ReplicationConfiguration enabled. The bucket configures the Amazon Resource Name (ARN) of the eu-west-1 bucket to replicate as the Destination. The bucket also defines a Role – a security configuration – that S3 assumes to get permissions to write to the eu-west-1 bucket.

  ReplicatedBucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: BucketOwnerFullControl
      VersioningConfiguration:
        Status: Enabled
      ReplicationConfiguration:
        Role: !GetAtt ReplicationRole.Arn
        Rules:
          - Prefix: ''
            Status: Enabled
            Destination:
              Bucket: !Ref ReplicatedBucketArn

  ReplicationRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: s3.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: Allow
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - s3:*
                Resource:
                  - '*'

The eu-west-1 bucket only has an enabled VersioningConfiguration.

  SimpleBucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: BucketOwnerFullControl
      VersioningConfiguration:
        Status: Enabled

To deploy the example type make deploy. To get information about the bucket names type make info.

Replicating Data

To replicate data, we need to copy data to us-east-1. To get the bucket name type make info that shows the BucketName for both regions. To copy data to us-east-1 we need the name of that bucket and type:

$ aws s3 cp aws s3 cp LICENSE s3://cross-region-replication-example-eu-simplebucket-1adlmia5ye8ji

After a second the file is replicated in eu-west-1:

$ aws s3 ls s3://cross-region-replication-example-eu-simplebucket-8eeig5mggnah
2018-11-14 05:19:50      11357 LICENSE

Conclusion

It is easy to configure S3 Cross Region Replication (CRR). Versioning has to be enabled and the bucket needs permissions to access the bucket in another region. With CRR we can copy log like access logs, error logs and so on to a central location for data analytics. Next time we’ll look at AWS Athena to do simple log analytics on S3 buckets.