S3 Bucket Notifications

Amazon Simple Storage Storage (S3) is a service that stores objects by means of a key and a value. The data associated with a key can easily be accessed by means of a web service API. S3 is a core service of AWS and is perfectly suited for storing any kind of data. S3 supports a feature called ‘Event Notifications’ that enables you to create event driven systems. When certain events happen in your bucket like eg. creating or deleting an object, an event can be sent to SNS or a Lambda function. In this blog we’ll see how to setup a Lambda as an event handler.

CloudFormation Configuration

The CloudFormation configuation is simple. We need to specify a name for the bucket by means of a custom provider that generates a random string. Next we need to specify a permission so the bucket may invoke the Lambda.

  InputBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref '${RandomName}'
      NotificationConfiguration:
        LambdaConfigurations:
        - Function: !GetAtt NotificationLambda.Arn
          Event: s3:ObjectCreated:*

  LambdaInvokePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt NotificationLambda.Arn
      Action: lambda:InvokeFunction
      Principal: s3.amazonaws.com
      SourceAccount: !Ref AWS::AccountId
      SourceArn: !Sub 'arn:aws:s3:::${RandomName}'

Custom Provider

To generate a random name we create a custom provider.

  RandomName:
    Type: Custom::RandomNameGenerator
    Properties:
      ServiceToken: !GetAtt 'RandomNameGenerator.Arn'

  RandomNameGenerator:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.lambda_handler
      Timeout: 30
      Role: !GetAtt 'LambdaBasicExecutionRole.Arn'
      Runtime: python3.6
      Code:
        ZipFile: |
          import string
          import random
          import cfnresponse
          def lambda_handler(event, context):
              if event['RequestType'] == 'Create':
                  event['PhysicalResourceId'] = ''.join(random.choice(string.ascii_lowercase) for _ in range(16))
              cfnresponse.send(event, context, cfnresponse.SUCCESS, {}, event['PhysicalResourceId'])

The Bucket Handler

The event handler for the bucket is a lambda function that prints the received event to CloudWatch logs.

def handler(event, ctx):
    print(event)

ObjectCreated Event

When we upload a file to the bucket, the following event is received by the lambda. The Records field will contain only a single entry and contains the details for the ObjectCreated event like the bucket and key name and where and when the event occurred.

{
  "Records": [
    {
      "eventVersion": "2.0",
      "eventSource": "aws:s3",
      "awsRegion": "eu-west-1",
      "eventTime": "2018-11-18T12:42:15.922Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "AWS:AIDAJSZKO5A3SBJFMYXTE"
      },
      "requestParameters": {
        "sourceIPAddress": "217.19.26.243"
      },
      "responseElements": {
        "x-amz-request-id": "B9CC7E95824DAC64",
        "x-amz-id-2": "qufqqLvaJ4w7e9UwHdBply4jPEJhwJXbL5qYr0fbhAtZPn0+lfixoN26JC/80uBGpHPxbvDL9XM="
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "8450cff2-771d-4a5f-9599-255bc97cbd47",
        "bucket": {
          "name": "qmuixbpinyhhlwpd",
          "ownerIdentity": {
            "principalId": "A97CX0CDO58US"
          },
          "arn": "arn:aws:s3:::qmuixbpinyhhlwpd"
        },
        "object": {
          "key": "README.md",
          "size": 215,
          "eTag": "2e59fd3b13286947aa9d4dcaa03bf568",
          "sequencer": "005BF15E27DB726B35"
        }
      }
    }
  ]
}

Conclusion

It is easy to setup S3 Event Notifications. The s3 bucket must have permission to invoke the lambda. The Lambda can contain arbitrary logic that will be executed when something happens with the bucket. Next time we’ll look at how to send received CloudFront log files from an S3 bucket to AWS Elasticsearch service.