How to keep AWS CloudWatch logs neat and tidy

AWS CloudWatch logs is an useful logging system, but in practice we run into two quircks: It does not allow you
to set a default retention period for newly created log groups. Secondly it does not delete empty log streams
that are older than the retention period. In this blog we introduce a utility that will fix these two issues. It can
be installed as a CLI or deployed as an AWS Lambda.
This utility provides two functions:

1. apply a retention policy on log groups that do not have one.
2. remove empty log streams older than the retention period of the log group.
   This utility can be used from the command line. You can also install it as an AWS Lambda function and have your
   logs kept in order, NoOps style!

Install the utility

to install the utility, type:

pip install aws-cloudwatch-log-minder

Set retention period

to set the default retention period on log groups without one, type:

cwlog-minder --dry-run set-log-retention --days 30

This will show you log groups which do not have a retention period set. Remove the --dry-run and
it will.

Delete empty log streams

To delete empty log streams older than the retention period, type:

cwlog-minder --dry-run delete-empty-log-streams

This will show you which empty log streams will be deleted. Remove the --dry-run and
these streams will be deleted.

Deploy as Lambda

To deploy the utility as an AWS Lambda, type:

git clone https://github.com/binxio/aws-cloudwatch-log-minder.git
cd aws-cloudwatch-log-minder
aws cloudformation create-stack \
        --capabilities CAPABILITY_IAM \
        --stack-name aws-cloudwatch-log-minder \
        --template-body file://./cloudformation/aws-cloudwatch-log-minder.yaml

aws cloudformation wait stack-create-complete  --stack-name aws-cloudwatch-log-minder

This will install the utility in your AWS account and run every hour. The added benefit
is that the maintenance of the log groups is automated and executed in parallel.

Conclusion

This utility provides a work-around for two quircks you will encounter when working with AWS CloudWatch.
it will set a retention policy on log groups created outside of control of the user, and it will
delete empty log streams older than the retention period. It would of course be cool if
AWS would fix the latter.