AWS | Cloud
How to keep AWS CloudWatch logs neat and tidy 
Mark van Holsteijn 12 Sep, 2019
Share 
You can protect yourself from losing logs on Amazon EC2 by using CloudWatch Logs. Configure the CloudWatch Agent to stream your logs to a LogGroup. This protects you from losing logs. For example, when the instance is replaced by autoscaling. You are also protected against tampering of the logs. An attacker who has gained access to your system can remove the logs. But the logs in the LogGroup will contain the original log lines.
Preparations
We will need the following resources:
InstanceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Security group for the test instance
VpcId: "{{resolve:ssm:/landingzone/vpc/vpc-id}}"
SecurityGroupEgress:
- Description: Allow outbound connectivity to port 443.
IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: 0.0.0.0/0
Role:
Type: AWS::IAM::Role
Properties:
PermissionsBoundary: !Sub arn:aws:iam::${AWS::AccountId}:policy/landingzone-workload-permissions-boundary
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service: ec2.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
- arn:aws:iam::aws:policy/AmazonInspector2ManagedCispolicy
- arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Roles:
- !Ref Role
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub ${AWS::StackName}-Instance
RetentionInDays: !!int 365
- InstanceSecurityGroup, allows an outbound connection on 443. The CloudWatch Agent needs to stream the logs to the CloudWatch endpoint.
- Role, holds the policies that allow the instance to send the logs to the LogGroup.
- InstanceProfile, the profile used to attach the role to the EC2 instances.
- LogGroup, the LogGroup used to store the logs in.
Setting up the EC2 Instance
In my previous blog I wrote about how you can create an EC2 instance using Infrastructure as Code. We will continue on that example. Instead of a single instance we will create an autoscaling group.
First we will need a LaunchTemplate. This LaunchTemplate will contain some metadata called AWS::CloudFormation::Init. The metadata has so called configSets and blocks of config. Here is an example that we will use:
LaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Metadata:
AWS::CloudFormation::Init:
configSets:
default:
- CloudFormationInit
- CloudWatchLogs
CloudFormationInit:
files:
/etc/cfn/cfn-hup.conf:
owner: root
group: root
mode: 000400
content: !Sub |-
[main]
stack=${AWS::StackId}
region=${AWS::Region}
/etc/cfn/hooks.d/cfn-auto-reloader.conf:
owner: root
group: root
mode: 000400
content: !Sub |-
[cfn-auto-reloader-hook]
triggers=post.update
path=Resources.LaunchTemplate.Metadata.AWS::CloudFormation::Init
action=/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --region ${AWS::Region} --resource LaunchTemplate
runas=root
services:
sysvinit:
cfn-hup:
enabled: true
ensureRunning: true
files:
- /etc/cfn/cfn-hup.conf
- /etc/cfn/hooks.d/cfn-auto-reloader.conf
CloudWatchLogs:
packages:
yum:
amazon-cloudwatch-agent: []
files:
/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json:
owner: root
group: root
mode: 000400
content: !Sub |-
{
"agent": {
"region": "${AWS::Region}",
"logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
"debug": false
},
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/user-data.log",
"log_group_name": "${LogGroup}",
"log_stream_name": "{instance_id}/user-data.log"
},
{
"file_path": "/var/log/cfn-hup.log",
"log_group_name": "${LogGroup}",
"log_stream_name": "{instance_id}/cfn-hup.log"
},
{
"file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
"log_group_name": "${LogGroup}",
"log_stream_name": "{instance_id}/amazon-cloudwatch-agent.log"
}
]
}
},
"log_stream_name": "default_log_stream"
},
"metrics": {
"append_dimensions": {
"AutoScalingGroupName": "${!aws:AutoScalingGroupName}"
},
"metrics_collected": {
"disk": {
"measurement": [
"used_percent"
],
"metrics_collection_interval": 60,
"resources": [
"/"
]
}
}
}
}
services:
sysvinit:
amazon-cloudwatch-agent:
enabled: true
ensureRunning: true
files:
- /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json
Properties:
LaunchTemplateData:
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
DeleteOnTermination: !!bool true
Encrypted: !!bool true
KmsKeyId: !GetAtt EncryptionKey.Arn
VolumeSize: !!int 32
VolumeType: gp2
DisableApiTermination: !!bool true
IamInstanceProfile:
Arn: !GetAtt InstanceProfile.Arn
ImageId: "{{resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2}}"
InstanceType: t3.micro
SecurityGroupIds:
- !Ref InstanceSecurityGroup
MetadataOptions:
HttpTokens: required
InstanceMetadataTags: enabled
UserData:
Fn::Base64: !Sub |-
#!/bin/bash -x
/opt/aws/bin/cfn-init --stack ${AWS::StackName} --region ${AWS::Region} --resource LaunchTemplate
/opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --region ${AWS::Region} --resource AutoScalingGroup
CloudFormationInit
The CloudFormationInit holds a configuration. This configuration is used by the cfn-init agent. It comes pre-installed with the AmazonLinux2 and AmazonLinux2023 AMIs. Under the files section you can see that we are defining 2 files:
/etc/cfn/cfn-hup.conf/etc/cfn/hooks.d/cfn-auto-reloader.conf
The syntax is quite explanatory. You set an owner, group, mode and the content of the file. The content of the tells the EC2 instance where it can find the metadata. During the initial boot this is given in the user-data. But you might change the metadata overtime. This makes sure that the running instances checks if there is an update available.
To recap this will make sure that the running EC2 instances stay in sync with the given metadata.
CloudWatchLogs
The Amazon Linux AMIs do not come with the CloudWatch Agent you need to install the agent first. You can do this by providing the amazon-cloudwatch-agent package, under packages.
Now it’s time to configure the agent to stream the log files to CloudWatch LogGroups. We have defined 3 files that will be watched by the agent:
/var/log/user-data.log/var/log/cfn-hup.log/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
You can extend this list to your own needs. In this example each EC2 instance will have 3 streams. 1 per file and the instance id is used as a prefix, example: i-0000000000/user-data.log.
You can also do measurements on the instance itself and expose this as a CloudWatch Metric. In this example we are capturing a percentage of the used space of the root volume.
We have now configured the agent. But we also need to make sure it’s running. When changes are made to the configuration file, we also need to restart it. We do this by pointing to the config file. Now cfn-init knows when it changes the configuration that it also needs to restart the agent.
User Data
The user data will make sure that the configuration is applied. Since we configure the cfn-init to check the configuration it will stay in sync from here on. The second thing it will do is it will send a signal to the AutoScalingGroup resource. If the cfn-init is executed successfully it will signal a success and if not it will signal a failure.
This means that if the configuration fails the autoscaling group will receive a failure. As a result it will rollback the stack.
AutoScalingGroup
The autoscaling group looks as followed:
AutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
CreationPolicy:
ResourceSignal:
Timeout: PT5M
UpdatePolicy:
AutoScalingRollingUpdate:
MinInstancesInService: 1
MaxBatchSize: 1
PauseTime: PT5M
WaitOnResourceSignals: !!bool true
Properties:
LaunchTemplate:
LaunchTemplateId: !GetAtt LaunchTemplate.LaunchTemplateId
Version: !GetAtt LaunchTemplate.LatestVersionNumber
MinSize: !!int 1
MaxSize: !!int 2
DesiredCapacity: !!int 1
VPCZoneIdentifier:
- "{{resolve:ssm:/landingzone/vpc/private-subnet-1-id}}"
- "{{resolve:ssm:/landingzone/vpc/private-subnet-2-id}}"
We will have 2 policies defined, one for creation and one for updates. If the signal configured in the user-data fails the resource will fail and trigger a rollback. If the instance is configured successfully it will continue. In the auto scaling group you can define the amount of instances required. In this example the maximum is 2 and the minimum 1. This makes sure that at least one EC2 instance is running. When a replacement is needed a new instance will be added first and if it is deployed successfully the old one is removed. This is why the max is set to 2 so that we can replace the old one gracefully.
Conclusion
Configuring a LogGroup to stream your log files is not hard. Using CloudFormation Init you can install and configure applications. In our example we configured the CloudWatch agent that will stream the logs.
When you have to deal with autoscaling events and/or malicious attackers. You can prevent the loss of logs by centralising your application log files.
Photo by Pixabay
by Joris Conijn
Joris has been working with the AWS cloud since 2009 and focussing on building event driven architectures. While working with the cloud from (almost) the start he has seen most of the services being launched.
Joris strongly believes in automation and infrastructure as code and is open to learn new things and experiment with them, because that is the way to learn and grow. In his spare time he enjoys running and runs a small micro brewery from his home. 
