In this blog we will show you how to replace awslogs with Vector when upgrading to Amazon Linux 2023, because awslogs is no longer available and the recommended unified Cloudwatch agent does not support journalctl.

Why Vector?

We chose Vector — an open-source observability pipeline as it supports both journalctl and CloudWatch Logs. We love the simplicity of the configuration files and as we are already using datadog for observability in our platform, we were pleased to know that Vector is maintained by the Datadog's Community Open Source Engineering team.

Configure AL2023 to forward journalctl logs to CloudWatch

To configure AL2023 to forward journalctl logs to CloudWatch using Vector, you need to do the following:

• Install Vector
• Configure journald log forwarding to CloudWatch log group
• Disable healthcheck validation on startup
• Ensure Vector restarts on environment file changes
• Configure the AWS log group name
• Enable the Vector systemd units
• Ensure EC2 instance has the proper permissions

The following paragraphs will provide details for each step.

Install Vector

Installing Vector is easy, just type:

bash -c "$(curl -L https://setup.vector.dev)"
sudo yum install -y vector

Configure the journalctl to CloudWatch Vector pipeline

To configure Vector to forward all the journalctl logs to a CloudWatch log group, create the following configuration in /etc/vector/vector.yaml:

---
sources:
  journald:
    type: journald
    exclude_matches:
      _SYSTEMD_UNIT: [ "ntpd.service", "vector.service" ]
      _TRANSPORT: [ "kernel" ]


transforms:
  journald-with-env:
    inputs:
      - journald
    type: "remap"
    source: |
      .log_group_name = "${AWS_LOG_GROUP_NAME:-vector}"

sinks:
  cloudwatch:
    type: aws_cloudwatch_logs
    inputs: [ "journald-with-env" ]
    create_missing_group: true
    create_missing_stream: true
    group_name: "{{ log_group_name }}"
    stream_name: "{{ host }}"
    retention:
      days: 30
      enabled: true
    encoding:
      codec: json

This configuration allows you to configure the CloudWatch Log group using the environment variable AWS_LOG_GROUP_NAME. This config can be used in a base image, and customized per instance via cloud-init. Read more about Vector pipeline configuration and aws_cloudwatch_logs for all options on AWS CloudWatch Log groups.

Disable healthcheck validation on startup

You should disable the healthcheck validation on startup, as Vector will not start up with it. This is because Vector runs a validation- and health check of the configuration on startup. In this validation step, the reference log_group_name is not resolved, which in turn causes the sink health check to fail. This failures is because {{ log_group_name }} is not a valid log group name.

To avoid this, override the startup validation of the Vector systemd configuration with the following content in /etc/systemd/system/vector.service.d/override.conf:

ExecStartPre=
ExecStartPre=/usr/bin/vector validate --skip-healthchecks

[Install]
WantedBy=multi-user.target

Ensure Vector restarts on environment file changes

To make sure that changes in the environment file /etc/default/vector are picked up, add the following systemd configuration files: vector-restart.path and vector-restart.service.

Create the file /etc/systemd/system/vector-restart.path, with the following content:

[Unit]
Description=Monitor Vector environment variable configuration changes
Requires=vector-restart.service

[Path]
PathChanged=/etc/default/vector

[Install]
WantedBy=multi-user.target

Create the file /etc/systemd/system/vector-restart.service, with the following content:

[Unit]
Description=Restart Vector service on config change

[Service]
Type=oneshot
ExecStart=/usr/bin/systemctl restart vector.service

Configure the AWS log group name

To configure the name of the desired log group set the environment variable AWS_LOG_GROUP_NAME in the Environment file of Vector in /etc/default/vector:

AWS_LOG_GROUP_NAME=bastion

Enable the Vector systemd units

To ensure Vector starts on boot, type:

systemctl daemon-reload
systemctl enable vector.service vector-restart.path vector-restart.service
systemctl start vector.service vector-restart.path

Ensure EC2 instance permissions

Make sure your EC2 instance has the following permissions:

• AmazonCloudWatchLogs:CreateLogGroup
• AmazonCloudWatchLogs:CreateLogStream
• AmazonCloudWatchLogs:DescribeLogGroups
• AmazonCloudWatchLogs:DescribeLogStreams
• AmazonCloudWatchLogs:PutLogEvents

Conclusion

This blogs shows how easy it is to configure the Vector log forwarder on your EC2 instance. By defining the target log group name in an environment variable, you can use this configuration in a base image and customize each instance by the variable AWS_LOG_GROUP_NAME in the file /etc/default/vector.

---

Image taken from https://github.com/vectordotdev/vector