Monitoring AWS EKS audit logs with Falco
AWS recently announced the possibility to send control plane logs from their managed Kubernetes service (EKS) to CloudWatch. Amongst those logs are the API server audit events, which provide an important security trail regarding interactions with your EKS cluster
Sysdig Falco is an open-source CNCF project that is specifically designed to monitor the behavior of containers and applications. Besides monitoring container run-time behavior, it can also inspect the Kubernetes audit events for non-compliant interactions based on a predefined set of rules.
Wouldn’t it be nice if you could automatically monitor your EKS audit events with Falco? In this blog post we will show you how to make this work.
There is no way for Falco to inspect the EKS audit events directly, since they are being sent to CloudWatch. Normally you would configure the API server to post the audit events to Falco directly, but since we are using a managed Kubernetes service there is no way to configure this. Instead, we need a way to present the CloudWatch logs to Falco via an alternative method. The solution consists of two parts:
- Transfer the CloudWatch logs to an S3 bucket using a default cloud-native AWS pattern
- Download the logs and transfer the audit events to Falco for compliance checking with a custom Falco-EKS bridge component
Schematically it looks like this:
EKS is sending the audit events to CloudWatch. From there we need to transfer them to an S3 bucket so that we can download and process them with Falco. We can do this the cloud-native way with AWS Kinesis Firehose. AWS have provided a nice guide on how to configure this. However, instead of clicking in the console or running one-off CLI commands you are going to use a proper infrastructure-as-code tool like Terraform or CloudFormation for this, right? You can check this repository for a working example with Terraform code. Please take some time to set this up, we’ll wait until you are ready…
Okay, now that your audit logs are flowing nicely into your S3 bucket it is time to inspect them with Falco for any non-compliant interactions. Since there is no standard way to do this we have created a dedicated tool for this: falco-eks-audit-bridge.
It is a Go based tool that is supposed to be run as a service in your EKS cluster. It will continuously monitor your S3 bucket for any new audit events and transfer them to Falco for inspection. Please review the instructions in the repository to learn how to configure and deploy the Falco EKS bridge.
We hope this blog and our custom Falco-EKS bridge application provide you with enough information to start monitoring your Kubernetes audit events right now. May all your EKS clusters be forever in compliance!