Automate Lambda Dependencies with Terraform

Recently I have been working on a few projects that involved Lambda functions. This meant defining and building these functions and all infrastructure around them via Terraform. When doing this previously I have always struggled to find simple ways to integrate the dependencies into the deployment to minimise manual intervention or unnecessary convoluted steps, so this time around I decided to invest some time into finding a better solution! So this is a quick article to cover the solution I found along with its pros and (frustrating) con.

Dir Structure

For this solution to work the project directory should be structured in a way so that terraform can find the dependencies you are looking to install to your Lambda. I utilise Terraform Cloud  for my deployments so have structured my directory in order to reflect this. Terraform Cloud works by isolating the terraform directory out of the project and working from there. This means all relevant files must be within this directory. If you are using a local deployment method, this will afford you more freedom in terms of where you can store your files. Remember to adjust the path names to the correct location where necessary though.

NOTE: This example uses a python function structure, but can be adjusted to fit other languages with a little know how.

The dir structure is as follows:

project/
|-README.md
|-terraform/
|  |-function/
|  |  |-main.py
|  |-layer/
|  |  |-requirements.txt
|  |-main.tf
|  |-variables.tf

Our requirements.txt file should be in the form of a standard python requirements text file eg:

boto3==1.33.8
requests>=2.0.0

With this structure in place we can put together our terraform file.

Terraform resources

I have omitted all of the config details of the resources not in scope so we can focus on the layer config:

terraform {
  # ... config
}

provider "aws" {
  # ... config
}

resource "null_resource" "install_layer_dependencies" {
  provisioner "local-exec" {
    command = "pip install -r layer/requirements.txt -t layer/python/lib/python3.9/site-packages"
  }
  triggers = {
    trigger = timestamp()
  }
}

data "archive_file" "layer_zip" {
  type        = "zip"
  source_dir  = "layer"
  output_path = "layer.zip"
  depends_on = [
    null_resource.install_layer_dependencies
  ]
}

resource "aws_lambda_layer_version" "lambda_layer" {
  filename = "layer.zip"
  source_code_hash = data.archive_file.layer_zip.output_base64sha256
  layer_name = "my_layer_name"

  compatible_runtimes = ["python3.9"]
  depends_on = [
    data.archive_file.layer_zip
  ]
}

data "archive_file" "function_zip" {
  type        = "zip"
  source_dir  = "function"
  output_path = "function.zip"
}

resource "aws_lambda_function" "lambda" {
  # ... other config
  filename      = "function.zip"
  source_code_hash = data.archive_file.functip_zip.output_base64sha256
  runtime = "python3.9"

  role = aws_iam_role.role.arn

  layers = [
    aws_lambda_layer_version.lambda_layer.arn
  ]
  depends_on = [
    data.archive_file.function_zip,
    aws_lambda_layer_version.lambda_layer
  ]
}

resource "aws_iam_role" "role" {
  # ... config
}

resource "aws_iam_role_policy" "role_policy" {
  # ... config
}

The key part to focus on here is the null_resource. This is where the dependencies are installed. In our example we are reading from the requirements.txt file and installing those dependencies into the layer/ directory, which we then zip this file with the archive_file.layer_zip data source so it can be used by the aws_lambda_function. To adjust this process for your runtime and dir structure, change the command within the null_resource to fit your install needs. It is also worth taking note of the depends_on variable within the resources, as they help make sure everything is in place before the next stage is deployed.

The next thing to point out is the triggers section in the null_resource. Due to how the state file works, in order to deploy this resource repeatedly a trigger needs to be added. Ideally this trigger would be a change in the requirements.txt file, however without being able to add this same trigger to the archive_file.layer_zip and aws_lambda_layer_version resources using the change in requirements.txt file as a trigger leads to improperly packaged layers down the line. At this current time the best we can use is timestamp(). Using this as a trigger causes a layer version to be built on every apply, which means the layer is always packaged correctly, but also causes our frustrating con: we have to wait to build a new layer version on every apply, even if there are no changes.

Layer vs Package

We have used a layer for our dependencies in this example, however we can easily adjust this to be a standard package. By adjusting the null_resource to install the dependencies into the function/ directory, we can remove the layer zip and layer version resources, and use the zipped function file directly. As mentioned previously though, this will cause the dependencies to be built on every apply, so having the function and dependencies packaged together means the function code will also be redeployed every apply. More info on this process can be found here.

Summary

Although using this method does lead to slower deployment times as the layer has to be built every time, I feel the tradeoff is worth it. Dependencies can easily be added to the requirements.txt and they will be automatically and correctly deployed to your function. Overall I feel you are trading speed for a "fire and forget" dependency installation method. On top of this, as Terraform evolves, hopefully triggers will be added to other resources in order to remove this speed hurdle. This post only covers AWS but with this concept you can adjust the method to fit any severless platform you use!