Deploying lambdas to AWS has always been painful when those Lambdas need more than just boto3 and when sticking to Infrastructure-as-Code. You can bring in the Serverless Framework, but it is complicated to bring into your CICD pipelines and has some issues with repeatability of deployments. So, what if you want to deploy a single Lambda?

In this post, I'll discuss the method I use to deploy a Lambda, written in Python 3, to AWS using Terraform and GitLab CI.

Project Structure

For this project, let's make some assumptions. First, it's that your project is configured with the following structure:

project_root
|  .gitlab-ci.yml
|
└--lambda/
|  |  requirements.txt
|  |  function.py
|
└--terraform/
   |  versions.tf
   |  variables.tf
   |  ...

Second, it's that you already have Terraform working in your GitLab CI setup using Terraform images from ECR Public Gallery or Docker Hub. A plan job should kick off automatically (perhaps after validation) and an apply job can run (manually?) after the terraform plan is completed. Basically, getting Terraform up and running in CI is out of the scope of this article, but I will show you how to make your apply job work with the Lambda.

Prep the Apply Job

My normal apply job looks like this:

terraform_apply:
  stage: apply
  before_script:
    - export TF_VAR_project_title
    - cd terraform/
    - rm -rf .terraform
  script:
    - terraform init
    - terraform apply -input=false $PLAN
  when: manual
  dependencies:
    - terraform_plan
  artifacts:
    name: $CI_COMMIT_REF_SLUG
    untracked: true
  only:
    - branches

I have globally defined TF_VAR_project_title to be equal to $CI_PROJECT_TITLE, so that I can tag AWS resources with the project name (making everything easier to find) and am using public.ecr.aws/hashicorp/terraform:1.1.2 as my global image.

We'll need to package up our Python dependencies, so Terraform will need access to pip. We can do this by installing the py3-pip package into the Terraform container. Add the following line to the before_script in the apply job:

- apk add py3-pip

Your validate, plan, and destroy jobs don't need this. Just apply.

Packaging the Lambda

So, we need our Lambda's dependencies to be packaged up with the source code so that aws_lambda_function can use it. Luckily, we can do this in Terraform using Terraform's null_resource and archive_file.

Let's ensure that we have the lambda_root variable set to the relative path to the Lambda source from the Terraform directory. My definition looks like this:

variable "lambda_root" {
  type        = string
  description = "The relative path to the source of the lambda"
  default     = "../lambda"
}

Now, we install our dependencies.

resource "null_resource" "install_dependencies" {
  provisioner "local-exec" {
    command = "pip install -r ${var.lambda_root}/requirements.txt -t ${var.lambda_root}/"
  }
  
  triggers = {
    dependencies_versions = filemd5("${var.lambda_root}/requirements.txt")
    source_versions = filemd5("${var.lambda_root}/function.py")
  }
}

This has pip install our Python dependencies into the lambda/ folder, so that they can be zipped up with our code. The triggers block determines when this will execute and our map is ensuring that happens when function.py or requirements.txt is changed. If you have additional files that should be watched, add them in with their own unique keys, such as:

helper_versions = filemd5("${var.lambda_root}/helpers.py")

Now, in order to ensure that cached versions of the Lambda aren't invoked by AWS, let's give our zip file a unique name for each version. We can do with by hashing our files together.

resource "random_uuid" "lambda_src_hash" {
  keepers = {
    for filename in setunion(
      fileset(var.lambda_root, "function.py"),
      fileset(var.lambda_root, "requirements.txt")
    ):
        filename => filemd5("${var.lambda_root}/${filename}")
  }

If you have any additional source files, include them in the setunion() block.

Now that we have a unique name for the zip file, let's create it.

data "archive_file" "lambda_source" {
  depends_on = [null_resource.install_dependencies]
  excludes   = [
    "__pycache__",
    "venv",
  ]

  source_dir  = var.lambda_root
  output_path = "${random_uuid.lambda_src_hash.result}.zip"
  type        = "zip"
}

We ensure that the zip file runs after the dependencies are installed, exclude our virtual environment (change that from venv to whatever you name yours) and pycache directories, and drop that zip file in the lambda/ directory.

Now we create the Lambda itself.

resource "aws_lambda_function" "lambda" {
  function_name    = "my_function"
  role             = aws_iam_role.lambda_role.arn
  filename         = data.archive_file.lambda_source.output_path
  source_code_hash = data.archive_file.lambda_source.output_base64sha256

  handler          = var.lambda_handler
  runtime          = var.lambda_runtime

  environment {
    variables = {
      LOG_LEVEL = var.lambda_log_level
    }
  }

  depends_on    = [
    aws_iam_role_policy_attachment.lambda_execution,
    aws_cloudwatch_log_group.task-creation,
  ]

  reserved_concurrent_executions = var.lambda_concurrent_executions

  tags = local.tags

I'm using a aws_iam_role created to give the Lambda its needed permissions. What these are are dependent on what your Lambda does and is outside of the scope of this article.

I'm also using a bunch of variables:

  • lambda_handler: your function reference. Assuming we were using a function called handler() in function.py, it would be function.handler.
  • lambda_runtime: the runtime for you Lambda. I'm using python3.9, but the full list is here.
  • lambda_log_level: the logging level I want to use for my Lambda. I set this to INFO, but can switch it to debug or whatever I feel. Note that this is set via setLevel in the Python code itself and isn't just some magic that Lambda provides.
  • lambda_concurrent_executions: the number of simultaneous executions of the lambda that I'm willing to have. This depends on your needs.

Triggering Terraform Jobs on Lambda Changes

My validate and plan jobs normally look like this:

terraform_validate:
  stage: validate
  before_script:
    - export TF_VAR_project_title
    - cd terraform/
    - rm -rf .terraform
    - terraform --version
    - terraform init
  script:
    - terraform validate
  only:
    refs:
      - branches
    changes:
      - '*'
      - terraform/**/*

terraform_plan:
  stage: plan
  before_script:
    - export TF_VAR_project_title
    - cd terraform/
    - rm -rf .terraform
    - terraform --version
    - terraform init
  script:
    - terraform plan -out=$PLAN -input=false
  artifacts:
    name: plan
    paths:
      - terraform/${PLAN}
  only:
    refs:
      - branches
    changes:
      - '*'
      - terraform/**/*
  needs:
    - terraform_validate

So, these trigger when I change something in the terraform/ directory or anything at the project root (like my .gitlab-ci.yml file). However, what happens when we make a change to our Lambda? The Terraform jobs won't run. So, let's fix that by adding the following line to the changes: list in each of those jobs:

- lambda/**/*

Now, when we change our Python source or update the requirements.txt file, our Terraform pipeline will kick off so that our new Lambda package can be deployed.