Terraform – Deploy Python Lambda (container image)

Andrei Maksimov

Andrei Maksimov

0
(0)

At the end of 2020 AWS announced support of container images for Lambda. This feature allows you to package and deploy Lambda functions as container images of up to 10 GB in size.  In this article we’ll cover how you can use Terraform to deploy Python Lambda functions backed by the container image.

One of the common tasks in the cloud world is to replicate source code repositories from on-premises to the cloud or between cloud environments. So, to illustrate the approach we decided to add Git and GitPython support to the Lambda function.

Project structure

Here’s a project structure which we’ll be using during this demo:

$ tree lambda_container
lambda_container
├── README.md
├── lambdas
│   └── git_client
│       ├── Dockerfile
│       └── index.py
└── main.tf

2 directories, 4 files
  • lambdas – the folder where we put Lambda functions source code
  • main.tf – Terraform demo code, which will build Docker container for git_client Lambda function and deploy the function afterwards

Dockerfile

Lets describe a Docker container which will host all dependencies for our lambda functions. Here’s the Dockerfile content:

FROM public.ecr.aws/lambda/python:3.8

RUN yum update -y && \
  yum install -y git && \
  rm -Rf /var/cache/yum && \
  pip install git-remote-codecommit boto3 GitPython awscli

COPY index.py ${LAMBDA_TASK_ROOT}

CMD [ "index.handler" ]

We’re taking Python3.8 public Docker image from Amazon as a base, then we’re installing Git, cleaning the yum caches to make the container smaller and installing required dependencies, which allow us to use Git with CodeCommit using IAM for authentication.

Next we’re copying index.py file to the folder where Lambda function code should reside. Check Using AWS Lambda environment variables for additional information.

Finally, we’re specifying to execute the handler method from the index.py file at the container launch.

Lambda code

As soon as the Lambda container declaration is finalized, we can write a Lambda function, which will use it. Here’s a code example, which will show how to clone Git repository. I’m sure, you’ll be able to to adjust this example for your personal needs:

import logging
import os
import git
 
TMP_DIR = "/tmp"
REPO_DIR = 'aws-config-rules'
REPO_URL = f'https://github.com/andreivmaksimov/{REPO_DIR}'
CLONE_PATH = os.path.join(TMP_DIR, REPO_DIR)
 
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
 
def clone(branch='master'):
   repo = git.Repo.clone_from(REPO_URL, CLONE_PATH, branch=branch)
 
   with repo.config_writer() as git_config:
       git_config.set_value('user', 'email', 'no-reply@hands-on.cloud')
       git_config.set_value('user', 'name', 'Git Lambda')
  
def handler(event, context):
   LOGGER.info('Event: %s', event)
 
   LOGGER.info('Cloning repo: %s', REPO_URL)
   clone()

In this code we’re declaring required Python libraries, some constants, configuring logger and couple of functions:

  • def clone(branch='master') – this function shows how to clone Git repository
  • def handler(event, context) – this function is the main entry point to the Lambda function, it logs the incoming event and calling clone function

Terraform code

As soon as we have Lambda code and declare its container, we can write some Terraform code to automate the deployment. Here it is:

variable region {
 default = "us-east-1"
}
 
provider aws {
 region = var.region
}
 
data aws_caller_identity current {}
 
locals {
 prefix = "git"
 account_id          = data.aws_caller_identity.current.account_id
 ecr_repository_name = "${local.prefix}-demo-lambda-container"
 ecr_image_tag       = "latest"
}
 
resource aws_ecr_repository repo {
 name = local.ecr_repository_name
}
 
resource null_resource ecr_image {
 triggers = {
   python_file = md5(file("${path.module}/lambdas/git_client/index.py"))
   docker_file = md5(file("${path.module}/lambdas/git_client/Dockerfile"))
 }
 
 provisioner "local-exec" {
   command = <<EOF
           aws ecr get-login-password --region ${var.region} | docker login --username AWS --password-stdin ${local.account_id}.dkr.ecr.${var.region}.amazonaws.com
           cd ${path.module}/lambdas/git_client
           docker build -t ${aws_ecr_repository.repo.repository_url}:${local.ecr_image_tag} .
           docker push ${aws_ecr_repository.repo.repository_url}:${local.ecr_image_tag}
       EOF
 }
}
 
data aws_ecr_image lambda_image {
 depends_on = [
   null_resource.ecr_image
 ]
 repository_name = local.ecr_repository_name
 image_tag       = local.ecr_image_tag
}
 
resource aws_iam_role lambda {
 name = "${local.prefix}-lambda-role"
 assume_role_policy = <<EOF
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Action": "sts:AssumeRole",
           "Principal": {
               "Service": "lambda.amazonaws.com"
           },
           "Effect": "Allow"
       }
   ]
}
 EOF
}
 
data aws_iam_policy_document lambda {
   statement {
     actions = [
         "logs:CreateLogGroup",
         "logs:CreateLogStream",
         "logs:PutLogEvents"
     ]
     effect = "Allow"
     resources = [ "*" ]
     sid = "CreateCloudWatchLogs"
   }
 
   statement {
     actions = [
         "codecommit:GitPull",
         "codecommit:GitPush",
         "codecommit:GitBranch",
         "codecommit:ListBranches",
         "codecommit:CreateCommit",
         "codecommit:GetCommit",
         "codecommit:GetCommitHistory",
         "codecommit:GetDifferences",
         "codecommit:GetReferences",
         "codecommit:BatchGetCommits",
         "codecommit:GetTree",
         "codecommit:GetObjectIdentifier",
         "codecommit:GetMergeCommit"
     ]
     effect = "Allow"
     resources = [ "*" ]
     sid = "CodeCommit"
   }
}
 
resource aws_iam_policy lambda {
   name = "${local.prefix}-lambda-policy"
   path = "/"
   policy = data.aws_iam_policy_document.lambda.json
}
 
resource aws_lambda_function git {
 depends_on = [
   null_resource.ecr_image
 ]
 function_name = "${local.prefix}-lambda"
 role = aws_iam_role.lambda.arn
 timeout = 300
 image_uri = "${aws_ecr_repository.repo.repository_url}@${data.aws_ecr_image.lambda_image.id}"
 package_type = "Image"
}
 
output "lambda_name" {
 value = aws_lambda_function.git.id
}

This Terraform code was tested using Terraform version 0.14.8.

In this example we’re using the following terraform resources:

  • aws_ecr_repository – creates an ECR registry where Terraform will save Docker container image, which will be later used by out Lambda function
  • null_resource – is used to build Docker container and push it to the ECR registry, triggers checks changes in the Lambda function code and Dockerfile and allows Terraform understand when to rebuild the image and update the Lambda function
  • aws_ecr_image – allows us to query information about published Docker image
  • aws_iam_role, aws_iam_policy_document and aws_iam_policy – declares a permissions (send logs to CloudWatch, CodeCommit access) for the Lambda function
  • aws_lambda_function – Lambda function declaration itself

Deployment

To test solution you need to deploy Terraform code first:

terraform init
terraform apply -auto-approve

Then you need to execute the Lambda function:

aws lambda invoke --function-name git-lambda out --log-type Tail --query 'LogResult' --output text |  base64 -d

Here’s an expected output:

START RequestId: b8b742d6-5bd6-4098-90e3-5e30f5c6e816 Version: $LATEST
[INFO]  2021-03-16T02:10:28.064Z        b8b742d6-5bd6-4098-90e3-5e30f5c6e816    Event: {}
[INFO]  2021-03-16T02:10:28.064Z        b8b742d6-5bd6-4098-90e3-5e30f5c6e816    Cloning repo: https://github.com/andreivmaksimov/aws-config-rules
END RequestId: b8b742d6-5bd6-4098-90e3-5e30f5c6e816
REPORT RequestId: b8b742d6-5bd6-4098-90e3-5e30f5c6e816  Duration: 4069.15 ms    Billed Duration: 6131 ms        Memory Size: 128 MB     Max Memory Used: 83 MB  Init Duration: 2061.73 ms

Cleaning up

To cleanup everything, execute the following command:

terraform destroy

Summary

In this article we built a Docker container for AWS Lambda function and deployed the entire solution using Terraform. We hope you found this article useful. If so, please, help us to spread it to the world. If you have any questions, please, feel free to ask them in the chat section below.

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Subscribe to our updates

Like this article?

Share on facebook
Share on Facebook
Share on twitter
Share on Twitter
Share on linkedin
Share on Linkdin
Share on pinterest
Share on Pinterest

Want to be an author of another post?

We’re looking for skilled technical authors for our blog!

Leave a comment

If you’d like to ask a question about the code or piece of configuration, feel free to use https://codeshare.io/ or a similar tool as Facebook comments are breaking code formatting.