At the end of 2020, AWS announced support of container images for Lambda. This feature allows you to package and deploy Lambda functions as container images of up to 10 GB in size. This Terraform Docker Lambda example covers how you can use Terraform to deploy Python Lambda functions backed by the container image.
Table of contents
One of the common tasks in the cloud world is to replicate source code repositories from on-premises to the cloud or between cloud environments. So, to illustrate the approach, we decided to add Git and GitPython support to the Lambda function.
Project structure
Here’s a project structure that we’ll be using during this demo:
$ tree lambda_container
lambda_container
├── README.md
├── lambdas
│ └── git_client
│ ├── Dockerfile
│ └── index.py
└── main.tf
2 directories, 4 files
lambdas
– the folder where we put Lambda functions source codemain.tf
– Terraform demo code, which will build a Docker container for the git_client Lambda function and deploy the function afterward
Dockerfile
Let’s describe a Docker container that will host all dependencies for our lambda functions. Here’s the Dockerfile
content:
FROM public.ecr.aws/lambda/python:3.8
RUN yum update -y && \
yum install -y git && \
rm -Rf /var/cache/yum && \
pip install git-remote-codecommit boto3 GitPython awscli
COPY index.py ${LAMBDA_TASK_ROOT}
CMD [ "index.handler" ]
We’re taking Python 3.8 public Docker image from Amazon as a base. Then we’re installing Git, cleaning the yum caches to make the container smaller, and installing required dependencies, which allow us to use Git with CodeCommit using IAM for authentication.
Next, we’re copying the index.py
file to the folder where the Lambda function code should reside. Check Using AWS Lambda environment variables for additional information.
Finally, we’re specifying to execute the handler method from the index.py file at the container launch.
Lambda code
Once the Lambda container declaration is finalized, we can write a Lambda function to use it. Here’s a code example showing how to clone the Git repository. I’m sure you’ll be able to adjust this example for your personal needs:
import logging
import os
import git
TMP_DIR = "/tmp"
REPO_DIR = 'aws-config-rules'
REPO_URL = f'https://github.com/andreivmaksimov/{REPO_DIR}'
CLONE_PATH = os.path.join(TMP_DIR, REPO_DIR)
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
def clone(branch='master'):
repo = git.Repo.clone_from(REPO_URL, CLONE_PATH, branch=branch)
with repo.config_writer() as git_config:
git_config.set_value('user', 'email', 'no-reply@hands-on.cloud')
git_config.set_value('user', 'name', 'Git Lambda')
def handler(event, context):
LOGGER.info('Event: %s', event)
LOGGER.info('Cloning repo: %s', REPO_URL)
clone()
In this code, we’re declaring required Python libraries, some constants, configuring logger, and a couple of functions:
def clone(branch='master')
– this function shows how to clone Git repositorydef handler(event, context)
– this function is the main entry point to the Lambda function, it logs the incoming event and callingclone
function
Terraform Docker Lambda code
Once we have Lambda code and declare its container, we can write Terraform code to automate the deployment. Here it is:
variable region {
default = "us-east-1"
}
provider aws {
region = var.region
}
data aws_caller_identity current {}
locals {
prefix = "git"
account_id = data.aws_caller_identity.current.account_id
ecr_repository_name = "${local.prefix}-demo-lambda-container"
ecr_image_tag = "latest"
}
resource aws_ecr_repository repo {
name = local.ecr_repository_name
}
resource null_resource ecr_image {
triggers = {
python_file = md5(file("${path.module}/lambdas/git_client/index.py"))
docker_file = md5(file("${path.module}/lambdas/git_client/Dockerfile"))
}
provisioner "local-exec" {
command = <<EOF
aws ecr get-login-password --region ${var.region} | docker login --username AWS --password-stdin ${local.account_id}.dkr.ecr.${var.region}.amazonaws.com
cd ${path.module}/lambdas/git_client
docker build -t ${aws_ecr_repository.repo.repository_url}:${local.ecr_image_tag} .
docker push ${aws_ecr_repository.repo.repository_url}:${local.ecr_image_tag}
EOF
}
}
data aws_ecr_image lambda_image {
depends_on = [
null_resource.ecr_image
]
repository_name = local.ecr_repository_name
image_tag = local.ecr_image_tag
}
resource aws_iam_role lambda {
name = "${local.prefix}-lambda-role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow"
}
]
}
EOF
}
data aws_iam_policy_document lambda {
statement {
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
]
effect = "Allow"
resources = [ "*" ]
sid = "CreateCloudWatchLogs"
}
statement {
actions = [
"codecommit:GitPull",
"codecommit:GitPush",
"codecommit:GitBranch",
"codecommit:ListBranches",
"codecommit:CreateCommit",
"codecommit:GetCommit",
"codecommit:GetCommitHistory",
"codecommit:GetDifferences",
"codecommit:GetReferences",
"codecommit:BatchGetCommits",
"codecommit:GetTree",
"codecommit:GetObjectIdentifier",
"codecommit:GetMergeCommit"
]
effect = "Allow"
resources = [ "*" ]
sid = "CodeCommit"
}
}
resource aws_iam_policy lambda {
name = "${local.prefix}-lambda-policy"
path = "/"
policy = data.aws_iam_policy_document.lambda.json
}
resource aws_lambda_function git {
depends_on = [
null_resource.ecr_image
]
function_name = "${local.prefix}-lambda"
role = aws_iam_role.lambda.arn
timeout = 300
image_uri = "${aws_ecr_repository.repo.repository_url}@${data.aws_ecr_image.lambda_image.id}"
package_type = "Image"
}
output "lambda_name" {
value = aws_lambda_function.git.id
}
This Terraform code was tested using Terraform version 0.14.8.
In this example, we’re using the following terraform resources:
aws_ecr_repository
– creates an ECR registry where Terraform will save Docker container image, which will be later used by our Lambda functionnull_resource
– is used to build a Docker container and push it to the ECR registry, triggers checks changes in the Lambda function code and Dockerfile, and allows Terraform to understand when to rebuild the image and update the Lambda functionaws_ecr_image
– allows us to query information about published Docker imageaws_iam_role
,aws_iam_policy_document
andaws_iam_policy
– declares permissions (send logs to CloudWatch, CodeCommit access) for the Lambda functionaws_lambda_function
– Lambda function declaration itself
Deployment
To test solution you need to deploy Terraform code first:
terraform init
terraform apply -auto-approve
Then you need to execute the Lambda function:
aws lambda invoke --function-name git-lambda out --log-type Tail --query 'LogResult' --output text | base64 -d
Here’s an expected output:
START RequestId: b8b742d6-5bd6-4098-90e3-5e30f5c6e816 Version: $LATEST
[INFO] 2021-03-16T02:10:28.064Z b8b742d6-5bd6-4098-90e3-5e30f5c6e816 Event: {}
[INFO] 2021-03-16T02:10:28.064Z b8b742d6-5bd6-4098-90e3-5e30f5c6e816 Cloning repo: https://github.com/andreivmaksimov/aws-config-rules
END RequestId: b8b742d6-5bd6-4098-90e3-5e30f5c6e816
REPORT RequestId: b8b742d6-5bd6-4098-90e3-5e30f5c6e816 Duration: 4069.15 ms Billed Duration: 6131 ms Memory Size: 128 MB Max Memory Used: 83 MB Init Duration: 2061.73 ms
Cleaning up
To clean up everything, execute the following command:
terraform destroy
Summary
This article built a Docker container for the AWS Lambda function and deployed the entire solution using Terraform. We hope you found this article useful. If so, please, help us to spread it to the world. If you have any questions, please, feel free to ask them in the chat section below.
Nice!