Using Terraform to deploy S3-SQS-Lambda integration

Terraform SQS Lambda Integration Example

Capturing S3 events with SQS and processing them with AWS Lambda allows for building simple and cost-effective solutions for processing user-uploaded content while ensuring that all events have been processed. SQS can store the messages until the Lambda function is polling them. This method is more reliable and efficient than directly subscribing to notifications from S3, especially when processing exceptions might happen.

In Terraform SQS Lambda integration example demonstrates how this integration works and how to deploy it using Terraform.

Here’s what the architecture diagram for this integration looks like.

Using Terraform to deploy S3-SQS-Lambda integration - Architecture

Events structures

The integration is very easy to understand as soon as you can see the event messages sent from one service, not another, as soon as the file is uploaded to the S3 bucket.

Jumping ahead, the S3 event is transferred within the SQS message body, and that’s how the AWS Lambda function code can get access there (will be shown below).

S3 event structure

S3-generated events are widely used and well-covered over the internet.

Here’s what this event structure looks like:


The S3 event consists of a list of records describing the object within the S3 bucket.

The most commonly used fields are:

  • event['Records'][*]['s3']['bucket']['name'] – gives us a bucket name where the file has been uploaded
  • event['Records'][*]['s3']['object']['key'] – gives us a file name and location within the S3 bucket

In addition to the file upload events, S3 sends test event to test, which has the following structure:

   "Service":"Amazon S3",

We’ll need to skip this event in our Lambda function code to avoid error messages.

SQS event structure

SQS events are widely used, and they have a similar structure.


SQS event consists of a list of records representing a message grabbed from the SQS queue.

The most common field here is:

  • event['Records'][*]['body'] – this field contains the text body of the SQS message

Demo Terraform SQS Lambda function

As soon as we know the event structures for both events, we can easily write a demo Lambda function to process those events. Check the Using Lambda with Amazon SQS documentation for more information.

Here’s what the code looks like (

#!/usr/bin/env python3
import logging
import json
LOGGER = logging.getLogger()
def handler(event, context):
    try:'SQS EVENT: %s', event)
        for sqs_rec in event['Records']:
            s3_event = json.loads(sqs_rec['body'])
  'S3 EVENT: %s', s3_event)
            # Skipping S3 test event
            if 'Event' in s3_event.keys() and s3_event['Event'] == 's3:TestEvent':
            for s3_rec in s3_event['Records']:
      'Bucket name: %s', s3_rec['s3']['bucket']['name'])
      'Object key: %s', s3_rec['s3']['object']['key'])
    except Exception as exception:
        LOGGER.error('Exception: %s', exception)

This Lambda function code processes SQS events.

The code extracts the S3 message from the SQS message body field for every received event.

We’re checking if we received the test event and skipping it.

Finally, the code logs events, S3 bucket name, and uploaded S3 object key.

This minimal working code is useful if you want to use it quickly in your project.

Note: try-except statement allows us to log Python exceptions if something goes wrong.

Terraform code

As soon as we have a Lambda function and understand how all the pieces need to be tied together, we can declare everything in Terraform.

Project structure

Here’s the project structure which we’ll use for our project:

 ├── lambda
 │   └──
 1 directory, 5 files

Provider configuration

Simple AWS provider configuration is declared in file:

# AWS provider configuration
provider "aws" {
    region = var.region


We parameterize our Terraform code with the following variables defined in file:

variable "region" {
    default = "us-east-1"
    description = "AWS Region to deploy to
variable "app_env" {
    default = "amaksimov"
    description = "Common prefix for all Terraform created resources"

S3 and SQS

Declaration and integration of the S3 bucket and SQS have been put to the file:

# SQS queue
resource "aws_sqs_queue" "queue" {
  name = "${var.app_env}-s3-event-notification-queue"
  policy = <<POLICY
  "Version": "2012-10-17",
  "Statement": [
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:*:*:${var.app_env}-s3-event-notification-queue",
      "Condition": {
        "ArnEquals": { "aws:SourceArn": "${aws_s3_bucket.bucket.arn}" }
# S3 bucket
resource "aws_s3_bucket" "bucket" {
  bucket = "${var.app_env}-s3-sqs-demo-bucket"
# S3 event filter
resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket =
  queue {
    queue_arn     = aws_sqs_queue.queue.arn
    events        = ["s3:ObjectCreated:*"]
# Event source from SQS
resource "aws_lambda_event_source_mapping" "event_source_mapping" {
  event_source_arn = aws_sqs_queue.queue.arn
  enabled          = true
  function_name    = aws_lambda_function.sqs_processor.arn
  batch_size       = 1

Lambda function

We’ve put Lambda function-related resources and the Lambda function CloudWatch Log Group to the file:

# Data resource to archive Lambda function code
data "archive_file" "lambda_zip" {
    source_dir  = "${path.module}/lambda/"
    output_path = "${path.module}/"
    type        = "zip"
# Lambda function policy
resource "aws_iam_policy" "lambda_policy" {
    name        = "${var.app_env}-lambda-policy"
    description = "${var.app_env}-lambda-policy"
    policy = <<EOF
  "Version": "2012-10-17",
  "Statement": [
      "Action": [
      "Effect": "Allow",
      "Resource": "${aws_s3_bucket.bucket.arn}"
      "Action": [
      "Effect": "Allow",
      "Resource": "${aws_sqs_queue.queue.arn}"
      "Action": [
      "Effect": "Allow",
      "Resource": "*"
# Lambda function role
resource "aws_iam_role" "iam_for_terraform_lambda" {
    name = "${var.app_env}-lambda-role"
    assume_role_policy = <<EOF
  "Version": "2012-10-17",
  "Statement": [
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": ""
      "Effect": "Allow"
# Role to Policy attachment
resource "aws_iam_role_policy_attachment" "terraform_lambda_iam_policy_basic_execution" {
    role =
    policy_arn = aws_iam_policy.lambda_policy.arn
# Lambda function declaration
resource "aws_lambda_function" "sqs_processor" {
    filename = ""
    source_code_hash = data.archive_file.lambda_zip.output_base64sha256
    function_name = "${var.app_env}-lambda"
    role = aws_iam_role.iam_for_terraform_lambda.arn
    handler = "index.handler"
    runtime = "python3.8"
# CloudWatch Log Group for the Lambda function
resource "aws_cloudwatch_log_group" "lambda_loggroup" {
    name = "/aws/lambda/${aws_lambda_function.sqs_processor.function_name}"
    retention_in_days = 14


As soon as we create all Terraform project files, we can deploy this integration using the following commands:

terraform init
terraform apply -auto-approve


After the deployment, we can upload any file to the created S3 bucket and see the following CloudWatch messages in place.

Using Terraform to deploy S3-SQS-Lambda integration - CloudWatch Logs

Cleaning up

To clean up everything, you need to delete all the uploaded files from the S3 bucket and then execute the following Terraform command:

terraform destroy -auto-approve


In this article, we’ve created a widely used integration building block that consists of an S3 bucket, SQS queue, and Lambda function. The purpose of this building block is to give you a guarantee of processing every S3 upload event with the Lambda function in a Serverless way.

We hope that you found this article useful. If so, please, help us to spread it to the world!

Similar Posts