This article covers building a Serverless AWS CRON solution that runs code, launches scheduled jobs in the AWS cloud using CloudWatch Events from the AWS services, and has AWS Lambda scheduled at no cost.
Table of contents
As usual, we’re providing some additional reading to those of you who are interested. If you’re looking for a solution only, please, jump here.
Benefits Of Using Scheduled AWS Lambda Functions
In the old-school world of on-prem Cron jobs, a significant amount of time went into managing the execution of these tasks. The crontab syntax is difficult to read, which frequently leads systems engineers to rely on online crontab formatters to translate the archaic time and date formats using online crontab generators – this leads systemd.timer to include a modified, more readable implementation.
However, neither of these solves the inherent problems associated with the state. If you run a Cron job with a cron expression on a single server, and that server goes down, the Cron job might never run or run at the wrong time. If you put the job on multiple servers, you need to implement some distributed state storage to ensure there isn’t a duplicate execution. Spread this across an entire organization, and you end up with a web of disparate Cron implementations.
Thankfully, we can solve both of these problems and more with the AWS Lambda function. Lambda is the AWS service that provides serverless functions. “Serverless” functions, despite what the name implies, do run on servers. However, you, the user, do not need to manage the underlying server. You write your code in the language of your choice, upload it to a Lambda, and it executes it! The application auto-scales based on the workload, billing per 100 milliseconds of execution time, and consistent performance between runs.
This means you don’t need to worry about hosts going down or maintaining a host, as the AWS control plane automatically ensures you get a high-availability platform for running the Lambda.
There are two ways of using Lambdas as Cron jobs. First, we can use rate() it to execute a lambda at a given interval. This is configured using the CloudWatch events rate expressions format rate(<value> <unit>. For example, if I want to execute a Lambda once every five minutes, I would use rate(5 minutes). For once a week, I would use rate(7 days). Simple!
Cron jobs use a slightly different format but provide much flexibility in implementation. The cron syntax is cron(<minutes> <hours> <day of month> <month> <day of week> <year>). This is a bit complicated at first glance, but let’s walk through it by opening the AWS Lambda console(AWS CLI).
To run at 09:15 UTC every day, use the following at your AWS management console:
cron(15 9 * * ? *)To run at 18:00 UTC (6:00 PM) every day, use this AWS management console:
cron(0 18 ? * * ? *)The ? character operates as a wildcard, matching all possible values.
Let’s say we want to execute every 10 minutes on weekdays, getting a little fancier. We can do that with short names for weekdays.
cron(0/10 * ? * MON-FRI *)For a more detailed look at scheduling cron jobs with Lambdas, check out the upstream AWS documentation on cron jobs with Lambda and CloudWatch Events scheduling expressions.
AWS CRON – CloudFormation
Let’s create a simple AWS Lambda that deletes outdated EC2 AMIs and EBS Snapshots daily.
Here’s a CloudFormation template sample code with implementation.
AWSTemplateFormatVersion: 2010-09-09
Description: >
This CloudFormation template creates a Lambda function triggered
by the CloudWatch Scheduled Events, which deletes old EC2 AMIs
Parameters:
pAmiMaxAge:
Description: Max age in days for AMI
Type: Number
Default: 14
MinValue: 1
MaxValue: 65535
Resources:
LambdaFunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Path: "/"
Policies:
- PolicyName: LambdaFunctionPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: '*'
- Effect: Allow
Action:
- ec2:DescribeImages
- ec2:DescribeSnapshotAttribute
- ec2:DescribeSnapshots
- ec2:DeleteSnapshot
- ec2:DescribeImages
- ec2:DescribeImageAttribute
- ec2:DeregisterImage
- ec2:DescribeInstances
Resource: '*'
LambdaFunction:
Type: AWS::Lambda::Function
Properties:
Runtime: python3.6
Timeout: 900
Handler: index.handler
Role: !GetAtt LambdaFunctionRole.Arn
Code:
ZipFile:
!Sub
- |-
"""
Lambda function to remove outdated AMIs
All credits: https://gist.github.com/luhn/802f33ce763452b7c3b32bb594e0d54d
"""
import logging
import os
import re
import sys
from datetime import datetime, timedelta
import boto3
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
LOGGER = logging.getLogger()
LOGGER.setLevel(logging.INFO)
ACCOUNT_ID = '${lambda_account_id}'
AMI_MAX_AGE = int(${lambda_ami_max_age})
EC2 = boto3.resource("ec2")
EC2_CLIENT = boto3.client("ec2")
def handler(event, context):
"""handler method"""
# pylint: disable=R0914,C0103,W0613
my_amis = EC2_CLIENT.describe_images(Owners=[ACCOUNT_ID])['Images']
used_amis = {
instance.image_id for instance in EC2.instances.all()
}
LOGGER.info('used_amis: %s', used_amis)
fresh_amis = set()
for ami in my_amis:
created_at = datetime.strptime(
ami['CreationDate'],
"%Y-%m-%dT%H:%M:%S.000Z",
)
if created_at > datetime.now() - timedelta(AMI_MAX_AGE):
fresh_amis.add(ami['ImageId'])
LOGGER.info('fresh_amis: %s', fresh_amis)
latest = dict()
for ami in my_amis:
created_at = datetime.strptime(
ami['CreationDate'],
"%Y-%m-%dT%H:%M:%S.000Z",
)
name = ami['Name']
if(
name not in latest
or created_at > latest[name][0]
):
latest[name] = (created_at, ami)
latest_amis = {ami['ImageId'] for (_, ami) in latest.values()}
LOGGER.info('latest_amis: %s', latest_amis)
safe = used_amis | fresh_amis | latest_amis
for image in (
image for image in my_amis if image['ImageId'] not in safe
):
LOGGER.info('Deregistering %s (%s)', image['Name'], image['ImageId'])
EC2_CLIENT.deregister_image(ImageId=image['ImageId'])
LOGGER.info('Deleting snapshots.')
images = [image['ImageId'] for image in my_amis]
for snapshot in EC2_CLIENT.describe_snapshots(OwnerIds=[ACCOUNT_ID])['Snapshots']:
LOGGER.info('Checking %s', snapshot['SnapshotId'])
r = re.match(r".*for (ami-.*) from.*", snapshot['Description'])
if r:
if r.groups()[0] not in images:
LOGGER.info('Deleting %s', snapshot['SnapshotId'])
EC2_CLIENT.delete_snapshot(SnapshotId=snapshot['SnapshotId'])
-
lambda_account_id: !Ref "AWS::AccountId"
lambda_ami_max_age: !Ref "pAmiMaxAge"
ScheduledRule:
Type: AWS::Events::Rule
Properties:
Description: "ScheduledRule"
ScheduleExpression: "rate(1 day)"
State: "ENABLED"
Targets:
-
Arn:
Fn::GetAtt:
- "LambdaFunction"
- "Arn"
Id: "TargetFunctionV1"
PermissionForEventsToInvokeLambda:
Type: AWS::Lambda::Permission
Properties:
FunctionName:
Ref: "LambdaFunction"
Action: "lambda:InvokeFunction"
Principal: "events.amazonaws.com"
SourceArn:
Fn::GetAtt:
- "ScheduledRule"
- "Arn"We first create a rule to be able to run our Lambda function. In this example, we’re declaring LaunctionRole with all necessary permissions to create a Lambda function, LambdaFunction, to create the AWS resources and manage Snapshots and Volumes. ScheduledRule CloudWatch Scheduled Event Rule is a scheduled rule/event from AWS CloudWatch to be executed daily. These event rules are allowed to execute our Lambda function because of PermissionForEventsToInvokeLambda. You can also start deleting AWS resources you might not need or use to prevent unnecessary charges to your AWS account. e
Lambda function code itself is also straightforward. Here’s the logic:
- Import all necessary libraries.
- Builds everal lists.
- All account AMIs.
- Used AMIs.
- Latest AMIs.
- Delete all AMIs which are safe to delete from this list.
- Deregistered unused snapshots.
CloudFormation deployment
To deploy this stack execute the following command:
aws cloudformation create-stack \
--stack-name "test-scheduled-lambda" \
--template-body "file://$(pwd)/cloudformation.yaml" \
--capabilities CAPABILITY_IAM
aws cloudformation wait stack-create-complete \
--stack-name "test-scheduled-lambda"CloudFormation cleanup
To clean up everything, we need to run the existing code below:
aws cloudformation delete-stack \
--stack-name "test-scheduled-lambda"AWS CRON – Terraform
We may implement the following code, but using Terraform
variable "aws_region" {
default = "us-east-1"
description = "AWS Region to deploy to
}
variable "env_name" {
default = "sheduled-lambda-cron"
description = "Terraform environment name"
}
variable "ami_max_age" {
default = "14"
description = "Max age in days for AMI"
}
data "archive_file" "delete_old_amis_lambda" {
source_dir = "${path.module}/lambda/"
output_path = "/tmp/lambda.zip"
type = "zip"
}
data "aws_caller_identity" "current" {}
provider "aws" {
region = "${var.aws_region}"
}
resource "aws_kms_key" "a" {}
resource "aws_kms_alias" "lambda" {
name = "alias/lambda"
target_key_id = aws_kms_key.a.key_id
}
resource "aws_iam_policy" "lambda_policy" {
name = "${var.env_name}_delete_old_amis_lambda_function"
description = "${var.env_name}_delete_old_amis_lambda_function"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"kms:ListAliases",
"kms:Decrypt"
],
"Effect": "Allow",
"Resource": "${aws_kms_alias.lambda.arn}"
},
{
"Action": [
"ec2:DescribeImages",
"ec2:DescribeSnapshotAttribute",
"ec2:DescribeSnapshots",
"ec2:DeleteSnapshot",
"ec2:DescribeImages",
"ec2:DescribeImageAttribute",
"ec2:DeregisterImage",
"ec2:DescribeInstances",
"kms:ListAliases",
"kms:Decrypt"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
EOF
}
resource "aws_iam_role" "delete_old_amis" {
name = "app_${var.env_name}_lambda_role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow"
}
]
}
EOF
}
resource "aws_lambda_function" "delete_old_amis" {
filename = "/tmp/lambda.zip"
source_code_hash = data.archive_file.delete_old_amis_lambda.output_base64sha256
function_name = "${var.env_name}_delete_old_amis_lambda"
role = "${aws_iam_role.delete_old_amis.arn}"
handler = "index.handler"
runtime = "python3.6"
timeout = 900
environment {
variables = {
ACCOUNT_ID = "${data.aws_caller_identity.current.account_id}",
AMI_MAX_AGE = "${var.ami_max_age}"
}
}
}
resource "aws_iam_role_policy_attachment" "delete_old_amis" {
role = "${aws_iam_role.delete_old_amis.id}"
policy_arn = "${aws_iam_policy.lambda_policy.arn}"
}
resource "aws_cloudwatch_event_rule" "delete_old_amis" {
name = "${var.env_name}_delete_old_amis"
description = "${var.env_name}_delete_old_amis"
schedule_expression = "rate(1 day)"
}
resource "aws_cloudwatch_event_target" "delete_old_amis" {
rule = "${aws_cloudwatch_event_rule.delete_old_amis.name}"
target_id = "lambda"
arn = "${aws_lambda_function.delete_old_amis.arn}"
}
resource "aws_lambda_permission" "cw_call_delete_old_amis_lambda" {
statement_id = "AllowExecutionFromCloudWatch"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.delete_old_amis.function_name}"
principal = "events.amazonaws.com"
source_arn = "${aws_cloudwatch_event_rule.delete_old_amis.arn}"
}
resource "aws_cloudwatch_log_group" "delete_old_amis" {
name = "/aws/lambda/${aws_lambda_function.delete_old_amis.function_name}"
retention_in_days = 14
}Terraform deployment
To deploy this stack execute the following command:
terraform init
terraform apply -auto-approveTerraform cleanup
To clean up everything, we need to run:
terraform destroy -auto-approveComplete source code
You can find the complete source code of the example on our GitHub
Summary
This article showed CloudFormation and Terraform for deploying and creating scheduled events/Lambda functions. We created a simple function that deleted outdated AMIs and Snapshots from your account.
We hope that this article will save you some time on your projects.
If you found this article useful, please, help us spread it to the world.
Stay tuned!
