Table of contents
- Processing Files Uploaded To Amazon S3
- Fan Out Ingested Data Stream
- Serverless Website
- Custom User Authentication Workflow
- Managing HTTP Requests and Responses
- Automation Of Customer Onboarding Journey
- Real-time Ingested Data Transformation
- Serverless CRON Jobs
- Real-Time Log Monitoring, Analysis And Alarming
- Building Serverless Chatbots
- Serverless IoT Backend
- Serverless Workflow Orchestration
- Auditing And Processing DB Data Changes
- Creating Alexa Skills
- Serverless Fan-Out Architecture
- Related articles
This article covers the Serverless articles series and offers a quick overview of possible AWS Lambda use cases. Every use case contains a link to a resource with more information.
Processing Files Uploaded To Amazon S3
When you upload something to the Amazon S3 bucket, it can notify AWS Lambda. AWS Lambda then can react to the notification event and something meaningful.
Working with AWS Lambda in Python using Boto3
The diagram below illustrates how the integration works.
The sequence of actions:
- The user uploads a file to the S3 bucket.
- Amazon S3 sends JSON events to AWS Lambda with the information about the uploaded file.
- Lambda function reacts to the received event.
This is a perfect architecture to solve the following problems:
- Generate thumbnails or GIFs from uploaded to S3 bucket videos.
- Generate PDF from Word file.
- Copy uploaded file to another S3 bucket in another AWS Region for disaster recovery.
More information: Tutorial: Using AWS Lambda with Amazon S3
Fan Out Ingested Data Stream
Amazon Kinesis, SQS, or DynamoDB can trigger Lambda functions. It invokes your code which can make something with the data in real-time.
The following diagram shows how you can process data ingested to a Kinesis.
The sequence of actions:
- The web, mobile, IoT, or custom application writes records to a Kinesis stream.
- Lambda function is invoked when the new records are detected in the stream.
- Lambda function processes the data and sends it to another service.
This is a perfect architecture to solve the following problems:
- Buffering of incoming data to reduce the number of Lambda executions processing.
- Real-time stream processing and monitoring solutions (ingested data can be analyzed using Kinesis Data Analytics).
- Real-time stream fan-out architecture allows you to save your data to multiple places like S3 and DynamoDB and send it to other Kinesis Data Streams for further processing.
Maintaining a dedicated server is not cool nowadays, even if it is a virtual server. Server maintaining operations like provisioning, updating, and patching take a lot of time. It distracts you from focusing on your business problems.
You can use Lambda with other services to build a web app without managing any servers. The templated example is the use of AWS API Gateway, DynamoDB, Amazon S3, and Amazon Cognito User Pool.
The main components here, which we’re using:
- API Gateway and AWS Lambda.
- DynamoDB is a NoSQL database that stores website data.
- Amazon Cognito is managing and authenticating users. It helps to secure backend APIs and provides unique features to your website users.
The architecture shows the basic serverless website. Adding other AWS services can enhance it into a more complex solution to serve your needs.
Extra reading: Build your first Serverless Web Application.
Custom User Authentication Workflow
Personalization of your users’ experience plays a big role in any project. Whether you’re running a personal blog or a complex website. Amazon Cognito and AWS Lambda can help with this personalization.
Let’s take a look at the example below. User management operations, like user registration, validation, or login, trigger different Lambda functions. You can protect your website from automatic account creation fraud. Start sending personalized account verification messages.
The common triggering sources where you can hook your Lambda function:
- Sign-up, confirmation, and sign-in.
- Pre and post-authentication.
- Custom authentication challenge.
- Pre token generation.
- Migrate user.
- Custom message.
Let’s take a look at how custom email or phone notifications work. Amazon Cognito triggers the Lambda function. Lambda function customizes the message for the user before sending it.
The triggering sources for the custom messages are:
- Confirmation code post-sign-up.
- The temporary password for new users.
- Resending confirmation code.
- Confirmation code to forget password request.
- A manual request for a new email/phone.
- Multi-factor authentication.
Extra information: Customizing User Pool Workflows with Lambda Triggers.
Managing HTTP Requests and Responses
As your web project grows, you may start experiencing a strong need to modify HTTP requests or responses. There’re a lot of possible use cases, and here are some of them:
- Resizing the image based on user queries to better handle mobile clients.
- Serving WebP images for Chrome and Firefox browsers and JPEG/PNG for the rest.
- Personalize Content by Country or Device Type Headers.
Lambda@Edge and CloudFront integration can help to solve those problems. You have four places where you can use Lambda in combination with CloudFront:
- Lambda 1 – Function can process HTTP requests before they get to CloudFront.
- Lambda 2 – Function can change HTTP requests after CloudFront has processed them.
- Lambda 3 – Function can process HTTP responses before they reach CloudFront.
- Lambda 4 – Function can change HTTP responses from the CloudFront.
Here’s what you can implement using those Lambda functions:
- Inspect cookies.
- Rewrite URLs.
- Perform A/B testing.
- Serve website content based on the User-Agent header.
- Put in place access control logic for incoming requests.
- Add, delete, or change HTTP headers.
- Support redirects for the old URLs.
- Make HTTP requests to other Internet resources and inject those results into your responses.
More information: Lambda@Edge Example Functions
Automation Of Customer Onboarding Journey
The architecture above illustrates how:
- New user registrations on your website can trigger the email onboarding process.
- New orders at your online shop can begin an upselling marketing email.
You can add CloudWatch Scheduled Events to send marketing campaigns by schedule.
Extra information: Amazon Pinpoint Journeys
Real-time Ingested Data Transformation
You can use Kinesis Firehose for ingesting real-time streaming data to S3, Redshift, or Elasticsearch. These services allow you to simplify data importing tasks. But what if you need to change data on the fly? Here’s where AWS Lambda comes into play.
Some of the most common problems which AWS Lambda helps to solve:
- Data normalization.
- Doing ETL transformations.
- Merging data from several data sources.
- Converting/transforming data by the destination requirements.
- Adding metadata to the ingested data.
The major benefit of using Lambda integration with Kinesis is almost unlimited scalability.
Lambda and Kinesis help you to transform the data from the following producers:
- Kinesis enabled applications on your EC2 instances.
- Mobile applications.
- Web applications.
- IoT devices.
Serverless CRON Jobs
CRON jobs are a common practice to automate routine IT operations in the cloud. Cloud CRON can save time and effort in managing distributed environments. CloudWatch Events integrated with AWS Lambda helps you to achieve this goal.
Create a Lambda function and execute it by the schedule using CloudWatch Events.
Some real-life examples are:
- Stopping unpaid subscriptions on your website.
- Sending out the newsletter on fixed timings.
- Cleaning up the database cache at a regular interval.
- Backing up your EC2 instances or EFS shares.
Extra information: Cloud CRON – Scheduled Lambda Functions.
Real-Time Log Monitoring, Analysis And Alarming
CloudWatch Events give you a near real-time stream of events. That events describe changes in your environment. And you can immediately react to any changes and take action if needed.
For example, you can react to such events in the following ways:
- Send informational or alarming messages to external messengers like Slack.
- Create Jira tickets.
- Do corrective changes.
- Capture state information.
Many different automation scenarios become available when using CloudWatch Event rules. Those rules are routing events to Lambda functions for further processing.
There are two different scenarios, which you need to be aware of:
- You can trigger the Lambda function when the SQS queue grows above a certain threshold.
- You can trigger the Lambda function when a particular text is found in the logs. For example, the log record contains the “Exception” keyword.
More information: Getting helpful CloudWatch alarms in Slack.
Building Serverless Chatbots
Building chatbots from scratch may be a time-consuming and expensive challenge. First, you write a chatbot code, deploy it, and run it at scale to support the business logic.
AWS did the heavy lifting and gave us all we needed to build and run a scalable chatbot for our clients with ease.
Here’s what you need:
- Amazon Lex allows us to create a conversational interface for our bot.
- AWS Lambda will enable us to fulfill the intent given by the Amazon Lex service with any action.
And again, no infrastructure. You’re paying for services only for requests you’re making to them.
More information: Bots Just Got Better with .NET and the AWS Toolkit for Visual Studio.
Serverless IoT Backend
Managing hundreds of thousands of IoT devices is challenging. Creating a simple and easy-to-use managing IoT devices interface for your clients is even harder.
All the devices or “things” like smart light bulbs, alarm buttons, door locks, or video cameras need to be registered centrally in a DB with additional meta-information (address, geolocation coordinates, placement in the room or at the floor plan, etc.).
Above, you can find examples of how to use Lambda, S3, DynamoDB, and Cognito to build a web application. If you add AWS IoT to them, you’ll get a serverless IoT device management platform.
Serverless Workflow Orchestration
Any modern apps have backend workflows executed by schedule or response to events.
To support those workflows, AWS has a service named AWS Step Functions. This service is the standard de-facto for workflow orchestration in the AWS cloud.
You are using JSON-based specification language for process declaration. Connecting different AWS services into a single workflow makes it very easy.
Here’s an example from our article AWS Step Functions – How to manage long-running tasks:
Some of the examples where you can use Step Functions and Lambda together:
- Simple and complex workflow orchestration.
- Coordinate tasks in distributed serverless applications.
- Automating Machine Learning workflows.
- Orchestrating ETL jobs.
Auditing And Processing DB Data Changes
We already covered the Serverless Website use-case. Now, let’s go a bit deeper and imagine that we need to react to our DynamoDB database changes. As soon as it becomes a case, it would be best to think about AWS Lambda and DynamoDB integration – DynamoDB Streams.
DynamoDB can call Lambda every time any data changes happen. Changes data is available in the Lambda function in a JSON data structure.
You use this data for:
- Data Filtering – you can confirm data input and correct or revert it based on your rules.
- Monitoring – you can react to events like new user registration or profile update.
- Auditing – you can review data changes and revert them if needed.
- Notifications – you can track your user’s activity and send reports about it. For example, send a report about the most performed employee during the last working hour.
More information: AWS Lambda – How to process DynamoDB streams
Creating Alexa Skills
If you’re unfamiliar with this technology, Alexa is an Amazon smart speaker device and voice assistant.
Every skill that Alexa has is powered by cloud technology.
The examples are:
- Check the weather.
- Read the news.
- Amazon delivery status notifications.
Alexa service does all the heavy lifting for you. You do not have to deal with natural language processing or understand your user’s meaning.
Every time Alexa asks about something, it launches a Lambda function to process the request, and you can write a Lambda function to do something meaningful in response.
Here are some examples of the Alexa skills that you can create:
- Ask Alexa about your website revenue.
- Turn on the lights in the room.
- Ask for your favorite pizza delivery.
More information: Rapidly Create Your Alexa Skill Backend with AWS CloudFormation
Serverless Fan-Out Architecture
One of the common challenges in the Serverless world is building fan-out architecture.
There are three ways of doing that:
Fan-out architecture is an ideal choice if you need to parallelize your workloads.
Here are some use-cases:
- Data processing data pipeline needs to save the data at various stages.
- Evaluating Machine Learning model with human results validation.
- Converting image files to different formats.
SNS is a great solution when you need to send the same message to many destinations. SNS integration with Lambda is a perfect building block for serverless fan-out architecture.
Here’s an example of converting an uploaded image to different formats. S3 can send its updates to the SNS topic so we can build the following solution.
EventBridge is a service bus and universal standard for events routing and processing. The idea is the same as in the case with SNS. You build a system that receives messages from your partners. EventBridge downstream it to your internal systems for further processing.
Step Functions allows you to orchestrate different workflows which have various complexities. The major component of almost any workflow is the Lambda function. Step Functions enable you to run several tasks in parallel.
Here’s an example of processing user registration on the website.
Here we’re doing the following:
- Updating website DB.
- Subscribing users to an onboarding email sequence.
- Confirming user about account creation.
Of course, you may change this workflow, whatever you like.
More information: Reducing custom code by using advanced rules in Amazon EventBridge
This article continued to cover the Serverless articles series and covered some possible AWS Lambda use cases.
We hope you found this article useful. If yes, please, help us to spread it to the world!
I’m a passionate Cloud Infrastructure Architect with more than 15 years of experience in IT.
Any of my posts represent my personal experience and opinion about the topic.