AWS Step Functions Tutorial

Automation is essential for any small company or large-scale enterprise to survive in the modern world. With cloud computing services such as AWS Lambda and Step Functions, you can automate any business or technical process easily and at no cost. This AWS Step Functions tutorial describes how to start using AWS Step Functions to automate your routine tasks using modern Serverless cloud technologies.

What are AWS Step Functions

AWS Step Functions is an orchestration service that allows you to use AWS Lambda functions and other AWS services to build and automate business-critical workflows. It allows you to describe your processes as a series of event-driven steps, each powered by AWS Lambda or other supported AWS services and responsible for its part of the process. Every workflow is described in JSON-like syntax.

Here’s a list of currently supported services, which you can directly call from Step Functions:

• AWS Lambda – Working with AWS Lambda in Python using Boto3
• DynamoDB – Working with DynamoDB in Python using Boto3
• AWS Batch – Working with AWS Batch in Python using Boto3
• Amazon SNS – Working with Amazon SNS in Python using Boto3
• Amazon SQS – Working with Amazon SQS in Python using Boto3
• Amazon ECS – Working with ECS in Python using Boto3
• Amazon EMR – Using Boto3 library to manage EMR clusters
• Amazon SageMaker
• AWS Step Functions – Working with Step Functions in Python using Boto3
• AWS Codebuild
• Amazon Athena – Working with Athena in Python using Boto3
• Amazon EKS – How to Set Up Amazon EKS Cluster Using Terraform
• API Gateway – Managing Amazon API Gateway using Terraform

For unsupported services, for example, when you need to call a third-party API or another cloud provider service, you need to use AWS Lambda.

AWS Step Functions tie all steps in a particular execution order to form a single automated workflow (state machine) and allow you to control execution flow based on the outputs of every executed step. The most straightforward workflow may contain only one step.

Here’s an example.

A more complex workflow might include many different steps and have complex execution logic, for example, a CICD automation workflow:

As soon as the workflow is described, you can launch it manually or in an automated way.

All Step Functions workflows can scale on-demand and handle a virtually unlimited amount of executions. If you’re using AWS Lambda as a step of Step Function workflow, remember that its execution time is limited to 15 minutes. If the step might take longer, use the asynchronous execution model or split your task into smaller subtasks.

The workflows for Step Functions can be easily created using an interactive AWS console. In addition to that, every workflow execution is easily traceable, allowing you to easily see which step is executing right now or which has failed.

Concepts of AWS Step Functions

Before we dive into implementing a new Step Function, we must understand certain terminologies we will use throughout the article.

A Step function defines an automated workflow or a state machine. Each state machine can receive input data (JSON format) and produce output data after the execution.

Every state machine consists of Tasks and States. Each Task can receive input data and produce output data. That’s how you can control data flow in your state machine.

Let’s look at simple Step Function workflow syntax and its graph.

{
  "Comment": "A Hello World example of the Amazon States Language using an AWS Lambda function",
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloFunction",
      "End": true
    }
  }
}

This declaration leads to the following state machine graph.

State

Each step (State) in a state machine executes a service or makes a decision. A state can be a task to execute, a conditional choice operator, or perform another action. Finally, it can lead to the next stage, the end of a state machine.

Each state is described as a JSON block in the state machine, as shown below.

For our HelloWord state machine, here is the State definition:

"HelloWorld": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloFunction",
      "End": true
}

Tasks

Tasks are the atomic unit of a state machine that does the actual job inside a State. A task is responsible for conditionally choosing the execution, performing the desired action, producing output, and returning it to the state. A task can call a supported AWS resource or service or take necessary input from humans. A task is contained inside a state definition as shown below:

{
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:HelloFunction",
      "End": true
}

State Machine

A state machine collects multiple states tied together to define an automated flow. The example state machine contains a single state – HelloWorld. This single state is used to define this state machine. The state machine is the “brain” that manages the orchestration of all the states defined inside it.

Getting started

To get started with AWS Step Functions, make sure you have the following prerequisites:

• You need to have an active AWS account
• You need to have basic knowledge of creating and executing a Lambda function
• You need to be familiar with JSON syntax

Once ready, you can go ahead and create your first step machine.

AWS Console

The easiest way to build a Step Functions state machine is to use an AWS Step Functions service and its wizard at the AWS console.

Click the “Get Started” button to move to the next screen.

It will display a sample function as shown below:

There are two main sections on the screen:

• Code – here, you’re defining your state machine. Please browse through the code and try to understand what it is supposed to do. Currently, the sample code is in read-only mode, but we’ll change it as soon as we walk through the wizard.
• Graph: Your code is shown as an image or directional graph. This graph gets dynamically updated as you change the code.

Once you have read the code and matched the graph, you can move to the next step to create your Step Function.

Click the Next button.

The next step will ask you to provide basic details like Step Function workflow name, IAM role, logging, tracing, and tagging configuration.

For this first function, let’s continue with the defaults.

Click Create state machine button.

Workflow execution

After successful creation, you will see a dialog for executing the state machine as shown below:

Click the Start execution button.

Congratulations! You just successfully created and ran your first AWS Step function.

Look through the output graph and click each green State block in the image to get additional details on the execution.

Each state will have a name, resource type, and execution state information.

Click the Step input and Step output tab after Details to see inputs and outputs for every step.

Integration with AWS Lambda

In the real world, this state machine is useless, and now is a good time to go ahead and create a real workflow powered by AWS Lambda. I recommend you use the AWS Step Functions – How to manage long-running tasks guide as the next step.

Benefits of using Step Functions

The benefits of a flexible and traceable workflow are endless.

Below are a few of the most important benefits:

• Virtually not limited to workflow execution – you can easily build long-running workflows. This service allows the execution of each workflow fully serverless, making it very cost-effective. A single workflow has a hard execution limit set to 1 year.
• Manual approvals – Require manual approvals in your workflows. I recommend you the Implementing Serverless Manual Approval Steps in AWS Step Functions and Amazon API Gateway article as a solution.
• Easy workflow traceability – without this service, you would go to every Lambda function log to gather the required information. This could get cumbersome with more Lambda functions getting involved. Step functions make it easy for you. In case of failure, you’ll get the Lambda function stack trace right in the failed State error message. Tracing workflow execution is also very easy.

Summary

In this article, we described the basics of the AWS Step Functions and built an example workflow. We also provided the most important benefits of using it. This service solves a large real-world problem of orchestrating automated workflows. With extensive traceability, serverless infrastructure, and ease of execution – AWS Step Functions is the future of Serverless applications and automation.

For a more hands-on experience using Step Functions, consider the “The AWS Step Functions Workshop” (provided by AWS for free).

We hope that the article was useful for you. If so, please, help us to spread it to the world!