AWS Step Functions – How to process arrays

Andrei Maksimov

Andrei Maksimov

5
(1)

Nowadays, the Serverless computing topic becomes extremely popular. It is so tasty to implement the business processes using several AWS Lambda functions and pay a low price for its execution. But everybody who tried to implement several Lambda functions interactions knows that it is not an easy task.

On the 1st of December of 2016 AWS announced, that they brought us a Step Functions – a visual workflow way to coordinate microservices. The main idea of the service is to give us the ability to describe the whole microservice architecture as a finite state machine and visualize it of cause.

One of the most common questions of using Step Functions is how to walk through and process arrays. In this post, I’ll show you how to do it.

The problem

Let’s imagine that we have a Lambda Function that returns an array of elements. We need to process these elements by another Lambda Function. How could we build such kind of microservice interaction using Step Functions?

Lambda Functions

Let’s create a Lambda Function that produces an array (fruits in our example):

exports.handler = (event, context, callback) => {
  var fruits = ['apple', 'orange', 'pinaple'];

  event.fruits = fruits;

  if (event.fruits.length > 0) {
    event.has_elements = true;
  } else {
    event.has_elements = false;
  }

  callback(null, event);
};

Pay attention that I’m adding has_elements flag to the function output. We’ll use this flag to check if we need to process more elements.

Let’s create another Lambda Function that will process array elements:

exports.handler = (event, context, callback) => {
  let fruit = event.fruits.shift();
  // Process array element
  console.log('Processing fruit: ' + fruit);

  // Array still has elements:
  if (event.fruits.length > 0) {
    event.has_elements = true;
  } else {
    event.has_elements = false;
  }

  // Log array elements (for demonstration purpose)
  console.log('Elements in array: ' + event.fruits);
  console.log('Array has more elements: ' + event.has_elements);

  callback(null, event);
};

The main idea is to remove processed elements from the array. We’re using .shift() method for that purpose here, which will remove the first element from the array.

And again, we’re setting up has_elements flag at the end of element processing.

Now we’re ready to create a Step Function algorithm.

Step Function

Here’s our algorithm visualization:

AWS-Step-Functions-How-to-process-arrays-Example

And our Step Function algorithm:

{
  "Comment": "Array processing example",
  "StartAt": "GetArray",
  "States": {
    "GetArray": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:012850762433:function:array_example",
      "ResultPath": "$",
      "Next": "CheckArrayForMoreElements"
    },
    "CheckArrayForMoreElements": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.has_elements",
          "BooleanEquals": true,
          "Next": "ProcessArrayElement"
        }
      ],
      "Default": "Done"
    },
    "ProcessArrayElement": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:012850762433:function:array_item_process_example",
      "ResultPath": "$",
      "Next": "CheckArrayForMoreElements"
    },

    "Done": {
      "Type": "Pass",
      "End": true
    }
  }
}

This algorithm consists of several steps:

  • GetArray – Step Function Task which will execute array_example Lambda functions, which will return fruits array:
{
  "fruits": ["apple", "orange", "pinaple"],
  "has_elements": true
}
  • CheckArrayForMoreElements – Conditional Step to check if we have more array elements to process. If yes, go to ProcessArrayElement Step, otherwise finish going to Step Done
  • ProcessArrayElement – Step Function Task that will process an array item. For example, it’s Input for last element:
{
  "fruits": ["pinaple"],
  "has_elements": true
}

And the final Output:

{
  "fruits": [],
  "has_elements": false
}

Summary

We’ve just implemented a simple Step Function algorithm to orchestrate the processing of an array of elements. I hope you’ll find this article useful. If you have any questions, feel free to contact me in the comments!

How useful was this post?

Click on a star to rate it!

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Top rated Udemy Courses to improve you career

Subscribe to our updates

Like this article?

Share on facebook
Share on Facebook
Share on twitter
Share on Twitter
Share on linkedin
Share on Linkdin
Share on pinterest
Share on Pinterest

Want to be an author of another post?

We’re looking for skilled technical authors for our blog!

Leave a comment

If you’d like to ask a question about the code or piece of configuration, feel free to use https://codeshare.io/ or a similar tool as Facebook comments are breaking code formatting.