AWS Step Function Loop Over Array Example

One of the most common questions about using AWS Step Functions is how to walk through and process arrays. In this AWS Step Function loop over array example, I’ll show you how to do it.

The problem

Let’s say AWS Lambda Function returns a simple array containing fruit elements. We need to process these elements by another Lambda Function. How could we build such kind of microservice interaction using Step Functions?

Lambda Functions

Let’s create a Lambda Function that produces an input array (fruits in our example):

exports.handler = (event, context, callback) => {
  var fruits = ['apple', 'orange', 'pinaple'];
  event.fruits = fruits;
  if (event.fruits.length > 0) {
    event.has_elements = true;
  } else {
    event.has_elements = false;
  }
  callback(null, event);
};

Pay attention to the Lambda function from the previous example that I’m adding has_elements flag from the input array to the output of the intrinsic function. We’ll use this flag to check if we need to process more elements using the map state, which will iterate over the JSON array.

Let’s create another Lambda Function that will process array elements:

exports.handler = (event, context, callback) => {
  let fruit = event.fruits.shift();
  // Process array element
  console.log('Processing fruit: ' + fruit);
  // Array still has elements:
  if (event.fruits.length > 0) {
    event.has_elements = true;
  } else {
    event.has_elements = false;
  }
  // Log array elements (for demonstration purpose)
  console.log('Elements in array: ' + event.fruits);
  console.log('Array has more elements: ' + event.has_elements);
  callback(null, event);
};

The main idea is to remove processed elements from the array. We’re using .shift() method for that purpose here for this intrinsic function, which will remove the first element from the array.

And again, we’re setting up has_elements flag at the end of element processing.

Now we’re ready to create a Step Function algorithm.

AWS Step Function Loop Over Array

Here’s the following example of our algorithm visualization:

And our AWS Step Functions workflow:

{
  "Comment": "Array processing example",
  "StartAt": "GetArray",
  "States": {
    "GetArray": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:012850762433:function:array_example",
      "ResultPath": "$",
      "Next": "CheckArrayForMoreElements"
    },
    "CheckArrayForMoreElements": {
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.has_elements",
          "BooleanEquals": true,
          "Next": "ProcessArrayElement"
        }
      ],
      "Default": "Done"
    },
    "ProcessArrayElement": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:012850762433:function:array_item_process_example",
      "ResultPath": "$",
      "Next": "CheckArrayForMoreElements"
    },
    "Done": {
      "Type": "Pass",
      "End": true
    }
  }
}

This algorithm has a state machine that consists of values, several steps, and intrinsic functions, which have the following purposes in the reference path:

• GetArray – Step Function Task, which will execute array_example Lambda function properly, which will return every single object(fruits) in the JSON array of objects with a passing state:

{
  "fruits": ["apple", "orange", "pinaple"],
  "has_elements": true
}

• CheckArrayForMoreElements – Conditional Step to check if we have more arrays containing the JSON objects to process. If yes, go to ProcessArrayElement Step as a response, otherwise finish going to Step Done
• ProcessArrayElement – Step Function Task state will process an array item with a map state that runs in parallel states. For example, it states input data for the last single object and passes the object input to its output without performing work using the pass state:

{
  "fruits": ["pinaple"],
  "has_elements": true
}

And the final output has a single value in it with the following parameters:

{
  "fruits": [],
  "has_elements": false
}

Summary

This article described the AWS Step Functions arrays from the AWS documentation and how to implement a simple Step Function algorithm to orchestrate an array of elements processing. I hope you’ll find this article useful. If you have any questions, contact me in the comments!