How to integrate Jenkins with CloudFormation and Step Functions

Andrei Maksimov
Andrei Maksimov

From my personal experience, many companies rely on Jenkins and Terraform while automating their cloud infrastructure. There’s no problem if you’d like to deploy something from Jenkins to the cloud, but what to do, if you want to call Jenkins job from to deploy a Terraform module as a Service Catalog product, or include Jenkins pipeline to a workflow orchestrated by StepFunctions? This article will describe a unified integration pattern, which will allow you to integrate Jenkins jobs with CloudFormation custom resources and Step Functions workflows.

Jenkins installation

There are many ways of deploying Jenkins to the AWS cloud. 

For the automated installation process of a single Jenkins server to AWS cloud use our How to Install Jenkins on the Latest Ubuntu in 5 Minutes article.

You may also be interested in Deploying Jenkins on Amazon EKS with Amazon EFS if you’re looking for a way to use Docker containers as Jenkins workers.

Whatever deployment method you use; as a result, you’ll get a Jenkins server whose workers can assume IAM roles in one or many AWS accounts to deploy required applications or services.

Now, we can integrate your Jenkins server with CloudFormation and Step Functions services.

Integration pattern

If you’re reading this article, you’re probably trying to solve one of the following problems:

  • How to call Jenkins APIs from other AWS services
  • How to provide access to your central Jenkins APIs from multiple AWS accounts

The solution is to use AWS Lambda, SQS, and SNS:

How to integrate Jenkins with CloudFormation and Step Functions - Integration pattern - Simple diagram

In this integration pattern:

  • AWS Lambda’s role is to make API calls to Jenkins using python-jenkins library.
  • Amazon SNS allows you to receive events from Step Function and CloudFormation services seamlessly.
  • Amazon SQS is responsible for sending, storing, and receiving messages for Jenkins at any volume without losing those messages if Jenkins (for any reason) becomes unavailable.

Here’s how the complete architecture diagram for one AWS account looks like:

How to integrate Jenkins with CloudFormation and Step Functions - Integration with Step Functions and Service Catalog

If you need to support muitiple AWS regions, you can easily expand such setup by putting SNS topics to the required AWS regions and integrate them directly to the SQS queue.

How to integrate Jenkins with CloudFormation and Step Functions - Integration with Step Functions and Service Catalog - Multi Region

You can configure SNS topic access policies to provide access to the topic from multiple AWS accounts and organizations. It is very easy to configure that because you can use Account IDs and AWS Organizations OUs:

Finally, as soon as the “Jenkins to AWS CloudFormation” integration is in place, you can use CloudFormation custom resources to run Jenkins jobs from the CloudFormation. That unlocks the capability of deploying Service Catalog products using Jenkins.


In this article, we’ve described a Jenkins integration with CloudFormation and Step Function AWS services, which allows you to use Jenkins jobs to deploy Service Catalog products or as a part of your Step Functions workflows.

We hope, this article was useful for you. If so, please, help us to spread it to the world!

Like this article?

Share on Facebook
Share on Twitter
Share on Linkdin
Share on Pinterest

Want to be an author of another post?

We’re looking for skilled technical authors for our blog!

Leave a comment

If you’d like to ask a question about the code or piece of configuration, feel free to use or a similar tool as Facebook comments are breaking code formatting.