Jenkins CloudFormation Integration Pattern

From my experience, many companies rely on Jenkins and Terraform while automating their cloud infrastructure with infrastructure code. There’s no problem if you’d like to deploy something from Jenkins to the cloud, but what to do if you want to call Jenkins job for creating a project, configuring resources, and deploy a Terraform module as a Service Catalog product, or include Jenkins pipeline to a workflow orchestrated by StepFunctions? This article will describe a unified Jenkins CloudFormation integration pattern, allowing you to integrate and manage Jenkins jobs with CloudFormation templates, custom resources, and Step Functions workflows.

Jenkins installation

There are many ways of deploying and installing Jenkins CloudFormation configuration to the AWS cloud. 

You may also be interested in a way to install Jenkins jobs through Deploying Jenkins on Amazon EKS with Amazon EFS if you’re looking for a way to be a Jenkins user for the deployment of Docker containers as Jenkins workers. You can also use AWS CodeDeploy as an alternative to AWS CloudFormation and Jenkins. AWS CodeDeploy is a service that makes it easier for you to release new features rapidly, avoid downtime during deployment, and handles the complexity of updating your applications.

Whatever deployment method you use to install. As a result, you’ll get a Jenkins server configured whose workers can assume an IAM role in one or many AWS accounts to deploy required applications or services.

Now, we can configure and integrate your Jenkins server with the CloudFormation template and Step Functions services through the CloudFormation stack.

Jenkins CloudFormation Integration Architecture

If you’re reading this article, you’re probably trying to solve one of the following problems:

• How to call Jenkins APIs and/or other AWS resources and have them configured from other AWS services
• How to provide access to your central Jenkins server APIs from multiple AWS accounts

The solution is to use AWS Lambda, SQS, and SNS:

In this example of an integration pattern:

• AWS Lambda’s role is to make API calls and trigger Jenkins jobs using python-jenkins library.
• Amazon SNS is responsible for receiving events from Step Function and CloudFormation services.
• Amazon SQS stores messages if the Jenkins server (for any reason) becomes unavailable.

Here’s what the complete architecture diagram for one AWS account looks like:

You can easily expand such a setup to support multiple AWS regions by putting SNS topics in the required regions.

You can configure SNS topic access policies to provide access to the topic from multiple AWS accounts and organizations. It is very easy to configure that because you can use Account IDs and AWS Organizations OUs: How do I allow AWS accounts in my organization to publish messages to an Amazon SNS topic in my account?

Finally, as soon as the “Jenkins to AWS CloudFormation” integration is in place, you can use CloudFormation custom resources to run Jenkins Master jobs from the CloudFormation and have them deployed. That unlocks the capability of deploying Service Catalog products using Jenkins.

Summary

In this article, we’ve described a Jenkins Master integration with CloudFormation and Step Function, AWS resources, and services, which allows you to use Jenkins jobs to deploy Service Catalog products or as a part of your Step Functions workflows.

We hope this article was useful for you. If so, please, help us to spread it to the world!