From my personal experience, many companies rely on Jenkins and Terraform while automating their cloud infrastructure. There’s no problem if you’d like to deploy something from Jenkins to the cloud, but what to do, if you want to call Jenkins job from to deploy a Terraform module as a Service Catalog product, or include Jenkins pipeline to a workflow orchestrated by StepFunctions? This article will describe a unified integration pattern, which will allow you to integrate Jenkins jobs with CloudFormation custom resources and Step Functions workflows.
There are many ways of deploying Jenkins to the AWS cloud.
For the automated installation process of a single Jenkins server to AWS cloud use our How to Install Jenkins on the Latest Ubuntu in 5 Minutes article.
You may also be interested in Deploying Jenkins on Amazon EKS with Amazon EFS if you’re looking for a way to use Docker containers as Jenkins workers.
Whatever deployment method you use; as a result, you’ll get a Jenkins server whose workers can assume IAM roles in one or many AWS accounts to deploy required applications or services.
Now, we can integrate your Jenkins server with CloudFormation and Step Functions services.
If you’re reading this article, you’re probably trying to solve one of the following problems:
- How to call Jenkins APIs from other AWS services
- How to provide access to your central Jenkins APIs from multiple AWS accounts
The solution is to use AWS Lambda, SQS, and SNS:
In this integration pattern:
- AWS Lambda’s role is to make API calls to Jenkins using python-jenkins library.
- Amazon SNS allows you to receive events from Step Function and CloudFormation services seamlessly.
- Amazon SQS is responsible for sending, storing, and receiving messages for Jenkins at any volume without losing those messages if Jenkins (for any reason) becomes unavailable.
Here’s how the complete architecture diagram for one AWS account looks like:
If you need to support muitiple AWS regions, you can easily expand such setup by putting SNS topics to the required AWS regions and integrate them directly to the SQS queue.
You can configure SNS topic access policies to provide access to the topic from multiple AWS accounts and organizations. It is very easy to configure that because you can use Account IDs and AWS Organizations OUs:
- How do I allow AWS accounts in my organization to publish messages to an Amazon SNS topic in my account?
Finally, as soon as the “Jenkins to AWS CloudFormation” integration is in place, you can use CloudFormation custom resources to run Jenkins jobs from the CloudFormation. That unlocks the capability of deploying Service Catalog products using Jenkins.
In this article, we’ve described a Jenkins integration with CloudFormation and Step Function AWS services, which allows you to use Jenkins jobs to deploy Service Catalog products or as a part of your Step Functions workflows.
We hope, this article was useful for you. If so, please, help us to spread it to the world!
I’m a passionate Cloud Infrastructure Architect with more than 15 years of experience in IT.
Any of my posts represent my personal experience and opinion about the topic.