Leveraging Ansible and Terraform for Efficient Infrastructure Automation
|

Leveraging Ansible and Terraform for Efficient Infrastructure Automation

DevOps is constantly evolving, with new tools and techniques being developed to help organizations streamline their processes and improve their infrastructure management. Ansible and Terraform are two tools that have gained widespread popularity for their ability to automate and manage infrastructure. In this article, we’ll explore what Ansible and Terraform are, how they differ, and how to use them together for maximum efficiency.

What is Ansible?

Ansible is an open-source automation tool that simplifies configuration management, application deployment, and task automation. It uses a simple, human-readable data serialization format called YAML to define automation tasks as playbooks. The Ansible automation platform is agentless, meaning it doesn’t require any additional software to be installed on the target nodes, and it communicates with them over SSH or WinRM.

What is Ansible used for in DevOps?

Ansible is commonly used for configuration management and application deployment in a DevOps environment. It helps ensure that your existing infrastructure and systems are consistently configured and maintained, allowing you to automate repetitive tasks and reduce human error. By using Ansible, you can increase efficiency, reduce downtime, and maintain a more stable and reliable infrastructure.

Terraform vs. Ansible: Key Differences

Both Ansible and Terraform can be used as orchestration and configuration management tools. But each tool has its purpose. Let’s dive deeper into the topic.

Is Terraform a Configuration Management Tool?

Terraform is an open-source infrastructure-as-code (IaC) tool that allows you to manage and provision your infrastructure using a declarative language called HCL (HashiCorp Configuration Language). While it is sometimes referred to as a configuration management tool, Terraform is more accurately described as an orchestration tool, focusing on provisioning and managing the underlying infrastructure components rather than configuring the individual components on those systems.

Terraform Versus Ansible: Use Cases

Terraform is best suited for creating, managing, and destroying new cloud infrastructure and resources like virtual machines, networks, and storage. It is particularly useful for managing infrastructure across multiple cloud providers, as it supports a wide range of providers through its plugin-based architecture.

On the other hand, Ansible excels at managing the configuration of individual systems and deploying applications. It can automate tasks such as installing packages, managing services, and configuring network settings. While Terraform provisions the infrastructure, Ansible ensures that the network devices and systems are properly configured and applications correctly deployed.

Getting Started with Terraform and Ansible

Installing the Tools

Before using Terraform and Ansible, you’ll need to install them on your local machine. You can download Terraform from the official website and follow the installation instructions for your platform. Ansible can be installed using the package manager of your choice, such as apt, yum, or pip.

You can use the following script on a Linux system to install everything automatically:

#!/bin/bash
# Create a Python virtual environment
python3 -m venv ansible_env
# Activate the virtual environment
source ansible_env/bin/activate
# Install the latest version of Ansible
pip install --upgrade pip ansible 
# Install the latest version of Terraform
LATEST_TERRAFORM_VERSION=$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')
curl -LO "https://releases.hashicorp.com/terraform/${LATEST_TERRAFORM_VERSION}/terraform_${LATEST_TERRAFORM_VERSION}_linux_amd64.zip"
unzip "terraform_${LATEST_TERRAFORM_VERSION}_linux_amd64.zip"
sudo mv terraform /usr/local/bin/
rm "terraform_${LATEST_TERRAFORM_VERSION}_linux_amd64.zip"
curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/linux_64bit/session-manager-plugin.rpm" -o "session-manager-plugin.rpm"
sudo yum install -y session-manager-plugin.rpm
# Verify the installation
ansible --version
terraform --version
session-manager-plugin --version

Project structure

Let’s create a folder on the file system to play with examples from this article. Here’s what the project structure looks like:

.
├── ansible
│   └── playbooks
│       └── nginx_install.yml
├── ansible.cfg
├── aws_ec2.yml
├── main.tf
└── user-data.tpl
2 directories, 5 files

In such a project structure:

  • ansible/playbooks – the location where we’ll store all required Ansible playbooks

  • ansible.cfg – general settings for Ansible

  • aws_ec2.yml – Dynamic Ansible inventory file which will allow us to run Ansible playbooks on AWS EC2 instances while connecting to them using Amazon Systems Manager Session Manager (SSM plugin)

  • main.tf – Basic example of using Terraform for provisioning EC2 instance in one of the public subnets in your VPC

  • user-data.tpl – if you need to run additional automation in your EC2 instances, this file will contain a template of user data for the cloud-init.

Basic Terraform Configuration

To start with Terraform, you must create one or more configuration files ending with a .tf extension. These files define the resources you want to deploy and the provider you’ll use to create them. Here’s a simple example that creates an AWS EC2 instance in the AWS cloud:

provider "aws" {
  region = "us-west-2"
}
resource "aws_instance" "example" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
  tags = {
    Name = "example-instance"
    Ansible = "true"
  }
}

We’ll extend the EC2 definition in a second. The most important part is the Ansible tag I’m adding to the instance. I will use this tag to dynamically select EC2 instances to run Ansible automation.

Ansible Tool Basics

To start with Ansible, you’ll need to create a playbook, a YAML file that defines the tasks you want to automate. Here’s a simple example that installs and starts the Nginx web server on a target system:

---
- name: Install and start Nginx on Amazon Linux 2
  hosts: all
  gather_facts: false
  vars:
    ansible_connection: aws_ssm
    ansible_aws_ssm_bucket_name: terraform-ansible-demo-hands-on-cloud
    ansible_aws_ssm_region: us-east-1
    ansible_aws_ssm_s3_addressing_style: virtual
  tasks:
    - name: Enable extras
      ansible.builtin.command:
        cmd: amazon-linux-extras enable nginx1
      become: yes
    - name: Install Nginx
      ansible.builtin.yum:
        name: nginx
        state: present
      become: yes
    - name: Start Nginx service
      ansible.builtin.systemd:
        name: nginx
        state: started
      become: yes

In the example above, I’m enabling Nginx through amazon-linux-extras on Amazon Linux 2 EC2 instance, installing it using YUM, and finally, starting the Nginx system service.

Several important Ansible configuration parameters with the community.aws plugin:

  • ansible_connection – enables Ansible connection to the EC2 instance using SSM instead of SSH. This is especially handy when configuring EC2 instances in the private VPC subnet.

  • ansible_aws_ssm_bucket_name – to support the SSM use case, you must have an S3 bucket in your AWS account dedicated to Ansible; provide the name of your bucket here. Also, don’t forget to use the same bucket name in Terraform configuration below.

Other configuration parameters can be found in the official Ansible community.aws plugin documentation.

Ansible Dynamic Inventory in AWS

Ansible has a variety of plugins and modules for managing AWS resources. Many of these plugins and modules are in the community.aws and amazon.aws collections. The only difference is that “amazon.aws” collection is maintained by AWS and contains many of the same modules and plugins as the community.aws collection. We’ll reply on both in our example of configuring a dynamic inventory file (aws_ec2.yml):

plugin: amazon.aws.aws_ec2
filters:
  tag:Ansible: true
  instance-state-name : running
compose:
  ansible_host: instance_id
  ansible_connection: 'community.aws.aws_ssm'

In the configuration above, we’re using amazon.aws.aws_ec2 plugin for getting EC2 instances where we need to run the Ansible playbook. The compose part configures Ansible to use SSM to connect to the instance.

Terraform-Ansible Provisioner: Using Ansible with Terraform

Let’s extend the Terraform code.

Configuring the Provisioner

To integrate Ansible with Terraform, you can use the local-exec provisioner to call the ansible-playbook command after creating a resource or run Ansible separately in your Terraform CICD pipeline. Here’s the complete content of the main.tf file:

locals {
  ami = data.aws_ami.amazon_linux_2.id
  region = "us-east-1"
  vpc_name = "demo-vpc"
  vpc_id = data.aws_vpcs.selected.ids[0]
  public_subnets = [for association in data.aws_route_table.selected.associations : association.subnet_id]
  ansible_s3_bucket = "terraform-ansible-demo-hands-on-cloud"
}
provider "aws" {
  region = local.region
}
data "aws_ami" "amazon_linux_2" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "name"
    values = ["amzn2-ami-hvm-*-x86_64-gp2"]
  }
  filter {
    name   = "architecture"
    values = ["x86_64"]
  }
  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }
}
data "aws_vpcs" "selected" {
  tags = {
    Name = local.vpc_name
  }
}
data "aws_internet_gateway" "selected" {
  filter {
    name   = "attachment.vpc-id"
    values = [local.vpc_id]
  }
}
data "aws_route_table" "selected" {
  filter {
    name = "route.gateway-id"
    values = [data.aws_internet_gateway.selected.id]
  }
}
resource "aws_iam_role" "ec2_instance_role" {
  name = "ec2_instance_role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
  tags = {
    Name = "ec2_instance_role"
  }
}
resource "aws_iam_policy" "ec2_instance_policy" {
  name        = "ec2_instance_policy"
  description = "Allows access to the DynamoDB table for EC2 instances"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ec2:DescribeInstances"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
        ]
        Resource = "arn:aws:s3:::${local.ansible_s3_bucket}/*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket"
        ]
        Resource = "arn:aws:s3:::${local.ansible_s3_bucket}"
      },
    ]
  })
}
resource "aws_iam_role_policy_attachment" "ec2_instance_policy_attachment" {
  policy_arn = aws_iam_policy.ec2_instance_policy.arn
  role       = aws_iam_role.ec2_instance_role.name
}
resource "aws_iam_role_policy_attachment" "ssm_managed_instance_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
  role       = aws_iam_role.ec2_instance_role.name
}
resource "aws_iam_instance_profile" "ec2_instance_profile" {
  name = "ec2_instance_profile"
  role = aws_iam_role.ec2_instance_role.name
}
resource "aws_security_group" "ec2_security_group" {
  name_prefix = "ec2_security_group_"
  vpc_id      = local.vpc_id
  
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}
data "template_file" "ec2_user_data" {
  template = "${file("user-data.tpl")}"
  
  vars = {
    my_variable = "test"
  }
}
resource "aws_instance" "ec2_instances" {
  depends_on = [
    aws_iam_role_policy_attachment.ec2_instance_policy_attachment
  ]
  
  ami           = data.aws_ami.amazon_linux_2.id
  
  instance_type = "t2.micro"
  
  user_data = data.template_file.ec2_user_data.rendered
  iam_instance_profile = aws_iam_instance_profile.ec2_instance_profile.name
  
  vpc_security_group_ids = [aws_security_group.ec2_security_group.id]
  
  
  subnet_id     = local.public_subnets[0]
  
  associate_public_ip_address = true
  tags = {
    Name = "demo"
    Ansible = "true"
  }
  
  provisioner "local-exec" {
    command = "ansible-playbook -i ${path.module}/aws_ec2.yml ${path.module}/ansible/playbooks/nginx_install.yml"
  }
}
output "website_url" {
  value = "http://${aws_instance.ec2_instances.public_dns}"
}

The Terraform example above will query a list of public VPC subnets by the VPC Name tag, the latest Amazon Linux 2 AMI, set up EC2 instance profile role with required permissions to the Ansible S3 bucket. And launch the EC2 instance.

The EC2 local-exec provisioner contains an Ansible command to run the playbook. You can easily extend this configuration by setting up your tagging schema for required EC2 instances.

The output block will contain a value of the website URL for convenience.

Executing Ansible Playbooks

With the provisioner configured, Terraform will automatically execute the specified Ansible playbook after creating the resource. When you run terraform apply, Terraform will create the AWS EC2 instance and then run the Ansible playbook to install and start Nginx on the newly created instance.

ansible-playbook -i ./aws_ec2.yml ./ansible/playbooks/nginx_install.yml

Advanced Techniques: Terraform and Ansible Integration

Immutable infrastructure approach

Immutable infrastructure is an approach where servers are never modified after deployment. Instead, any configuration changes are made by deploying a new server with the updated configuration. This helps to ensure consistency and reliability across your infrastructure.

To follow the immutable infrastructure approach using Ansible and Terraform, you can use the following strategy:

  1. Create a base image: Use a tool like Packer to create a base image with the core software and necessary dependencies. For example, you could create an Amazon Machine Image (AMI) with your base operating system, core packages, and other common software for all instances.

  2. Use Terraform to provision infrastructure: Use Terraform to create and manage your infrastructure resources like VPCs, subnets, security groups, and instances. Instead of modifying existing instances, create new instances with the updated configuration when needed.

  3. Configure instances with Ansible: Use Ansible to apply server-specific configurations on your base image. Create Ansible playbooks for each type of server in your infrastructure. When you provision a new instance with Terraform, use the appropriate Ansible playbook to configure it.

  4. Automate deployment: Use a Continuous Integration/Continuous Deployment (CI/CD) pipeline to automate the deployment process. When changes are made to your infrastructure or server configurations, your pipeline can automatically build new images with Packer, deploy new resources with Terraform, and configure instances with Ansible.

  5. Use version control: Track your Terraform and Ansible configurations in a version control system like Git. This allows you to maintain a history of your infrastructure changes and easily roll back to previous configurations if needed.

  6. Clean up old resources: As you deploy new instances with updated configurations, don’t forget to clean up the old instances that are no longer needed. You can use Terraform to destroy old resources or create scripts to terminate instances that are no longer in use automatically.

Here’s a high-level example of how you can use Terraform and Ansible together to create infrastructure to follow the immutable infrastructure approach:

  1. Create a base image using Packer with your core software and dependencies.

  2. Write Terraform configurations to provision your infrastructure, including instances that use the base image you created with Packer.

  3. Write Ansible playbooks to configure instances based on their role in your infrastructure.

  4. Use the “null_resource” and provisioner “local-exec” blocks to execute Ansible playbooks on newly created instances in your Terraform configuration.

  5. Set up a CI/CD pipeline to automate the deployment process. When changes are made to your infrastructure or server configurations, your pipeline should:

    • Build new images with Packer.

    • Deploy new resources with Terraform.

    • Configure instances with Ansible.

    • Terminate old instances and clean up resources.

Dynamic Inventory Management

One of the challenges when using Terraform and Ansible together is managing inventory. You can use dynamic inventory scripts to automatically generate an inventory based on the current state of the virtual machine in your infrastructure. Terraform can output the necessary information for the dynamic inventory script, making it easy to integrate with Ansible.

Alternatively, you can use dynamic inventory, as in the example above.

Instance lifecycle management

EC2 instance lifecycle management involves the various stages an EC2 instance goes through, from creation to termination. Properly managing the instance lifecycle helps ensure optimal resource usage, cost control, and a consistent and reliable infrastructure. Here are some key aspects to consider for EC2 instance lifecycle management:

  1. Provisioning: Use tools like Terraform or AWS CloudFormation to create and manage infrastructure resources, including EC2 instances. In your infrastructure-as-code templates, define the instance configurations, such as the instance type, AMI, security groups, and other options.

  2. Configuration Management: Use tools like Ansible, Chef, or Puppet to manage the configuration of your instances consistently. Apply server-specific configurations on top of the base image, and ensure that the instances are correctly set up according to their roles in the infrastructure.

  3. Health Monitoring: Implement monitoring and alerting for your instances using services like Amazon CloudWatch, Datadog, or New Relic. Monitor key metrics like CPU utilization, memory usage, disk space, and network traffic to ensure your instances are healthy and performing well.

  4. Scaling: Use Auto Scaling Groups (ASGs) or other scaling mechanisms to adjust the number of instances in your infrastructure based on demand. You can configure scaling policies to automatically add or remove instances based on metrics like CPU utilization or network traffic.

  5. Backup and Recovery: Implement backup strategies for your instances, such as creating Amazon EBS snapshots or backing up application data to Amazon S3. Ensure that you can quickly recover instances of failure or data loss. Consider using AWS Backup to simplify the entire process.

  6. Patch Management: Keep your instances updated with the latest security patches and software updates. Use tools like AWS Systems Manager, Ansible, or Chef to automate patch management and ensure your instances run the latest software versions.

  7. Immutable Infrastructure: Instead of modifying existing instances, create new instances with updated configurations when changes are required. This approach ensures consistency and reliability across your infrastructure.

  8. Instance Termination: When instances are no longer needed or have reached the end of their lifecycle, terminate them to free up resources and reduce costs. Use tools like Terraform or AWS CloudFormation to destroy resources when they’re no longer needed. You can also implement automation to terminate instances based on age or other criteria.

  9. Cost Optimization: Regularly review your instance usage and costs to identify areas for optimization. Right-size your instances based on actual resource usage, and use Spot Instances or Reserved Instances to reduce costs where appropriate.

Properly managing the EC2 instance lifecycle ensures a reliable, consistent, and cost-effective infrastructure. Incorporate automation and monitoring tools to help streamline the management cloud deployment process and maintain a robust environment.

Deploying Multi-Cloud Infrastructure

Terraform’s multi-provider support makes it an excellent choice for deploying infrastructure across multiple cloud providers. Combining Terraform with Ansible allows you to create a unified automation workflow that provisions infrastructure and manages configuration across various platforms.

Ansible Tower vs Terraform: A Comparison

While this article focuses on integrating Terraform and Ansible, it’s worth mentioning Ansible Tower as a potential alternative. Ansible Tower is a web-based interface for managing Ansible that includes additional features such as role-based access control, job scheduling, and a REST API. However, it lacks the same infrastructure provisioning capabilities and multi-provider support Terraform offers, making it better suited for configuration management and application deployment rather than full infrastructure automation tools.

Conclusion

By combining the strengths of Ansible and Terraform, you can create a powerful and efficient infrastructure automation solution. Terraform excels at provisioning infrastructure resources across multiple cloud providers, while Ansible ensures that your systems are consistently configured and applications are properly deployed. Together, they provide a seamless, flexible, and robust solution for managing your infrastructure in a DevOps environment.

Similar Posts