Are you looking for a way to simplify the process of deploying and managing Kubernetes? If so, you should consider using Amazon Elastic Kubernetes Service (Amazon EKS). Amazon EKS is a managed service that allows you to run Kubernetes on AWS without installing, operating, or maintaining your own Kubernetes control plane or nodes. This article will show how to deploy the Amazon EKS cluster, EKS Cluster Autoscaler, and Horizontal Pod Autoscaler (HPA) using Terraform.
Table of contents
Prerequisites
To follow along, you need the following:
- Basic knowledge of Terraform and Kubernetes
- An AWS account with a suitable IAM User (preferably with an admin role)
- AWS CLI should be installed and configured on your local machine
- Kubectl installed on your local machine to interact with the EKS Control Plane
The source code for all examples for this article is located at our GitHub repository: managing-amazon-eks-using-terraform.
Deployment architecture overview
In this article, we’ll deploy the Amazon EKS Cluster and self-managed EKS Worker Node Group in the Auto Scaling group in a separate VPC across two availability zones using Terraform. A custom VPC will have public and private subnets to allow you to launch public and private Kubernetes workloads. The control plane of EKS (master nodes) is managed by AWS, and you do not need to worry about them.
EKS worker nodes will host Kubernetes PODs, Services, Daemons, and other Kubernetes resources.
Load balancers for EKS services depending on the Kubernetes application definition can be deployed either in the public or private subnets to distribute traffic to the pods inside the worker nodes.

We will split our project into several parts:
- Base VPC infrastructure
- EKS Cluster
- EKS Cluster Autoscaler
- EKS demo applications
Setting up the VPC infrastructure
The networking infrastructure consists of a VPC, two public & private subnets, NAT Gateways, route tables, and route table associations. Setting up all of these from scratch could be very tedious, so we will leverage the AWS VPC module from Terraform to simplify the deployment.
Complete module source code can be found in our GitHub repository folder: 1_vpc.

If you’re interested in how to set up VPCs from scratch, check out the following articles:
- Terraform – Managing AWS VPC – Creating Public Subnet
- Terraform – Managing AWS VPC – Creating Private Subnets
First, we need to create a project folder called managing-amazon-eks-using-terraform
. Within the project folder, we create a Terraform module (folder) for managing VPC called 1_vpc
.
In our implementation, we’re using Terraform S3 backend for storing state files and DynamoDB for Terraform execution locks. You can easily deploy the S3 bucket and DynamoDB table for your own Terraform-managed AWS infrastructure using our already existing module: managing-amazon-eks-using-terraform/0_remote_state.
terraform {
backend "s3" {
bucket = "hands-on-cloud-terraform-remote-state-s3"
key = "managing-eks-terraform-vpc.tfstate"
region = "us-west-2"
encrypt = "true"
dynamodb_table = "hands-on-cloud-terraform-remote-state-dynamodb"
}
}
Here you’re specifying the following parameters for Terraform S3 backend:
bucket
– the name of the S3 bucket to store Terraform state fileskey
– Terraform state file name (for the current Terraform module)region
– the region of the S3 bucket for storing Terraform stateencrypt
– optional feature, allowing you to encrypt Terraform state file using S3 server-side encryptiondynamodb_table
– optional DynamoDB table which is used to lock Terraform module executions from different machines at the same time
If you’d like to pin Terraform provider version, you can do it in providers.tf
file:
# Set up Terraform provider version (if required)
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.9"
}
}
}
# Defining AWS provider
provider "aws" {
region = var.aws_region
}
Next, let’s add the variables.tf
file for storing project resources common prefix, AWs region, and VPC CIDR range:
variable "prefix" {
default = "managing-eks-terraform"
description = "Common prefix for AWS resources names"
}
variable "aws_region" {
default = "us-east-1"
description = "AWS Region to deploy VPC"
}
variable "vpc_cidr" {
default = "10.10.0.0/16"
description = "AWS VPC CIDR range"
}
Also, we need to define several Terraform local variables which we’ll use within the module:
locals {
prefix = "managing-eks-terraform"
vpc_name = "${local.prefix}-vpc"
vpc_cidr = var.vpc_cidr
common_tags = {
Environment = "dev"
Project = "hands-on.cloud"
}
}
Now, we’re ready to create a file here called vpc.tf
, which will contain the VPC definition for the EKS cluster:
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
name = local.vpc_name
cidr = local.vpc_cidr
azs = ["${var.aws_region}a", "${var.aws_region}b"]
public_subnets = [
cidrsubnet(local.vpc_cidr, 8, 0),
cidrsubnet(local.vpc_cidr, 8, 1)
]
private_subnets = [
cidrsubnet(local.vpc_cidr, 8, 2),
cidrsubnet(local.vpc_cidr, 8, 3)
]
enable_nat_gateway = true
single_nat_gateway = false
enable_dns_hostnames = true
tags = merge(
{
Name = local.vpc_name
},
local.common_tags
)
public_subnet_tags = {
"kubernetes.io/role/elb" = "1"
"kubernetes.io/cluster/${local.prefix}-cluster" = "owned"
}
}
Public subnets need to have the required tags for load balancer placement (the public_subnet_tags
argument).
Finally, we need to define some outputs which we’ll use in Terraform modules managing EKS cluster deployment:
output "prefix" {
value = local.prefix
description = "Exported common resources prefix"
}
output "common_tags" {
value = local.common_tags
description = "Exported common resources tags"
}
output "vpc_id" {
value = module.vpc.vpc_id
description = "VPC ID"
}
output "public_subnets" {
value = module.vpc.public_subnets
description = "VPC public subnets' IDs list"
}
output "private_subnets" {
value = module.vpc.private_subnets
description = "VPC private subnets' IDs list"
}
To apply this module, you need to execute the following Terraform commands from the 1_vpc
folder:
terraform init
terraform apply -auto-approve
Setting up the EKS cluster
In this part of the article, we will define and set up the Amazon EKS Cluster with all required IAM roles, the EKS cluster worker nodes, etc. Create a 2_eks
folder inside the project folder, which will house all required Terraform code.
Complete module source code can be found in our GitHub repository folder: 2_eks.

Note: In this article, we’re showing how to define the EKS cluster using Terraform from scratch. At the same time, we highly recommend you to check out an excellent official AWS terraform-aws-modules/eks which has more functionality than we’re covering in this article.
EKS module variables
To be able to deploy the EKS cluster, we need to define several local Terraform variables for storing VPC module outputs:
locals {
remote_state_bucket_region = "us-west-2"
remote_state_bucket = "hands-on-cloud-terraform-remote-state-s3"
infrastructure_state_file = "managing-eks-terraform-vpc.tfstate"
prefix = data.terraform_remote_state.vpc.outputs.prefix
common_tags = data.terraform_remote_state.vpc.outputs.common_tags
vpc_id = data.terraform_remote_state.vpc.outputs.vpc_id
public_subnets = data.terraform_remote_state.vpc.outputs.public_subnets
private_subnets = data.terraform_remote_state.vpc.outputs.private_subnets
}
data "terraform_remote_state" "vpc" {
backend = "s3"
config = {
bucket = local.remote_state_bucket
region = local.remote_state_bucket_region
key = local.infrastructure_state_file
}
}
In addition to that, let’s define an external module variable for managing the AWS region for EKS cluster deployment:
variable "aws_region" {
default = "us-east-1"
description = "AWS Region to deploy VPC"
}
Creating an IAM Role for the EKS cluster using Terraform
Next, we need to create an IAM service role for our cluster and attach an AWS Managed IAM policy called AmazonEKSClusterPolicy
to it.
AmazonEKSClusterPolicy
is an IAM service role that provides Amazon EKS with the permissions it needs to create and delete AWS resources on your behalf. AmazonEKSClusterPolicy
is used by Amazon EKS to create the following AWS resources when you create an Amazon EKS cluster:
- An Amazon EKS control plane environment in your account, including the VPC subnets, security groups, and route tables that Amazon EKS creates for the control plane servers
- An Amazon Elastic Load Balancing network load balancer in your account for each Availability Zone in the region to allow west-east traffic between your worker nodes and the Amazon EKS control plane environment
- An Amazon EFS file system in your account if you enable self-managed worker nodes (nodes that are not launched and managed by an Amazon EKS node group)
When you delete an Amazon EKS cluster, Amazon EKS deletes these AWS resources. For more information, see Amazon Resource Names (ARNs), IAM Roles, Service Principals, and Policies in the AWS Identity and Access Management User Guide. If you create multiple clusters in a single account, we recommend that you use a separate IAM service role for each cluster.
To define an EKS IAM service role using Terraform, create a file called iam.tf
inside the 2_eks
module folder and add the following code:
# EKS Terraform - IAM Role for EKS Cluster (Control Plane)
resource "aws_iam_role" "eks_iam_role" {
name = "eks-service-role"
assume_role_policy = jsonencode({
Version = "2012-10-17",
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "eks.amazonaws.com"
}
}]
})
tags = {
"Terraform" = "true"
}
}
resource "aws_iam_role_policy_attachment" "eks_iam_policy_attachment" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
role = aws_iam_role.eks_iam_role.name
}
Creating an IAM Role for the EKS worker nodes using Terraform
Next, we need to create a role for EC2 instances used in the EKS managed node group. This EC2 worker node will contain the following AWS IAM Managed policies:
AmazonEKSWorkerNodePolicy
is an IAM policy that you can attach to an Amazon EKS worker node’s instance profile role. Amazon EKS uses theAmazonEKSWorkerNodePolicy
to allow your worker nodes to join the Amazon EKS cluster as specified in the Amazon EKS ClusterEndpoint. After you create anAmazonEKSWorkerNodePolicy
, you must apply it to your Amazon EKS worker node instance profile roles for the nodes to function properly within your Amazon EKS cluster.AmazonEKS_CNI_Policy
is an Amazon EKS-managed policy that you can attach to your worker node instance profile role to grant Amazon EKS permission to manage the Amazon EKS pod networking for your worker nodes. Amazon EKS customers use Amazon EKS to run Kubernetes pods on top of AWS compute resources such as Amazon EC2 instances.AmazonEC2ContainerRegistryReadOnly
is an Amazon Elastic Container Registry (Amazon ECR) permission level that lets Amazon EKS operators grant read-only access to the Amazon ECR resources in their account to Amazon EKS worker nodes. Amazon EKS worker nodes are Amazon EC2 instances that are registered with an Amazon EKS cluster and that run the Docker daemon and the Amazon EKS container agent.
Add the following code to the same (iam.tf
) file:
# EKS Terraform - IAM Role for EKS Worker Nodes
resource "aws_iam_role" "eks_node_role" {
name = "eks-node-role"
assume_role_policy = jsonencode({
Version = "2012-10-17",
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
}]
})
tags = {
Terraform = "true"
}
}
resource "aws_iam_role_policy_attachment" "worker_node_policy_attachment" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
role = aws_iam_role.eks_node_role.name
}
resource "aws_iam_role_policy_attachment" "cni_policy_attachment" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
role = aws_iam_role.eks_node_role.name
}
resource "aws_iam_role_policy_attachment" "ecr_readonly_policy_attachment" {
policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
role = aws_iam_role.eks_node_role.name
}
Creating EKS cluster using Terraform
Now we’re ready to define the EKS cluster and EKS managed worker node group. Create a file called eks.tf
in the 2_eks
module folder with the following code:
locals {
cluster_name = "${local.prefix}-cluster"
aws_account_id = data.aws_caller_identity.current.account_id
}
data "aws_caller_identity" "current" {}
# EKS Terraform - EKS Cluster
resource "aws_eks_cluster" "eks_cluster" {
depends_on = [ aws_iam_role_policy_attachment.eks_iam_policy_attachment ]
name = local.cluster_name
role_arn = aws_iam_role.eks_iam_role.arn
vpc_config {
subnet_ids = concat(local.public_subnets, local.private_subnets)
}
tags = merge(
{
Name = local.cluster_name
},
local.common_tags
)
}
# EKS Terraform - creating the private EKS Node Group
resource "aws_eks_node_group" "private_node_group" {
depends_on = [
aws_iam_role_policy_attachment.worker_node_policy_attachment,
aws_iam_role_policy_attachment.cni_policy_attachment,
aws_iam_role_policy_attachment.ecr_readonly_policy_attachment
]
cluster_name = aws_eks_cluster.eks_cluster.name
node_group_name = "${local.cluster_name}-private-ng"
node_role_arn = aws_iam_role.eks_node_role.arn
subnet_ids = local.private_subnets
ami_type = "AL2_x86_64"
capacity_type = "ON_DEMAND"
instance_types = ["m4.medium"]
disk_size = 8
scaling_config {
desired_size = 2
max_size = 6
min_size = 1
}
update_config {
max_unavailable = 2
}
tags = merge(
{
Name = local.cluster_name
},
local.common_tags
)
}
data "template_file" "config" {
template = file("${path.module}/templates/config.tpl")
vars = {
certificate_data = aws_eks_cluster.eks_cluster.certificate_authority[0].data
cluster_endpoint = aws_eks_cluster.eks_cluster.endpoint
aws_region = var.aws_region
cluster_name = local.cluster_name
account_id = local.aws_account_id
}
}
resource "local_file" "config" {
content = data.template_file.config.rendered
filename = "${path.module}/${local.cluster_name}_config"
}
In the code above, we’ve defined two local variables for storing AWS account ID (obtained from the aws_caller_identity data resource) and EKS cluster name.
Next, we’re using the aws_eks_cluster resource to define the EKS cluster control plane (managed by the AWS).
The aws_eks_node_group resource allows us to define an EKS cluster self-managed worker node group (EC2 instances) managed by the Auto Scaling Group. EC2 instances in this Auto Scaling group will be managed by the Cluster Autoscaler (defined in the next section).
In addition to that, we’re using a template_file Terraform resource to generate the content of the kubectl config file and the local_file resource to save this config to the local filesystem.
To apply this module and deploy the Amazon EKS cluster using Terraform, you need to execute the following commands from the 2_eks
folder:
terraform init
terraform apply -auto-approve
Once the configurations are successfully applied, you can verify the creation of the cluster by executing the following AWS CLI command:
aws eks list-clusters
The above command returns a list of control plane EKS clusters in your account. You would get an output like this:
{
"clusters": [
"managing-eks-terraform-cluster"
]
}
Connecting to the EKS cluster
In the previous section, we deployed the Amazon EKS cluster and generated kubectl
config. Set up the KUBECONFIG
with the location path to this file:
export KUBECONFIG=$(pwd)/managing-eks-terraform-cluster_config
Test connection to the cluster, let’s try to get the information about all worker nodes connected to the EKS cluster:
kubectl get nodes
You should get a similar output:
NAME STATUS ROLES AGE VERSION
ip-10-10-2-183.ec2.internal Ready <none> 13m v1.21.5-eks-9017834
ip-10-10-3-179.ec2.internal Ready <none> 13m v1.21.5-eks-9017834
Nginx deployment
Now, let’s create a deployment manifest for a simple Nginx server. We will be using a separate nginx-ns
namespace for this demo application. Create a folder called 3_k8s_apps
at the root level of the project folder and the nginx
subfolder within it. Create a nginx.yml
file within the nginx
folder with the following resources definitions:
# K8s example - Nginx Namespace
kind: Namespace
apiVersion: v1
metadata:
name: nginx-ns
labels:
name: nginx-ns
---
# K8s example - Nginx Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: nginx-ns
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- name: web
containerPort: 80
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 256Mi
cpu: 250m
---
# K8s example - Nginx Load Balancer (NLB)
apiVersion: v1
kind: Service
metadata:
name: nginx-nlb
namespace: nginx-ns
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
type: LoadBalancer
selector:
app: nginx
ports:
- protocol: TCP
port: 80
targetPort: web
It’s a simple manifest that defines a separate Kubernetes Namespace for Nginx Deployment and LoadBalancer service to expose it to the outside world.
Load Balancers are created using the Service and Ingress resources. For simplicity, we’re exposing Nginx POD through a Network Load Balancer (NLB) defined as a Service resource. You may also set up an Application Load Balancer (ALB) using the ALB Ingress Controller.
To apply the above manifest, run the following command:
kubectl apply -f 3_k8s_apps/nginx/nginx.yml
Once applied, we can wait for a few seconds and inspect the EKS cluster PODs using the following command:
kubectl get pods --namespace nginx-ns
The output would be something like this:

You can get the URL of the Network Load Balancer for the deployed Nginx PODs using the following command:
kubectl get service --namespace nginx-ns
Here’s an example output:

Now, you can use the Load balancer EXTERNAL-IP (URL) information to connect to the Nginx PODs:

We have successfully created an EKS Cluster with the self-managed worker node group in private subnets and an internet-facing load balancer in public subnets.
Scaling the EKS Cluster
In the event of increased traffic or requests to our application, we can configure our Kubernetes cluster to scale horizontally to handle sudden traffic spikes. Scaling can happen at two levels – at the pod level (using HPA) and at the node level (using Cluster Autoscaler). In this section, we shall discuss and implement both!
Horizontal Pod Autoscaler (HPA)
The Horizontal Pod Autscaler (HPA) is a special Kubernetes object that automatically scales up or down the number of pods in a deployment/replica set. It works by tracking different metrics such as CPU utilization, memory utilization, etc. Whenever the utilization of these resources exceeds a certain percentage, a scale-out action is triggered, adding more identical pods to the deployment. Similarly, when the utilization percentage reduces, a scale in action is triggered, and unnecessary pods are terminated.
HPA Manifest
To track the above-mentioned metrics, we need to install the metrics-server
in the kube-system
namespace of our EKS Cluster. You can install that by following the documentation here.
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
Once the metrics-server
is installed on the cluster, we can add the HPA manifest (the hpa.yml
file) inside our 3_k8s_apps/hpa
folder:
# EKS Terraform - Nginx HPA
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
namespace: nginx-ns
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: nginx
minReplicas: 2
maxReplicas: 8
metrics:
- type: Resource
resource:
name: cpu #scaling based on CPU
targetAverageUtilization: 5 #setting low value - so that HPA scale out can be seen quickly on running load generator script
To apply the HPA manifest, use the following command:
kubectl apply -f 3_k8s_apps/hpa/hpa.yml
Testing the HPA
Let’s verify the HPA installation on our cluster by running the following command:
kubectl get hpa --namespace nginx-ns
The above command would provide us with the following output :

The ‘TARGETS’ column would show 0% – 1% CPU utilization as the pods are not being subjected to any load. To test the HPA, keep two terminal windows opened – side by side. We will use the first one to constantly monitor the state of the HPA using the following command:
kubectl get hpa --namespace nginx-ns -w
We will use the second terminal to apply load to our nginx
deployment, which would, in turn, trigger a scale-out action via the HPA. In the second terminal, execute the following command :
kubectl run -i --tty load-generator --image=busybox /bin/sh
The above command will create a POD called load-generator
on the EKS Cluster using the busybox
image and would provide us with a bash terminal from the POD container. Use the following command inside the POD terminal to put a load on the nginx
deployment:
while true; do wget -q -O- http://<loadbalancer-url>; done
The above script infinitely sends requests to the Nginx deployment via the load balancer. Do not forget to replace the loadbalancer-url
with your current load balancer URL.
You should see something like this:

As the load increases, the HPA triggers a scale-out action to increase the number of PODs in the replica and decrease target utilization.
You may now stop the processes in both terminals (using CTRL+C). Note that the scale-in action takes about 5 mins to kick in on reducing load. So you would see 6 pods running during that time, after which it would again reduce to 2.
To delete the HPA, use the following command:
# kubectl delete hpa <hpa-name>
kubectl delete hpa nginx-hpa --namespace nginx-ns
Cluster Autoscaler
As discussed earlier, HPA is a Kubernetes object that scales PODs in a node, while the Cluster Autoscaler is a Kubernetes object that scales nodes in a cluster. The implementation of Cluster Autoscaler in AWS EKS is based on Autoscaling Groups in AWS. The open-source Cluster Autoscaler K8s component asks AWS Autoscaling Groups to scale the number of nodes in the cluster when the requested CPU usage goes above the available CPU. We need to have an authentication mechanism to enable this communication between the K8s Cluster Autoscaler component and AWS components. This authentication mechanism is provided by the IAM OIDC provider (OpenID Connect Identity Provider), which grants a service account with the necessary permissions to the Cluster Autoscaler to interact with the AWS ASG component. Let’s deploy Cluster Autoscaler prerequisites using Terraform.
First, let’s create a separate 4_eks_cluster_autoscaler
subfolder in the project root folder to store Terraform module code. Now, we can easily pin Terraform provider version if needed:
# Set up Terraform provider version (if required)
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 4.9"
}
}
}
# Defining AWS provider
provider "aws" {
region = var.aws_region
}
The Terraform backend configuration for the module will be defined as:
terraform {
backend "s3" {
bucket = "hands-on-cloud-terraform-remote-state-s3"
key = "managing-eks-terraform-eks-cluster-autoscaler.tfstate"
region = "us-west-2"
encrypt = "true"
dynamodb_table = "hands-on-cloud-terraform-remote-state-dynamodb"
}
}
We’ll define the AWS region as an external module variable:
variable "aws_region" {
default = "us-east-1"
description = "AWS Region to deploy VPC"
}
Also, we need to grab some information from other modules’ outputs from their state files:
locals {
remote_state_bucket_region = "us-west-2"
remote_state_bucket = "hands-on-cloud-terraform-remote-state-s3"
vpc_state_file = "managing-eks-terraform-vpc.tfstate"
eks_state_file = "managing-eks-terraform-eks-cluster.tfstate"
prefix = data.terraform_remote_state.vpc.outputs.prefix
common_tags = data.terraform_remote_state.vpc.outputs.common_tags
vpc_id = data.terraform_remote_state.vpc.outputs.vpc_id
public_subnets = data.terraform_remote_state.vpc.outputs.public_subnets
private_subnets = data.terraform_remote_state.vpc.outputs.private_subnets
}
data "terraform_remote_state" "vpc" {
backend = "s3"
config = {
bucket = local.remote_state_bucket
region = local.remote_state_bucket_region
key = local.vpc_state_file
}
}
data "terraform_remote_state" "eks" {
backend = "s3"
config = {
bucket = local.remote_state_bucket
region = local.remote_state_bucket_region
key = local.eks_state_file
}
}
As soon as we’ll use the Cluster Autoscaler IAM role ARN, let’s output it from the module:
output "eks_ca_iam_role_arn" {
value = aws_iam_role.cluster_autoscaler.arn
description = "AWS IAM role ARN for EKS Cluster Autoscaler"
}
Creating Cluster Autoscaler IAM Role using Terraform
Now we need to create an IAM role for the Cluster Autoscaler and allow it to use the service account to get STS credentials via Federated Login. After that, we would provide it with the permissions necessary for autoscaling operations. Let’s define the Terraform configuration for that:
# Policy document allowing Federated Access for IAM Cluster Autoscaler role
data "aws_iam_policy_document" "cluster_autoscaler_sts_policy" {
statement {
actions = ["sts:AssumeRoleWithWebIdentity"]
effect = "Allow"
condition {
test = "StringEquals"
variable = "${replace(aws_iam_openid_connect_provider.eks_ca_oidc_provider.url, "https://", "")}:sub"
values = ["system:serviceaccount:kube-system:cluster-autoscaler"]
}
principals {
identifiers = [aws_iam_openid_connect_provider.eks_ca_oidc_provider.arn]
type = "Federated"
}
}
}
# IAM Role for IAM Cluster Autoscaler
resource "aws_iam_role" "cluster_autoscaler" {
assume_role_policy = data.aws_iam_policy_document.cluster_autoscaler_sts_policy.json
name = "${local.prefix}-cluster-autoscaler"
}
# IAM Policy for IAM Cluster Autoscaler role allowing ASG operations
resource "aws_iam_policy" "cluster_autoscaler" {
name = "${local.prefix}-cluster-autoscaler"
policy = jsonencode({
Statement = [{
Action = [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup",
"ec2:DescribeLaunchTemplateVersions"
]
Effect = "Allow"
Resource = "*"
}]
Version = "2012-10-17"
})
}
resource "aws_iam_role_policy_attachment" "eks_ca_iam_policy_attach" {
role = aws_iam_role.cluster_autoscaler.name
policy_arn = aws_iam_policy.cluster_autoscaler.arn
}
Creating the IAM OIDC provider using Terraform
Finally, we need to get a TLS certificate (the identity provider’s server certificate) using the tls_certifcate data source from our EKS cluster. After that, we can reference the certificate from the IAM OIDC provider resource (aws_iam_openid_connect_provider):
locals {
eks_cluster_name = data.terraform_remote_state.eks.outputs.eks_cluster_name
}
data "aws_eks_cluster" "eks_cluster" {
name = local.eks_cluster_name
}
# Obtain TLS certificate for the OIDC provider
data "tls_certificate" "tls" {
url = data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
}
# Create OIDC Provider using TLS certificate
resource "aws_iam_openid_connect_provider" "eks_ca_oidc_provider" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.tls_certificate.tls.certificates[0].sha1_fingerprint]
url = data.aws_eks_cluster.eks_cluster.identity[0].oidc[0].issuer
}
We have completed the configuration required to set up an IAM OIDC provider for Cluster Autoscaler. This would allow authentication of the Cluster Authentication via STS, and allow it to interact with AWS Autoscaling Group.
To apply this module and deploy the Amazon EKS cluster using Terraform, you need to execute the following commands from the 2_eks
folder:
terraform init
terraform apply -auto-approve
You’ll get the Cluster Autoscaler IAM Role ARN as the module output:

Deploying the Cluster Autoscaler
We have got an EKS cluster service account to interact with the AWS Auto Scaling group. Now, let’s install the Cluster Autoscaler onto the cluster. First, we need to prepare the Kubernetes manifest. As the manifest is quite elaborate and complicated, we will use an existing example manifest from the open-source Kubernetes repository on GitHub. Execute the following command at the root level of your project :
curl https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples
/cluster-autoscaler-autodiscover.yaml -o 3_k8s_apps/cluster_autoscaler/cluster_autoscaler.yml
The above command will download the example manifest from the GitHub repository into a file called cluster-autoscaler.yml
inside the 3_k8s_apps/cluster_autoscaler
directory. We need to make a couple of minor changes to the code:
- Replace
<YOUR CLUSTER NAME>
with the name of your K8s cluster - Replace
<YOUR CLUSTER AUTOSCALER ROLE ARN>
with the Cluster Autoscaler IAM Role ARN obtained from the Terraform module output
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
name: cluster-autoscaler
namespace: kube-system
annotations:
eks.amazonaws.com/role-arn: <YOUR CLUSTER AUTOSCALER ROLE ARN>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
- apiGroups: [""]
resources: ["events", "endpoints"]
verbs: ["create", "patch"]
- apiGroups: [""]
resources: ["pods/eviction"]
verbs: ["create"]
- apiGroups: [""]
resources: ["pods/status"]
verbs: ["update"]
- apiGroups: [""]
resources: ["endpoints"]
resourceNames: ["cluster-autoscaler"]
verbs: ["get", "update"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["watch", "list", "get", "update"]
- apiGroups: [""]
resources:
- "namespaces"
- "pods"
- "services"
- "replicationcontrollers"
- "persistentvolumeclaims"
- "persistentvolumes"
verbs: ["watch", "list", "get"]
- apiGroups: ["extensions"]
resources: ["replicasets", "daemonsets"]
verbs: ["watch", "list", "get"]
- apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
verbs: ["watch", "list"]
- apiGroups: ["apps"]
resources: ["statefulsets", "replicasets", "daemonsets"]
verbs: ["watch", "list", "get"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses", "csinodes", "csidrivers", "csistoragecapacities"]
verbs: ["watch", "list", "get"]
- apiGroups: ["batch", "extensions"]
resources: ["jobs"]
verbs: ["get", "list", "watch", "patch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["create"]
- apiGroups: ["coordination.k8s.io"]
resourceNames: ["cluster-autoscaler"]
resources: ["leases"]
verbs: ["get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create","list","watch"]
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["cluster-autoscaler-status", "cluster-autoscaler-priority-expander"]
verbs: ["delete", "get", "update", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-autoscaler
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-autoscaler
subjects:
- kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
k8s-addon: cluster-autoscaler.addons.k8s.io
k8s-app: cluster-autoscaler
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cluster-autoscaler
subjects:
- kind: ServiceAccount
name: cluster-autoscaler
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cluster-autoscaler
namespace: kube-system
labels:
app: cluster-autoscaler
spec:
replicas: 1
selector:
matchLabels:
app: cluster-autoscaler
template:
metadata:
labels:
app: cluster-autoscaler
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '8085'
spec:
priorityClassName: system-cluster-critical
securityContext:
runAsNonRoot: true
runAsUser: 65534
fsGroup: 65534
serviceAccountName: cluster-autoscaler
containers:
- image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.21.0
name: cluster-autoscaler
resources:
limits:
cpu: 100m
memory: 600Mi
requests:
cpu: 100m
memory: 600Mi
command:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=least-waste
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>
volumeMounts:
- name: ssl-certs
mountPath: /etc/ssl/certs/ca-certificates.crt #/etc/ssl/certs/ca-bundle.crt for Amazon Linux Worker Nodes
readOnly: true
imagePullPolicy: "Always"
volumes:
- name: ssl-certs
hostPath:
path: "/etc/ssl/certs/ca-bundle.crt"
Now, we can apply the manifest to the EKS cluster:
kubectl apply -f 3_k8s_apps/cluster_autoscaler/cluster_autoscaler.yml
Verify the installation by inspecting the pods in the kube-system
namespace:
kubectl get pods -n kube-system

Thus, the Cluster Autoscaler is successfully installed on our cluster!
Testing the Cluster Autoscaler
Open two terminal windows side by side to monitor the number of pods and nodes in real-time:
kubectl get pods --namespace nginx-ns -w
kubectl get nodes -w
To test the Cluster Autoscaler, we would need to increase the replicas
for our deployment
to a higher number (say 10). This would increase the requested CPU and make the cluster autoscaler trigger a scale-out action, and add more nodes to our cluster :
kubectl scale --replicas=60 -f 3_k8s_apps/nginx/nginx.yml
On applying the manifest, you would notice an increase in the number of pods and nodes in the two open terminal windows:

We have a working Cluster Autoscaler at this stage!
Cleanup
To delete the demo Nginx Kubernetes application, we need to execute the following command:
kubectl delete -f 3_k8s_apps/nginx/nginx.yml
To destroy Terraform provisioned infrastructure, execute the following command for the 4_eks_cluster_autoscaler
, 2_eks
, and 1_vpc
Terraform modules:
terraform destroy --auto-approve
Summary
In this article, we’ve covered how to deploy the Amazon EKS cluster, EKS Cluster Autoscaler, and Horizontal Pod Autoscaler (HPA) using Terraform. Also, we’ve tested our K8s deployment configuration using the Nginx demo application.
Related articles
- Docker – How to setup Jupyter behind Nginx proxy
- Serverless Framework – Run your Kubernetes Workloads on Amazon EC2 Spot Instances with Amazon EKS – Part 1
- Serverless Framework – Run your Kubernetes Workloads on Amazon EC2 Spot Instances with Amazon EKS – Part 2
- Managing AWS IAM using Terraform
- Managing Amazon ECS Using Terraform
How useful was this post?
Click on a star to rate it!
We are sorry that this post was not useful for you!
Let us improve this post!
Tell us how we can improve this post?
DevOps Engineer. Fullstack Developer. Web3.0 Enthusiast. AWS and Terraform Certified.