Managing AWS CloudWatch using Terraform

Laptop with code
Related Content

Amazon CloudWatch is a service that monitors resources and applications in AWS (Amazon Web Services). With Amazon CloudWatch, you can track the resources and application performance, collect and monitor log files details, and enable your resources’ alarms and notifications to be triggered on specific events. This article will use Terraform to automate alarms, dashboards, and logs in the AWS CloudWatch service.

Prerequisites

To start managing any AWS services, including AWS IAM, using Terraform, you need to install Terraform on your machine and set up access to your AWS account using the AWS access key. You can also use aws-vault for managing access to multiple AWS environments.

Working with Amazon CloudWatch in Python using Boto3

Working with Amazon CloudWatch in P...
Working with Amazon CloudWatch in Python using Boto3

Alternatively, you can set up and launch a Cloud9 IDE Instance.

Project Structure

This article comprises four sections, including managing AWS CloudWatch alarms, dashboards, and logs using Terraform. Each section of this article has an example that you can execute independently. You can put every example in a separate Terraform *.tf file to achieve the results shown in this article.

Some of the examples in this article are based on open-source Terraform modules. These modules hide and encapsulate your Terraform code’s implementation logic into a reusable resource. You can reuse modules in multiple places of your Terraform project without repeatedly duplicating lots of Terraform code.

Note: you can find every open-source Terraform module code on GitHub. For example, here’s a source code of the terraform-aws-modules/cloudwatch/aws module.

All Terraform files are in the same folder and belong to the same Terraform state file.

Make sure to use commands to avoid unnecessary errors while following the article:

  • terraform validate to verify your Terraform HCL file
  • terraform plan to check out the desired changes on every Terraform file creation
  • terraform apply to create the resources in AWS

AWS Terraform provider

To start managing the AWS CloudWatch service, you need to declare the AWS Terraform provider in a providers.tf file.

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "3.69.0"
    }
  }
}

provider "aws" {
  profile = "default"
  region  = "us-east-2"
}

Run the terraform init command to initialize the Terraform working directory with the AWS plugins.

Here is the execution output.

Initialize AWS Terraform provider plugins

Create an AWS CloudWatch dashboard using Terraform

To create an AWS CloudWatch dashboard using Terraform, you can use the aws_cloudwatch_dashboard resource and assign the required arguments: the dashboard_name and the dashboard_body. Then, you can refer to the official AWS CloudWatch API reference documentation that defines the recommended dashboard body structure and syntax to build your dashboards.

In the following example, we will create a dashboard that monitors the Maximum CPU utilization of all running EC2 instances in your AWS account.

resource "aws_cloudwatch_dashboard" "EC2_Dashboard" {
  dashboard_name = "EC2-Dashboard"

  dashboard_body = <<EOF
{
    "widgets": [
        {
            "type": "explorer",
            "width": 24,
            "height": 15,
            "x": 0,
            "y": 0,
            "properties": {
                "metrics": [
                    {
                        "metricName": "CPUUtilization",
                        "resourceType": "AWS::EC2::Instance",
                        "stat": "Maximum"
                    }
                ],
                "aggregateBy": {
                    "key": "InstanceType",
                    "func": "MAX"
                },
                "labels": [
                    {
                        "key": "State",
                        "value": "running"
                    }
                ],
                "widgetOptions": {
                    "legend": {
                        "position": "bottom"
                    },
                    "view": "timeSeries",
                    "rowsPerPage": 8,
                    "widgetsPerRow": 2
                },
                "period": 60,
                "title": "Running EC2 Instances CPUUtilization"
            }
        }
    ]
}
EOF
}

After running terraform apply, you can log in to the AWS console and access the CloudWatch service dashboard, where you should be able to see the maximum CPU utilization of all the running EC2 instances in your AWS account.

CloudWatch Dashboard

Manage AWS CloudWatch metrics using Terraform

This article section will cover how to manage AWS CloudWatch metrics using Terraform.

Create an AWS CloudWatch metric alarm

To create an AWS CloudWatch metric alarm, you can use the aws_cloudwatch_metric_alarm resource and pass the required arguments, such as the alarm_name, comparison_operator, and evaluation_periods.

resource "aws_cloudwatch_metric_alarm" "EC2_CPU_Usage_Alarm" {
  alarm_name                = "EC2_CPU_Usage_Alarm"
  comparison_operator       = "GreaterThanOrEqualToThreshold"
  evaluation_periods        = "2"
  metric_name               = "CPUUtilization"
  namespace                 = "AWS/EC2"
  period                    = "60"
  statistic                 = "Average"
  threshold                 = "70"
  alarm_description         = "This metric monitors ec2 cpu utilization exceeding 70%"
}

Alternatively, you can create an AWS CloudWatch metric using the official Terraform AWS CloudWatch module.

module "metric_alarm" {
  source  = "terraform-aws-modules/cloudwatch/aws//modules/metric-alarm"
  version = "~> 2.0"

  alarm_name          = "EC2_CPU_Usage"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 2
  threshold           = 70
  period              = 60
  unit                = "Count"

  namespace   = "AWS/EC2"
  metric_name = "CPUUtilization"
  statistic   = "Average"

  alarm_actions = ["arn:aws:sns:us-east-2:585584209241:cpu_utilization"]
}

Create an AWS CloudWatch metric alarm with EC2 Autoscaling

You can create an AWS CloudWatch metric alarm that triggers multiple actions depending on the defined CloudWatch metric alarm conditions. In the following example, we shall create an AWS CloudWatch metric alarm that monitors the average CPU usage and triggers an AWS AutoScaling policy to spin up more EC2 instances in an AWS AutoScaling group, should the average CPU Utilization average at 70 % for one minute. The depends_on meta-argument is used to ensure that the creation of one resource depends first on the creation of the referenced resource.

The aws_launch_template resource provides an EC2 launch template resource to use with the AutoScaling group. While the aws_autoscaling_group resource is used to create and define an AWS AutoScaling group’s properties, the aws_autoscaling_policy resource is used to create and define the AutoScaling policy properties of your EC2 instances.


resource "aws_launch_template" "EC2_Launch_Template" {
  name_prefix   = "EC2-Launch-Template"
  image_id      = "ami-0f540030bb04d884a"
  instance_type = "t2.micro"
}

resource "aws_autoscaling_group" "EC2_AutoScaling_Group" {
  availability_zones = ["us-east-2b"]
  desired_capacity   = 1
  max_size           = 5
  min_size           = 1

  launch_template {
    id      = aws_launch_template.EC2_Launch_Template.id
    version = "$Latest"
  }
  depends_on = [
    aws_launch_template.EC2_Launch_Template,
  ]
}


resource "aws_autoscaling_policy" "EC2_AutoScaling_Policy" {
  name                   = "EC2-AutoScaling-Policy"
  scaling_adjustment     = 2
  adjustment_type        = "ChangeInCapacity"
  cooldown               = 60
  autoscaling_group_name = aws_autoscaling_group.EC2_AutoScaling_Group.name
  depends_on = [
    aws_autoscaling_group.EC2_AutoScaling_Group,
  ]
}

resource "aws_cloudwatch_metric_alarm" "EC2_metric_alarm" {
  alarm_name          = "EC2-metric-alarm"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "1"
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = "60"
  statistic           = "Average"
  threshold           = "70"
  depends_on = [
    aws_autoscaling_group.EC2_AutoScaling_Group,
  ]
  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.EC2_AutoScaling_Group.name
  }
  

  alarm_description = "This metric monitors ec2 cpu utilization"
  alarm_actions     = [aws_autoscaling_policy.EC2_AutoScaling_Policy.arn]
}

Create an AWS CloudWatch metric stream

AWS CloudWatch metric streams send metrics to a real-time destination such as Amazon S3 (Simple Storage Service). For example, you can create a metric stream and connect it to an Amazon Kinesis Data firehose delivery stream that delivers your metrics to Amazon S3. The data sent can later be used for your purposes, such as analyzing resource performance.

To create an AWS CloudWatch metric stream, use the aws_cloudwatch_metric_stream and assign the required arguments, such as the firehose_arn, role_arn, and output_format. The include_filter argument is used to specify the namespaces you want to send metrics to S3. In this case, we are using AWS/EC2 and AWS/EBS.

In the following example, the aws_iam_role and aws_iam_role_policy terraform resources are used to provide the required permissions to the roles required by AWS CloudWatch metrics and the firehose delivery stream to access Amazon S3. The aws_s3_bucket resource is used to create the destination Amazon S3 bucket.

resource "aws_cloudwatch_metric_stream" "main" {
  name          = "my-metric-stream"
  role_arn      = aws_iam_role.metric_stream_to_firehose.arn
  firehose_arn  = aws_kinesis_firehose_delivery_stream.s3_stream.arn
  output_format = "json"

  include_filter {
    namespace = "AWS/EC2"
  }

  include_filter {
    namespace = "AWS/EBS"
  }
}

resource "aws_iam_role" "metric_stream_to_firehose" {
  name = "metric_stream_to_firehose_role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "streams.metrics.cloudwatch.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "metric_stream_to_firehose" {
  name = "default"
  role = aws_iam_role.metric_stream_to_firehose.id

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:PutRecord",
                "firehose:PutRecordBatch"
            ],
            "Resource": "${aws_kinesis_firehose_delivery_stream.s3_stream.arn}"
        }
    ]
}
EOF
}

resource "aws_s3_bucket" "bucket" {
  bucket = "test-metric-stream-bucket"
  acl    = "private"
}

resource "aws_iam_role" "firehose_to_s3" {
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "firehose_to_s3" {
  name = "default"
  role = aws_iam_role.firehose_to_s3.id

  policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:PutObject"
            ],
            "Resource": [
                "${aws_s3_bucket.bucket.arn}",
                "${aws_s3_bucket.bucket.arn}/*"
            ]
        }
    ]
}
EOF
}

resource "aws_kinesis_firehose_delivery_stream" "s3_stream" {
  name        = "metric-stream-test-stream"
  destination = "s3"

  s3_configuration {
    role_arn   = aws_iam_role.firehose_to_s3.arn
    bucket_arn = aws_s3_bucket.bucket.arn
  }
}

Create an AWS CloudWatch composite alarm

AWS CloudWatch Composite alarms are used to monitor two or more alarms to determine the alarm state and take action. The most common use of Composite alarms is to reduce alarm notifications by grouping multiple alarms and defining which metrics within’ alarms should trigger the different alarm states. When you set up a composite alarm, you will only be notified when the specified metrics from the different alarms are reached.

To create a Composite alarm using Terraform, you can use the aws_cloudwatch_composite_alarm resource and pass the required argument, which is the alarm_name

The aws_sns_topic and the aws_sns_topic_subscription resources are used to configure notifications sent to the User when the composite alarm reaches the desired alarm state.

resource "aws_cloudwatch_composite_alarm" "EC2_and_EBS" {
  alarm_description = "Composite alarm that monitors CPUUtilization and EBS Volume Write Operations"
  alarm_name        = "EC2_&EBS_Composite_Alarm"
  alarm_actions = [aws_sns_topic.EC2_and_EBS_topic.arn]

  alarm_rule = "ALARM(${aws_cloudwatch_metric_alarm.EC2_CPU_Usage_Alarm.alarm_name}) OR ALARM(${aws_cloudwatch_metric_alarm.EBS_WriteOperations.alarm_name})"


  depends_on = [
    aws_cloudwatch_metric_alarm.EC2_CPU_Usage_Alarm,
    aws_cloudwatch_metric_alarm.EBS_WriteOperations,
    aws_sns_topic.EC2_and_EBS_topic,
    aws_sns_topic_subscription.EC2_and_EBS_Subscription
  ]
}


resource "aws_cloudwatch_metric_alarm" "EC2_CPU_Usage_Alarm" {
  alarm_name          = "EC2_CPU_Usage_Alarm"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "2"
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = "60"
  statistic           = "Average"
  threshold           = "70"
  alarm_description   = "This metric monitors ec2 cpu utilization exceeding 70%"
}


resource "aws_cloudwatch_metric_alarm" "EBS_WriteOperations" {
  alarm_name          = "EBS_WriteOperations"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "2"
  metric_name         = "VolumeReadOps"
  namespace           = "AWS/EC2"
  period              = "120"
  statistic           = "Average"
  threshold           = "1000"
  alarm_description   = "This monitors the average read operations on EBS Volumes in a specified period of time"
}


resource "aws_sns_topic" "EC2_and_EBS_topic" {
  name = "EC2_and_EBS_topic"
}

resource "aws_sns_topic_subscription" "EC2_and_EBS_Subscription" {
  topic_arn = aws_sns_topic.EC2_and_EBS_topic.arn
  protocol  = "email"
  endpoint  = "kelvingalabuzi@gmail.com"

  depends_on = [
    aws_sns_topic.EC2_and_EBS_topic
  ]
}

Managing AWS CloudWatch Logs using Terraform

This article section will cover how to manage AWS CloudWatch logs using Terraform.

Create an AWS CloudWatch log group

A log group in AWS CloudWatch defines a collection of log streams that share the same retention, monitoring, and access control settings. To create an AWS CloudWatch log group using Terraform, use the aws_cloudwatch_log_group resource and assign arguments such as the name of the log group.

resource "aws_cloudwatch_log_group" "EC2_logGroup" {
  name = "EC2_logGroup"
  retention_in_days = 30
}

Alternatively, you can create an AWS CloudWatch log group using the official Terraform AWS CloudWatch module.

module "log_group" {
  source  = "terraform-aws-modules/cloudwatch/aws//modules/log-group"
  version = "~> 2.0"

  name              = "lambda_logGroup"
  retention_in_days = 120
}

Create an AWS CloudWatch log stream

A log stream is a sequence of events sharing the same source. AWS CloudWatch log groups can have one or more log streams. To create an AWS CloudWatch log stream, use the aws_cloudwatch_log_stream resource and assign the required arguments, which are the name of the log stream and the log_group_name

resource "aws_cloudwatch_log_group" "ebs_log_group" {
  name = "ebs_log_group"
}

resource "aws_cloudwatch_log_stream" "ebs_log_stream" {
  name           = "ebs_log_stream"
  log_group_name = aws_cloudwatch_log_group.ebs_log_group.name
}

Create an AWS CloudWatch log metric filter

AWS CloudWatch metric filters are used to extract information from events. They define the terms and patterns to look for in log data as it’s sent to CloudWatch logs. The data from the metric filters can be turned into CloudWatch metrics, which can then be graphed or used to set alarms. To create an AWS CloudWatch metric filter, use the aws_cloudwatch_log_metric_filter resource and assign all the required arguments.

resource "aws_cloudwatch_log_metric_filter" "ElasticBeanStalk_metric_filter" {
  name           = "ElasticBeanStalk_metric_filter"
  pattern        = "ERROR"
  log_group_name = aws_cloudwatch_log_group.ElasticBeanStalk_log_group.name

  metric_transformation {
    name      = "ErrorCount"
    namespace = "ElasticBeanstalk"
    value     = "1"
  }
}

resource "aws_cloudwatch_log_group" "ElasticBeanStalk_log_group" {
  name = "ElasticBeanStalk_log_group"
}

Create an AWS CloudWatch log subscription filter

Amazon CloudWatch log subscription filters associate events that match a specific pattern found in a log group with Kinesis, Lamba, or Firehose for additional processing. To create an AWS CloudWatch log subscription filter, you can use the aws_cloudwatch_log_subscription_filter resource and assign the required arguments, such as the name of the subscription filter, destination_arn, and filter_pattern

resource "aws_cloudwatch_log_subscription_filter" "route53_logfilter" {
  name            = "route53_logfilter"
  role_arn        = aws_iam_role.kinesis_subscription_filter_role.arn
  log_group_name  = "route53_logs"
  filter_pattern  = "testpattern"
  destination_arn = aws_kinesis_stream.route53_kinesis_stream.arn
  distribution    = "Random"
  depends_on = [
  aws_kinesis_stream.route53_kinesis_stream,
  aws_iam_role.kinesis_subscription_filter_role
  ]
}


resource "aws_kinesis_stream" "route53_kinesis_stream" {
  name             = "route53_kinesis_stream"
  shard_count      = 1
  retention_period = 30

  shard_level_metrics = [
    "IncomingBytes",
    "OutgoingBytes",
  ]

  stream_mode_details {
    stream_mode = "PROVISIONED"
  }

}

resource "aws_iam_role" "kinesis_subscription_filter_role" {
  name = "kinesis_subscription_filter_role"
  managed_policy_arns = [aws_iam_policy.subscription_filter_policy.arn]
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = ["logs.amazonaws.com",
          "kinesis.amazonaws.com"
          ]
        }
      },
    ]
  })

  }
  
  resource "aws_iam_policy" "subscription_filter_policy" {
  name        = "subscription_filter_policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["logs:*", "route53:*", "kinesis:*"]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

Create an AWS CloudWatch log resource policy

An AWS CloudWatch log resource policy allows an AWs service to log events to a log group. In the following example, we will create an AWS CloudWatch log resource policy enabling Route53 to log to a log group. To create the resource policy, you can use the aws_cloudwatch_log_resource_policy resource and assign the required arguments, which are the policy_document and the policy_name

data "aws_iam_policy_document" "route53-logging-policy" {
  statement {
    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]

    resources = ["rn:aws:logs:us-east-2:585584209241:log-group:route53_logs:*"]

    principals {
      identifiers = ["route53.amazonaws.com"]
      type        = "Service"
    }
  }
}

resource "aws_cloudwatch_log_resource_policy" "route53-logging-policy" {
  policy_document = data.aws_iam_policy_document.route53-logging-policy.json
  policy_name     = "route53-logging-policy"
}

Create an AWS CloudWatch log destination

An AWS CloudWatch log destination allows you to create a destination to which CloudWatch logs can send logs. To create an AWS CloudWatch log destination, you can use the aws_cloudwatch_log_destination resource and assign the required arguments, such as the name of the log destination, role_arn and target_arn. The log destination resource depends on creating two more AWS resources, such as the aws_iam_role to grant CloudWatch logs to send logs to a destination, and the aws_kinesis_stream resource.

resource "aws_iam_role" "cloudwatch_log_destination_role" {
  name = "cloudwatch_log_destination_role"
  managed_policy_arns = [aws_iam_policy.cloudwatch_log_destination_policy.arn]
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = [
          "logs.amazonaws.com"
          ]
        
        }
      },
    ]
   })
  }
  
  resource "aws_iam_policy" "cloudwatch_log_destination_policy" {
  name        = "cloudwatch_log_destination_policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["logs:*", "kinesis:*"]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}


resource "aws_kinesis_stream" "Kinesis_stream" {
  name             = "Kinesis_stream"
  shard_count      = 1
  retention_period = 48

  shard_level_metrics = [
    "IncomingBytes",
    "OutgoingBytes",
  ]

  stream_mode_details {
    stream_mode = "PROVISIONED"
  }

}

resource "aws_cloudwatch_log_destination" "EC2_log_destination" {
  name       = "EC2_log_destination"
  role_arn   = aws_iam_role.cloudwatch_log_destination_role.arn
  target_arn = aws_kinesis_stream.Kinesis_stream.arn
  depends_on = [
    aws_iam_role.cloudwatch_log_destination_role,
    aws_kinesis_stream.Kinesis_stream
  ]
}

Create an AWS CloudWatch query definition using Terraform

To create an AWS CloudWatch query definition, you can use the aws_cloudwatch_query_definition resource and assign the required arguments: the name of the query and the query_string.

resource "aws_cloudwatch_query_definition" "demo_definition" {
  name = "demo_query"

  log_group_names = [
    "route53_logs"
  ]

  query_string = <<EOF
fields @timestamp, @message
| sort @timestamp desc
| limit 25
EOF
}

Create an AWS CloudWatch event rule using Terraform

Amazon CloudWatch events stream system events performed on AWS resources in real-time. Rules are used to match incoming events and route them to targets. To create an AWS CloudWatch event rule, you can use the aws_cloudwatch_event_rule resource and assign some optional parameters. Then, the event rule is used with the aws_cloudwatch_event_target resource using an SNS topic target. In the following example, we will create an event rule that monitors AWS console sign-ins and sends the events to an SNS topic.

resource "aws_cloudwatch_event_rule" "console" {
  name        = "capture-aws-sign-in"
  description = "Capture each AWS Console Sign In"

  event_pattern = <<EOF
{
  "detail-type": [
    "AWS Console Sign In via CloudTrail"
  ]
}
EOF
}

resource "aws_cloudwatch_event_target" "sns" {
  rule      = aws_cloudwatch_event_rule.console.name
  target_id = "SendToSNS"
  arn       = aws_sns_topic.aws_logins.arn
}

resource "aws_sns_topic" "aws_logins" {
  name = "aws-console-logins"
}

resource "aws_sns_topic_policy" "default" {
  arn    = aws_sns_topic.aws_logins.arn
  policy = data.aws_iam_policy_document.sns_topic_policy.json
}

data "aws_iam_policy_document" "sns_topic_policy" {
  statement {
    effect  = "Allow"
    actions = ["SNS:Publish"]

    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }

    resources = [aws_sns_topic.aws_logins.arn]
  }
}

Summary

This article used Terraform to automate alarms, dashboards, and logs in the AWS CloudWatch service.

LIKE THIS ARTICLE?
Facebook
Twitter
LinkedIn
Pinterest
WANT TO BE AN AUTHOR OF ANOTHER POST?

We’re looking for skilled technical authors for our blog!

Table of Contents