How to inspect VPC, subnet, and EC2 instance traffic in AWS

Chris Noon

Chris Noon

0
(0)

Running a complex network is not an easy job. In addition to simply keeping it up and running, you need to keep an ever-watchful eye out for unusual traffic patterns or content that could signify a network intrusion, a compromised instance, or some other anomaly. This article covers how to inspect and monitor AWS VPCs, Subnets, and Network Interfaces traffic using VPC Flow Logs, Traffic Mirroring, Amazon Athena, OpenSearch (Elasticsearch), Suricata, and Guard Duty deployed by the Terraform.

Prerequisites

In this article, we’ll use Terraform to set up the following AWS services:

  • Amazon VPC (2 public, 2 private subnets, Internet Gateway, NAT Gateways)
  • VPC Flow Logs (VPC, Subnet, Interface)
  • Amazon Athena
  • Amazon OpenSearch (Elasticsearch)
  • Amazon EC2
  • AWS Systems Manager Session Manager
How to inspect VPC, subnet, and EC2 instance traffic in AWS - Entire setup

We’ll use Amazon EC2 instances with the following roles:

  • Demo EC2 instance to analize traffic from
  • Suricata – an open source-based intrusion detection system and intrusion prevention system (sends alerts to ElasticSearch, deployed as a Docker container)
  • Windows Jumphost – to get access to Kibana (Elasticsearch Dashboard)

You can find all source code in our GitHub repository:

Note: all Terraform modules rely on remote state stored in S3 bucket and use DynamoDB for execution lock. You can deploy this infrastructure using the 0_remote_state module.

We recommend accomplishing the MKT210 – Amazon VPC Traffic Mirroring Builder Session workshop before automating everything using Terraform.

VPC environment

Before enabling VPC Flow Logs and VPC Traffic mirroring, we need to set up a base demo VPC environment. Here’s how it will look like:

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Base infrastructure

To deploy this environment, you need to apply the 1_base_infrastructure module from our amazon-vpc-traffic-inspection-and-monitoring GitHub Repository:

cd 1_base_infrastructure
terraform init
terraform apply -auto-approve

If you’re interested in more in-depth information on creating a VPC infrastructure using Terraform, please, check out our previous posts:

Once you’ve deployed the module, you can connect to the Demo EC2 instance using AWS Systems Manager Session Manager.

Select your EC2 instance and click the Connect button:

SSM Session Manager - connect to EC2 instance

Choose the Session Manager tab and hit the Connect button:

SSM Session Manager - Connect to EC2 instance - Step 2

Now, you can test the Internet connectivity by executing the following commands:

bash
ping google.com -c 3

Here’s an execution output:

SSM Session Manager - EC2 instance - Test Internet connectivity

As soon as we have our VPC infrastructure in place, we can enable VPC Flow Logging.

VPC Flow Logs

VPC Flow Logs is a helpful feature that allows you to capture traffic flows within a VPC. Traffic logs are sent to an endpoint of your choosing, most commonly to the Amazon S3 bucket or Amazon CloudWatch. We’ll move forward with the S3 bucket as a log destination.

You can choose what traffic to capture: accepted, rejected traffic, or both.

You don’t have any controls to filter the traffic by source, destination, protocol, or other parameters. We’ll show you how to do it later by using Amazon Athena.

Usually, the generated traffic flow logs appear in your destination with some delay (from 5 to 15 mins). You can speed up this process by reducing the aggregation time up to 1 minute.

Entire VPC

You can enable VPC Flow Logging for the entire VPC:

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Flow Logs - VPC

To make this happen, you need to apply the 2_vpc_logging module from our demo repository.

This module uses the aws_flow_log Terraform resource to enable VPC Flow Logs.

Subnet

If you need more fine granular control over which traffic to capture, you can enable VPC Flow Logs at the subnet level:

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Flow Logs - Subnet

To make this happen, you need to apply the 3_subnet_logging module from our demo repository.

This module uses the aws_flow_log Terraform resource to enable VPC Flow Logs.

Network interface (ENI)

Finally, you can enable VPC Flow Logs for a specific network interface:

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Flow Logs - Interface

To make this happen, you need to apply the 4_interface_logging module from our demo repository.

This module uses the aws_flow_log Terraform resource to enable VPC Flow Logs.

VPC Flow Logs analysis using Amazon Athena

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. As soon as our VPC Flow Logs are stored in the S3 bucket, Amazon Athena will be a perfect tool to query information from them.

Here’s what we will set up:

How to inspect VPC, subnet, and EC2 instance traffic in AWS - VPC Flow Logs - Athena

To set up Amazon Athena, you need to apply the 5_athena_setup module from our demo repository.

Note: Terraform can’t execute SQL commands, so we’ll use it to create saved queries to simplify Athena table creation and partitioning setup.

As soon as you’ve applied the module, go to Athena web-console and switch the current workgroup to the amazon-vpc-traffic-mirroring-vpc-flow-logs:

Amazon Athena - Switch current workgroup

Now, go to the Query editor and execute the Athena table creation query:

Amazon Athena - Create VPC Flow Logs Table

To stimulate some traffic, let’s run the curl command in a simple Bash while-loop at Demo EC2 instance:

while true; do curl google.com; sleep 30; done

Within a few minutes, logs should start appearing in our S3 bucket:

VPC Flow Logs in S3

Now, you can begin querying the information from the VPC Flow Logs using Amazon Athena.

Amazon Athena - Query VPC Flow Logs Table

In our example, we’ve queried 20 records from VPC Flow Logs starting from the 25th of October 2021:

SELECT * FROM vpc_flow_logs WHERE day > '2021/10/25' limit 20;

You’re using the basic SQL syntax to query any traffic information you’d like.

For example, to filter by the source IP address, execute the following query:

SELECT * FROM vpc_flow_logs WHERE sourceaddress='10.0.1.101';

To filter by the destination address, run the following query:

SELECT * FROM vpc_flow_logs WHERE destinationaddress='54.246.214.20';

You can combine query conditions using AND statement in your SQL WHERE block:

SELECT * FROM vpc_flow_logs WHERE sourceaddress='10.0.1.101' AND destinationport=80;

You can go further and integrate Amazon Athena with QuickSight to make any VPC Flow Logs data visualizations.

Here’s an example of Demo EC2 instance traffic grouped by destination visualized in QuickSight:

VPC Flow Logs visualization using Amazon QuickSight

VPC Traffic Mirroring

There are some situations when VPC Flow Logs information is not enough, and we need to see more detail about the packets flowing around your VPC. This is a perfect use case for setting up Traffic Mirroring. Setting up a Traffic Mirroring for Network Interface within a VPC is the equivalent of configuring a SPAN port on a Network Switch.

The traffic mirroring feature provides several benefits:

  • No agent is required on the EC2 instances
  • Traffic captures can be forwarded to anywhere inside or outside the VPC
  • Advanced traffic analysis can be done in combination with a third-party tools

Traffic Mirroring filters can be attached to any (EC2 instance or Network Load Balancer) ENI interfaces.

Traffic Mirroring in comparison with VPC Flow Logs allows filtering on the packet captures. If you are interested only in a specific destination or port, you can dramatically cut down the amount of traffic captured and forwarded to the analytics platform.

You can configure two deployment models forwarding packet captures to the target.

The first deployment model assumes that you’re configuring traffic forwarding to a single target. The target can be either a single instance or multiple instances behind a Network Load Balancer.

To simplify an article a bit, we’ll use precisely this deployment model and configure Suricata open source solution to capture mirrored traffic flows:

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Traffic mirroring

If you are processing significant traffic volumes, it makes sense to put Suricata or any other network traffic analysis tools behind the Load Balancer.

For more information about this deployment model, consider reading the following articles from the AWS blog:

Using VPC Traffic Mirroring to monitor and secure your AWS infrastructure

Scaling network traffic inspection using AWS Gateway Load Balancer

The second deployment model is when the same traffic is sent to multiple network traffic analysis platforms. In this instance, you would need a Cloud Broker platform to replicate the traffic and forward it to various endpoints:

Replicating mirrored traffic to multiple analytics platforms

Amazon OpenSearch (Elasticsearch)

There are two open-source solutions mentioned in official AWS documentation for processing mirrored traffic right now: Zeek and Suricata.

Both solutions require the Elasticsearch setup for traffic analysis.

To deploy the Amazon OpenSearch (Elasticsearch), you need to apply the 6_elk_stack module from our demo repository.

This module will deploy the Amazon OpenSearch (Elasticsearch) instance in the private subnet of our VPC.

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Elasticsearch

If you’d like to deploy ElasticSearch in HA mode, check out our article Terraform recipe – How to create AWS ElasticSearch cluster.

Suricata

There are two open-source solutions mentioned in official AWS documentation for processing mirrored traffic right now: Zeek and Suricata.

In this article, we’ll use the Suricata solution for demo purposes. The Zeek configuration is quite similar.

Zeek or Suricata are capturing mirrored traffic flows and logging them based on configured rules. The Filebeats logs shipper sends logged information to the Elasticsearch instance or cluster.

To deploy the Suricata EC2 instance, you need to apply the 7_suricata module from our demo repository.

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Suricata

During the bootstrap, the Suricata EC2 instance will install the Suricata Docker container built from scratch. We slightly modified the original solution to use Amazon OpenSerach (Elasticsearch) service instead of keeping ELK stack on the Suricata instance to make it lighter.

Here’s how the Suricata EC2 instance is designed:

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Suricata instance

Enabling Traffic Mirroring

As soon as we set up the infrastructure for traffic analysis, we can enable Traffic Mirroring from the Demo EC2 instance to the Suricata EC2 instance.

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Traffic mirroring

To enable Traffic Mirroring, you need to deploy the 7_suricata Terraform module from our demo repository.

This module will mirror all HTTPS traffic from the Demo EC2 instance to the Suricata EC2 instance.

Windows Jumpbox

Finally, to access to ElasticSearch Kibana dashboard (URL provided in the 6_elk_stack Terraform module output), we need to deploy the Windows Jumpbox EC2 instance.

How to inspect VPC, subnet, and EC2 instance traffic in AWS - Windows Jumpbox

Note: you need to provide your own SSH Key Name value to deploy this module successfully.

Deploy the 9_windows_jumpbox Terraform module, which will run Windows EC2 instance and automatically install Google Chrome using Chocolatey.

Kibana initial configuration

Log in to your Windows JumpHost, open the Kibana dashboard using Google Chrome (URL provided in the 6_elk_stack Terraform module output), and hit Add data button.

Kibana - Add data

Click on the Suricata logs module:

Kibana - Suricata logs

Scroll down the page and hit Create button.

Kibana - Enable Suricata logs module

Hit Create index pattern button:

Kibana - Create index pattern for Suricata

Put filebeat-* to the Index pattern name field. Hit the Next step button.

Kibana - Filebeat index patter for Suricata logs

Specify @timestamp as the Time field and hit the Create index pattern button.

Kibana - Filebeat index patter for Suricata logs - Timestamp

You’ll see a successfully created Elasticsearch Index pattern for Suricata logs:

Kibana - Filebeat index patter for Suricata logs - Created index

Return to the Kibana homepage and click on the Kibana widget:

Kibana - Homepage

Click on the Discover widget:

Kibana - Discover widget

In the Search field type src_ip = '10.0.1.101' to get Suricata traffic events captured from the Demo EC2 instance:

Kibana - Source IP filtering

Amazon GuardDuty

Amazon GuardDuty uses ML (Machine Learning), Anomaly Detection, and Integrated Threat Intelligence to identify threats in your environment. It accomplishes this by analyzing log data from your environment. Coming full circle, GuardDuty can use VPC Flow Logs data to examine what it believes to be “normal” traffic flows. Any anomalies can be flagged for review or have an action created against them with either CloudWatch or Lambda.

We enabled Amazon GuardDuty when we deployed our demo VPC using the 1_base_infrastructure Terraform module.

GuardDuty automatically checks S3 buckets data for the VPC Flow Logs data, analyzes it, and generates alerts.

You can test GuardDuty by executing the following command from our Demo EC2 instance:

dig GuardDutyC2ActivityB.com any

The above DNS query command to a known fake domain generates a GuardDuty finding:

GuardDuty - VPC Flow Logs findings

You may find some additional GuardDuty tests in the amazon-guardduty-tester repository.

Another interesting use-case of using GuardDuty is enabling automatic Traffic Mirroring from the instances that generated the GuardDuty events to your traffic network analysis platform.

Check out the VPC Traffic Mirroring Source Automation Application GitHub repository for more information.

Summary

In this article, we’ve covered how to automate traffic inspection and monitoring for AWS VPCs, Subnets, and Network Interfaces using VPC Flow Logs, Traffic Mirroring, Amazon Athena, OpenSearch (Elasticsearch), Suricata, and Guard Duty deployed by the Terraform.

How useful was this post?

Click on a star to rate it!

As you found this post useful...

Follow us on social media!

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Top rated Udemy Courses to improve you career

Subscribe to our updates

Like this article?

Share on facebook
Share on Facebook
Share on twitter
Share on Twitter
Share on linkedin
Share on Linkdin
Share on pinterest
Share on Pinterest

Want to be an author of another post?

We’re looking for skilled technical authors for our blog!

Leave a comment

If you’d like to ask a question about the code or piece of configuration, feel free to use https://codeshare.io/ or a similar tool as Facebook comments are breaking code formatting.