Cloud infrastructure logs are extremely important because they help organizations troubleshoot problems and improve cloud infrastructure security. They can also provide insights into how an organization’s users interact with the cloud infrastructure. Centralized AWS logging improves your logging infrastructure’s efficiency and performance. You can easily identify and troubleshoot issues by consolidating your logs in a centralized location. In addition, centralized logging can also help you comply with corporate or regulatory requirements. Finally, this article is worth reading if you’re preparing for AWS Certified Solution Architect Professional certification.
In this post, we’ll explain how you can build a centralized logging infrastructure in AWS. Let’s get started!
Table of contents
Central logging account (AWS Landing Zone)
Regarding central aggregation and analysis of AWS logs, you must start from the proper AWS Landing Zone implementation – a solution that helps organizations manage a secure and scalable multi-account AWS environment.
AWS Landing Zone (LZ) can be implemented using the following services and technologies:
- AWS Control Tower – native AWS LZ solution
- AWS CloudFormation (custom-built LZ)
- Terraform (custom-built LZ)
Usually, every Landing Zone dedicates a central logging account for storing and analyzing log data generated by various AWS services, cloud, and on-premises applications.
A central logging account allows you to store and analyze logs centrally, apply centralized policies for logs’ archiving and deleting processes, and perform centralized logs monitoring.
Depending on the AWS Organization structure complexity, you may have separate logging accounts for infrastructure, security, or application logs.
Central logging architecture
Let’s start with the AWS Organizations and AWS accounts’ level first. Here’s an idea of a centralized log data aggregation approach.
Next, we must understand what logs we need to collect and the options to monitor and analyze them centrally. Let’s start with logs monitoring and analysis first.
Monitoring logs in the Logging account
A few solutions can be used to monitor and analyze logs and implement a log analytics strategy in your AWS infrastructure.
Amazon CloudWatch
This service allows you to collect, monitor, and analyze log data from AWS services and other sources. You can also use the AWS Logging agent to collect logs from EC2 instances and send them to Amazon CloudWatch Logs. After that, you can use CloudWatch Logs Insights for logs analysis. Amazon CloudWatch service is a great option for monitoring and analysis of your logs, but it is usually replaced by more feature-reach solutions like ELK stack or Splunk.
Elasticsearch, Logstash, Kibana (ELK stack)
ELK (Elasticsearch, Logstash, Kibana) stack can collect and analyze logs from multiple sources, including AWS services. You can deploy the ELK stack in the central logging account in several ways:
- Hosted ELK stack powered by Elastic
- Amazon OpenSearch Service – an AWS offering of ELK stack based on the OpenSearch project (Amazon OpenSearch Service), a community-driven, ALv2 licensed fork of open-source Elasticsearch and Kibana.
Whatever deployment option you choose, the raw log data digesting architecture in the central logging AWS account will be the following:
Note: you can send the same stream to the OpenSearch cluster and S3 bucket, depending on your needs – “Write Streaming data to multiple Data Stores- AWS Kinesis.”
Note: Check the “Stream Amazon CloudWatch Logs to a Centralized Account for Audit and Analysis” article for automating CloudWatch Logs Subscription management.
For every other account in your AWS Organization, you’re digesting logs for centralized log management using the following approach:
Check the “Centralized Logging on AWS” implementation guide for more in-depth information.
Splunk
Splunk is a search engine and data analytics platform that enables users to search, monitor, and analyze data from various sources. It indexes data stored in various formats, including log files, application performance data, and clickstream data. Splunk offers a freemium pricing model, with a limited feature set available for free and a full feature set available for paid customers.
Here’s an official reference deployment of Splunk Enterprise on the AWS cloud.
Splunk uses the same log digestion architecture as ELK stack, but instead of Kinesis Data Firehose and OpenSearch, Lambda should send logs to Splunk HTTP Event Collector:
Check the “How to stream AWS CloudWatch Logs to Splunk” article for more information.
SaaS log monitoring solutions
Several great SaaS platforms are currently available for organizations ready to export their data outside the organization’s perimeter.
New Relic
New Relic is a cloud-based SaaS platform for application and cloud performance management and observability. It offers a variety of products helping developers and operations teams measure and monitor the performance of their applications and monitors cloud infrastructure.
New Relic | Monitor, Debug and Improve Your Entire Stack
DataDog
DataDog – a SaaS-based platform that enables users to collect and analyze data from their applications and infrastructure. It offers a variety of features, such as monitoring, alerting, reporting, and collaboration tools. DataDog also integrates with a wide range of services and applications, making it an ideal solution for organizations that want to consolidate all of their data into one platform.
Dynatrace
Dynatrace platform combines logs with other data sources, including metrics, traces, and real user data, all in context with dependency mapping, for smarter AI-powered full-stack observability that delivers answers and continuous automation.
FAQ
Which AWS services should send logs to the central AWS account?
Almost any service or application can send logs to the central AWS account, depending on your needs and compliance requirements. You may consider the following services as a baseline for security purposes:
- AWS CloudTrail logs
- AWS Config logs
- VPC (flow logs)
For compute workloads, you may consider collecting logs from:
- Amazon EC2 (through CloudWatch Agent)
- AWS Lambda functions
- AWS Step Functions
- Amazon ECS
- Amazon EKS
Finally, many AWS organizations prefer to log S3 buckets’ events.
Free hands-on AWS workshops
For gaining hands-on experience with Amazon CloudWatch, we strongly suggest you check out CloudWatch AWS workshops:
Summary
Amazon CloudWatch Logs service allows you to aggregate logs in the central Logging AWS account and stores them in the central S3 bucket. You can use the AWS OpenSearch cluster or several 3rd-party solutions for monitoring, root-cause analysis, and operational troubleshooting.
