Collecting and Processing Logs

CloudWatch Logs can be used to monitor, store, and access your log files from Amazon EC2 instances, AWS CloudTrail, Route 53, and other sources. See the AWS Services That Publish Logs to CloudWatch Logs documentation page.

Logs information includes, for example:

Custom application metrics and logs can also be published to CloudWatch Logs by installing the CloudWatch Agent on Amazon EC2 instances or on-premises servers.

Logs can be analyzed interactively using CloudWatch Logs Insights, performing queries to help you respond more efficiently and effectively to operational issues.

CloudWatch Logs can be processed in near real-time by configuring subscription filters and delivered to other services such as an Amazon OpenSearch Service (OpenSearch Service) cluster, an Amazon Kinesis stream, an Amazon Data Firehose stream, or Lambda for custom processing, analysis, or loading to other systems. CloudWatch metric filters can be used to define patterns to look for in log data, transform them into numerical CloudWatch metrics, and set up alarms based on your business requirements. For example, following the AWS recommendation not to use the root user for everyday tasks, it is possible to set up a specific CloudWatch metric filter on a CloudTrail log (delivered to CloudWatch Logs) to create a custom metric and configure an alarm to notify the relevant stakeholders when root user credentials are used to access your AWS account.

Logs such as Amazon S3 server access logs, Elastic Load Balancing access logs, VPC flow logs, and AWS Global Accelerator flow logs can be delivered directly to an Amazon S3 bucket. For example, when you enable Amazon Simple Storage Service server access logs, you can get detailed information regarding the requests that are made to your Amazon S3 bucket. An access log record contains details about the request, such as the request type, the resources specified in the request, and the time and date the request was processed. For more information about the contents of a log message, see Amazon Simple Storage Service Server Access Log Format in the Amazon Simple Storage Service User Guide. Server access logs are useful for many applications because they give bucket owners insight into the nature of requests made by clients that are not under their control. By default, Amazon S3 does not collect service access logs, but when you enable logging, Amazon S3 usually delivers access logs to your bucket within a few hours. If you require a faster delivery or need to deliver logs to multiple destinations, consider using CloudTrail logs or a combination of both CloudTrail logs and Amazon S3. Logs can be encrypted at rest by configuring default object encryption in the destination bucket. The objects are encrypted using server-side encryption with either Amazon S3-managed keys (SSE-S3) or KMS keys (formerly AWS KMS Key) stored in AWS Key Management Service (AWS KMS).

Logs stored in an Amazon S3 bucket can be queried and analyzed using Amazon Athena. Amazon Athena is an interactive query service that enables you to analyze data in S3 using standard SQL. You can use Athena to run ad-hoc queries using ANSI SQL, without the need to aggregate or load the data into Athena. Athena can process unstructured, semi-structured, and structured data sets and integrates with Amazon Quick Sight for easy visualization.

Logs are also a useful source of information for automated threat detection. Amazon GuardDuty is a continuous security monitoring service that analyzes and processes events from several sources, such as VPC Flow Logs, CloudTrail management event logs, CloudTrail Amazon S3 data event logs, and DNS logs. It uses threat intelligence feeds, such as lists of malicious IP addresses and domains, and machine learning to identify unexpected and potentially unauthorized and malicious activity within your AWS environment. When you enable GuardDuty in a Region, it immediately starts analyzing your CloudTrail event logs. It consumes CloudTrail management and Amazon S3 data events directly from CloudTrail through an independent and duplicative stream of events.

Amazon Security Lake can be used to automatically centralize security data from AWS environments, SaaS providers, on-premises, and cloud sources into a purpose-built data lake stored in your AWS account. With Security Lake, you can get a more complete understanding of your security data across your entire organization. Security Lake has adopted the Open Cybersecurity Schema Framework (OCSF), an open standard. With OCSF support, the service normalizes and combines security data from AWS and a broad range of enterprise security data sources.