# SEC 4. How do you detect and investigate security events?
Capture and analyze events from logs and metrics to gain visibility. Take action on security events and potential threats to help secure your workload.
**Topics**
+ [SEC04-BP01 Configure service and application logging](sec_detect_investigate_events_app_service_logging.md)
+ [SEC04-BP02 Analyze logs, findings, and metrics centrally](sec_detect_investigate_events_analyze_all.md)
+ [SEC04-BP03 Automate response to events](sec_detect_investigate_events_auto_response.md)
+ [SEC04-BP04 Implement actionable security events](sec_detect_investigate_events_actionable_events.md)
# SEC04-BP01 Configure service and application logging
Retain security event logs from services and applications. This is a fundamental principle of security for audit, investigations, and operational use cases, and a common security requirement driven by governance, risk, and compliance (GRC) standards, policies, and procedures.
**Desired outcome:** An organization should be able to reliably and consistently retrieve security event logs from AWS services and applications in a timely manner when required to fulfill an internal process or obligation, such as a security incident response. Consider centralizing logs for better operational results.
**Common anti-patterns:**
+ Logs are stored in perpetuity or deleted too soon.
+ Everybody can access logs.
+ Relying entirely on manual processes for log governance and use.
+ Storing every single type of log just in case it is needed.
+ Checking log integrity only when necessary.
**Benefits of establishing this best practice:** Implement a root cause analysis (RCA) mechanism for security incidents and a source of evidence for your governance, risk, and compliance obligations.
**Level of risk exposed if this best practice is not established:** High
## Implementation guidance
During a security investigation or other use cases based on your requirements, you need to be able to review relevant logs to record and understand the full scope and timeline of the incident. Logs are also required for alert generation, indicating that certain actions of interest have happened. It is critical to select, turn on, store, and set up querying and retrieval mechanisms and alerting.
**Implementation steps**
+ **Select and use log sources.** Ahead of a security investigation, you need to capture relevant logs to retroactively reconstruct activity in an AWS account. Select log sources relevant to your workloads.
The log source selection criteria should be based on the use cases required by your business. Establish a trail for each AWS account using AWS CloudTrail or an AWS Organizations trail, and configure an Amazon S3 bucket for it.
AWS CloudTrail is a logging service that tracks API calls made against an AWS account capturing AWS service activity. It’s turned on by default with a 90-day retention of management events that can be [retrieved through CloudTrail Event history](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html) using the AWS Management Console, the AWS CLI, or an AWS SDK. For longer retention and visibility of data events, [create a CloudTrail trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) and associate it with an Amazon S3 bucket, and optionally with a Amazon CloudWatch log group. Alternatively, you can create a [CloudTrail Lake](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-lake.html), which retains CloudTrail logs for up to seven years and provides a SQL-based querying facility
AWS recommends that customers using a VPC turn on network traffic and DNS logs using [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) and [Amazon Route 53 resolver query logs](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-query-logs.html), respectively, and streaming them to either an Amazon S3 bucket or a CloudWatch log group. You can create a VPC flow log for a VPC, a subnet, or a network interface. For VPC Flow Logs, you can be selective on how and where you use Flow Logs to reduce cost.
AWS CloudTrail Logs, VPC Flow Logs, and Route 53 resolver query logs are the basic logging sources to support security investigations in AWS. You can also use [Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) to collect, normalize, and store this log data in Apache Parquet format and Open Cybersecurity Schema Framework (OCSF), which is ready for querying. Security Lake also supports other AWS logs and logs from third-party sources.
AWS services can generate logs not captured by the basic log sources, such as Elastic Load Balancing logs, AWS WAF logs, AWS Config recorder logs, Amazon GuardDuty findings, Amazon Elastic Kubernetes Service (Amazon EKS) audit logs, and Amazon EC2 instance operating system and application logs. For a full list of logging and monitoring options, see [Appendix A: Cloud capability definitions – Logging and Events](https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/logging-and-events.html) of the [AWS Security Incident Response Guide](https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/detection.html).
+ **Research logging capabilities for each AWS service and application:** Each AWS service and application provides you with options for log storage, each of which with its own retention and life-cycle capabilities. The two most common log storage services are Amazon Simple Storage Service (Amazon S3) and Amazon CloudWatch. For long retention periods, it is recommended to use Amazon S3 for its cost effectiveness and flexible lifecycle capabilities. If the primary logging option is Amazon CloudWatch Logs, as an option, you should consider archiving less frequently accessed logs to Amazon S3.
+ **Select log storage:** The choice of log storage is generally related to which querying tool you use, retention capabilities, familiarity, and cost. The main options for log storage are an Amazon S3 bucket or a CloudWatch Log group.
An Amazon S3 bucket provides cost-effective, durable storage with an optional lifecycle policy. Logs stored in Amazon S3 buckets can be queried using services such as Amazon Athena.
A CloudWatch log group provides durable storage and a built-in query facility through CloudWatch Logs Insights.
+ **Identify appropriate log retention:** When you use an Amazon S3 bucket or CloudWatch log group to store logs, you must establish adequate lifecycles for each log source to optimize storage and retrieval costs. Customers generally have between three months to one year of logs readily available for querying, with retention of up to seven years. The choice of availability and retention should align with your security requirements and a composite of statutory, regulatory, and business mandates.
+ **Use logging for each AWS service and application with proper retention and lifecycle policies:** For each AWS service or application in your organization, look for the specific logging configuration guidance:
+ [ Configure AWS CloudTrail Trail ](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html)
+ [ Configure VPC Flow Logs ](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html)
+ [ Configure Amazon GuardDuty Finding Export ](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_exportfindings.html)
+ [ Configure AWS Config recording ](https://docs.aws.amazon.com/systems-manager/latest/userguide/quick-setup-config.html)
+ [ Configure AWS WAF web ACL traffic ](https://docs.aws.amazon.com/waf/latest/developerguide/logging.html)
+ [ Configure AWS Network Firewall network traffic logs ](https://docs.aws.amazon.com/network-firewall/latest/developerguide/firewall-logging.html)
+ [ Configure Elastic Load Balancing access logs ](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html)
+ [ Configure Amazon Route 53 resolver query logs ](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-query-logs.html)
+ [ Configure Amazon RDS logs ](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_LogAccess.html)
+ [ Configure Amazon EKS Control Plane logs ](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html)
+ [ Configure Amazon CloudWatch agent for Amazon EC2 instances and on-premises servers ](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html)
+ **Select and implement querying mechanisms for logs:** For log queries, you can use [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html) for data stored in CloudWatch log groups, and [Amazon Athena](https://aws.amazon.com/athena/) and [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/) for data stored in Amazon S3. You can also use third-party querying tools such as a security information and event management (SIEM) service.
The process for selecting a log querying tool should consider the people, process, and technology aspects of your security operations. Select a tool that fulfills operational, business, and security requirements, and is both accessible and maintainable in the long term. Keep in mind that log querying tools work optimally when the number of logs to be scanned is kept within the tool’s limits. It is not uncommon to have multiple querying tools because of cost or technical constraints.
For example, you might use a third-party security information and event management (SIEM) tool to perform queries for the last 90 days of data, but use Athena to perform queries beyond 90 days because of the log ingestion cost of a SIEM. Regardless of the implementation, verify that your approach minimizes the number of tools required to maximize operational efficiency, especially during a security event investigation.
+ **Use logs for alerting:** AWS provides alerting through several security services:
+ [AWS Config](https://aws.amazon.com/config/) monitors and records your AWS resource configurations and allows you to automate the evaluation and remediation against desired configurations.
+ [Amazon GuardDuty](https://aws.amazon.com/guardduty/) is a threat detection service that continually monitors for malicious activity and unauthorized behavior to protect your AWS accounts and workloads. GuardDuty ingests, aggregates, and analyzes information from sources, such as AWS CloudTrail management and data events, DNS logs, VPC Flow Logs, and Amazon EKS Audit logs. GuardDuty pulls independent data streams directly from CloudTrail, VPC Flow Logs, DNS query logs, and Amazon EKS. You don’t have to manage Amazon S3 bucket policies or modify the way you collect and store logs. It is still recommended to retain these logs for your own investigation and compliance purposes.
+ [AWS Security Hub CSPM](https://aws.amazon.com/security-hub/) provides a single place that aggregates, organizes, and prioritizes your security alerts or findings from multiple AWS services and optional third-party products to give you a comprehensive view of security alerts and compliance status.
You can also use custom alert generation engines for security alerts not covered by these services or for specific alerts relevant to your environment. For information on building these alerts and detections, see [Detection in the AWS Security Incident Response Guide](https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/detection.html).
## Resources
**Related best practices:**
+ [SEC04-BP02 Analyze logs, findings, and metrics centrally](sec_detect_investigate_events_analyze_all.md)
+ [SEC07-BP04 Define data lifecycle management](sec_data_classification_lifecycle_management.md)
+ [SEC10-BP06 Pre-deploy tools](sec_incident_response_pre_deploy_tools.md)
**Related documents:**
+ [AWS Security Incident Response Guide ](https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-response-guide/aws-security-incident-response-guide.html)
+ [ Getting Started with Amazon Security Lake ](https://aws.amazon.com/security-lake/getting-started/)
+ [ Getting started: Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_GettingStarted.html)
+ [Security Partner Solutions: Logging and Monitoring](https://aws.amazon.com/security/partner-solutions/#logging-monitoring)
**Related videos:**
+ [AWS re:Invent 2022 - Introducing Amazon Security Lake ](https://www.youtube.com/watch?v=V7XwbPPjXSY)
**Related examples:**
+ [ Assisted Log Enabler for AWS](https://github.com/awslabs/assisted-log-enabler-for-aws/)
+ [AWS Security Hub CSPM Findings Historical Export ](https://github.com/aws-samples/aws-security-hub-findings-historical-export)
**Related tools:**
+ [ Snowflake for Cybersecurity ](https://www.snowflake.com/en/data-cloud/workloads/cybersecurity/)
# SEC04-BP02 Analyze logs, findings, and metrics centrally
Security operations teams rely on the collection of logs and the use of search tools to discover potential events of interest, which might indicate unauthorized activity or unintentional change. However, simply analyzing collected data and manually processing information is insufficient to keep up with the volume of information flowing from complex architectures. Analysis and reporting alone don’t facilitate the assignment of the right resources to work an event in a timely fashion.
A best practice for building a mature security operations team is to deeply integrate the flow of security events and findings into a notification and workflow system such as a ticketing system, a bug or issue system, or other security information and event management (SIEM) system. This takes the workflow out of email and static reports, and allows you to route, escalate, and manage events or findings. Many organizations are also integrating security alerts into their chat or collaboration, and developer productivity platforms. For organizations embarking on automation, an API-driven, low-latency ticketing system offers considerable flexibility when planning what to automate first.
This best practice applies not only to security events generated from log messages depicting user activity or network events, but also from changes detected in the infrastructure itself. The ability to detect change, determine whether a change was appropriate, and then route that information to the correct remediation workflow is essential in maintaining and validating a secure architecture, in the context of changes where the nature of their undesirability is sufficiently subtle that they cannot currently be prevented with a combination of AWS Identity and Access Management (IAM) and AWS Organizations configuration.
Amazon GuardDuty and AWS Security Hub CSPM provide aggregation, deduplication, and analysis mechanisms for log records that are also made available to you via other AWS services. GuardDuty ingests, aggregates, and analyzes information from sources such as AWS CloudTrail management and data events, VPC DNS logs, and VPC Flow Logs. Security Hub CSPM can ingest, aggregate, and analyze output from GuardDuty, AWS Config, Amazon Inspector, Amazon Macie, AWS Firewall Manager, and a significant number of third-party security products available in the AWS Marketplace, and if built accordingly, your own code. Both GuardDuty and Security Hub CSPM have an Administrator-Member model that can aggregate findings and insights across multiple accounts, and Security Hub CSPM is often used by customers who have an on- premises SIEM as an AWS-side log and alert preprocessor and aggregator from which they can then ingest Amazon EventBridge through a AWS Lambda-based processor and forwarder.
**Level of risk exposed if this best practice is not established:** High
## Implementation guidance
+ Evaluate log processing capabilities: Evaluate the options that are available for processing logs.
+ [Find an AWS Partner that specializes in logging and monitoring solutions ](https://aws.amazon.com/security/partner-solutions/#Logging_and_Monitoring)
+ As a start for analyzing CloudTrail logs, test Amazon Athena.
+ [ Configuring Athena to analyze CloudTrail logs ](https://docs.aws.amazon.com/athena/latest/ug/cloudtrail-logs.html)
+ Implement centralize logging in AWS: See the following AWS example solution to centralize logging from multiple sources.
+ [Centralize logging solution ](https://aws.amazon.com/solutions/centralized-logging/)
+ Implement centralize logging with partner: APN Partners have solutions to help you analyze logs centrally.
+ [ Logging and Monitoring ](https://aws.amazon.com/security/partner-solutions/#Logging_and_Monitoring)
## Resources
**Related documents:**
+ [AWS Answers: Centralized Logging ](https://aws.amazon.com/answers/logging/centralized-logging/)
+ [AWS Security Hub CSPM](https://docs.aws.amazon.com/securityhub/latest/userguide/what-is-securityhub.html)
+ [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/)
+ [Amazon EventBridge ](https://aws.amazon.com/eventbridge)
+ [ Getting started: Amazon CloudWatch Logs ](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_GettingStarted.html)
+ [Security Partner Solutions: Logging and Monitoring](https://aws.amazon.com/security/partner-solutions/#logging-monitoring)
**Related videos:**
+ [ Centrally Monitoring Resource Configuration and Compliance ](https://youtu.be/kErRv4YB_T4)
+ [Remediating Amazon GuardDuty and AWS Security Hub CSPM Findings ](https://youtu.be/nyh4imv8zuk)
+ [ Threat management in the cloud: Amazon GuardDuty and AWS Security Hub CSPM](https://youtu.be/vhYsm5gq9jE)
# SEC04-BP03 Automate response to events
Using automation to investigate and remediate events reduces human effort and error, and allows you to scale investigation capabilities. Regular reviews will help you tune automation tools, and continuously iterate.
In AWS, investigating events of interest and information on potentially unexpected changes into an automated workflow can be achieved using Amazon EventBridge. This service provides a scalable rules engine designed to broker both native AWS event formats (such as AWS CloudTrail events), as well as custom events you can generate from your application. Amazon GuardDuty also allows you to route events to a workflow system for those building incident response systems (AWS Step Functions), or to a central Security Account, or to a bucket for further analysis.
Detecting change and routing this information to the correct workflow can also be accomplished using AWS Config Rules and [Conformance Packs](https://docs.aws.amazon.com/config/latest/developerguide/conformance-packs.html). AWS Config detects changes to in-scope services (though with higher latency than EventBridge) and generates events that can be parsed using AWS Config Rules for rollback, enforcement of compliance policy, and forwarding of information to systems, such as change management platforms and operational ticketing systems. As well as writing your own Lambda functions to respond to AWS Config events, you can also take advantage of the [AWS Config Rules Development Kit](https://github.com/awslabs/aws-config-rdk), and a [library of open source](https://github.com/awslabs/aws-config-rules) AWS Config Rules. Conformance packs are a collection of AWS Config Rules and remediation actions you deploy as a single entity authored as a YAML template. A [sample conformance pack template](https://docs.aws.amazon.com/config/latest/developerguide/operational-best-practices-for-wa-Security-Pillar.html) is available for the Well-Architected Security Pillar.
**Level of risk exposed if this best practice is not established:** Medium
## Implementation guidance
+ Implement automated alerting with GuardDuty: GuardDuty is a threat detection service that continuously monitors for malicious activity and unauthorized behavior to protect your AWS accounts and workloads. Turn on GuardDuty and configure automated alerts.
+ Automate investigation processes: Develop automated processes that investigate an event and report information to an administrator to save time.
+ [ Lab: Amazon GuardDuty hands on ](https://hands-on-guardduty.awssecworkshops.com/)
## Resources
**Related documents:**
+ [AWS Answers: Centralized Logging ](https://aws.amazon.com/answers/logging/centralized-logging/)
+ [AWS Security Hub CSPM](https://docs.aws.amazon.com/securityhub/latest/userguide/what-is-securityhub.html)
+ [ Amazon CloudWatch ](https://aws.amazon.com/cloudwatch/)
+ [Amazon EventBridge ](https://aws.amazon.com/eventbridge)
+ [ Getting started: Amazon CloudWatch Logs ](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_GettingStarted.html)
+ [Security Partner Solutions: Logging and Monitoring](https://aws.amazon.com/security/partner-solutions/#logging-monitoring)
+ [ Setting up Amazon GuardDuty ](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_settingup.html)
**Related videos:**
+ [ Centrally Monitoring Resource Configuration and Compliance ](https://youtu.be/kErRv4YB_T4)
+ [Remediating Amazon GuardDuty and AWS Security Hub CSPM Findings ](https://youtu.be/nyh4imv8zuk)
+ [ Threat management in the cloud: Amazon GuardDuty and AWS Security Hub CSPM](https://youtu.be/vhYsm5gq9jE)
**Related examples:**
+ [Lab: Automated Deployment of Detective Controls ](https://wellarchitectedlabs.com/Security/200_Automated_Deployment_of_Detective_Controls/README.html)
# SEC04-BP04 Implement actionable security events
Create alerts that are sent to and can be actioned by your team. Ensure that alerts include relevant information for the team to take action. For each detective mechanism you have, you should also have a process, in the form of a [runbook](https://wa.aws.amazon.com/wat.concept.runbook.en.html) or [playbook](https://wa.aws.amazon.com/wat.concept.playbook.en.html), to investigate. For example, when you use [Amazon GuardDuty](http://aws.amazon.com/guardduty), it generates different [findings](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_findings.html). You should have a runbook entry for each finding type, for example, if a [trojan](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_trojan.html) is discovered, your runbook has simple instructions that instruct someone to investigate and remediate.
**Level of risk exposed if this best practice is not established:** Low
## Implementation guidance
+ Discover metrics available for AWS services: Discover the metrics that are available through Amazon CloudWatch for the services that you are using.
+ [AWS service documentation](https://aws.amazon.com/documentation/)
+ [Using Amazon CloudWatch Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/working_with_metrics.html)
+ Configure Amazon CloudWatch alarms.
+ [Using Amazon CloudWatch Alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html)
## Resources
**Related documents:**
+ [ Amazon CloudWatch ](https://aws.amazon.com/cloudwatch/)
+ [Amazon EventBridge ](https://aws.amazon.com/eventbridge)
+ [Security Partner Solutions: Logging and Monitoring](https://aws.amazon.com/security/partner-solutions/#logging-monitoring)
**Related videos:**
+ [ Centrally Monitoring Resource Configuration and Compliance ](https://youtu.be/kErRv4YB_T4)
+ [Remediating Amazon GuardDuty and AWS Security Hub CSPM Findings ](https://youtu.be/nyh4imv8zuk)
+ [ Threat management in the cloud: Amazon GuardDuty and AWS Security Hub CSPM](https://youtu.be/vhYsm5gq9jE)