View a markdown version of this page

Operational Analytics Pipeline on AWS - Operational Analytics Pipeline on AWS
Operational Analytics Pipeline on AWSFurther readingDiagram history

Operational Analytics Pipeline on AWS

Publication date: 2021 (Diagram history)

With this architecture, you can perform operational analytics in batch and real time. You use log information from operational data sources such as AWS CloudTrail and Amazon VPC flow logs. You visualize operational insights in an OpenSearch dashboard and distribute alerts through Amazon Simple Notification Service (Amazon SNS).

Operational Analytics Pipeline on AWS

Architecture diagram for an operational analytics pipeline on AWS with Amazon CloudWatch, Amazon Data Firehose, Amazon OpenSearch Service, and AWS Lambda.

The following steps describe the architecture:

  1. Logs stream to Amazon CloudWatch from various data sources. Example sources include AWS CloudTrail and Amazon VPC flow logs.

  2. Amazon Managed Service for Apache Flink receives logs intended for real-time analysis.

  3. An Amazon S3 bucket receives logs intended for batch processing. This bucket serves as a backup for storing the raw logs.

  4. Amazon CloudWatch subscription filters stream log data to Amazon Data Firehose for distribution.

  5. Amazon Managed Service for Apache Flink derives metrics and sends them to a AWS Lambda function for processing. The Lambda function transforms the metrics and ingests them into Amazon OpenSearch Service. If a condition requires alerting, the Lambda function publishes a message to Amazon SNS for distribution.

  6. AWS Glue collects the logs from Amazon S3 and performs bulk processing, transformation, and analysis. AWS Glue uses the Amazon OpenSearch Service connector to ingest the results into Amazon OpenSearch Service.

  7. An OpenSearch dashboard visualizes the metrics from Amazon OpenSearch Service.

  8. Amazon SNS notifies consumers of alerts by publishing to email and SMS notification topics.

Further reading

For additional information, see the following resources:

Diagram history

To be notified about updates to this reference architecture diagram, subscribe to the RSS feed.

ChangeDescriptionDate

Initial publication

Reference architecture diagram first published.

January 1, 2021

RSS subscription

To subscribe to RSS updates, you must have an RSS plugin enabled for the browser that you are using.