View a markdown version of this page

Build Your Own Anomaly Detection ML Pipeline - Build Your Own Anomaly Detection ML Pipeline
Build Your Own Anomaly Detection ML PipelineFurther readingDiagram history

Build Your Own Anomaly Detection ML Pipeline

Publication date: June 1, 2021 (Diagram history)

This architecture shows how to build an end-to-end ML pipeline that detects anomalies. You can ingest real-time streaming data, perform transformations, and continuously retrain ML models.

Build Your Own Anomaly Detection ML Pipeline

Architecture diagram showing an anomaly detection ML pipeline with SageMaker AI, Kinesis, and AWS CodePipeline.

The following steps describe the architecture:

  1. Device telemetry data is ingested from field devices on a near real-time basis through calls to the API through Amazon API Gateway. Amazon Cognito authenticates and authorizes the requests.

  2. Amazon Kinesis Data Firehose ingests the data in real time and invokes AWS Lambda to transform the data into Parquet format. Kinesis Data Firehose automatically scales to match the data throughput.

  3. AWS Glue jobs aggregate the telemetry data on an hourly basis. The jobs re-partition data based on year, month, date, and hour. Additional steps such as transformations and feature engineering prepare the data for training. The training dataset is stored in the Amazon Simple Storage Service data lake.

  4. Training code checked into AWS CodeCommit triggers an MLOps pipeline by using AWS CodePipeline. CodePipeline builds the Amazon SageMaker AI training and inference containers. It triggers the SageMaker AI training job, deploys the trained model in the testing environment, and upon approval, deploys the model into production.

  5. The ML models generated by training jobs are registered in the SageMaker AI Model Repository. The deploy pipeline selects the best ML model to deploy by using SageMaker AI hosting.

  6. You can classify telemetry data as anomalous or not through an HTTPS API by using API Gateway and Lambda functions. The Lambda function invokes the SageMaker AI endpoint to predict the anomaly.

  7. SageMaker AI Model Monitor monitors the inference quality. Requests with ambiguous prediction scores are sent for re-labeling by using Amazon CloudWatch events. These events trigger the SageMaker AI A2I workflow through AWS Step Functions.

Further reading

For additional information, refer to the following resources:

Diagram history

To be notified about updates to this reference architecture diagram, subscribe to the RSS feed.

ChangeDescriptionDate

Initial publication

Reference architecture diagram first published.

June 1, 2021

Note

To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.