Tracing Data Lake Ingest and Processing
Publication date: July 15, 2021 (Diagram history)
This architecture shows how to trace your data lake ingestion and processing using AWS X-Ray. You can emit traces from key checkpoints in the flow to diagnose data lake processing lifecycle issues and identify performance bottlenecks.
Tracing Data Lake Ingest and Processing Using AWS X-Ray
The following steps describe the architecture:
-
Diverse producers deliver data into the data lake built on Amazon Simple Storage Service.
-
The data first lands in an Amazon S3 bucket for raw data. This is the first trace since AWS X-Ray enables trace messages for Amazon S3 Event Notifications.
-
Data lake curation blocks prepare the data for analytics. AWS Step Functions triggers these processing blocks. AWS X-Ray integrates with Step Functions to help you identify performance bottlenecks and troubleshoot requests that resulted in an error.
-
AWS X-Ray can trace AWS Lambda functions. Lambda runs the X-Ray daemon and records a segment with details about the function invocation and results. For further instrumentation, bundle the X-Ray SDK with the function to record outgoing calls and add annotations and metadata.
-
Data curation continues with the Step Functions state machine triggering the next curation block. Like the first block, X-Ray can capture Step Functions and Lambda functions.
-
The data lands in an aggregated bucket ready for analytics. This is the last trace in the data lake ingestion flow. With traces emitting from all key checkpoints, you can diagnose data lake processing lifecycle issues.
-
Diverse analytics tools such as Amazon Quick Sight, Amazon Redshift, and Amazon Athena enable you to visualize the data from the data lake built on Amazon S3.
Further reading
For additional information, refer to the following resources:
Diagram history
To be notified about updates to this reference architecture diagram, subscribe to the RSS feed.
| Change | Description | Date |
|---|---|---|
Initial publication | Reference architecture diagram first published. | July 15, 2021 |
Reference architecture diagram first published. | July 15, 2021 |
Note
To subscribe to RSS updates, you must have an RSS plugin enabled for the browser you are using.
