This whitepaper is for historical reference only. Some content might be outdated and some links might not be available.
Monitoring
At a high-level, there are two types of monitoring within your data lake:
-
Monitoring your resources
-
Monitoring your data quality in those resources
There are a couple of options in this space:
-
Monitoring your AWS resources — Amazon CloudWatch collects and tracks metrics, monitors logs, sets thresholds, and triggers alarms per your environment. CloudWatch can monitor AWS resources such as Amazon EC2 instances, Amazon S3, Amazon EMR, Amazon Redshift, Amazon DynamoDB, and Amazon Relational Database Service (RDS) database instances, as well as custom metrics generated by other data lake applications and services. CloudWatch provides system-wide visibility into resource utilization, application performance, and operational health. You can use these insights to proactively react to issues and keep your data lake applications and workflows running smoothly.
For more information, refer to Monitoring and optimizing the data lake environment and the Data Lake on AWS
implementation guide. AWS Partners such as Datadog
, New Relic, and Splunk offer similar capabilities as Amazon CloudWatch, and are available as options to our customers. -
Monitoring your data quality — Everyday customers accumulate petabytes of data on AWS. With this constant flow of data, it is a struggle for these customers to maintain the quality of data they manage. Fortunately, AWS has some options in this space. For example, on the data layer, you can use Deequ with AWS Glue, AWS Amplify, and DynamoDB to create a data quality and analysis framework
. On the ML side, you can use SageMaker AI to automatically monitor ML models in a production environment, and get notifications when data quality issues arise.
