RAIDP04-BP01 Create a dataset registry
Create a registry to track dataset versions, metadata, and usage across training, evaluation, and operational contexts. Store datasets with version control, including local copies of public benchmarks to assist builders with reproducibility as external datasets evolve. Document the provenance, characteristics, and intended use of each dataset version to enable others to understand appropriate usage and limitations. Link dataset versions to specific system training events and evaluation results to maintain traceability between data changes and performance outcomes.
Level of risk exposed if this best practice is not established: High
Implementation considerations
-
Build a centralized registry system that captures essential metadata for each dataset including version numbers, creation dates, source information, and intended use cases. Start with a simple database or structured file system that can track when datasets were created, who created them, and what they're designed to test.
-
Create version control workflows that automatically snapshot datasets whenever changes are made like a version-controlled code repository. Test your versioning system by making small changes to a dataset and verifying you can retrieve both the current and previous versions reliably.
-
Set up local storage for copies of external benchmarks and public datasets you use, rather than pulling from external sources. Test this by comparing results from your local copy against the original source to catch differences that could affect reproducibility.
-
Build linking mechanisms that connect specific dataset versions to the training runs and evaluations that used them. Test this traceability by picking a model performance result and verifying you can trace back to the exact dataset version that produced it.
Resources
Related documents:
Related tools:
-
Accelerate generative AI development with Amazon SageMaker AI AI and MLflow
-
Amazon SageMaker AI Unified Studio
