RAIDP04-BP05 Document the characteristics of each dataset using a datasheet
Create datasheets that document the intended uses, composition, and collection process for each dataset. Include information about data sources, collection methodologies, potential unwanted biases, and recommended and prohibited use cases to assist others to understand appropriate applications. Document the characteristics of data contributors and annotators, including demographic information and potential sources of unwanted bias that could affect system behavior. Maintain datasheets as living documents that are updated when datasets change or when new insights about their characteristics or limitations are discovered.
Level of risk exposed if this best practice is not established: High
Implementation considerations
-
If needed, create standardized datasheet templates that capture essential information about each dataset. Your template should cover basic information such as intended uses, inappropriate uses, data sources, data labels, collection methods, volumes, formats, as well as more nuanced aspects like known limitations and potential biases.
-
Complete the template. As appropriate, capture distributions of sources by label types, and note unexpected distribution skews, gaps in representation, and missing data. Characterize the types of human or machine annotators (for example, experience, training, and potential sources of bias). This assists others understand who's represented in your data and what perspectives shaped the labels or annotations.
-
Set up processes to keep your datasheets current as you learn more about your datasets or make changes to them. Schedule regular reviews to update datasheets when you discover new limitations, modify the data, or find better ways to describe the dataset's characteristics and appropriate uses.
Resources
Related documents
-
ISO/IEC 42001:2023
A.7.5 Data provenance
