View a markdown version of this page

GENREL04-BP02 Implement a model catalog - Generative AI Lens
Implementation guidanceResources

GENREL04-BP02 Implement a model catalog

Model catalogs store and manage model versions. They act as a reliable store for models which may need to be deployed or rolled back at any time. They also facilitate decoupled deployment automation.

Desired outcome: When implemented, this best practice improves the reliability of your generative AI workload by helping to make sure the deployed model is the appropriate model for the given use case.

Benefits of establishing this best practice: Manage change through automation - Implementing a model catalog helps to automate the process of deploying and rolling back model versions.

Level of risk exposed if this best practice is not established: Low

Implementation guidance

Model catalogs provide a centralized location to review models, model versions, and model cards. Traditionally, model catalogs are meant to store model artifacts developed by customers. Foundation models are rarely developed from scratch, and as a result, foundation model catalogs should maintain first-party models, third-party models, and custom models developed from third-party models.

Consider implementing a model catalog for foundation models that records and tracks model access, model versions, and model card information. Maintain a model catalog in your environment to track available models. Model catalogs should provide a central location for model management, particularly if there is a need to roll back to a particular model or model version.

AI policy documents should provide clear details regarding the usage, maintenance, and updating of the model catalog. The AI policy document is intended to be the central authority for operational questions pertaining to AI workloads and supporting infrastructure. Keep this document up to date with the appropriate materials necessary to scale the usage of the model catalog throughout the organization.

Implementation steps

  1. Set up catalog structure:

    • Create model classification system (by type, purpose, and provider)

    • Define model metadata schema

    • Establish versioning conventions

    • Design access control framework

  2. Configure model tracking:

    • Record model lineage and dependencies

    • Track model versions and updates

    • Document model customizations

    • Maintain performance benchmarks

  3. Implement model cards:

    • Define required model information

    • Document model capabilities and limitations

    • Record training data characteristics

    • Specify intended use cases and constraints

    • Include ethical considerations and biases

  4. Establish model governance:

    • Create model approval workflows

    • Define deployment procedures

    • Set up model monitoring

    • Implement security controls

    • Track model usage and access

  5. Create maintenance procedures:

    • Define model update process

    • Establish deprecation policies

    • Create archival procedures

    • Set up backup and recovery

  6. Implement validation framework:

    • Create model testing procedures

    • Define acceptance criteria

    • Set up performance benchmarking

    • Establish quality gates

Resources

Related best practices:

Related documents:

Related examples: