Updating model versions

Post-deployment of our models, it is important to track business metrics that we are aiming to optimize. The business metrics are usually tied to revenue generation, sales, number of user clicks, purchases, or as in cases found in this whitepaper, on the efficacy of the workforce productivity recommendations or sales forecast shown to C-suite users. In this solution, AWS recommends storing the business metrics in Amazon DynamoDB.

When new versions of the models are available, it is important to see which variant performs better before routing all traffic to the new model. Models tested during the engineering process and the same models running “in the wild” may produce different results, and you don’t want to switch your use base to the new models without first testing them sufficiently with live, unseen, real traffic. Use Canary testing, A/B testing, and Multi-Armed Bandit (MAB) strategies in a champion-challenger framework, to choose the best new model among the various SageMaker AI hosted production variants.

Deploying production variants and continuous MLOps in production