RAIRC02-BP01 Select metrics to measure the properties tested by the release criteria

Take each yes or no release criterion and identify what specific measurements you need to answer that question. For example, if your criterion is "Does the system respond to queries quickly?", you need response time metrics, or if it's "Does the system block toxic content?", you need toxicity detection scores. Break down abstract criteria into concrete, measurable criteria.

Look for existing automated metrics that can measure what you need, such as accuracy scores, response time tracking, or toxicity detection tools. Check open source options like scikit-learn or Hugging Face libraries as well as paid services such as Amazon Bedrock Evaluations. Automated metrics save time and provide consistent measurements you can run repeatedly.

Consider using LLM-as-a-judge for criteria that require understanding context, quality, or appropriateness. For example, you can prompt an LLM to evaluate whether responses are helpful, coherent, or follow specific guidelines by giving it examples and scoring rubrics. LLM judges work well for subjective assessments that are too complex for simple automated metrics and are more scalable than human review.

Identify which criteria need human evaluation because neither automated metrics nor LLM judges can capture what you're trying to measure. For example, measuring whether user interface designs are intuitive may require actual users to test the interface to better capture the real user experience and preferences. Human evaluation catches the most nuanced issues and is more representative of your user experience but is slower and more expensive.

If you find yourself needing multiple different metrics to test one criterion because the criterion itself is complex, consider splitting the criterion into separate yes or no questions. For example, change "Does the system provide a good user experience?" into "Does the system respond quickly?", "Does the system give accurate results?", and "Does the system have an intuitive interface?" This makes each criterion simple to measure definitively.

Track which metric you'll use for each release criterion. This gives you a clear testing plan and creates a mapping from your measurements to your release criteria.