RAIRC02-BP03 Design a custom metric if no suitable metric exists

Clearly define what you're trying to measure and write down its key characteristics, focusing on how it directly impacts your stakeholders. For example, if you need to measure how natural a conversation is, define what makes a conversation feel natural or robotic to your specific users. This foundation assists to build an accurate metric.

Break complex concepts down into smaller pieces that you can count or score. For example, split user satisfaction into task completion rate, time to complete, and user survey scores, as you can measure each of these objectively. This makes abstract concepts concrete and measurable.

Identify what good and bad results look like, including edge cases that might confuse your metric. Define that a helpful response should be accurate, relevant, and actionable. Clear examples reduce confusion during measurement.

Test your custom metric on examples where you already know what the right answer should be. Run your metric on obviously good and obviously bad examples to see if it gives the results you expect. This catches major problems with your metric design before you use it on real data.

Choose the types of scores your measurement needs. Continuous scores give you more nuanced information and let you track gradual improvements, while categorical ratings are simpler for humans and LLM judge models to assign consistently and binary scores simplify the metric but can hide performance nuance.

Document exactly how to calculate your metric, including step-by-step instructions that someone else could follow to get the same results. This blocks inconsistency when different team members apply your metric and assists you to spot problems in your methodology

Plan to refine your metric based on real testing since custom metrics often need adjustment after you see how they perform. Start with small tests and be ready to modify the metric if it doesn't work well in practice or gives misleading results on new types of data.