RAIRC03-BP03 Measure veracity of outputs

Identify metrics for potential hallucination, omission, and misemphasis harms that you identified in your risk assessment (RAIBR02).

Plan expert human evaluations where domain specialists review sample outputs for factual accuracy and appropriateness within their area of expertise. Have subject matter experts evaluate claims in their field to catch subtle inaccuracies that automated tools might miss. Human experts can assess context, nuance, and domain-specific accuracy that automated systems often overlook.

Measure groundedness, i.e. the degree to which your system's outputs can be traced back to reliable source material when sources are available. Check if claims in generated content align with the source documents and whether citations are accurate and relevant. Groundedness testing blocks your system from making claims that aren't supported by its reference materials.

Measure consistency by asking your system the same questions multiple times and across different phrasings to see if answers remain factually consistent. Also test related questions to see if responses contradict each other across different interactions. Consistency testing reveals when your system generates conflicting information about the same topics.