RAIRC03-BP02 Measure fairness as unwanted bias across stakeholder groups

Measure individual fairness by testing whether similar individuals get similar treatment regardless of their demographic characteristics.

Measure group fairness by comparing your system's performance across the demographic groups you identified in your risk assessment (RAIBR02) using metrics like accuracy, precision, and recall. Calculate performance differences between groups and compare them to your acceptable thresholds to identify potential biases. Group-level measurement reveals systemic unwanted bias that may have larger impacts (like bias across entire groups).

Test for representational fairness by analyzing whether your system's outputs reinforce harmful stereotypes or misrepresent different groups. Use existing tools like stereotype detection classifiers or analyze generated content for biased language patterns. This catches subtle bias that may not show up in performance metrics but still causes harm.

Consider testing your system on pairs of similar inputs that differ only in demographic attributes to see if outputs change inappropriately. This reveals potential bias where demographic factors inappropriately influence decisions.

Consider testing your system on intersectional groups that combine multiple demographic characteristics, using the same metrics you applied to single-group analysis. Compare results across these intersectional groups to identify potential bias that might be hidden when looking at single demographics alone.

Consider experimenting with complementary fairness metrics like demographic parity, equal opportunity, and equalized odds to get multiple perspectives on your system's fairness. For example, measure whether different groups receive similar positive prediction rates and whether the system correctly identifies positive cases at similar rates across groups. Multiple metrics reveal different types of bias since systems can appear fair on one measure but not on another.

Identify which fairness metrics conflict with each other for your system and make explicit decisions about which to prioritize based on your use case context and stakeholder values established in your risk characterization (RAIBR02). Record your reasoning for these trade-offs since optimizing for one type of fairness often reduces performance on others. Clear prioritization assists you to make consistent decisions when fairness measures conflict.