RAIRC03-BP01 Measure safety harms and harmful outputs

Write clear and objective definitions of what counts as safe and unsafe content for your specific use case by creating measurable criteria and concrete examples of acceptable and unacceptable outputs. Include both direct harms like violence or toxicity and contextual problems like inappropriate tone for your audience, with specific thresholds and boundaries that evaluators can apply consistently. Objective definitions reduce subjective interpretation and assist evaluators apply consistent standards.

Identify the specific harm categories that your system could potentially produce, such as toxicity, violence, misinformation, or inappropriate content for your target users. Focus on harms that are realistic given your system's purpose and capabilities rather than trying to cover every possible risk. This targeted approach assists you to allocate evaluation resources effectively.

Choose quantitative metrics like automated toxicity classifiers or content filtering tools that can measure your identified harm categories at scale. Test popular tools like Detoxify or Perspective API on sample outputs to see how well they detect the types of harmful content your system might produce. Automated metrics give you consistent measurement across large datasets.

Plan qualitative evaluation methods like human red-teaming where experts try to get your system to produce harmful outputs through adversarial prompting. Have safety experts or domain specialists review sample outputs for harms that automated tools might miss. Human evaluation catches nuanced safety issues that automated systems may overlook.

Supplement your custom evaluation with open-source benchmarks like ToxiGen or AdvBench that test for common safety problems. Run these standard tests alongside your custom evaluation to compare your system's performance against known safety baselines. This provides additional validation and assists to identify blind spots in your custom evaluation approach.

Match your evaluation intensity to your system's risk level by using more thorough testing for higher-risk applications. For example, consider using basic automated screening for low-risk creative tools but adding human red-teaming for systems that might influence important decisions. Appropriate evaluation depth blocks both over-testing low-risk systems and under-testing higher-risk ones.