RAISP03-BP03 Implement output filtering to catch unsafe content before it reaches users
Build screening mechanisms that automatically review and filter your system's outputs to catch potentially harmful content before users see it, acting as a final safety check regardless of what your system generates. This means implementing safety classifiers that can identify toxic, harmful, or inappropriate content in real-time, content filtering systems that block or modify unsafe outputs based on your safety definitions, and automated screening processes that evaluate each response against your safety criteria before delivery.
For example, if your release criteria include safety standards for blocking harmful outputs, you might implement toxicity detection models that score each response, build content filters that automatically block responses containing harmful information, or create screening systems that flag suspicious outputs for human review before they reach users.
Level of risk exposed if this best practice is not established: High
Implementation considerations
-
Based upon release criteria, identify types of potentially harmful content, including adversarial inputs and categories outside of your system's use case
-
Map the types of potentially harmful content to available content filters, such as those already built-in to safeguard services. Additional safety concerns may require custom safeguards, either through system prompting, fine tuning, exact match word configuration, or non-AI solutions that control critical access and permissions to resources.
-
Configure or customize safety filters that can identify and block potentially harmful content across multiple categories such as violence, self-harm, illegal activities, and other context-specific safety concerns.
Resources
Related documents:
Related video:
Related tools:
