RAIBR02-BP07 Identify potential harmful events impacting explainability

Consider scenarios where your users might be confused or frustrated by your AI system's outputs, especially when those outputs could lead to significant decisions. For example, if your system recommends against a loan application or insurance coverage or flags content for removal, consider the information that users would want to contest or improve the result.

Identify situations where users could take corrective action if they understood your system's reasoning but might give up or make things worse without that understanding. This includes cases where users provided incorrect information, missed required fields, or could improve their outcomes by adjusting their inputs or approach.

Consider whether your organization has requirements around AI system outputs, and whether AI system outputs could fail to meet those requirements.

Consider how a lack of explanation might amplify other problems with your system by making it harder for users to provide feedback, for operators to troubleshoot issues, or for your team to identify when the system is making systematic errors.

Look for situations where misunderstanding your system's outputs could lead users to make harmful decisions themselves, such as ignoring important warnings, over-relying on uncertain recommendations, or losing trust in legitimate system outputs. Think about both immediate harms to individual users and broader impacts if many people misunderstand how your system works.