RAIRC03-BP04 Measure robustness of outputs to input variation

Build controlled robustness tests that vary one input factor at a time while keeping the content meaning the same, using the input variations your RAIBR02 risk assessment found most likely in your deployment environment. Create paired test cases where you change only one thing, such as converting formal business language to casual speech or adjusting image lighting conditions. Controlled variation testing shows you which specific input factors cause performance drops and by how much.

Apply the same metrics you selected in RAIRC02-BP01 to measure performance across different input variations, comparing how your system performs on standard inputs versus challenging variations. Use controlled comparisons where you test the same content with only one input characteristic changed at a time, such as measuring accuracy on both formal and casual versions of the same question. This approach reveals which specific input factors cause performance drops and by how much.

Calculate performance variance and degradation across known input variations to quantify how much your system's reliability fluctuates under different conditions. Identify the worst-case performance drops across input types.

Test combinations of multiple input variations together, such as processing accented speech with background noise or analyzing low-quality images with poor lighting, since real users often provide challenging inputs with several issues simultaneously. Focus on combinations most likely to occur in your deployment environment based on your use case analysis. Combined variation testing catches failure modes that only emerge when multiple challenging factors interact.