The Most Common AI Validation Failure
Why post-hoc acceptance criteria appear in most FDA AI warning letters.
The most common finding in every FDA AI warning letter to date: acceptance criteria defined after testing was complete. This practice is the AI equivalent of changing your test protocol after seeing the results — and the FDA considers it a fundamental validation failure.
Why This Happens
AI model development is inherently iterative. Teams build a model, evaluate performance, and adjust. The natural instinct is to set the acceptance threshold at whatever the model achieved. The GAMP AI Guide requires acceptance criteria to be defined before testing — based on the clinical or process requirement, not the model output.
How to Get It Right
Start with the user requirement: what does this AI system need to achieve? For a visual inspection AI: sensitivity ≥ 99.5%, specificity ≥ 98%. These numbers come from the quality requirement — write them into the URS and validation protocol before model development begins.