Impact Assessment: Patients and Borrowers Are Not Users

Product thinking asks what a user wants. Regulated thinking asks who is affected, in what way, and what happens to them when the model is wrong.

Product language is a liability in a regulated context. Calling someone a 'user' frames them as a party who can simply choose to leave. A patient cannot choose a different hospital halfway through treatment, and a borrower cannot un-apply for the mortgage that was declined automatically at 3am.

What an impact assessment asks Who is affected by this system's output, what changes for them, whether they can contest the outcome, and what failure looks like at the level of an individual rather than a segment. The unit of analysis is a person, not a cohort.

Why aggregate metrics are not enough A model can improve on every headline measure and still produce worse outcomes for a narrow group. In lending that is a fairness problem with a legal dimension. In clinical triage it is a safety problem with a reporting dimension. Neither is visible in an overall accuracy figure, which is exactly why the assessment cannot be delegated to a metric.

The practice both sectors need A standing rule: no system that affects an individual decision goes into production without a written assessment, a named owner and a review date. The template can be shortened for lower-risk systems. It cannot be skipped for high-risk ones, and the classification of which is which should be written down too.

Conclusion Impact assessment is the point where governance stops being a policy document and becomes a decision about a specific person. That is what makes it worth the effort.