Model Risk Management

Model risk management came from banking, where regulators have required it for well over a decade, and it is now the framework most enterprises are adapting for AI systems — which means architects in regulated industries will meet it whether or not they have heard the term.

Its three sources of risk are the useful part, and they generalise well beyond finance. The model may be fundamentally wrong — poor assumptions, unsuitable data, methodological error. It may be implemented incorrectly, so the deployed version differs from the validated one, which is a software engineering failure rather than a data science one. And it may be used inappropriately — applied to a population or condition outside its design, which is the most common failure in practice and the least detected.

The control framework that follows: an inventory of every model in use with an owner, independent validation by people who did not build it, documented limitations and approved use, ongoing performance monitoring, and periodic revalidation.

The architectural implications are concrete and often missed. Independent validation requires the ability to reproduce a model exactly — versioned data, code and parameters. Implementation verification requires that the deployed artifact be provably the validated one, which is a signing and provenance problem. And monitoring requires that inputs and outputs be logged in a form that supports drift detection, decided before deployment rather than after.