Model Risk Management
Inventory, validation, monitoring and challenge for models that make consequential decisions.
5 to work through
-
advanced
A deployed model performed well in validation and its business metric has declined over four months. Nothing has been deployed. What do you investigate?
2 min answer -
advanced
A recommendation model is retrained monthly. The challenger is promoted automatically when it beats the champion on an offline metric computed from the last 30 days of logged interactions. It passes every month, and live engagement has been flat for a year. What is happening?
2 min answer -
advanced
A vendor SaaS product embeds a model that scores customers, and its output drives an automated decision in your process. Your model governance framework covers models you build. What do you do?
2 min answer -
advanced
An independent validator asks you to reproduce a model's training run from eight months ago. Can you? What is needed?
2 min answer -
advanced
An organisation deploys machine learning models in decisions affecting customers. What does model risk management require?
1 min answer
5 terms in this topic
Champion-Challenger Contamination
The condition where a model's offline evaluation data was generated by the model it is being compared against, so the metric rewards imitation and im…
practiceModel Inventory
A complete record of every model deployed in the organisation with its owner, purpose, risk tier and review date - the precondition for any AI govern…
practiceModel Inventory
A maintained register of every model making or informing decisions - the prerequisite without which no other model governance control can be applied.
practiceModel Risk Management
The discipline of governing the risk that a model is wrong, is used incorrectly, or is applied outside the conditions it was built for.
conceptValidated Envelope
The explicit conditions a model's validation actually covers - input schema, population, feature sources, thresholds, upstream versions - outside whi…
Neighbouring topics
Assurance, Audit & Model Risk
General material on assurance, architectural governance and risk oversight.
Control Design vs Operation
A control that is well designed and never runs fails exactly like one that is absent.
Audit Evidence
Producing durable, tamper-evident proof as a by-product rather than as a project.
Certification Impact on Architecture
What SOC 2 and ISO 27001 actually require of a design, and what they do not.
Continuous Controls Monitoring
Testing controls continuously instead of sampling them once a year.
Segregation of Duties
Splitting authority so no single actor can both make and approve a change.
Change Advisory vs Automated Gates
Replacing a weekly board with evidence a machine produces on every change.
Risk Appetite
The stated tolerance that tells you which risks you are allowed to accept.
Risk Assessment Methods
Qualitative matrices, FAIR and scenario analysis, and the illusion of a precise score.
Security Design Review
Reviewing an architecture for security while changing it is still cheap.
Architecture Compliance Checks
Automating conformance to standards so review effort goes to the genuinely novel.
Exception & Waiver Management
Time-boxed, owned deviations with a remediation date, rather than permanent silence.
Design Authority
How an ARB should decide, what it should not review, and how it avoids becoming a queue.
Three Lines Model
Ownership, oversight and independent assurance, and where architecture sits in it.
AI Risk Tiering
Classifying a use case by potential harm, and the obligations each tier triggers.
Model Documentation
Model cards, intended use, limitations, and the record a regulator will ask for.
Model Evaluation & Red-Teaming
Adversarial testing of a probabilistic system with no fixed expected output.
Bias & Fairness Controls
Measuring disparate outcomes, choosing a fairness definition, and living with the trade-off.
Human-in-the-Loop Design
Meaningful review rather than a rubber stamp, and designing against automation bias.