What this course adds, and to whom
Two courses in this catalogue already cover ground adjacent to this one, and it is worth being exact about the difference before you spend two and a half hours here.
The AI risk management course is written for the person answerable for an AI system. It has a lesson on validating before go-live and a lesson on documentation a supervisor accepts, and those lessons give you the checklist. The fraud models course covers model risk as it applies to fraud detection specifically, including the current US and UK supervisory guidance as it bears on that use.
This course is for the validator, and for whoever has to satisfy a supervisor afterwards. It works through the supervisory and legal texts themselves — what they require, in their own words, clause by clause — rather than summarising them into a checklist. If you have read the other two and want to know why the checklist says what it says, and what a reviewer will hold you to, this is the course.
The texts it is built on, because the whole course is an argument from them:
- The revised US interagency guidance on model risk management, issued April 2026, which replaced the 2011 guidance that shaped the discipline — and which contains a scope exclusion most readers have not noticed. That is the next lesson.
- The UK prudential regulator's model risk principles, in their current version effective April 2026.
- The European Central Bank's guide to internal models, which is about credit and market risk models rather than AI, and whose requirements on validation independence, benchmarking frequency and the model register are the most concrete anywhere.
- European banking guidelines on estimation and on machine learning, which name the technical failure modes supervisors actually look for.
- The EU AI Act, Articles 9 to 15, Annex IV and the post-market provisions, which turn much of the above into statutory documentation requirements for high-risk systems.
- Model cards and datasheets, the two papers that created the documentation formats everybody now cites.
- And the NIST publication on bias, for the vocabulary.
One framing to carry throughout. Validation is not testing. Testing asks whether the model performs. Validation asks whether it is fit for this purpose, in this process, for these people — and whether you can demonstrate that to somebody who was not there and is not obliged to be sympathetic.
And one caution about scope creep. Not everything called AI is a model in the supervisory sense, and not every model becomes high risk because it uses a neural network. The texts in this course apply by what a system decides and whom it affects. Lesson two starts there, because getting scope wrong in either direction is the most expensive mistake available: too narrow and you have an ungoverned system, too wide and you exhaust the function that was supposed to govern the ones that matter.
What the current guidance covers, and what it excludes
The document most people mean when they say "model risk management" was replaced in 2026, and the replacement contains two statements that change how it should be used.
The replacement. The revised guidance states on its face that it "supersedes and replaces SR letter 11-7, Guidance on Model Risk Management (issued April 4, 2011)". Independently verified for this course: it also supersedes a 2021 interagency statement on model risk management for bank systems supporting anti-money laundering compliance — so two documents went, not one.
If your model risk policy cites the 2011 guidance as its authority, it cites a superseded document, and the original URL now returns nothing.
The first statement that matters: it is not enforceable standards. "This guidance does not set forth enforceable standards or prescriptive requirements; accordingly, non-compliance with this guidance will not result in supervisory criticism against a banking organization."
That is unusually explicit and it cuts both ways. It is not a rulebook to be complied with line by line, which should relieve anyone who has been treating it as one. And it remains the articulation of what supervisors consider sound practice, which means departing from it is a decision to be justified rather than a non-event.
The second statement, which is the one to sit with. "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance."
Read that again if you have been assuming the current model risk guidance governs your language-model deployment. It does not. The guidance that the discipline rests on explicitly excludes the category of system most organisations are now deploying.
It does not leave you without an obligation. The same footnote continues that the organisation's own risk management and governance practices should guide the determination of appropriate governance and controls for tools, processes or systems not covered — which is to say: you are expected to govern it, using your own framework, without this document telling you how.
What follows practically. Three things:
- Do not claim conformance to that guidance for a generative system. It is outside its scope, and asserting otherwise in a validation report is a misstatement a reviewer can check.
- Say which framework you are using instead, and why. The AI Act provisions in lessons nine and ten, the NIST framework, or your own documented approach.
- And expect this to change. A guidance that names a category as novel and rapidly evolving is signalling a future revision.
On which organisations it is aimed at. The guidance describes itself as most relevant to banking organisations with over thirty billion dollars in total assets, and addresses smaller ones separately. Scale is part of proportionality here, stated in the text rather than inferred.
Independence, and what it actually requires
Every supervisory text in this field says validation must be independent. The useful question is what independence consists of, and two regulators answer it concretely.
The UK principle. The prudential regulator's model risk principles: "Firms should have a validation process that provides ongoing, independent, and effective challenge to model development and use."
Three adjectives, each load-bearing. Ongoing, so not a one-off gate at go-live. Independent, which the next quote defines. And effective challenge, which is a standard about outcome rather than process — a validation that never changes anything is not providing it.
What independence means organisationally. The European Central Bank's guide is the most specific: "To ensure the effective independence of the internal validation function from the model development process (i.e. model design, development, implementation and monitoring), institutions should have appropriate organisational arrangements in place."
Note the scope of what validation must be independent from: design, development, implementation and monitoring. The last one is often missed. An arrangement where the development team runs the ongoing monitoring and validation reviews their reports has an independence gap in the part of the lifecycle that lasts longest.
Why this is not pedantry. Validation's only product is a judgement, and a judgement is worth what the independence of the person making it is worth. A validator who reports to the model owner, whose objection can be overruled without record, or whose findings are negotiated before issue, is performing review. The distinction shows up in exactly one observable: whether validation has ever stopped something.
Three arrangements that look independent and are not:
- Independence of person but not of incentive. A separate team, measured on how quickly models reach production.
- Independence of reporting but not of competence. A validator who cannot read the model's design can only check that documents exist, which is a compliance check wearing a validation badge. The European banking authority's work on machine learning names this directly, listing among the main challenges in development and validation "the skills of developers and validators".
- Independence at first validation but not afterwards. The initial review is done properly and the annual re-review is a form signed by the owner.
And a resourcing point that follows. Effective challenge of a modern model needs a validator who could have built it. That is an expensive person, in a role with less visible output than the development team's, and it is the first role cut when a programme is under pressure. An institution's real commitment to model risk management is visible in who holds that job and what they are paid, not in the policy.
What to do when full independence is not available. A small institution may have one person who understands the model, and pretending otherwise produces a validation performed by someone who cannot challenge it. The honest arrangements, in descending order of strength: commission the validation externally; have it performed by a competent person in another function with no stake in the outcome; or have the developer document the validation and a second institution's validator review it under a reciprocal arrangement.
What is not available is to record the constraint and carry on. If the review was not independent, the report should say so, name what was done instead, and state what that leaves unexamined — which is a defensible position, where an unmarked claim of independence is not.
Conceptual soundness: reading the design, not the test set
The part of validation that finds the serious problems is not a measurement. It is reading the design and asking whether the approach can answer the question being asked of it.
Why it comes first. A model can perform excellently against a test set and be unfit, because the test set shares the design's mistake. If the target variable does not represent what the business believes it represents, every performance figure is a measurement of the wrong thing, computed correctly.
The questions that find it:
- What is the target variable, actually? A model predicting "fraud" is usually predicting "cases the previous process flagged as fraud", which is a different quantity with the previous process's blind spots built in.
- Does the population the model was built on resemble the one it will meet? Lesson five is about this, and it is a requirement rather than a preference.
- What does the model do when the input is outside anything it has seen? A confident answer in that case is worse than an abstention.
- What is the feature doing that you did not intend? A variable that correlates with a protected characteristic will act as a proxy for it whether or not anybody meant it to.
- And what happens to the people the model is wrong about? Which is the question the AI Act turns into a documentation requirement in lesson nine.
The complexity trade, stated by a supervisor. The European banking authority's follow-up report on machine learning for internal models puts it plainly: "A higher level of complexity may lead to better model performance but at the cost of lower explainability and comprehension of the model's functioning."
That is the trade every validator has to price, and the report is careful not to resolve it in the abstract. The question is whether the gain in performance is worth the loss in explicability for this decision — which is a different answer for a marketing propensity score than for a credit refusal a customer may contest.
The failure modes the same report names. Its consultation responses "confirmed that issues related to the overfitting, the skills of developers and validators and the interpretation issues are the main challenges during the development and validation phase."
Three named problems: overfitting, which lesson six addresses; validator skill, which lesson three addressed; and interpretation. None of them is found by running a test set.
What conceptual soundness produces. Not a score. A written argument that the chosen approach is appropriate for the problem, with the reasons, the alternatives considered, and the limitations accepted. A validation report whose first section is a table of performance metrics has skipped the part where the real findings are, and a reviewer who knows this field will notice.
Data, and the representativeness requirement
The most common cause of a model that works in development and fails in production is that the data it learned from did not resemble the data it would meet. Two texts turn that observation into an obligation.
The banking supervisor's version. The European guidelines on estimation require that "Institutions should select an appropriate dataset for the purpose of model development to ensure that the performance of the model on the application portfolio, in particular its discriminatory power, is not significantly hindered by insufficient representativeness of data."
Note the construction. It is not "the data should be representative" in the abstract — it is that performance on the portfolio the model will actually be applied to must not be hindered by the development data being unrepresentative. The test is downstream, not upstream.
The statutory version. The EU AI Act, for high-risk systems: "Training, validation and testing data sets shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose."
Four properties, and the qualifiers matter. Sufficiently representative, and free of errors to the best extent possible — the drafters knew that perfect data does not exist. What is not qualified is "in view of the intended purpose": representativeness is judged against what the system is for, which means a change of purpose can invalidate a data assessment without a line of code changing.
What a validator should actually examine:
- The time period. A model trained on a period that contained no stress has not been shown to work in one. The reverse is also true for a model trained only on a crisis.
- The segments. Which customer types, channels, products and geographies are in the development data in what proportions, against the proportions in the application portfolio.
- The exclusions. What was filtered out, by whom, and why. Exclusions are where representativeness quietly disappears, because each one was individually reasonable.
- The labels. Who decided the outcome, on what basis, and whether that decision process has changed since. A label produced by a previous rules engine encodes that engine.
- And the missing. The population the data cannot contain — declined applicants whose performance was never observed, which is the oldest selection problem in credit and reappears in every domain.
The reason this is validation's work rather than development's. A development team assesses its data against the problem as it understands it. A validator's job is to ask whether that understanding matches the deployment, and the answer is frequently held by a third party — the business owner who knows which segments the product will be sold to next year.
And one practical test. Ask for the distribution of the development sample and the distribution of the live population, side by side, on the five variables that matter most. If nobody can produce that comparison, the representativeness assessment has not been done, whatever the document says.
Out-of-sample, out-of-time, and overfitting
A model that performs better on the data it was built from than on data it has not seen has learned the sample rather than the phenomenon. Detecting that is the most mechanical part of validation and it is routinely done badly.
The supervisory expectation, stated specifically. The European banking authority's machine learning report: "financial institutions are expected to put particular attention on the comparison of the model performances measured within the development sample with those obtained using out-of-sample and out-of-time data sample".
Three samples, and the distinction between the last two matters:
- In-sample performance, on the data used to fit the model. Always the best, and informative only as a baseline.
- Out-of-sample, on data held back from the same period. Tests whether the model generalises to unseen cases.
- Out-of-time, on data from a different period. Tests whether the relationship it learned is stable. A model can generalise perfectly within a period and decay immediately outside it, which no out-of-sample test will reveal.
Out-of-time is the one most often skipped, because holding back a period costs data and because the result is frequently unflattering.
Why overfitting is worse in modern models. Flexible models can fit arbitrarily complex relationships, including noise. The report names overfitting first among the main development and validation challenges, and the mechanism is simple: with enough parameters, a model can memorise. The guard is not a philosophical position about complexity; it is a held-back sample the development team never saw.
And the discipline that makes the held-back sample meaningful. If a team tunes the model until the held-back set performs well, that set has become a development set and no longer tests anything. The only honest arrangement is a sample the developers cannot see, held by validation, used once. An institution whose validation sample has been scored forty times during development has no out-of-sample evidence at all, however carefully it was constructed.
What else to test for stability. The AI Act requires that high-risk systems achieve "an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle" — consistently, throughout, which is an out-of-time requirement by another name.
And it is prescriptive about how testing is done: "Testing shall be carried out against prior defined metrics and probabilistic thresholds that are appropriate to the intended purpose of the high-risk AI system."
Prior defined. The metrics and the thresholds are set before the test, not chosen after the results are in. That single word closes the most common way a validation reaches a favourable conclusion, and it is worth quoting at anybody who proposes to "see how it performs and then decide what good looks like".
And the same discipline applied annually. The European guidelines on estimation require that the reviews of estimates performed at least annually "should be performed taking into account the metrics, standards and thresholds defined by the institution", with predefined thresholds and significance levels, and predefined actions to be taken where the review produces adverse results.
Three things are predefined there: what you will measure, what counts as a bad result, and what you will do about it. The third is the one almost always missing. A monitoring regime that detects deterioration and has no agreed consequence produces a paper trail showing the institution knew — which is a worse position than not having measured, and it is the shape of several public enforcement actions.
Challenger models and benchmarking
A performance figure on its own is not evidence of fitness, because there is nothing to compare it with. Benchmarking supplies the comparison, and two supervisors specify how.
What a challenger model is. The UK principles define it precisely: comparisons of the model output with the output of "available challenger models, which are alternative implementations of the same theory, or implementations of alternative theories and assumptions".
Two distinct kinds in that definition. A different implementation of the same theory tests whether the result is an artefact of this build. A model resting on different assumptions tests whether the theory itself is doing the work. The second is more informative and less often attempted.
And the simpler version that is often enough. The same text describes benchmarking as "comparing model estimates with comparable but alternative estimates". A simple, interpretable model — a logistic regression, a small decision tree, a well-chosen rule set — is a legitimate challenger, and if it performs within a small margin of a complex model, the complexity has not earned the explicability it cost. That is a finding, and it is one validators are often reluctant to write down.
The frequency, which one supervisor fixes. The European Central Bank's guide: "Benchmarking analyses should be performed at initial validation and at an adequate frequency thereafter, but at least every three years."
At least every three years, with an adequate frequency being whatever the model's risk and volatility require. That is the only hard periodicity in any of the texts in this course, and it is worth adopting as a floor even outside its formal scope — because a model that has not been compared with an alternative since it was built has been assumed rather than assessed.
Monitoring frequency, tied to tiering. The UK principles: "Model monitoring should be conducted on an ongoing basis with a frequency determined by the model tier."
Which presupposes a tiering, and that is the point. A single monitoring cadence applied to every model spends the same effort on the trivial and the consequential. The tier decides the frequency, and the tier is a governance decision that has to exist before the monitoring schedule can be defensible.
What to compare against, beyond another model. Three comparisons that are cheaper than building a challenger and are frequently more persuasive:
- The current process. Not against zero. Including the cost of the errors the current process makes, which is the comparison the business actually faces.
- The model's own earlier version, on the same sample, which isolates what the change did.
- A deliberately naive baseline — predict the base rate, or predict last month. A model that cannot clearly beat predicting last month is not yet a model.
Bias, and what can honestly be claimed
Fairness assessment is where validation reports are weakest, usually because the topic is treated as a single question with a yes or no answer. The standards treat it as three different problems.
The taxonomy. NIST's publication on identifying and managing bias in AI "identifies three categories of bias in AI — systemic, statistical, and human". Keeping them apart is most of the analytical work:
- Systemic bias is in the institution and the data it generated: historic decisions, who was marketed to, who was declined before the model existed.
- Statistical bias is in the sample and the method: unrepresentative data, label noise, a metric that optimises for the majority.
- Human bias is in the people building and using it: how the problem was framed, which features seemed sensible, how the output is interpreted by the person acting on it.
A validation that measures outcome differences across groups has examined one consequence of all three and diagnosed none of them. The mitigation differs by category, which is why the diagnosis matters.
And the claim that cannot honestly be made. The same publication is unambiguous: "it is not possible to achieve zero risk of bias in an AI system".
That sentence should appear in validation reports more often than it does. The defensible position is a measured, bounded and disclosed differential with a stated rationale — not an assertion of fairness. A report claiming a model is unbiased has made a claim the field's own standards body says cannot be supported.
Where the challenges lie. NIST groups them as "three broad challenges for mitigating bias — datasets, testing and evaluation, and human factors", which maps onto the three categories and tells you that a purely technical mitigation addresses at most one of them.
The measurement problem, and it is a real constraint. Testing performance by group requires holding data about the group, and data protection law restricts collecting exactly those categories. The AI risk management course covers that catch-22 and the routes through it. What belongs here is the validator's position: the inability to measure is a limitation to be recorded, not a reason to assert the absence of a problem.
Where the statutory requirement bites. The AI Act requires that "The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use."
Declared, in the instructions. Not held internally — stated to the deployer, which means a provider must commit to figures a deployer can hold it to. And Annex IV requires documentation of "the validation and testing procedures used, including information about the validation and testing data used and their main characteristics; metrics used to measure accuracy, robustness and compliance".
Read those two together and the direction is clear: the metrics, the data they were measured on, and its characteristics all become disclosed artefacts rather than internal working papers.
When documentation becomes a legal requirement
For high-risk systems under the EU AI Act, much of what the previous lessons described as good validation practice becomes a statutory documentation obligation. The shift is worth understanding precisely, because the Act specifies artefacts rather than intentions.
Risk management is continuous, not a gate. The Act requires the risk management system to be "a continuous iterative process planned and run throughout the entire lifecycle of a high-risk AI system, requiring regular systematic review and updating".
Planned and run throughout the lifecycle. A validation performed once before go-live, however thorough, does not satisfy that — and nor does an annual review that is a signature on last year's document.
Human oversight, specified as a capability. Among the measures required is that the person overseeing the system is able "to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system".
Four verbs: not use, disregard, override, reverse. A deployment where the reviewer can only approve or escalate does not provide the third and fourth. This is a design requirement with a user-interface consequence, and it is the clause that most often fails on inspection because nobody tested whether the override actually works end to end.
What Annex IV demands of the technical documentation. Two items bear directly on validation. The documentation must describe "the validation and testing procedures used, including information about the validation and testing data used and their main characteristics; metrics used to measure accuracy, robustness and compliance".
And it must contain "A detailed description of the system in place to evaluate the AI system performance in the post-market phase".
Read the second one carefully. The documentation prepared before placing the system on the market must describe the arrangements for monitoring it afterwards. You cannot complete the pre-market file without having designed the post-market monitoring, which is lesson twelve — and that sequencing is deliberate.
Post-market monitoring, as its own obligation. "Providers shall establish and document a post-market monitoring system". Established and documented, not merely intended.
And the quality management system that holds it together. The Act requires the provider's quality management system to be documented in written policies, procedures and instructions — which is the clause that turns a set of good practices performed by capable individuals into something an auditor can examine in their absence.
One practical consequence for a validator. Your validation report is no longer an internal working paper. For a high-risk system it is a component of a technical file that a market surveillance authority may request, and its contents are specified. Writing it to the Annex IV headings from the outset costs nothing and saves a reconstruction later — and a reconstruction, performed months afterwards by people who were not there, is where the gaps become visible.
Logs, retention, and the ten-year file
Documentation obligations are usually discussed without their duration, which is where the operational difficulty actually lives.
The logging capability is a design requirement. The AI Act: "High-risk AI systems shall technically allow for the automatic recording of events (logs) over the lifetime of the system."
Technically allow, over the lifetime. That is a statement about the system's architecture rather than about an operational policy, and it has to be true at build time. A system that cannot record its own events cannot be retrofitted to comply without being changed.
Two retention periods, and they differ by an order of magnitude.
The technical documentation: a provider must, "for a period ending 10 years after the high-risk AI system has been placed on the market or put into service, keep at the disposal of the national competent authorities: (a) the technical documentation".
The logs: kept "for a period appropriate to the intended purpose of the high-risk AI system, of at least six months".
Ten years for the reasoning; a floor of six months for the record of what the system actually did. Both were re-verified against the Official Journal text for this course.
Why the asymmetry makes sense, and what it costs you. The documentation explains why the system was considered acceptable, and that question can be asked for as long as decisions made by the system have consequences. The logs are evidence of individual operations, which is a far larger volume and a shorter useful life.
The practical problem is that ten years exceeds the life of almost everything around the file: the team, the vendor, the storage platform, the document management system, and frequently the model itself. So the question to settle at the start is not what the file contains but where it will live and who will be able to open it in 2036 — in a format that will still be readable, with the context needed to interpret it.
A validation report that references "the standard thresholds" or "the usual sample" is unreadable once the people who knew what those were have gone. Write the numbers down.
Six months is a floor, not a target. "Appropriate to the intended purpose" means the period is a reasoned decision. A system making decisions a person may contest a year later needs logs that outlast the contest, and six months will not serve. That reasoning belongs in the file.
And the reason this gets attention. The Act's penalties for non-compliance with its obligations run to "up to EUR 15 000 000 or, if the offender is an undertaking, up to 3 % of its total worldwide annual turnover". Figures at that scale are what move documentation from a professional preference to a board matter, and they are the argument to use when the retention budget is questioned.
Model cards, datasheets, and the register
Three documentation artefacts recur in every modern framework, and all three come from outside the regulatory world. Knowing what each was designed to do stops them being filled in as forms.
The model card. The 2019 paper that introduced them describes them as "short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions".
Benchmarked evaluation in a variety of conditions — not a summary of a single headline figure. The paper is specific about how the results should be presented: "Quantitative analyses should be disaggregated, that is, broken down by the chosen factors." And: "Quantitative analyses should provide the results of evaluating the model according to the chosen metrics, providing confidence interval values when possible."
Two requirements rarely met in practice. Disaggregated, which is the fairness analysis of lesson eight presented rather than filed. And confidence intervals, which are almost always omitted — and whose absence lets a difference between two figures be read as real when the samples cannot support it.
The datasheet. The companion idea for datasets states the problem it exists to solve: "a model is unlikely to perform well in the wild if its deployment context does not match its training or evaluation datasets" — which is lesson five's representativeness requirement, expressed as a documentation need.
Its purpose is reflective rather than administrative: "For dataset creators, the primary objective is to encourage careful reflection on the process of creating, distributing, and maintaining a dataset". And the timing instruction is the one that makes it work: "Dataset creators should read through these questions prior to any data collection and then provide answers once data collection is complete."
Read the questions before collecting. A datasheet completed afterwards records what happened; one read beforehand changes what happens. That is the same distinction the impact assessment carries in the AI risk management course, and it is the difference between a document and a control.
The register. The most concrete requirement comes from the European Central Bank's guide: "Adequate controls of the register of the institution's internal models, together with an inventory of the documentation, including an annual review, should be in place."
Three things in one sentence: a register of models, an inventory of the documentation for each, and an annual review of both. And the scope note that catches people: "The register should also include models purchased from third-party vendors."
Vendor models in the register, which is lesson twelve's subject. An institution whose model inventory lists only what it built has an inventory of its own work rather than of its model risk — and this is the single most common finding in this area, because a purchased model arrives through procurement and never passes the desk that maintains the register.
Vendor models, monitoring, and what a supervisor asks
Most models an institution relies on were built by somebody else, and the texts are unambiguous that this changes nothing about the obligation.
Validation applies to purchased models. The revised US guidance: "An important element of model risk management is the validation of vendor products, either by internal or outside parties."
And the UK principles set the standard: firms should "satisfy themselves that the vendor models have been validated to the same standards as their own internal MRM expectations".
The same standards. Not a lighter touch because the model is a product, and not reliance on the vendor's own assurance as a substitute. Where a vendor will not provide what validation needs, the institution is left with three options — test enough itself, confine the model to uses where being wrong is tolerable, or not use it. Recording the gap and proceeding is not among them, which is the same conclusion the third-party cyber risk course reaches from a different direction.
Why this is getting harder. The Financial Stability Board's assessment names the mechanism: "the difficulty of assessing data quality for widely used AI models could increase model risk for FIs that lack robust AI governance", and "The use of opaque training data sources for these models also complicates data quality assessments."
Opaque training data defeats lesson five. You cannot assess representativeness against a deployment population if you do not know what the model was trained on — and for most large pre-trained models you do not. That is a limitation to record explicitly, with the compensating controls you are relying on instead.
The same assessment notes the structural consequence: "The reliance on specialised hardware, cloud services, and pre-trained models has increased the potential for AI-related third-party dependencies."
Ongoing monitoring of the relationship. The interagency guidance on third-party relationships describes what monitoring is for: it "enables a banking organization to: (1) confirm the quality and sustainability of a third party's controls and ability to meet contractual obligations", across a lifecycle of "planning, due diligence and third-party selection, contract negotiation, ongoing monitoring, termination".
Negotiation is in that list, which is the moment the validation access you will need in year three has to be secured.
What a supervisor actually asks, and the honest state of most answers:
- Which models do you run, including purchased ones? The register usually omits the bought ones.
- Who validated this, independently of whom? Often independent of development and not of monitoring.
- Show me the out-of-time test. Frequently absent.
- What did you compare it against? Often nothing.
- What is the performance differential across groups, with intervals? Rarely with intervals.
- What happens when it is wrong, and can the reviewer actually override it? Seldom tested end to end.
- And where will this file be in ten years? Almost never answered.
The judgement to take away. Validation's product is a defensible written argument, made by someone able to disagree, against a comparison, on data that resembles the deployment, with its limitations stated rather than omitted. Every text in this course is a different way of saying that.
Fifty questions
Answer all fifty questions, then check your answers. You need 45 out of 50 to pass and receive a certificate. If you score less, you will see which answers were right and wrong, and then go through the course again before you retake the check. Your answers, progress and times are kept only in this browser.
Your answers
Your certificate of completion
Enter your name as you want it to appear, then save the certificate as a PDF. In the print window, choose Save as PDF. A certificate is issued once per completion of the course.
Rate this course
How would you rate this course? Your rating is anonymous — just a number, with no name and no account.
Bring this to your team
Your colleagues can take this course too — it is free, needs no sign-up, and nothing they enter leaves their browser. If your institution wants the same ground covered as structured training for a team, with an instructor and your own cases, we run courses in a classroom or online for groups of ten or more.
Talk to us about training your team → See the paid courses →
Saolix does not record who takes this course, so it cannot verify these certificates. The certificate confirms completion of a free self-paced course and is not an accredited qualification.
Standards and frameworks covered
The published frameworks this course teaches toward. Saolix is not affiliated with, endorsed by or accredited by any of these bodies; naming a standard is a citation, not a claim about its author.
Sources
The official documents this course relies on. Laws and guidance change, so check the current version.
- SR 26-2: Revised Guidance on Model Risk Management (17 April 2026), replacing SR 11-7 and SR 21-8 · Board of Governors of the Federal Reserve System, FDIC and OCC
- SS1/23: Model risk management principles for banks (current version, effective 23 April 2026) · Prudential Regulation Authority, Bank of England
- ECB guide to internal models, General topics chapter (February 2024) · European Central Bank, Banking Supervision
- EBA/GL/2017/16: Guidelines on PD estimation, LGD estimation and the treatment of defaulted exposures · European Banking Authority
- EBA/REP/2023/28: Follow-up report on machine learning for IRB models (August 2023) · European Banking Authority
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 9 to 19, 72 and 99, and Annex IV · Official Journal of the European Union, via EUR-Lex
- Regulation (EU) 2026/1744 (Digital Omnibus on AI), amending the application dates for high-risk systems · Official Journal of the European Union, via EUR-Lex
- Model Cards for Model Reporting (Mitchell et al., 2019) · ACM Conference on Fairness, Accountability and Transparency, via arXiv
- Datasheets for Datasets (Gebru et al.) · Communications of the ACM, via arXiv
- NIST SP 1270: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (March 2022) · US National Institute of Standards and Technology
- NIST AI 100-1: Artificial Intelligence Risk Management Framework (AI RMF 1.0), the Measure function · US National Institute of Standards and Technology
- NIST AI 600-1: AI Risk Management Framework, Generative AI Profile (July 2024) · US National Institute of Standards and Technology
- Interagency Guidance on Third-Party Relationships: Risk Management (SR 23-4 attachment, June 2023) · Board of Governors of the Federal Reserve System, FDIC and OCC
- The Financial Stability Implications of Artificial Intelligence (14 November 2024) · Financial Stability Board
- Regulation (EU) 2016/679 (GDPR), Articles 22 and 35 · Official Journal of the European Union, via EUR-Lex
- Data Protection Act, 2019 (No. 24 of 2019), sections 31 and 35 · Office of the Data Protection Commissioner, Kenya



