Know what you actually run
Every AI risk programme begins in the same place and most of them stop there: an inventory nobody can complete.
Why the list is always wrong. AI arrives in an organisation through four doors, and only one of them is the one people watch:
- Built. Models your teams trained or fine-tuned. Visible, usually governed.
- Bought. A feature inside a product you already licence — the fraud score in the payments platform, the summariser in the service desk, the matching engine in the screening tool. Procured as a product, so nobody recorded that it contains a model.
- Called. An API to a model provider, added by a developer in an afternoon. Appears in the cloud bill, not the risk register.
- Brought. Staff using consumer tools on work data. Invisible by construction, and the most common source of real incidents.
What an entry must contain to be useful. A name in a spreadsheet is not an inventory. For each system: what decision or output it produces; who owns it; what data it uses, including the stores that are easy to forget; whether a person is affected by the output and how; what happens if it is wrong; whether a human reviews it and with what authority; and the date of its last assessment.
The question that finds the missing ones. Not "what AI do we have?" — people answer that with the projects they are proud of. Ask instead: "where does a decision about a person get made, and what informs it?" Then follow each answer backwards. This finds the model inside the vendor product, the score nobody calls a model, and the rule that was replaced by a classifier two years ago.
A caution about scope. A statistical model that has existed for twenty years is not automatically in scope because somebody started calling it AI, and a large language model summarising internal documents is not automatically high risk because it is fashionable. Scope by what the system decides and whom it affects, not by what the technology is called. The next lesson is how.
Shadow usage is a tooling problem. Staff paste customer data into consumer tools because the approved route is worse. Policy alone does not fix this and training alone does not either; providing something good enough to use is most of the answer, and monitoring for the rest.
Sort systems by risk
Treating every system alike is the most common failure in this field, and it fails in both directions at once: trivial systems are burdened, and consequential ones get the same thin review.
Risk here is not model risk in the statistical sense. It is the combination of what the system decides, who is affected and what happens when it is wrong. A highly accurate model making an irreversible decision about a person is higher risk than an inaccurate one suggesting search results.
The factors that actually separate tiers:
- Consequence for a person. Credit, employment, insurance, access to a service, a fraud block, an account closure, a report to an authority. These are the systems that attract legal duties.
- Reversibility. Can the outcome be undone, and how quickly? A declined transaction is recoverable in minutes; a closed account and a fraud marker may not be recoverable for years.
- Degree of automation. Suggestion, decision-with-review, or decision alone. The third category is where most regimes attach specific rights, and it is the category organisations most often cannot enumerate.
- Reach. One customer at a time, or every customer at once. Scale turns a small error rate into a systemic event.
- Contestability. Can the person affected find out why and challenge it? A system whose output cannot be explained to the person it affects cannot lawfully make some decisions about them.
- Data sensitivity. Special categories, children, biometrics.
The regulatory backdrop, without overstating it. Several regimes now tier AI by use rather than by technique, the European Union's AI Act being the most developed, with obligations that bite hardest on uses affecting people's rights and access to services. Kenya's Data Protection Act addresses decisions made solely by automated means directly. International standards — the NIST AI Risk Management Framework, ISO/IEC 42001 for management systems and ISO/IEC 42005 for impact assessment — give structure that works wherever you operate. None of them removes the need to write down, for your own systems, which decisions are solely automated. Most organisations have never produced that list, and it is usually the first thing asked for.
Make the tier do something. A tier that changes no obligation is theatre. Each tier should determine the depth of validation required, who approves deployment, how often it is reviewed, whether a human must be in the loop, and what documentation must exist before launch.
Validate it before it goes live
Validation asks a different question from evaluation. Evaluation asks how well does it perform. Validation asks is it fit for this purpose, in this process, for these people, and can we show that.
Independence is the point. Validation performed by the team that built the system is review, not validation. The reviewer needs enough independence to say no, and enough standing for that to matter. Supervisory expectations on model risk — the US Federal Reserve's guidance being the most quoted — are built around exactly this separation, and they apply to AI systems as much as to the statistical models they were written for.
What a validation should establish:
- Conceptual soundness. Is the chosen approach appropriate for the problem? Does the target variable represent what the business thinks it represents? This is where most deep problems are found, and it requires reading the design, not running a test set.
- Data fitness. Is the training and evaluation data representative of the population the system will meet? Does it encode historical decisions that were themselves biased, so the model learns to reproduce them?
- Performance, on the right slices. Overall accuracy conceals everything that matters. Performance by customer segment, by channel, by product, by demographic group where lawful to measure, and on the hard cases rather than the average.
- Stability. Behaviour under input drift, missing data, adversarial input and edge cases.
- Benchmarks. Against the current process, not against zero. "Better than nothing" is not the comparison; "better than what we do today, including its cost of errors" is.
- Failure behaviour. What it does when uncertain — a system that is confidently wrong is worse than one that abstains.
For generative and agentic systems, the same questions with harder answers: outputs are open-ended, so the test set has to be constructed rather than sampled; behaviour changes when the underlying model is updated by the provider, often without notice; and the system's own prompt is part of the artefact under test, so a prompt change is a change requiring re-validation.
The decision validation produces. Not a score — a recommendation with conditions: approve, approve with limits (volume caps, mandatory review, excluded segments), or decline. Conditions have to be enforceable in the system, not merely written in a report nobody reads again.
One catch worth naming, because it stops teams measuring at all. Testing performance by demographic group requires holding the demographic data, and data protection regimes restrict collecting exactly those categories. The result is a programme that cannot prove its system is fair and cannot prove it is not.
There are routes through — using data already held lawfully for another purpose, aggregate or proxy analysis, a one-off assessment under a specific basis, or a third party doing the measurement and returning only the result. None is free, and the wrong answer is to conclude that measuring is forbidden and therefore unnecessary.
Documentation a supervisor accepts
Documentation is where most AI governance effort is spent and most of it is wasted, because it is written for an audit file rather than for a reader with a question.
The test for every document here: could someone who was not involved reconstruct what the system does, why it was considered acceptable, and what would make it unacceptable? If not, it is a record of activity, not documentation.
What should exist for a consequential system:
- A system description in plain language. What it does, where it sits in the process, what it does not do. One page, no jargon, and the first thing anyone asks for.
- An impact assessment, covering the effect on people rather than only on the business. ISO/IEC 42005 gives a structure for this, and data protection regimes separately require one for high-risk processing — Kenya's Office of the Data Protection Commissioner has published guidance on data protection impact assessments. Done at the start it changes the design; done at the end it is paperwork, and a reviewer can tell which it was by asking what changed because of it.
- A model card or equivalent. Intended use, out-of-scope use, training data provenance, evaluation results including by slice, known limitations. The practice comes from the model-card proposal and is now widely expected.
- The validation report, with its conditions.
- The decision record. Who approved deployment, on what date, with what information in front of them, and under what conditions. In practice this is the document that matters most and the one most often missing.
- Change history. Model version, prompt version, threshold changes, retraining dates, and what each change was expected to do.
Write down which decisions are solely automated. Repeated deliberately: most regimes attach rights to those decisions — to be informed, to obtain human intervention, to contest — and you cannot honour a right you have not identified. The list is usually short, and almost nobody has it.
On explanation. Being able to name the algorithm is not an explanation. For a person affected, the explanation has to say what about their case drove the outcome, in terms they can argue with. If that cannot be produced, the system should not be making that decision alone, whatever its accuracy.
And how long to keep it. Documentation has a retention period like anything else, and the periods here are long. Where the EU AI Act applies to a high-risk system, its provider must keep the technical documentation at the disposal of the national competent authorities for ten years after the system is placed on the market or put into service, and the automatically generated logs for a period appropriate to the purpose and in any case at least six months.
Read those two together and they describe the real obligation: keep the reasoning for a decade, keep the record of what the system actually did for at least months — and resolve now where each will live, because the team that built it will not exist for ten years.
AI you did not build
Most of the AI an organisation is accountable for was written by somebody else. Everything in the previous lessons still applies, and almost none of it is under your control.
Three shapes, with different problems:
- A model you procured deliberately — a fraud engine, a screening matcher, a credit scorecard. You chose it, so you can demand things before signing.
- A model that arrived inside something you already own. A feature added in an update to software licensed years ago. Nobody procured it, no assessment was done, and your inventory was correct until the vendor shipped.
- A model you call over an API. Behaviour can change when the provider updates the model, often without notice and sometimes without a version change you can see.
What accountability does not do is transfer. The supervisor's question is about your decision, not your vendor's product, and "the vendor validated it" is not an answer. Supervisory guidance on model risk is explicit that vendor products require validation, by internal or outside parties, to the standard you would apply to your own.
What to ask for, before signing rather than after:
- What the model does, on what data it was trained, and its known limitations — in writing, specific to the version you are buying.
- Performance evidence on a population like yours, not on the vendor's demonstration set. Where they will not provide it, the answer is to test it yourself on your own data before go-live.
- Notice of change. What constitutes a material change, how much warning you get, and whether you can stay on a version while you re-validate.
- The right to audit or to receive independent assurance, and the right to your own logs — needed for an access request, an incident and a supervisor.
- What happens at the end: your data returned or deleted, and whether the model's behaviour on your data survives the contract.
Where the vendor will not answer. That is itself information, and it leaves three options: test enough yourself to be satisfied, confine the system to uses where being wrong is tolerable, or do not use it. The option that is not available is to record the gap and proceed as though it were closed.
And put procurement inside the programme. If AI risk is triggered only by a project, every model that arrives through a licence renewal bypasses it entirely.
Putting an agent into a regulated process
Operating-model diagrams for AI agents circulate widely, and most share a shape: boxes for the steps, shaded to show which run unattended and which keep a human. The shading is the part worth arguing with, because it is usually drawn from what is technically possible rather than from what the process owes the customer.
The reframing. In a regulated process the question is not "can an agent do this step?" It is: what duty attaches to this step, to whom, and can it be discharged without a person? The answer differs sharply between steps that look similar on a diagram.
A worked example: a customer in arrears. Consider the steps in any collections or customer-assistance flow, and the duties that attach:
- Segmentation and prioritisation. Ranking accounts by risk or by likelihood of contact. Low direct duty, well suited to automation. This is a scoring problem the industry has run for decades; calling the component an agent changes nothing about it.
- Channel and timing selection. Which channel, when, how often. Constrained by contact rules, permitted hours and harassment provisions in many regimes. Automatable, within hard limits that must be enforced in code rather than in a prompt.
- Outbound contact with the customer. This is the step that diagrams most often mark as running unattended, and it is the most conduct-sensitive interaction in the whole flow. The person is in financial difficulty. Some are vulnerable. The duties engaged include identifying vulnerability, not applying undue pressure, giving accurate information about consequences, and recognising when to stop and offer support instead. An agent that cannot reliably detect distress, or that pursues its objective through it, is a conduct incident generator — and the fact that it is cheap to run is precisely what makes it dangerous.
- Negotiation, forbearance and settlement. Decisions about affordability and a person's ability to repay. These are judgement calls with direct consequences for the customer, and in most regimes they require a human with authority.
- Documentation and summarisation. Low duty, high value, highest adoption — and the least discussed, because it is unglamorous.
The inversion to watch for. A diagram that automates contact and keeps humans for negotiation has it backwards from a conduct standpoint: it automates the first touch with a distressed person and reserves people for the commercial decision.
What such diagrams routinely omit, and what you should add before adopting one: how vulnerability is identified and what happens then; affordability and forbearance; consent and permitted contact; complaint routes; record-keeping of what the agent actually said and for how long; the failure path when the agent is wrong; and whether any step is a decision made solely by automated means.
The honest summary. Agents are well suited to ranking, routing, drafting and summarising. They are poorly suited, today, to interactions where the duty is to notice how a person is and respond to it. That boundary is not a limitation of the technology that will be engineered away soon; it is where the obligation sits.
A human in the loop that is load-bearing
"Human in the loop" appears in nearly every AI governance document and is, in most deployments, decorative. It is worth being precise about what makes it real, because a decorative human is worse than none: the organisation believes a control exists, and the person carries the responsibility without the means to exercise it.
Four conditions, all of which have to hold:
- Authority. The reviewer can say no, and saying no is not career-limiting. If the metric the reviewer is measured on is throughput, they will approve.
- Information. They can see what the system concluded and why, the inputs it used, its confidence, and what the alternatives were. Presenting a decision for approval without its basis is asking for a signature, not a decision.
- Time. The review time is in the process design and in the staffing model. A reviewer with four seconds per case is a rubber stamp with a job title.
- Consequence awareness. They know what happens if they approve and are wrong — and, equally, what happens if they decline and are wrong, because a reviewer who only fears one of those is not neutral.
Automation bias is the quiet failure. People defer to a confident machine, more so when the system is usually right and when the reviewer is accountable for slow throughput. Rates of review-and-override close to zero are not evidence that the model is excellent; they are evidence that nobody is reviewing. Measure the override rate and investigate it in both directions — a rate near zero and a rate implausibly high are both findings.
Design patterns that help:
- Have the human decide, with the system advising, rather than the system deciding with the human confirming. The order changes behaviour.
- Sample cases for blind review, where the reviewer does not see the system's answer first, to measure genuine agreement.
- Make declining as easy as approving, with the same number of clicks.
- Route the uncertain cases rather than a random sample, and tell the reviewer why this case arrived.
- Record the reviewer's reasoning where they disagree, because that is the most valuable training data in the whole system.
On post-hoc review. Reviewing a sample after the fact is a legitimate control for reversible decisions and no control at all for irreversible ones. Match the control to what can be undone.
And the governance point. Where a human is the reason a system is permitted to operate, the quality of that oversight is a control that must itself be tested, evidenced and reported — not an assumption written once in a policy.
Monitor, respond and retire
Deployment is the beginning of the risk, not the end of the project. Most AI governance stops at launch, which is where the measurable harms start.
What to monitor, in order of how often it is missing:
- Outcomes, not just outputs. Not the score distribution — what happened to the people. Complaints, overturns on appeal, downstream losses, and the fate of the cases the system declined or flagged.
- Performance by slice, continuously. Aggregate performance can hold steady while a segment degrades badly.
- Drift. In the inputs, in the population, and in the relationship between them. Also prompt and model drift for generative systems, where the provider may change the model under you.
- The human layer. Override rates, review times, and whether reviewers are agreeing more over time, which is usually automation bias rather than improvement.
- Volume and cost, which catch both runaway loops and silent failures.
Thresholds decided in advance. What level of degradation triggers investigation, restriction or withdrawal, decided before launch and written into the decision record. Deciding it afterwards means deciding it under pressure, with someone's revenue attached.
Incidents. An AI incident is not only an outage. A system producing confidently wrong outputs at normal availability is the harder case, because nothing alerts. Have a defined route to restrict, roll back or disable; know who can authorise it out of hours; and be able to identify and remediate the decisions made while it was wrong — which requires the logging decided in lesson four.
Appeals as a monitoring signal. Contested decisions are free, high-quality evidence. An appeals process with no feedback into the model is a complaint-handling function; one that feeds back is a control.
Retirement, the stage nobody plans. Systems are decommissioned when they are replaced, superseded or no longer justified — and the obligations do not stop: data retention and deletion across every store, records kept for decisions already made, and a decision about whether past outputs should be revisited if the system turns out to have been wrong.
What good looks like, compressed. You can say what you run and what it decides; the consequential systems are tiered and the tier changes something; each has been validated by someone who could have said no; documentation would let a stranger reconstruct the decision; the humans in the loop have authority, information and time, and their override rate is measured; outcomes are monitored by slice with thresholds set in advance; and somebody owns each system by name. None of that is exotic. It is ordinary control discipline applied to a technology whose failure modes are quieter than most.
Thirty questions
Answer all thirty questions, then check your answers. You need 27 out of 30 to pass and receive a certificate. If you score less, you will see which answers were right and wrong, and then go through the course again before you retake the check. Your answers, progress and times are kept only in this browser.
Your answers
Your certificate of completion
Enter your name as you want it to appear, then save the certificate as a PDF. In the print window, choose Save as PDF. A certificate is issued once per completion of the course.
Rate this course
How would you rate this course? Your rating is anonymous — just a number, with no name and no account.
Bring this to your team
Your colleagues can take this course too — it is free, needs no sign-up, and nothing they enter leaves their browser. If your institution wants the same ground covered as structured training for a team, with an instructor and your own cases, we run courses in a classroom or online for groups of ten or more.
Talk to us about training your team → See the paid courses →
Saolix does not record who takes this course, so it cannot verify these certificates. The certificate confirms completion of a free self-paced course and is not an accredited qualification.
Standards and frameworks covered
The published frameworks this course teaches toward. Saolix is not affiliated with, endorsed by or accredited by any of these bodies; naming a standard is a citation, not a claim about its author.
Sources
The official documents this course relies on. Laws and guidance change, so check the current version.
- AI Risk Management Framework (AI RMF 1.0) · US National Institute of Standards and Technology
- NIST AI 600-1: Generative Artificial Intelligence Profile · National Institute of Standards and Technology
- ISO/IEC 42005:2025 AI system impact assessment · ISO
- ISO/IEC 42001:2023 AI management systems · ISO
- SR 26-2: Revised Guidance on Model Risk Management · Board of Governors of the Federal Reserve System
- Model Cards for Model Reporting · Mitchell and others (arXiv)
- Data Protection Act, 2019 (sections 31, 35, 42 and 43) · Kenya Law
- Guidance Note on Data Protection Impact Assessment · Office of the Data Protection Commissioner (Kenya)
- Regulation (EU) 2024/1689 (Artificial Intelligence Act) · European Union (EUR-Lex)
- OWASP Top 10 for Agentic Applications for 2026 · OWASP Gen AI Security Project
- FSI Insights No 63: Regulating AI in the financial sector · Bank for International Settlements
- Machine learning in UK financial services (2019) · Bank of England and Financial Conduct Authority



