What an agent is, and when you need one
An agent is a loop: a model decides on an action, something executes it, the result comes back, and the model decides again. That is the whole of it, and most of the difficulty is in deciding whether you want one.
The minimal definition. A model, a set of tools it may call, a goal, and a loop that continues until the goal is met or a limit is reached. Everything else - planning, memory, reflection, multiple agents - is an elaboration of those four parts.
What distinguishes it from a pipeline. In a pipeline you decided the sequence of steps. In an agent the model decides, at run time, which step comes next. You have traded predictability for the ability to handle cases you did not enumerate.
That trade is the whole engineering question. A pipeline is cheaper, faster, testable and auditable. An agent is none of those things and it handles the long tail. If you can enumerate the steps, do not build an agent; a workflow with a model in two or three places is almost always the better system.
The cases that genuinely need one. The number of steps is not known in advance. The next step depends on what was found in the previous one. The task requires exploring until something is located. A person would describe the work as investigating rather than processing.
The cases that do not. Classify this. Extract these fields. Summarise that. Route this to a queue. Each of these is one call, sometimes two, and wrapping it in a loop adds cost, latency and failure modes for nothing.
The published starting point is worth reading. The interleaving of reasoning with acting - a thought, an action, an observation, repeated - was the pattern that made this practical, and most current frameworks are a version of it. Knowing it helps you recognise how little is actually novel in any given framework.
The honest state of the field. A widely cited critique argued that agent evaluations have been dominated by accuracy figures with no attention to cost, that benchmarks are often not reproducible, and that the resulting claims do not survive contact with production. That paper is the right corrective to read next to any vendor demonstration.
What this course covers. The loop, tools, context, planning, memory, failure modes, cost control, evaluation, the human in the loop, multi-agent arrangements, and operation. It does not cover security against agents arriving at your systems, which is a separate course.
And the sentence to keep. An agent is a system that will take many actions on your behalf based on the output of a model that cannot be relied upon. Every design decision in this course follows from taking that seriously.
The loop, concretely
Before the elaborations, it is worth being precise about the mechanism, because most agent bugs are in the plumbing rather than in the model.
The prompt contains the tools. Tool definitions - names, descriptions, parameter schemas - are text in the context. The model does not call anything; it produces a structured output naming a tool and its arguments, and your code performs the call.
So a tool call is a parsing problem with a model in front of it. Current providers support constrained structured output for this, which removes the class of failures where the arguments cannot be parsed. Use it rather than asking for a format in the prose.
The result goes back in as text. The tool returns something - a record, a page, an error - which is appended to the context as an observation. The context therefore grows with every step, which is the origin of most cost and most degradation.
The loop needs termination conditions, plural. The model declaring it is finished. A maximum number of steps. A maximum token budget. A wall-clock timeout. A repeated-action detector. Any agent without all five will, eventually, run until something else stops it.
Errors are information, not exceptions. A tool that fails should return a message the model can act on: what went wrong and what might work instead. Agents recover from informative errors surprisingly well and loop endlessly on bare failures.
Observations need trimming. A tool returning ten thousand tokens of raw output will consume the context in three steps. Summarise, truncate with a marker, or return a reference the model can ask about. This single decision has more effect on whether an agent works than the choice of model.
State lives in your code, not in the model. What has been tried, what has been found, what is still outstanding. Keeping that in a structure you own, and rendering a view of it into the prompt, is far more reliable than hoping the transcript remains coherent.
Idempotency belongs in the tools. Agents retry. A tool that creates a record must be safe to call twice with the same arguments, which means an idempotency key rather than an instruction not to retry.
And every step should be logged in full. The prompt sent, the tool chosen, the arguments, the result, the tokens used. An agent without this log is undebuggable, because the interesting failures are three steps back and you cannot reconstruct them.
Designing the tools
Tool design is where most of the quality of an agent is determined, and it is usually treated as an afterthought. A good tool set makes a modest model effective; a bad one defeats a strong model.
Fewer tools, better described. A model choosing among six well-named tools is far more reliable than one choosing among thirty with overlapping purposes. If two tools could plausibly do the same job, the model will sometimes choose the wrong one, and that is your fault rather than its.
Describe the tool for somebody who cannot see your system. What it does, when to use it, when not to use it, what it returns, and what it costs. The description is a prompt, and it is the highest-leverage prose in the whole system.
Make the parameters hard to get wrong. Enumerated values rather than free text. Explicit units. Required fields that are genuinely required. Defaults that are safe. A parameter whose meaning depends on another parameter will be misused.
Return structured, compact results. A tool should return what is needed to decide the next step, not everything it knows. Pagination, field selection and summarisation belong inside the tool, where you control them, rather than in the model's context.
Design tools at the level of a task, not of an interface. Exposing twelve low-level operations and expecting the model to compose them produces long, fragile sequences. A single tool that performs the composite task reliably is better than a model orchestrating twelve calls, every time.
Separate reading from writing. Read-only tools can be given freely. Tools that change something need a different standard: narrow scope, validation inside the tool, limits the model cannot raise, and in some cases a confirmation outside the loop.
Validate inside the tool, always. Never rely on the model to respect a constraint. The tool checks the amount, the entitlement, the state and the limit, and refuses with an informative error. The model is an untrusted caller of your own system, which is a useful framing even when the agent is entirely internal.
Give the agent a way to ask for help. A tool that escalates to a person, with the context assembled, turns a dead end into a handover. Agents without one produce confident nonsense at exactly the moments they should stop.
And test the tools independently. Each one, with its own unit tests, called directly. A surprising share of agent failures are a tool that returns something unexpected on an edge case, which is cheaper to find without a model in the way.
Context, the real constraint
An agent's context fills up. Everything about managing that is the difference between a system that completes long tasks and one that degrades after five steps.
The growth is mechanical. Each step adds the model's reasoning, the tool call and the observation. A twenty-step task with verbose observations exceeds a large window easily, and cost grows with the whole accumulated context on every call.
The symptom of overflow is not an error. It is an agent that forgets its goal, repeats a step it already did, or contradicts something it established earlier. If an agent becomes less coherent as a task proceeds, the context is the first place to look.
Summarise the history, deliberately. Replace older steps with a compact record of what was learned. This is lossy and it is better than truncation, because truncation removes the beginning, which is usually where the goal was stated.
Keep a structured scratchpad outside the model. Facts established, actions taken with their outcomes, open questions, and the current plan. Render it as a compact block at the top of each prompt. The transcript then becomes recoverable history rather than the only state.
Put the goal where it will be read. At the beginning of the prompt and, for long tasks, restated near the end. The published finding on long contexts applies directly: material in the middle of a long input is used less reliably.
Observations are where the budget goes. Tool results dominate an agent's context. Returning a summary and a reference, rather than everything, is the highest-value optimisation available, and it belongs in the tool.
Separate working context from retrieval. If the agent needs documents, retrieve them for the step that needs them and drop them afterwards rather than carrying them for the rest of the task.
Budget the steps as well as the tokens. A token budget per task, a step limit, and a limit on tokens per observation. Enforced in code, logged, and surfaced when hit so the agent can wind up rather than being cut off.
And measure context length against outcome. Plot task success against the context size at which the task completed. Most teams find a length beyond which success falls sharply, and that number is a design constraint worth knowing rather than discovering.
Planning
Letting a model choose one action at a time works for short tasks and degrades on long ones. Planning is the family of techniques for giving it more structure, and the evidence for each varies considerably.
Intermediate reasoning is the foundation. The published chain-of-thought result showed that prompting a model to produce its reasoning before its answer substantially improves performance on multi-step problems. In an agent, the reasoning step before each action is this technique applied in a loop.
Explicit up-front plans help on structured tasks. Asking the model to produce a plan, then executing it step by step, gives you something to inspect, something to show a user, and a way to detect when execution has diverged. It is less useful where the first discovery invalidates the plan, which is common.
Re-planning matters more than planning. The question is not whether the agent made a plan but whether it notices that the plan is wrong. A step that compares what happened with what was expected, and revises, is where the value is.
Decomposition into sub-tasks is reliable and underused. Breaking a task into parts, completing each with a fresh and narrow context, and combining the results avoids the context growth in lesson four and makes each part testable.
Reflection has mixed evidence. Published work on verbal self-critique showed gains on certain benchmarks by having the agent evaluate its own attempt and retry with that critique in context. It helps where there is an external signal - a test that fails, a tool that errors - and helps much less where the agent is grading its own unverifiable output.
Search over possible steps is powerful and expensive. Exploring several candidate continuations and evaluating them, as in the published tree-structured approach, improves results on problems with a verifiable objective. The cost is multiplied calls, which rules it out for most interactive uses.
The strongest practical lever is external verification. A test suite, a schema, a checksum, a database constraint, a second tool that confirms the result. An agent with a way to know it was wrong is in a different class from one without, and this matters more than any planning method.
Beware of plans as theatre. A visible plan reassures users and does not make the work more likely to succeed. If the plan is not inspected, compared against execution, or used to constrain the next step, it is output rather than mechanism.
And prefer the simplest structure that passes your evaluation. Each layer - planning, reflection, search - multiplies cost and latency. Add one, measure, keep it only if the number moved.
Memory
Agents are often described as having memory. It is worth separating what that means, because three different mechanisms get the same name and they fail in different ways.
Within-task memory is the context, and lesson four covered it. The transcript, the scratchpad, the current plan. It lasts as long as the task and should be managed rather than accumulated.
Across-task memory is a store you build. Facts about a user, preferences, prior decisions, outcomes of previous tasks. The model has none of this unless you retrieve it and put it in the prompt, which makes it a retrieval problem with the properties of the previous course.
Which means the hard parts are retrieval's hard parts. What to store, how to find it when relevant, how to avoid filling the context with things that are not, and what to do when two stored facts conflict. The published work on simulated agents with long-term memory is explicit that retrieval and summarisation of memories, rather than storage, is where the difficulty lies.
Deciding what to remember is a judgement you should not delegate blindly. An agent that writes everything it encounters into memory produces a store full of transient detail. A short, explicit list of what is worth keeping - a stated preference, a confirmed fact, a decision and its reason - works better than a general instruction to remember.
Stale memory is worse than no memory. A remembered fact that has since changed will be used confidently. Timestamp everything, prefer recent over old, and re-verify anything consequential rather than trusting the store.
Memory is personal data in most systems. Preferences, history and inferences about a person are exactly what data protection law is about. Retention, access, correction and deletion all apply, and an agent memory store with no deletion path is a problem waiting to be raised.
Tell the user what is remembered. A visible, editable memory is better product and better compliance than an invisible one. It also gives you free correction, because users will fix what is wrong if they can see it.
Distinguish memory from configuration. A durable preference belongs in a settings record you can show and edit, not in a free-text memory the model may or may not retrieve. Teams discover this after a user asks why their stated preference was ignored.
And measure whether memory helps. It costs tokens on every call and adds a failure mode. On many tasks a well-written prompt and the current request outperform a memory store, and that is worth establishing rather than assuming.
How agents fail
The failure modes are consistent across systems and worth knowing by name, because each has a specific countermeasure.
The loop. The agent repeats the same action, or alternates between two, indefinitely. Countermeasure: detect repeated action signatures and intervene, either by injecting a note that the action has been tried or by terminating.
The plausible wrong turn. An action that looks reasonable and takes the task in the wrong direction, with every subsequent step compounding it. Countermeasure: verification at each step against the goal, and a cheap way to backtrack.
Compounding error. If each step is ninety-five per cent reliable, twenty steps are about a third reliable. This arithmetic is the central fact of agent engineering, and the answer is fewer steps, verification between steps, or both.
Goal drift. The agent ends up doing something adjacent to what was asked, usually after a long detour. Countermeasure: restate the goal in the prompt and compare the current action against it.
Premature completion. The agent declares success without having done the work, often because the output looks like a finished answer. Countermeasure: a completion check that is external - a test, a schema, a confirmation call - rather than the model's own assertion.
Tool misuse. The right tool with wrong arguments, or the wrong tool for the step. Countermeasure: better descriptions, enumerated parameters, validation inside the tool and an informative error.
Hidden failure. A tool returns an empty result, the agent proceeds as though it got an answer, and the final output is confident and unfounded. Countermeasure: tools that distinguish nothing found from an error, and an instruction about what to do with each.
Cost explosion. A task that should take five steps takes ninety, with the full context resent each time. Countermeasure: hard budgets, enforced in code, per task.
The injected instruction. Text the agent reads during the task steers it. Countermeasure: treat all tool output as data, keep consequential actions behind something the content cannot satisfy, and read the course on securing systems against agents.
And the organisational failure, which is the most common. Nobody owns the agent, nobody looks at the logs, and it degrades quietly while continuing to produce output that looks like work. Countermeasure: a named owner and a weekly number.
Cost and latency
An agent's economics are different in kind from a single call, and they are the usual reason a working prototype is not deployed.
The multiplier is the step count. A twenty-step task is twenty model calls, each carrying the accumulated context. The cost is not twenty times a single call; it is closer to the sum of a growing series, which is substantially worse.
So the context curve is the cost curve. Anything that keeps the context small - compact observations, summarised history, sub-tasks with fresh contexts - reduces cost more than switching to a cheaper model.
Latency is the step count times the per-step latency. Interactive agents are therefore limited to a handful of steps unless the user is prepared to wait. Streaming the reasoning gives the impression of progress and does not make it faster.
Mixed model use is the standard pattern. A cheaper model for routing, extraction, summarising observations and deciding whether a step succeeded; a stronger one for the decisions that need judgement. Measure the substitution on your own task rather than assuming it degrades.
Parallelise what is independent. Several read-only tool calls that do not depend on each other can be issued together. Many frameworks support this and many implementations do not use it, which leaves easy latency on the table.
Cache what repeats. The system instruction and the tool definitions are identical on every call of every task, which makes them ideal for prefix caching. Putting them first and keeping them stable is a direct saving on every step.
Set three budgets and enforce them. Tokens per task, steps per task, and currency per task if your provider exposes it. When a budget is hit, the agent should be told so it can summarise what it achieved, rather than being terminated silently.
Measure cost per completed task, not per call. The only number that matters is what it costs to finish one unit of real work, including the failures and retries. A system with a cheap per-call cost and a low completion rate is expensive.
Compare against the alternative honestly. The relevant comparison is not against a different model; it is against a workflow, a script, or a person. The critique paper on agent evaluation made exactly this point: reported accuracy without cost tells you nothing about whether a system is worth running.
And watch the long tail. The median task may be cheap and the ninety-ninth percentile fifty times worse. Budget for the tail, because that is what appears on the invoice.
Evaluating an agent
Evaluating a loop is harder than evaluating a call, because the same task can succeed by several routes and fail at any step. The methods here are what make an agent improvable rather than merely demonstrable.
Score the outcome, not the trajectory. Did the task get done, judged by something external: the record created, the file produced, the test passing, the field correctly populated. Trajectory scoring - did it take the steps we expected - penalises better routes and rewards imitation.
But record the trajectory. You need it for diagnosis: at which step did it go wrong, how many steps did it take, which tool misfired. Score outcomes; inspect trajectories.
Build a task set with verifiable completion. Twenty to fifty real tasks, each with a programmatic check for success. The checkability is the hard part and it is what makes the set useful, because it allows unattended runs.
Report cost alongside success, always. Success rate at a given budget, not success rate alone. This is the central argument of the published critique of agent evaluation: a system can reach a high score by spending unboundedly, and a figure without a cost is not a result.
Run each task several times. Agents are stochastic and a single run tells you little. Three to five runs per task, reporting the success rate and the variance, is the minimum to distinguish an improvement from noise.
Keep the public benchmarks in proportion. The published agent benchmarks - multi-environment suites, assistant tasks requiring tool use, and the repository-issue benchmark for code - are useful for comparing models and nearly useless for predicting performance on your workflow. They are a model-selection signal, not an evaluation of your system.
Watch for contamination and overfitting. A benchmark that has been public for a year may be in training data, and a task set you tune against stops measuring. Hold some tasks back and refresh the set from production failures.
Measure the failure categories from lesson seven. Count loops, premature completions, tool errors and budget exhaustions separately. These counts tell you what to fix; a single success rate does not.
Evaluate the hand-off too. Where the agent escalates to a person, measure how often it does so appropriately and how useful the assembled context is. An agent that escalates well is more valuable than one that escalates rarely.
And re-run before every change goes out. Model version, prompt, tool description, budget, planning structure. Each of these changes behaviour, and without a suite you are shipping on anecdote.
The person in the loop
Most successful deployments of agents are not autonomous. Deciding precisely where a person sits is a design decision with more effect on value than any model choice.
Propose rather than do, as a default. The agent prepares the action and a person approves it. The tedious part - gathering, drafting, checking, assembling - is where the time goes, and the judgement stays with somebody accountable.
Put the approval where the consequence is. Irreversible, outward-facing or high-value actions need a person. Reading, drafting and internal preparation generally do not. The classification is about the action, not about the agent.
Make the approval meaningful. A reviewer shown a conclusion will approve it. A reviewer shown the action, the evidence it rests on and what else was considered can actually review. If approvals are being granted in two seconds, the control is decorative.
Design for the reviewer's throughput. If an agent produces forty proposals an hour and a person can review eight, the system has moved the bottleneck rather than removed it. Either raise the confidence threshold for automatic handling or reduce the volume.
Escalation needs a good handover. What was asked, what was tried, what was found, what is uncertain and what the agent recommends. An escalation that says it could not complete the task wastes the work already done.
Let people correct, and capture the correction. Every edit a reviewer makes is a labelled example of what the agent got wrong. Systems that capture corrections improve; systems that discard them do not.
Be careful about automation complacency. A reviewer who approves a hundred correct proposals will approve the hundred and first without reading it. Periodic deliberate sampling, and occasional injected cases, are how aviation and medicine handle this and the same logic applies.
Tell the user they are dealing with software. Where an agent interacts with customers, saying so is increasingly a legal requirement as well as a decent one, and it changes expectations usefully.
And keep accountability with a person. A named owner for the agent, a record of who approved what, and a route to challenge an outcome. Where a decision affects somebody, the rules on automated decision-making may engage, and that is a design-time question.
Several agents
Multi-agent designs are fashionable and frequently unnecessary. There are cases where separation genuinely helps, and knowing them saves a great deal of complexity.
The honest default is one agent with good tools. Most tasks described as needing a team of agents are a single loop with a sensible tool set. Adding agents adds calls, context duplication, coordination failures and a debugging problem.
Separation helps when the contexts should differ. A researcher that reads widely and a writer that only sees the findings keeps the writer's context clean. That is a real benefit and it is the most defensible reason to split.
It helps when the privileges should differ. An agent that reads untrusted material and an agent that can act, with a deliberate boundary between them, is the containment pattern from the security course. Here the separation is a control rather than an efficiency.
It helps when the work is genuinely parallel. Twenty independent documents to analyse is twenty concurrent single-agent tasks, which is parallelism rather than collaboration and is the easy case.
It helps when a verifier should be independent. One agent produces, another checks against criteria it can evaluate. This works where the check is concrete and adds little where the verifier is simply agreeing in different words.
The patterns worth naming. A supervisor delegating to specialists with a fixed set of roles. A pipeline where each stage is an agent with its own tools. Parallel workers with an aggregator. Each is a workflow with agents inside it, which is the arrangement that actually works.
What does not work well. Open-ended conversation between peers until they agree. It is expensive, it converges slowly, it produces confident consensus on wrong answers, and it is almost impossible to debug.
Coordination costs are real. Each hand-off loses context, and information passed as free text between agents degrades. Pass structured state rather than prose, and keep the shared state in your code rather than in a conversation.
Debugging gets much harder. A failure in a five-agent system requires reconstructing five transcripts and the messages between them. Trace with a shared identifier through every call, or accept that you will not find the cause.
And the test to apply. Could this be a workflow with one model call per stage? If yes, build that. Multi-agent systems earn their place on tasks where the branching is genuinely unknown, and nowhere else.
Running one in production
An agent in production is a system that takes actions continuously on the basis of a component you did not build. The operational requirements follow from that.
Trace every task end to end. One identifier through every model call, tool call and hand-off, with the prompt, the arguments, the result and the tokens at each step. This is not optional instrumentation; it is the only way to answer what happened.
Keep the numbers that matter visible. Completion rate, steps per task, tokens per completed task, escalation rate, budget exhaustions, loop detections and tool error rates by tool. A weekly glance at those tells you whether the system is healthy.
Watch for silent degradation. A provider model update, a changed tool response format, a corpus that grew, a new category of request. The symptom is usually a slow fall in completion rate or a slow rise in steps per task, neither of which produces an alert unless you built one.
Version everything and record it with the task. Prompt, tool definitions, model version, budgets, planning configuration. A task complained about three weeks later cannot be reproduced otherwise.
Have a switch. The ability to turn the agent off, or to put it into propose-only mode, without a deployment. Every team that has operated one has wanted this at short notice.
Rate-limit the agent against your own systems. An agent with an interface credential can generate a load pattern no human workflow would. Limits on its calls protect your systems from your own software.
Review a sample of completed tasks by hand. Weekly, a handful, read properly. Metrics tell you about the distribution; reading tells you about the kind of mistakes being made, which is what metrics never show.
Treat the tool set as the change surface. Most behaviour changes come from tool descriptions and tool behaviour rather than from the prompt. Review changes to them as carefully as code, because they are code.
Plan the incident. What you do when an agent has taken a hundred wrong actions: how to find them, how to reverse them, who is told. Deciding this in advance is the difference between an afternoon and a week.
And keep the scope small and the owner named. The agents that survive in production are narrow, well-instrumented and owned. The ones that disappear are broad, autonomous and nobody's.
Deciding not to build one
The most valuable judgement in this subject is recognising when an agent is the wrong answer. This lesson is the checklist, stated plainly.
If you can enumerate the steps, build a workflow. Deterministic orchestration with a model call at the two or three points that need language understanding. Cheaper, faster, testable, auditable, and it will outperform an agent on the task you specified.
If the task is one decision, make one call. Classification, extraction, routing and summarising are single calls. A loop around them adds latency and failure modes and buys nothing.
If the answer is a calculation over structured data, write a query. An agent reasoning over records will produce confident arithmetic. A database query will produce the answer.
If the consequences are irreversible and the volume is low, use a person. The economics of automation depend on volume. Ten high-stakes decisions a week do not justify a system whose failure mode is a wrong action taken quickly.
If you cannot verify completion, be very cautious. An agent without an external signal of success cannot be evaluated, cannot be improved systematically, and will report success either way. Find a verifiable form of the task or reconsider.
If the task requires completeness, retrieval and loops will not give it. Every obligation in a contract, every instance in a corpus. These need an exhaustive pass over the material, which is a pipeline.
If nobody will own it, do not deploy it. An unowned agent degrades and keeps producing output. This is the most common way these systems end up causing harm, and it is entirely organisational.
Where an agent genuinely is right. Unknown number of steps, branching that depends on findings, exploration until something is located, and a verifiable end state. Investigation, triage, reconciliation of discrepancies, and multi-system lookups where the next lookup depends on the last.
And the staged route is almost always correct. Start with the workflow. Add a model where language understanding is needed. Add tools. Add a loop only when you have evidence that the fixed sequence cannot cover the cases that matter. Teams that start with an agent spend their first quarter rebuilding it as a workflow.
One agent, end to end
A finance team spends a great deal of time on one task: a supplier payment arrives that does not match any open invoice, and somebody has to work out what it is. Several systems, an unknown number of lookups, a verifiable end state. This is the shape that justifies an agent.
The task as a person does it. Check the amount against open invoices, including partial payments. Check the remittance reference against invoice numbers and purchase orders. Search the supplier's recent history for a credit note. Look for two invoices summing to the amount. If still unmatched, check whether the supplier has another account. Then either match it, or write a note explaining what was found and pass it on.
Why a workflow was tried first. The team built the obvious sequence of checks as a pipeline. It resolved about 55 per cent of cases, and the remainder were exactly the ones where the next check depended on what the last one found. That evidence - a measured ceiling on the fixed sequence - is what justified the loop.
The tools. Six, each at the level of a task: find invoices by amount and tolerance; find by reference with fuzzy matching; supplier history for a period; find combinations of invoices summing to an amount; list related supplier accounts; and record a match or escalate with a note. All read-only except the last, which validated everything inside the tool and refused a match outside tolerance.
The budgets. Twelve steps, forty thousand tokens, and a wall-clock limit. Observations capped, with pagination inside the tools rather than truncation outside them.
The verifiable end state. A match is checkable: the amounts reconcile. That made the task set programmatically scorable, which made everything else possible.
The evaluation. Eighty historical unmatched payments with known correct resolutions, including nine that genuinely had no match. Each run three times. The first version resolved 71 per cent correctly, escalated 18 per cent and matched 11 per cent wrongly, which was the unacceptable number.
What fixed the wrong matches. Not a better model. The combination tool was returning any set of invoices summing to the amount, including implausible sets of seven, and the agent accepted them. Restricting it to combinations of two or three, and requiring the dates to be within a window, took wrong matches to 2 per cent.
What the second iteration added. A tolerance that could not be exceeded, enforced in the recording tool. A repeated-action detector, after a case looped between two lookups eleven times. And an escalation note template, because the first version's notes were not useful to the person receiving them.
Where it settled. 84 per cent resolved, 14 per cent escalated with a useful note, 2 per cent wrong, at a cost per resolved payment of roughly a twentieth of the staff time it replaced. Every wrong match was reversible, and the recording tool wrote an audit entry naming the agent and its version.
What the team said afterwards. The hard parts were the tool that returned implausible combinations, the observation sizes, and getting the escalation note right. The model was changed once, late, and it made less difference than the combination fix.
Fifty questions
Answer all fifty questions, then check your answers. You need 45 out of 50 to pass and receive a certificate. If you score less, you will see which answers were right and wrong, and then go through the course again before you retake the check. Your answers, progress and times are kept only in this browser.
Your answers
Your certificate of completion
Enter your name as you want it to appear, then save the certificate as a PDF. In the print window, choose Save as PDF. A certificate is issued once per completion of the course.
Rate this course
How would you rate this course? Your rating is anonymous — just a number, with no name and no account.
Bring this to your team
Your colleagues can take this course too — it is free, needs no sign-up, and nothing they enter leaves their browser. If your institution wants the same ground covered as structured training for a team, with an instructor and your own cases, we run courses in a classroom or online for groups of ten or more.
Talk to us about training your team → See the paid courses →
Saolix does not record who takes this course, so it cannot verify these certificates. The certificate confirms completion of a free self-paced course and is not an accredited qualification.
Standards and frameworks covered
The published frameworks this course teaches toward. Saolix is not affiliated with, endorsed by or accredited by any of these bodies; naming a standard is a citation, not a claim about its author.
Sources
The official documents this course relies on. Laws and guidance change, so check the current version.
- ReAct: Synergizing Reasoning and Acting in Language Models · Yao and others, arXiv 2210.03629
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models · Wei and others, arXiv 2201.11903
- Toolformer: Language Models Can Teach Themselves to Use Tools · Schick and others, arXiv 2302.04761
- Reflexion: Language Agents with Verbal Reinforcement Learning · Shinn and others, arXiv 2303.11366
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models · Yao and others, arXiv 2305.10601
- Generative Agents: Interactive Simulacra of Human Behavior · Park and others, arXiv 2304.03442
- A Survey on Large Language Model based Autonomous Agents · Wang and others, arXiv 2308.11432
- AgentBench: Evaluating LLMs as Agents · Liu and others, arXiv 2308.03688
- GAIA: a benchmark for General AI Assistants · Mialon and others, arXiv 2311.12983
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? · Jimenez and others, arXiv 2310.06770
- AI Agents That Matter, on cost-aware evaluation and reproducibility · Kapoor and others, arXiv 2407.01502
- Agentic AI - Threats and Mitigations · OWASP GenAI Security Project
- AI 600-1: Artificial Intelligence Risk Management Framework - Generative AI Profile · US National Institute of Standards and Technology (NIST)



