Leaving this course?

Lesson 1 of 14

One operation, repeated

A language model does one thing. Given a sequence, it produces a probability distribution over what could come next. Everything a system built on it appears to do is that operation, repeated.

The unit is a token, not a word. The model's vocabulary is a fixed list of pieces of text, typically tens of thousands of them. Input text is cut into those pieces, and the output is a score for every piece in the list.

The output is a distribution, not an answer. For each position the model produces a number for every token in its vocabulary, and those numbers are turned into probabilities that sum to one. Something outside the model then chooses one token from that distribution, appends it to the sequence, and runs the model again.

Which means generation is a loop with a choice in it. The model is deterministic given the same input and weights; the variation you see comes from the selection step, which is a separate piece of software with its own settings. This distinction causes more confusion than any other in the subject, and lesson five is about it.

Nothing in the operation refers to truth. The model was trained to assign high probability to text that resembles its training data. A sentence that is fluent, well-formed and wrong is exactly what the objective rewards, and no part of the architecture checks a claim against anything.

There is no memory between calls unless you supply it. The model sees the sequence it is given and nothing else. What looks like a conversation is the whole conversation being resent each time, which is why the context window in lesson six is a budget rather than a background detail.

The same operation produces every behaviour. Translation, summarising, classification, writing code, answering a question, following an instruction. None of these is a separate capability; each is a sequence whose likely continuation happens to be the thing you wanted.

This is why prompting works at all. Changing the beginning of a sequence changes the distribution over its continuation. A prompt is not a command to a system that understands commands; it is a context chosen to make the desired continuation the likely one.

And it is why the failures have the shape they have. A model that cannot count characters, invents a citation, loses an instruction in a long document, or produces different answers to the same question is not malfunctioning. Each of those follows directly from predicting tokens one at a time from a distribution, and the rest of this course is that sentence worked out in detail.

Lesson 2 of 14

How text becomes tokens

Tokenisation is the least glamorous part of the subject and the source of a surprising number of practical problems. It is worth understanding properly once.

The naive options both fail. One token per word gives a vocabulary that is enormous, incomplete and helpless with anything unseen. One token per character gives a tiny vocabulary and sequences so long that the model spends its capacity on spelling.

So models use subword pieces. Common words become single tokens; rare words are built from fragments. The vocabulary is learned from a corpus, not designed by hand, and it reflects the statistics of that corpus.

Byte pair encoding is the usual method. Start from individual characters or bytes, count adjacent pairs, merge the most frequent pair into a new symbol, and repeat until the vocabulary reaches the target size. The published method for neural translation adapted an older compression algorithm, and variants of it are behind most current tokenisers.

Bytes rather than characters, in most modern designs. Working on raw bytes means any text can be represented without an unknown-token escape, at the cost of more tokens for scripts whose characters take several bytes.

The consequences are uneven and worth knowing. English prose is efficient, often close to one token for every three or four characters. Text in a language poorly represented in the training corpus costs considerably more tokens for the same content, which means a longer sequence, a higher price and less room in the window for the same meaning. That is a fairness issue as well as an engineering one.

Numbers and code tokenise awkwardly. A long number may split into pieces that have nothing to do with its digits, which is part of why arithmetic is unreliable. Code tokenises better than you might expect, because whitespace and common identifiers are frequent in the corpus.

Names, identifiers and misspellings fragment. An unfamiliar name may become four or five tokens, and a typo produces a completely different token sequence from the correct spelling, which is why small input errors sometimes have outsized effects.

And the model cannot see letters. Asking how many times a letter appears in a word asks the model to reason about something it was never given: the word arrived as one or two opaque tokens. The failure is a representation problem, not a reasoning problem.

Practical consequence for anybody building. Count tokens, do not estimate from word counts; measure in the languages you actually serve; and never assume a character-level operation will work just because it is easy for a person.

Lesson 3 of 14

Embeddings, where meaning enters

A token is an integer, and an integer carries no information about what it means. The first thing the model does is replace it with a vector, and that step is where meaning starts.

An embedding is a list of numbers. Each token in the vocabulary has one, learned during training, typically with hundreds or thousands of dimensions. The model never works with the token identifier again; it works with the vector.

Similar tokens end up with similar vectors. Nothing told the model that two words were related. Tokens appearing in similar contexts acquire similar vectors because that is what minimises prediction error, and relatedness emerges as a by-product.

Direction matters more than distance. Comparison is usually by cosine similarity, the angle between two vectors, rather than by how far apart they are. This is the measure used in retrieval, and lesson nine of the retrieval course depends on it.

A token embedding is context-free; the layers make it contextual. The vector for a word is the same wherever it appears. What distinguishes the two senses of a word is what the model does with the vector afterwards, using the surrounding tokens, which is the subject of the course on attention.

Position has to be added separately. A set of vectors has no order. Information about where each token sits in the sequence is injected deliberately, because otherwise the model cannot distinguish one arrangement of the same words from another.

Sentence and document embeddings are a different thing. A single vector representing a whole passage is produced by a model trained for that purpose, and it is what retrieval systems store and compare. Good embeddings for retrieval and good embeddings inside a generative model are not the same objects and are not interchangeable.

Dimensionality is a trade. More dimensions capture more distinctions and cost more to store and compare. Retrieval systems holding millions of vectors feel this directly, which is why reduced-dimension and quantised variants exist.

Embeddings carry the training data's associations. Including the unwanted ones. Stereotypical associations present in a corpus appear in the geometry, and no amount of careful prompting removes something that is built into the representation.

And they are not secret. An embedding is derived from text and retains a great deal about it. Treating a vector store as though it were anonymised data is a mistake with data protection consequences, because text can often be substantially reconstructed from embeddings.

Lesson 4 of 14

Logits, probabilities and what they mean

The model's actual output is a vector of raw scores. Understanding what happens to those scores, and what they do and do not tell you, is the difference between using a model and guessing at it.

Logits are unnormalised scores. One number for every token in the vocabulary, for each position. They are not probabilities; they are inputs to the function that produces probabilities.

Softmax converts them. Exponentiate each score and divide by the total, which produces positive numbers summing to one. Because the exponential amplifies differences, a small gap in logits becomes a large gap in probability, and this sensitivity is what temperature manipulates.

Log-probabilities are what you actually inspect. Providers expose them because they are numerically well behaved and additive: the log-probability of a sequence is the sum of the log-probabilities of its tokens. That additivity is useful and also a trap, because longer sequences accumulate lower totals and need normalising before comparison.

A high probability is a statement about text, not about the world. The model is saying this continuation is likely given the training distribution. A confidently stated falsehood can carry a high probability, because fluent false text is common in the corpus and the objective never consulted reality.

Calibration is the real question, and it is partial. A well-calibrated model assigns probabilities that match observed frequencies. Large models are better calibrated than small ones on some tasks and poorly calibrated after certain kinds of fine-tuning, which can make a model more confident without making it more accurate.

So token probabilities are a weak signal, usefully applied. Low probability on a critical token is worth noticing. A distribution spread across several plausible answers indicates genuine uncertainty. Neither is a fact-check, and a system that routes low-confidence outputs to a person is using the signal correctly.

For classification, read the distribution rather than the text. If you want one of five labels, inspecting the probability mass on each label token is more reliable and cheaper than parsing a sentence, and it gives you a confidence number for free.

Constrained decoding uses the same mechanism. Forcing valid output - a choice from a list, well-formed structured data - is done by zeroing the probability of tokens that cannot come next. This is a better way to get a valid format than asking politely in the prompt.

And the vocabulary limits what can be expressed. A token the model does not have cannot be produced, and anything outside the vocabulary has to be assembled from pieces, which is why rare strings and unusual identifiers are generated less reliably.

Lesson 5 of 14

Choosing the next token

The model gives you a distribution. Something has to pick from it, and that choice is a separate piece of software with settings that change behaviour more than most prompt edits do.

Greedy decoding takes the highest-probability token every time. Deterministic, and prone to flat, repetitive text. It is the right default for classification and extraction and the wrong one for anything that should read naturally.

Sampling draws from the distribution. Each token is chosen with its own probability, which produces variety and occasionally nonsense, because a token with a small probability is still sometimes chosen.

Temperature reshapes the distribution before sampling. Divide the logits by a number: above one flattens the distribution and increases variety; below one sharpens it towards the most likely tokens; at zero the choice is effectively greedy. Temperature does not make the model more creative or more accurate, it changes how much of the tail is reachable.

Top-k truncates to a fixed number of candidates. Keep the k most likely tokens and renormalise. Simple, and badly behaved when the distribution is very peaked or very flat, because k is fixed and the distribution is not.

Nucleus sampling truncates by probability mass instead. Keep the smallest set of tokens whose probabilities sum to a threshold, then sample from it. The published motivation was that a fixed cut-off produces degenerate text, and this method adapts the candidate set to how confident the model is at each step.

Repetition and presence penalties discourage reuse. They reduce the score of tokens already produced. Useful against loops, and heavy-handed in technical text where a term legitimately recurs.

Beam search optimises the whole sequence. It keeps several partial sequences and extends the most likely. Valuable in translation, where there is one best answer, and it tends to produce dull, generic text in open-ended generation.

Determinism is not fully available. Fixing the sampling parameters removes one source of variation, and the same prompt may still give different outputs because of batching, hardware arithmetic ordering and provider-side changes. Build systems that tolerate variation rather than depending on its absence.

And stop conditions are part of the configuration. Maximum output tokens, stop sequences and the model's own end-of-sequence token. Most truncated outputs in production are a maximum-token setting somebody forgot, not a model limitation.

The practical summary. For extraction, be deterministic and constrain the output. For drafting, sample with a modest temperature and a nucleus threshold. And record the settings with the output, because an answer without its decoding parameters is not reproducible even in principle.

Lesson 6 of 14

The context window as a budget

The context window is the number of tokens the model can take into account at once. Treating it as a budget rather than a limit is the difference between a system that works and one that degrades mysteriously.

Everything shares the same space. The system instruction, the conversation so far, retrieved documents, tool definitions, examples and the output all come from one allowance. Adding retrieved context removes room for the conversation, and nothing warns you.

There is no memory outside it. A model has no recollection of earlier calls. Continuity is produced by resending material, which means a long conversation costs more on every turn and eventually has to drop something.

What gets dropped is a design decision. Truncating the oldest turns, summarising them, or keeping a running set of facts are all choices with different failure modes. Silent truncation is the worst of them, because the model then answers as though the missing material never existed.

Position within the window matters. Published work on long contexts has found that material in the middle of a long input is used less reliably than material at the beginning or the end. Put instructions and the most important evidence where they will be read.

Long windows are not free. Attention cost grows with the square of sequence length, so a much longer input costs disproportionately more and takes longer. A system that routinely fills a very large window is usually one that has not been asked to select.

Caching changes the arithmetic. Where a provider supports reusing the processed prefix of a prompt, a stable system instruction followed by variable content is substantially cheaper than the reverse. That is an argument for ordering the prompt with the fixed parts first.

Count tokens, do not estimate them. Use the tokeniser for the model you are calling. Estimates from word counts are wrong in exactly the cases that matter: non-English text, code, tables and long identifiers.

Leave room for the answer. The output comes out of the same budget. A request that fills the window with input leaves nothing to generate into, and the result is a truncated response rather than an error.

And measure what you actually send. In most systems that go wrong, the context assembled at run time is substantially different from what the developer believes it to be. Logging the exact assembled prompt, with token counts by component, is the single most useful piece of instrumentation in this subject.

Lesson 7 of 14

Why it is confidently wrong

Fabrication is not a defect bolted onto an otherwise truthful system. It is what the training objective produces, and understanding why tells you which mitigations can work.

The objective rewards plausibility. Training maximises the probability of the next token in text that already exists. A continuation that looks right in form is what the objective is about, and there is no term in it for whether a statement corresponds to anything.

So the error mode is fluent and specific. A fabricated citation has plausible authors, a plausible journal and a plausible year, because that is what citations look like. The surface features are exactly the ones the model learned.

It is worst where the training data was thin. Questions about obscure entities, recent events, small jurisdictions or private facts have no well-supported continuation, and the model produces the shape of an answer instead. Published surveys of the problem distinguish fabrication that contradicts the source material from fabrication that merely goes beyond it, and the second is more common and harder to catch.

Asking for a source does not help by itself. The reference is generated by the same process as the claim. Only a source the system actually retrieved and can show is evidence, which is the argument for retrieval and for citations that resolve.

Nor does asking the model to check its own work, on its own. A second pass with the same model on the same context will often repeat the error, because the same distribution produced it. Independent evidence, a different model, a tool or a person adds something; a rephrased self-review frequently does not.

Confidence is a property of the text, not of the knowledge. Models are trained on writing that states things plainly, so hedging is under-represented. An output's assertive tone carries no information about its reliability, and users read it as though it does.

Calibration helps where it exists. Token probabilities on the decisive part of an answer carry some signal, and routing low-confidence cases to review is a reasonable use of it. It is a triage mechanism and not a verifier.

The controls that actually reduce it. Retrieval with citation. Constrained output formats. Tools for anything computable, such as arithmetic and dates. Refusal paths for questions outside the supplied material. And a product design that shows the evidence rather than only the conclusion.

And the one that matters most is scope. A system asked to answer from supplied documents and to say when they do not contain the answer fabricates far less than one asked to answer from everything it has ever read. Narrowing the question is the cheapest available mitigation.

Lesson 8 of 14

Counting the cost

Token arithmetic decides whether a feature is viable. It is simple arithmetic, it is almost never done before building, and it is the most common reason a working prototype cannot be deployed.

Input and output are priced separately. Output is usually the more expensive per token and the smaller quantity; input is cheaper and, in retrieval-heavy systems, vastly larger. The ratio between them determines where optimisation effort pays.

Build the per-call model explicitly. Fixed instruction, variable user input, retrieved context, examples, tool definitions, and expected output. Multiply by the call volume and by the number of calls per user action, which in an agent is not one.

Latency is driven by output length. Tokens are produced one at a time, so a long answer takes proportionally longer to finish. Processing the input is comparatively fast and largely parallel. If a feature feels slow, the question is usually how much it is writing, not how much it is reading.

Streaming changes perception, not cost. Showing tokens as they arrive makes a response feel immediate while the total time is unchanged. It is a user-experience decision with no effect on the bill.

Caching a stable prefix is the largest easy saving. Where supported, the processed form of an unchanged prompt prefix can be reused across calls. It rewards putting the fixed instruction and examples first and the variable content last, which is the opposite of how prompts are usually assembled by accretion.

Model choice is a cost lever with a quality cost. A smaller model for classification, extraction and routing, and a larger one only where judgement is needed, is the standard pattern. Measure the quality difference on your own task rather than on a published benchmark.

Retries and failures are part of the cost. A system that retries on malformed output pays twice for the worst cases. Constrained decoding reduces both the error rate and the spend.

Batching helps throughput, not individual latency. Where work is not interactive, processing many items together is substantially cheaper per item, and several providers price asynchronous batches lower.

Set a budget per request and enforce it. A maximum token count for input and output, enforced in code. Without it, one unusual input - a pasted document, a long conversation, a loop in an agent - produces a bill nobody predicted.

And log tokens as a first-class metric. Tokens in and out per request, by component and by feature. Teams that log only latency and errors discover their cost problem in an invoice.

Lesson 9 of 14

What this means for prompting

Most prompting advice is folklore. The parts that hold up follow from the mechanics in the previous lessons, and they are worth separating from the parts that do not.

Put the stable material first. It reads better, it caches better, and the instruction is in the position the model attends to most reliably. A prompt that ends with the instruction and begins with a large document is the harder arrangement for both cost and attention.

Delimit sections unambiguously. Clear markers between instruction, document and question reduce the chance that content is read as instruction. This matters for correctness and it is also the first line against injected text, which the course on securing systems against agents covers properly.

Ask for a format you can parse, and constrain it. Describing the format in the prompt helps; enforcing it at the decoding step is what makes it reliable. A system that parses free text and hopes will fail on the inputs you did not test.

Examples work, and they cost tokens. A small number of well-chosen examples usually improves format adherence and edge-case handling more than a longer description. The published few-shot result is about exactly this, and the practical limit is the budget.

Be explicit about what to do when the answer is not available. A model asked to answer will answer. A model told to reply with a specific marker when the supplied material does not contain the answer will frequently do so, and that single instruction removes a large class of fabrication.

Ask for the work where the work matters. Requesting the intermediate steps before the conclusion improves accuracy on multi-step problems, because each step conditions the next. It also costs output tokens and slows the response, so it belongs where the task needs it rather than everywhere.

Avoid asking for things the representation cannot support. Counting characters, reversing strings, exact arithmetic on long numbers, or precise date differences. Use a tool. This follows directly from tokenisation rather than from any limitation of reasoning.

Keep prompts in version control and test them. A prompt is code. It has inputs, behaviour and regressions, and a change to it should be reviewable and revertible in the same way.

And measure rather than argue. Almost every prompting question - does this phrasing help, does this example earn its tokens, is the shorter instruction as good - is answerable in an afternoon with a set of cases and a score. The alternative is a team with opinions and no evidence.

Lesson 10 of 14

The limits you will actually meet

These are the failures that appear once real traffic arrives. None is subtle and all of them are cheaper to design for than to discover.

Context limit exceeded. A long conversation, a pasted document or an accumulating agent transcript crosses the window. Handle it deliberately: measure before sending, and decide whether to summarise, truncate from the oldest, or refuse with a clear message.

Output truncated at the maximum. The response stops mid-sentence because the output cap was reached. This is the most common production fault in text generation and it is a configuration value, not a model problem. Check whether the response ended because it finished or because it ran out.

Rate and concurrency limits. Requests per minute, tokens per minute, and concurrent calls. A feature that works for one user fails for a hundred. Back off, queue, and treat these as capacity planning rather than as errors.

Non-determinism in a test suite. Tests that assert exact output strings fail intermittently. Assert on structure, on constraints and on scores against a threshold instead.

Degradation in the middle of long inputs. Material in the centre of a long context is used less reliably. Where evidence matters, select and place it rather than supplying everything and hoping.

Prompt drift. The prompt is edited repeatedly without evaluation until nobody can say why a clause is there. Keep a case set, score changes against it, and remove clauses that do not earn their place.

Model updates change behaviour. A provider's model is a moving target. Pin a version where that is offered, keep a regression set, and re-run it before changing version. A system with no regression set cannot adopt an improvement safely.

Knowledge cut-offs and silent staleness. The model knows nothing after its training data ends and will not say so unprompted. Anything time-sensitive needs retrieval or a tool, and the system should state the date basis of its answer.

Cost spikes from loops. An agent that retries, re-reads and re-plans can consume an unbounded number of tokens for one request. A hard ceiling on calls and tokens per request is the control.

And multilingual cost and quality asymmetry. The same content costs more tokens and is handled less well in languages that were thinly represented in training. If you serve such a market, measure it there rather than assuming the English result transfers.

Lesson 11 of 14

How the model got this way

You do not need to train a model to use one well, but you do need to know what training did, because it explains the behaviour you will be arguing about.

Pre-training is next-token prediction at scale. A very large corpus of text, a very large network, and one objective: predict the next token. No labels, no task definitions, no human instructions. Everything the model appears to know was acquired as a side effect of getting better at that prediction.

Capability emerges from the objective rather than being specified. Translation, summarising and arithmetic were not trained as tasks. They are present because text containing them was present, and the model that predicts such text well has to represent something about how they work.

Scale behaves predictably, within limits. Published work established that loss falls smoothly as model size, data and compute increase, following regular relationships over several orders of magnitude. That predictability is why the field pursued scale: the result of the next larger run was forecastable.

Then the balance was corrected. Later work on compute-optimal training showed that the models of the time were substantially larger than their training data justified, and that for a fixed compute budget, model size and the number of training tokens should be scaled in roughly equal measure. The practical consequence was a generation of smaller models trained on far more data, performing better and costing less to serve.

Data quality turned out to matter as much as quantity. Deduplication, filtering and curation change outcomes significantly. A corpus is not a neutral sample of the world; it is a sample of what was written down, digitised, accessible, and chosen - which is where a model's blind spots and its assumptions come from.

Cut-off dates are structural. The corpus ends at a point, and after that the model knows nothing. It will not volunteer this, and it will answer questions about later events with the shape of an answer. Any time-sensitive system needs retrieval, a tool, or an explicit statement of the basis of its knowledge.

Context length is partly a training decision. A model is trained at a certain sequence length and extended by various techniques. Performance at the extreme end of a very long advertised window is frequently worse than at moderate lengths, which is why the advertised number is a ceiling rather than a promise.

Nothing in pre-training made the model helpful. A purely pre-trained model continues text; it does not follow instructions, answer questions or decline anything. Everything that makes a model usable as an assistant comes afterwards, which is the next lesson.

Lesson 12 of 14

After pre-training

The model you call is not the model that came out of pre-training. Several further stages shaped it, and each one explains a category of behaviour you will encounter.

Supervised fine-tuning teaches the format of being useful. A smaller, curated set of examples - an instruction and a good response - trains the model to respond to requests rather than to continue text. This is where the assistant shape comes from.

Preference training optimises for what people chose. Human raters compare candidate responses, and the model is trained towards the preferred ones, through reinforcement learning from feedback or one of the more direct preference-optimisation methods. The objective becomes approximating human preference rather than reproducing a corpus.

Which introduces its own distortions. Preferred responses tend to be longer, more structured, more agreeable and more confident, because raters prefer those qualities. Verbosity, over-hedging on safe topics and a tendency to agree with the user are artefacts of the optimisation, not accidents.

Alignment and refusal behaviour are trained in. What a model declines, how it declines, and the disclaimers it adds are the product of this stage. They are therefore inconsistent at the margins, adjustable by the provider between versions, and not a property of the underlying capability.

Calibration can get worse. A model that was reasonably calibrated after pre-training may become more confident without becoming more accurate, because confident answers were preferred. This is why probability signals should be validated on your own task rather than trusted generically.

Adaptation to your domain has several options. A prompt with examples is the cheapest and most flexible. Retrieval supplies knowledge without changing weights and keeps it current. Fine-tuning changes the model and is appropriate for a consistent format, tone or narrow task, and is a poor way to install facts. Lightweight methods that train a small number of additional parameters make fine-tuning affordable.

Choose by what is actually wrong. If the model lacks facts, retrieve. If it will not hold a format, constrain the decoding. If it misunderstands a specialist vocabulary consistently, fine-tuning may be the answer. Fine-tuning to fix a problem that was a prompt problem is the most expensive mistake available in this subject.

Evaluation has to come before any of it. Each option costs money and time, and without a measured baseline there is no way to tell whether a change helped. The next lesson is about building that measurement, because it is the step teams skip and then regret.

Lesson 13 of 14

Measuring whether it works

Generation is hard to evaluate because there is rarely one right answer. That difficulty is why most teams do not evaluate at all, and why the ones that do move faster.

Start with a case set, not a metric. Thirty to a hundred real inputs, with what a good output would contain. Drawn from actual use where possible, including the awkward cases. This artefact is worth more than any choice of scoring method.

Score what the task actually requires. For extraction, exact or field-level accuracy. For classification, the usual measures and a confusion matrix. For retrieval, whether the right passage was found. For free text, a rubric: does it answer the question, is it grounded in the source, does it avoid claims the source does not support, is the format right.

Reference-overlap metrics are weak for generation. Measures of word overlap with a reference answer penalise good paraphrases and reward bad ones. They are usable as a cheap regression tripwire and poor as a measure of quality.

A model can score outputs, with discipline. Using a model as a judge works reasonably where the criterion is concrete and the rubric is explicit. It has known biases: towards longer answers, towards its own style, and towards the first option presented. Mitigate by randomising order, using a different model from the one under test, and checking agreement against human scores on a sample.

Humans score the sample that matters. A small number of carefully judged cases anchors everything else. Use clear criteria, more than one rater on a subset, and measure their agreement; if raters do not agree, the rubric is the problem.

Separate the components. In a retrieval system, measure retrieval and generation separately, or you will tune the wrong one. Most quality complaints in such systems are retrieval failures presented as generation failures.

Run it as a regression suite. Every prompt change, model version, parameter change and retrieval change runs against the case set, with the scores recorded. This is what makes it safe to accept an upgrade or revert a change.

Watch cost and latency alongside quality. A change that improves a score and triples the spend is a decision, not an improvement, and the figures should be in the same table.

Add the failures you find in production. Every real complaint becomes a case. A suite that grows from live failures converges on the things that actually matter, which no benchmark will do for you.

And keep a small set you never tune against. Held-out cases, looked at occasionally, to tell you whether you have improved the system or merely fitted the suite.

Lesson 14 of 14

One feature, end to end

A team is asked to summarise incoming customer emails into a short structured note for a support queue. It is a small feature and it exercises everything in this course.

What was asked for. For each email, a summary of two or three sentences, a category from a fixed list of nine, an urgency rating, and the customer's specific request as a single line. Eight thousand emails a day.

Step one, measure the input. A sample of two thousand emails showed a median of 310 tokens and a ninety-fifth percentile of 1,900, with a maximum above 20,000 where a long thread had been forwarded. The maximum, not the median, is what sizes the system.

Step two, write the budget. Fixed instruction and category definitions, around 600 tokens. The email, capped at 4,000 with the oldest quoted material dropped first. Output, capped at 200. That gives a predictable worst case and a known average, and the cap decision was recorded with its reason.

Step three, choose the decoding. Deterministic, with the four fields produced as constrained structured output rather than parsed from prose. The category is selected from the nine permitted values at the decoding step, so an invalid category is not possible rather than merely unlikely.

Step four, build the case set before building the feature. One hundred emails, each with a human-written target: the correct category, a reasonable urgency, and the key points a summary had to contain. Two people rated a subset and compared, which changed the urgency definition twice before any code was written.

Step five, choose the model by measurement. The smaller model reached 94 per cent category accuracy against the set; the larger one reached 96. The difference was concentrated in two categories, which were the ones the team had defined ambiguously. Fixing the definitions brought the smaller model to 96, and it was the one deployed, at a fifth of the cost.

Step six, order the prompt for caching. The instruction and category definitions were put first and the email last, so the processed prefix could be reused. Measured saving on the input side was substantial, and it required no change to behaviour.

Step seven, instrument it. Tokens in and out per call, by component. Finish reason on every response. Category distribution per day. A daily sample of twenty summaries scored against the rubric.

What went wrong in the first week. Forwarded threads hit the 4,000-token cap and the summary described the oldest message rather than the newest, because the truncation kept the wrong end. The fix was to truncate from the top of the thread rather than the bottom, found by looking at the finish-reason and token logs.

What it ended up costing. Under the original estimate, because the cap bounded the worst case and the smaller model did the work. The cost model written in step two was within ten per cent of the first month's actual spend.

The point. Nothing in this feature required a novel technique. It required counting tokens before building, constraining the output instead of parsing it, writing the case set first, and logging enough to find a truncation bug in a week rather than a quarter.

Knowledge check

Fifty questions

Answer all fifty questions, then check your answers. You need 45 out of 50 to pass and receive a certificate. If you score less, you will see which answers were right and wrong, and then go through the course again before you retake the check. Your answers, progress and times are kept only in this browser.

Standards and frameworks covered

The published frameworks this course teaches toward. Saolix is not affiliated with, endorsed by or accredited by any of these bodies; naming a standard is a citation, not a claim about its author.

Sources

The official documents this course relies on. Laws and guidance change, so check the current version.

  1. Neural Machine Translation of Rare Words with Subword Units, the byte pair encoding method · Sennrich, Haddow and Birch, arXiv 1508.07909
  2. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing · Kudo and Richardson, arXiv 1808.06226
  3. The Curious Case of Neural Text Degeneration, which introduced nucleus sampling · Holtzman, Buys, Du, Forbes and Choi, arXiv 1904.09751
  4. Scaling Laws for Neural Language Models · Kaplan and others, arXiv 2001.08361
  5. Training Compute-Optimal Large Language Models · Hoffmann and others, arXiv 2203.15556
  6. Language Models are Few-Shot Learners · Brown and others, arXiv 2005.14165
  7. Survey of Hallucination in Natural Language Generation · Ji and others, arXiv 2202.03629
  8. Attention Is All You Need · Vaswani and others, arXiv 1706.03762
  9. AI 600-1: Artificial Intelligence Risk Management Framework - Generative AI Profile · US National Institute of Standards and Technology (NIST)