The problem attention was built to solve
Before attention, sequence models read text one position at a time and carried everything they had learned in a single fixed-size state. Understanding why that failed explains what attention is for.
Recurrence processes in order. A recurrent network reads a token, updates a hidden state, reads the next, updates again. The state is the model's only memory of what came before, and it has a fixed size regardless of how long the input is.
Which creates a bottleneck. An entire sentence, paragraph or document has to be compressed into that one vector before the output begins. The longer the input, the more has to be discarded, and what gets discarded is decided implicitly.
And it makes long-range dependence hard to learn. Information from early in a sequence must survive many update steps to influence a later output. Gradients travelling back through those steps shrink or grow, which is what the gated architectures were designed to mitigate rather than solve.
Order also forbids parallelism. Position twenty cannot be computed until position nineteen is finished. On hardware built for doing thousands of things at once, that is the expensive constraint, and it limited how large such models could practically be trained.
The first fix kept recurrence and added a lookup. Published work on machine translation let the decoder, at each output step, compute a weighted combination of all the encoder's hidden states rather than relying on the final one. The weights were learned, and inspecting them showed the model aligning each output word with the input words it drew on.
That weighted lookup is attention. The output at a position is a weighted average of representations elsewhere, with the weights computed from the content rather than fixed by position. Nothing about it requires recurrence.
Then recurrence was removed entirely. The transformer architecture kept the attention mechanism, dropped the recurrent state, and processed every position at once. The title of that paper was the claim: attention was sufficient, and the sequential bottleneck was not necessary.
What that bought was not only accuracy. Removing the sequential dependence made training parallel across the sequence, which made much larger models and much larger corpora practical. The scaling results that followed depended on this architectural fact.
And what it cost is the subject of lesson nine. Letting every position look at every other position means the work grows with the square of the sequence length. The modern engineering of long context is almost entirely about managing that one consequence.
What an attention operation computes
Attention sounds abstract and is arithmetically simple. It is worth following once, concretely, because every later lesson refers back to it.
Three roles, derived from the same input. For each position, the model computes a query, a key and a value, each by multiplying the position's representation by a learned matrix. The same token produces all three; they are different projections of it.
The query asks, the keys answer. The relevance of position j to position i is the dot product of i's query with j's key. A large dot product means the two projections point in a similar direction, which the model has learned to mean relevant.
The scores are scaled. The dot products are divided by the square root of the key dimension. Without that, scores grow with dimension, the softmax saturates and gradients become tiny. It is a numerical correction with a large practical effect.
Softmax turns scores into weights. The scaled scores across all positions are exponentiated and normalised, giving weights that are positive and sum to one. Each position therefore has a distribution over the positions it is reading from.
The output is a weighted average of values. Multiply each position's value vector by its weight and add them up. That sum replaces the representation at position i for the next stage of processing.
So attention is retrieval by content, differentiably. It is a soft lookup: instead of fetching one entry, it blends all of them in proportion to how well each key matches the query. Because every step is differentiable, the matrices that produce queries, keys and values are learned by ordinary gradient descent.
Nothing in it is sequential. Every position's query can be matched against every position's key at the same time, which is why the whole operation is a few large matrix multiplications and why it suits the hardware.
The matrix view is the one to hold. Stack the queries into a matrix, the keys into another, multiply, scale, softmax each row, then multiply by the stacked values. That one line is scaled dot-product attention in full, and everything else in the architecture surrounds it.
And the shape of the cost is visible in that line. The query-key product is a matrix with one entry for every pair of positions. For a thousand tokens that is a million entries; for a hundred thousand tokens it is ten billion. The arithmetic is simple and it is quadratic.
Self-attention and where context comes from
Attention between two different sequences was the original use. Pointing it at a single sequence, so that every position attends to every other position in the same sequence, is what produced the models we now use.
Self-attention means queries, keys and values come from one sequence. Each token looks at all the tokens, including itself, and rebuilds its own representation from what it finds.
This is where a word acquires a sense. The embedding of a token is the same wherever it appears, as the previous course set out. After a self-attention layer it is not: the representation at that position now mixes in information from the surrounding tokens, so the same word in two sentences becomes two different vectors.
Resolution happens by content, not by distance. A pronoun can attend strongly to a noun thirty tokens earlier and weakly to the adjacent word, because the weights come from the projections rather than from proximity. Long-range dependence costs the same as short-range dependence, which was precisely the problem with recurrence.
Causal masking is what makes generation possible. In a model that predicts the next token, a position must not see positions after it. The mask sets those scores to negative infinity before the softmax, so their weights become zero. That single modification turns self-attention into something that can be trained on ordinary text and used to generate.
Without the mask, attention is bidirectional. Every position sees the whole sequence, which is better for understanding tasks such as classification and worse for generation, because the model has seen the answer. Lesson eight is about the two families this produces.
Training is parallel because of the mask. With causal masking, a single forward pass computes the prediction at every position simultaneously, each conditioned only on what precedes it. One pass over a document therefore yields as many training signals as it has tokens, which is the efficiency the whole field rests on.
Attention mixes positions; it does not transform them individually. The operation moves information between positions and leaves the per-position work to the feed-forward part of the block in lesson six. This division - mix, then process - is the architecture in one phrase.
And one layer is not enough. A single round of mixing lets each position see the original representations of the others. Stacking layers lets a position see representations that have themselves already been mixed, which is how dependencies that require several steps get built.
Several heads at once
One attention operation produces one weighted average. Models run many in parallel, and the reason is both intuitive and worth stating precisely.
Multi-head attention splits the representation. Rather than one attention over the full dimension, the model projects into several lower-dimensional subspaces, runs attention independently in each, concatenates the results and projects back. The original architecture used eight such heads with a model dimension of 512.
Each head can attend to something different. One weighted average can only emphasise one pattern of relevance per position. With several, a position can simultaneously draw on the subject of its clause, the preceding punctuation and a term defined earlier, without those demands competing for the same weights.
The cost is roughly unchanged. Because each head works in a smaller subspace, the total arithmetic is similar to a single full-dimension attention. Heads are close to free, which is why there are many of them.
Some heads do appear to specialise. Analyses of trained models have found heads that track syntactic relations, heads that attend to the previous token, and heads that appear to copy. This is real and it is also easy to over-read: specialisation is a tendency observed in particular models, not a designed property, and heads are not modules with defined jobs.
Heads are also redundant. Published work has shown that a substantial fraction of heads in a trained model can be removed with little loss, which tells you that the division of labour is diffuse rather than crisp.
Different layers use heads differently. Early layers tend towards local and positional patterns and later layers towards longer-range and more abstract ones. The tendency is well attested and the boundaries are soft, so it is a useful intuition rather than a specification.
Serving changed the head design. Keys and values have to be stored for every position during generation, as lesson ten explains, and that store is dominated by the number of heads. Published work proposed sharing a single set of keys and values across all query heads, and a later method grouped query heads to share a smaller number of key-value sets, trading a little quality for a large reduction in memory.
So the modern head count is two numbers, not one. How many query heads, and how many key-value heads they share. If you read a model's architecture and see those differ, that is why.
And the practical reading is this. Heads buy the ability to attend to several things at once, cheaply; they do not buy interpretable components, and their configuration is now as much a serving decision as a modelling one.
Telling the model where it is
Attention has no notion of order. A set of vectors attended to by another set produces the same result whichever way round the inputs are listed, so position must be supplied deliberately.
The original approach added fixed waves. Sinusoidal functions of different frequencies were added to the input embeddings, giving each position a distinct pattern and giving relative offsets a consistent relationship. No parameters to learn, and in principle defined for any position.
Learned absolute positions are the simplest alternative. A trained vector for position one, position two and so on. Effective, and it fails completely beyond the longest position seen in training, because those vectors do not exist.
Rotary embeddings are the current default in most models. Instead of adding a position signal to the input, the query and key vectors are rotated by an angle proportional to position. The dot product between a rotated query and a rotated key then depends on the difference between the two positions, so relative position falls out of the arithmetic rather than being approximated.
Linear biases take a different route. A published method adds a penalty to each attention score proportional to the distance between positions, with no position embeddings at all. Its central result was extrapolation: a model trained on short sequences handled much longer ones at inference.
Which is why extrapolation is a live engineering topic. A model has been trained at some sequence length, and running it beyond that length without degradation needs either a scheme designed for it or an adjustment. Interpolating the rotary frequencies to stretch the effective range, with a short period of further training, is a common approach.
And why advertised context lengths deserve scepticism. A window extended by such a method works, and quality at the far end is usually below quality in the range the model was actually trained on. Measure on your own task at the length you intend to use rather than trusting the number on the specification.
Relative beats absolute for most purposes. What matters linguistically is usually how far apart two tokens are, not where they sit in the document. Schemes that encode the difference generalise better, which is the practical reason rotary embeddings won.
One consequence for prompt design. Because position enters the arithmetic, where you place material in a long prompt is not neutral, and the empirical finding in lesson eleven about the middle of long inputs is partly a consequence of how position and attention interact at length.
Inside one block
A transformer is a stack of identical blocks. Each block does two things and wraps both in machinery that makes deep stacking possible.
First the mixing. Multi-head self-attention moves information between positions, as lessons two to four described. Nothing in it transforms a position's content on its own.
Then the per-position processing. A feed-forward network is applied to each position separately and identically: expand the dimension by a factor of several, apply a non-linearity, project back. This is where most of a model's parameters live, and it is the part that does the per-token computation attention does not.
The gating variants are a small, real improvement. Current models commonly use a gated form of the feed-forward layer, in which one projection modulates another. Published comparisons found it consistently better at equal parameter count, and it is now the usual choice.
Residual connections make depth trainable. The output of each sub-layer is added to its input rather than replacing it. The effect is that the identity function is the default behaviour and each sub-layer learns a correction, which keeps gradients from vanishing across many layers.
Normalisation keeps the scales usable. Layer normalisation rescales each position's vector to a consistent mean and variance, stabilising training. Its placement matters: applying it before each sub-layer rather than after is now standard, because it trains more reliably at depth without careful warm-up.
So the block is four pieces in a fixed arrangement. Normalise, attend, add; normalise, process, add. Repeat that for every layer and you have the architecture, with the attention mask and the position scheme deciding the family it belongs to.
Where the parameters sit is worth knowing. Roughly two thirds of a typical block's parameters are in the feed-forward part and one third in the attention projections. If a model's size surprises you, the feed-forward width is usually the reason.
Where the time goes is a different question. At short sequence lengths the feed-forward layers dominate the arithmetic; as the sequence grows, the attention term grows quadratically and eventually dominates. This crossover is why optimisation advice differs between short-prompt and long-context workloads.
And the uniformity is the point. Every block is the same shape, so depth is a parameter rather than a redesign. That regularity is what made it possible to scale the architecture by orders of magnitude without inventing anything new, which is the subject of the scaling results in the previous course.
Depth, and what the layers do
Stacking identical blocks produces behaviour nobody specified. What is known about how the layers divide the work is genuinely useful, and it is easy to overstate.
Depth composes operations. Each layer's attention reads representations that previous layers have already mixed. A dependency that needs two steps - find the clause, then find its subject - can be built across two layers in a way one layer cannot express.
The observed gradient is from form to meaning. Probing studies on trained models have repeatedly found that information about surface features and syntax is most accessible in earlier layers, and information about meaning and task-relevant abstractions in later ones. The pattern is robust across models; the boundaries are not sharp and the ordering is a tendency.
The residual stream is the useful mental model. Because every sub-layer adds to its input, the representation at a position can be read as a running sum that each layer contributes to. Attention heads write information into it and later layers read from it, which is the framing most current interpretability work uses.
Depth and width trade off. For a fixed parameter count, more layers or wider layers are both options, and the published scaling work found performance relatively insensitive to the exact shape over a broad range. That insensitivity is why architectures from different labs look so similar.
Very deep models need the stabilisers. Residual connections and pre-normalisation are not decorations; without them, stacks of this depth do not train. Most of the architectural refinements since the original paper are about making depth reliable rather than about attention itself.
Not every layer is equally necessary. Experiments removing or reordering middle layers in trained models often show modest degradation, which suggests a degree of redundancy. Layer-skipping and early-exit methods exploit this for speed.
Be careful what you conclude from any of it. These are empirical observations about particular trained models, obtained with particular probing methods. They are a good basis for intuition and engineering decisions, and a poor basis for confident claims about how a model reasons.
The practical upshot for a builder. Depth is why a model can do multi-step work within a single forward pass, and that capacity is finite. Asking for intermediate steps in the output gives the model more passes rather than more depth, which is the mechanical reason that technique helps on problems that need several steps.
Encoder, decoder, and the one that won
The original architecture had two halves. Most models you will use today have one of them, and knowing which explains what a model is good for.
The encoder reads. Bidirectional self-attention, every position seeing every other. It produces a representation of the input and nothing else. Published work in this family trains by masking tokens and predicting them, which gives a model strong at classification, extraction and similarity and incapable of generating text left to right.
The decoder writes. Causal self-attention, each position seeing only what precedes it, trained to predict the next token. It generates, and because the objective is just next-token prediction, it trains on any text at all.
The original design used both. An encoder for the source sentence and a decoder for the target, with a cross-attention layer in each decoder block whose queries come from the decoder and whose keys and values come from the encoder. That arrangement remains the natural one for translation and for any task with a distinct input and output.
Decoder-only architectures became the default for general models. The reasons are practical: one objective, any text as training data, no architectural distinction between the prompt and the continuation, and a straightforward path to scale. A task with an input and an output is handled by putting the input in the prompt.
Cross-attention has not disappeared. It is how models that take images, audio or other modalities bring that information into a text decoder, and it is used in several retrieval and speech architectures. When you see it in a model description, look for two different sources of information.
The encoder family is still the right tool for several jobs. Embedding text for retrieval, classifying at high volume and low cost, and extracting spans. Using a large generative model for a task a small bidirectional encoder does better and far cheaper is a common and expensive mistake.
Masking is the whole difference at the level of code. Bidirectional and causal attention are the same operation with a different mask. That is worth remembering because it means the distinction is about training and use, not about mechanism.
Prefix arrangements sit in between. Some designs attend bidirectionally over the prompt and causally over the generated continuation, which recovers some of the encoder's advantage for the input while remaining generative.
And the practical reading of a model card. Whether attention is causal or bidirectional tells you what the model can be asked to do. Whether cross-attention is present tells you whether it consumes something other than its own text. Those two facts place almost any model in the landscape.
The quadratic problem
Every position attending to every position means the attention cost grows with the square of the sequence length. Nearly all long-context engineering is about this one fact.
The arithmetic is unavoidable in the exact form. A sequence of n tokens produces an n-by-n matrix of scores. Double the length and the attention work quadruples. The feed-forward layers grow only linearly, so attention dominates as sequences get long.
Memory is the binding constraint before arithmetic is. The score matrix is large and, in a naive implementation, written to and read from memory repeatedly. Published work on input-output-aware attention reorganised the computation to keep blocks in fast on-chip memory and never materialise the full matrix, producing exactly the same result several times faster and with far less memory traffic.
That is worth stating precisely because it is often misdescribed. That method is exact, not an approximation. It changed what sequence lengths are practical without changing what the model computes.
Approximate methods restrict who attends to whom. Local or sliding-window attention lets each position see a fixed neighbourhood. Strided and dilated patterns add long-range links cheaply. Global tokens give a few positions full visibility so information can route through them. Each reduces the cost to something close to linear and each loses some exact long-range interaction.
Linear attention changes the algebra. By replacing the softmax with a factorisable similarity, the computation can be reordered so cost grows linearly with length. These methods work well on some tasks and have generally been a little behind exact attention on language modelling quality, which is why they appear more often in hybrids than alone.
Hybrid stacks are the current pragmatic answer. Alternating a few layers of full attention with many layers of local attention keeps most of the long-range capability at a fraction of the cost, and several current long-context models are built this way.
State-space and recurrent alternatives have returned. Architectures with a fixed-size recurrent state have linear cost in sequence length and constant memory during generation, and recent variants are competitive on language modelling. Hybrids combining them with a few attention layers are an active direction.
For a builder, the implications are concrete. Long context is expensive in a way that scales badly, the quality at extreme lengths is not uniform, and selecting what to send is usually cheaper and better than sending everything. That is the engineering case for retrieval, which is the next course in this sequence.
The key-value cache
Generation would be hopelessly slow without one optimisation, and understanding it explains most of what you observe about latency, memory and cost in production.
Generating token by token would repeat everything. Each new token's attention needs the keys and values of all previous positions. Recomputing them for the whole sequence at every step would make a thousand-token output a thousand passes over a growing prefix.
So keys and values are cached. After processing the prompt, the keys and values for every position in every layer are kept. Each new token computes only its own query, key and value, attends against the cache, and appends to it.
Which splits generation into two phases with different characteristics. Prefill processes the whole prompt at once: parallel, arithmetic-bound, fast per token. Decode produces one token at a time: sequential, memory-bandwidth-bound, and the reason output length drives latency.
The cache is large. Its size grows with the number of layers, the number of key-value heads, the dimension per head, the sequence length and the batch size. For long contexts and many concurrent users it becomes the dominant consumer of accelerator memory, often exceeding the model weights.
This is why the head design changed. Sharing one set of keys and values across all query heads cuts the cache by the head count, at some cost to quality. The grouped compromise, with several query heads sharing each key-value set, recovers most of the quality while keeping most of the saving. Both published methods exist because of the cache, not because of accuracy.
Serving systems manage it like memory, because it is. Paged allocation of cache blocks, sharing a common prompt prefix between requests, and evicting or recomputing under pressure are all standard. Prompt prefix caching, which the previous course described as a cost lever, is this mechanism exposed to the caller.
Quantising the cache is a routine trade. Storing keys and values at lower precision reduces memory substantially with modest quality cost, and is widely used to extend practical context length.
The consequences you will actually notice. A long prompt costs mostly once, at prefill. A long output costs on every token. Many concurrent long-context sessions exhaust memory before they exhaust compute. And throughput improves with batching while individual latency does not.
So the design advice follows. Keep outputs short where you can, reuse stable prefixes, batch anything that is not interactive, and treat context length as a memory budget shared with every other user of the same hardware.
Long context in practice
A large advertised window is not the same as a usable one. This lesson is what to expect and how to find out for yourself.
The trained length and the advertised length are different numbers. Models are trained at some sequence length and extended afterwards by position interpolation, continued training at longer lengths, or an architecture designed to extrapolate. Behaviour inside the trained range is what the published evaluations usually reflect.
Retrieval from a long context is uneven. The published study of this found that performance was best when relevant information was at the beginning or the end of the input and noticeably worse when it was in the middle, in models with ample window to spare. The effect has been reproduced widely enough to design around.
So placement is a design decision. Put the instruction where it will be read, put the most important evidence at the edges rather than buried, and do not assume a model has weighted a document uniformly because it fitted.
Benchmarks that only test recall are too easy. Finding a single distinctive sentence in a long document is a weak test. Tasks requiring several facts to be combined from different places in the context are much harder and much more representative of real use.
Quality degrades before capacity does. A model will accept a very long input and answer, and the answer may quietly rely on less of the input than you assume. There is no error to catch, which is why evaluation at your intended length is not optional.
Cost and latency scale against you at the same time. The quadratic attention term and the cache growth both bite, so the longest inputs are the slowest and most expensive as well as the least reliable. Three reasons to select rather than to dump.
Selection beats stuffing at almost every size. A retrieval step that supplies two relevant pages generally outperforms sending two hundred, on quality as well as cost. This is the practical conclusion of the whole course, and the mechanics of doing it well are the next one.
Compression is a middle route. Summarising earlier conversation, extracting the facts worth keeping, or caching a distilled form of a long document all reduce the burden while retaining continuity. Each is lossy and each should be evaluated rather than assumed.
And measure three things at your working length. Accuracy on a task that needs material from several positions, latency at the ninety-fifth percentile, and tokens consumed per request. A long-context decision taken without those three numbers is a guess.
Sparse models and what they change
Not every parameter in a large model is used for every token. Mixture-of-experts designs are now common enough that the distinction matters when reading a model's description or its price.
The idea is conditional computation. The feed-forward part of a block is replaced by several parallel copies, called experts, and a small router chooses which one or two to use for each token. Only the chosen experts do any work.
Which separates two parameter counts. Total parameters, which determine memory and capacity, and active parameters, which determine the arithmetic per token. A model can have very many of the first and comparatively few of the second, which is the whole appeal.
The published route to very large models used this. Work on switch-style routing scaled to a trillion parameters by sending each token to a single expert, with a training recipe that made the routing stable. The claim was not that it was better at equal total size but that it was better at equal compute.
Routing has its own failure modes. Experts can collapse so that a few receive most of the traffic, which wastes capacity, so auxiliary objectives are added to balance the load. Capacity limits mean tokens can be dropped when an expert is full, and routing decisions can be unstable between near-identical inputs.
Serving is harder than for a dense model. All the experts must be resident in memory even though only a fraction are used, so memory requirements track total parameters while speed tracks active ones. At small batch sizes the hardware is often poorly utilised.
For a user of a hosted model, the visible effects are indirect. Price and latency reflect active parameters; capability reflects total. A model that seems unusually capable for its speed and price is quite often sparse, and that is a reasonable thing to infer from the numbers.
Other sparsity exists at inference. Early exit for easy tokens, layer skipping, speculative decoding with a small model proposing and a large one verifying, and quantisation. Each reduces cost for a given quality, and speculative decoding in particular is widely deployed because it is exact.
The reason this belongs in a course on attention. It is the clearest current example of the architecture being changed to move cost around rather than to add capability. Attention gave the field something that scaled; most engineering since has been about paying for that scale, and sparsity and the cache are the two largest levers.
Reading a model, and what attention does not explain
Two practical skills close the architectural half of this course: extracting what matters from a model's description, and knowing what attention weights can and cannot tell you.
What to look for in a model's architecture. Whether attention is causal or bidirectional. Layer count and model dimension. Query heads and key-value heads, which tell you about the cache. The position scheme and the length actually trained on. Whether the feed-forward layers are dense or routed. The vocabulary size and the tokeniser, because that determines your token costs.
Those facts answer practical questions. The head ratio predicts memory pressure under concurrency. The trained length predicts where quality falls away. The tokeniser predicts your bill in the languages you serve. None of it requires reading the weights.
Then the interpretability question, where care is needed. It is tempting to treat attention weights as an explanation: the model attended to these words, therefore it used them. Published work put that claim under pressure, showing that alternative attention distributions could often produce the same prediction, which means the observed weights were not the only possible account of the output.
The response was a qualification rather than a dismissal. A follow-up paper argued the standard was too strict and that attention weights do carry information, while agreeing they are not a faithful explanation on their own. Both papers are worth knowing by name, because the disagreement between them is the state of the question.
So treat attention maps as a hypothesis generator. They show where information could have flowed. Establishing that it did requires intervention: changing the input, ablating a head, patching an activation and observing whether the output changes.
Mechanistic work has produced some real results. Specific circuits have been identified in small models, including heads that copy a token from earlier in the sequence to complete a repeated pattern. These are genuine findings about particular models, obtained with careful intervention rather than by reading weights.
What none of this supports. Claims that a model reasoned in a particular way, or that an attention map constitutes an audit trail of a decision. For a regulated process, an explanation has to be something defensible to somebody who will not accept a heat map.
Which has a design consequence. If a decision must be explainable, do not rely on interpreting the model. Structure the system so the explanation is the retrieved evidence, the tool output and the recorded intermediate steps, all of which are artefacts you control.
Sizing one deployment
A team must serve a document question-answering feature for an internal audience of about two thousand users. The decision is what to run, at what context length, on what hardware. This is the arithmetic.
The requirement. Up to forty concurrent requests at peak. A ninety-fifth percentile response time under eight seconds. Documents averaging twelve pages, occasionally sixty. Answers of a paragraph.
First, the context decision. A sixty-page document is roughly forty thousand tokens. Sending whole documents would mean a long window, a quadratic attention cost at every request, and a cache per session large enough to constrain concurrency. The team instead retrieved the four most relevant passages, about three thousand tokens, and set the working context at eight thousand including instruction and history.
Second, the cache arithmetic. The cache size is layers times key-value heads times head dimension times two, for keys and values, times sequence length, times batch size, times bytes per value. With the chosen model's grouped key-value heads and the eight-thousand-token context, forty concurrent sessions fitted comfortably in the available memory. At the forty-thousand-token alternative, the same hardware supported eight.
That single comparison decided the architecture. Not a quality argument and not a preference: five times the concurrency on the same hardware, because the cache grows with context length and the number of users multiplies it.
Third, the latency budget. Prefill of eight thousand tokens was well under a second. The answer, at roughly two hundred output tokens, dominated the rest. Streaming was enabled so the user saw text immediately, and the maximum output was capped so a runaway answer could not breach the budget.
Fourth, the measurement at the working length. A set of sixty real questions, each needing facts from two different places in the supplied passages, scored against written targets. The retrieval step was scored separately, which is what identified that two failures were missing passages rather than bad answers.
What they tested and rejected. Sending whole documents with no retrieval scored slightly better on three questions and worse on eleven, at four times the cost - the middle-of-the-context effect, visible in their own data rather than taken on trust.
What they instrumented. Tokens in and out per request by component, finish reason, prefill and decode time separately, cache occupancy, and the retrieval score. When latency later rose, the cause was visible in one chart: conversation history had been accumulating without a limit.
The point. Every decision here came from the mechanics in this course - quadratic attention, the cache, grouped heads, position effects at length - and each was settled by arithmetic done before the system was built rather than by an argument about models.
Fifty questions
Answer all fifty questions, then check your answers. You need 45 out of 50 to pass and receive a certificate. If you score less, you will see which answers were right and wrong, and then go through the course again before you retake the check. Your answers, progress and times are kept only in this browser.
Your answers
Your certificate of completion
Enter your name as you want it to appear, then save the certificate as a PDF. In the print window, choose Save as PDF. A certificate is issued once per completion of the course.
Rate this course
How would you rate this course? Your rating is anonymous — just a number, with no name and no account.
Bring this to your team
Your colleagues can take this course too — it is free, needs no sign-up, and nothing they enter leaves their browser. If your institution wants the same ground covered as structured training for a team, with an instructor and your own cases, we run courses in a classroom or online for groups of ten or more.
Talk to us about training your team → See the paid courses →
Saolix does not record who takes this course, so it cannot verify these certificates. The certificate confirms completion of a free self-paced course and is not an accredited qualification.
Sources
The official documents this course relies on. Laws and guidance change, so check the current version.
- Attention Is All You Need, the transformer architecture · Vaswani and others, arXiv 1706.03762
- Neural Machine Translation by Jointly Learning to Align and Translate, the first attention mechanism · Bahdanau, Cho and Bengio, arXiv 1409.0473
- Layer Normalization · Ba, Kiros and Hinton, arXiv 1607.06450
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · Devlin, Chang, Lee and Toutanova, arXiv 1810.04805
- RoFormer: Enhanced Transformer with Rotary Position Embedding · Su and others, arXiv 2104.09864
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation · Press, Smith and Lewis, arXiv 2108.12409
- Fast Transformer Decoding: One Write-Head is All You Need, on multi-query attention · Shazeer, arXiv 1911.02150
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · Ainslie and others, arXiv 2305.13245
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness · Dao, Fu, Ermon, Rudra and Re, arXiv 2205.14135
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity · Fedus, Zoph and Shazeer, arXiv 2101.03961
- Lost in the Middle: How Language Models Use Long Contexts · Liu and others, arXiv 2307.03172
- Attention is not Explanation · Jain and Wallace, arXiv 1902.10186
- Attention is not not Explanation · Wiegreffe and Pinter, arXiv 1908.04626



