How an AI Model Turns Input Into Output
A step-by-step explanation of tokens, embeddings, attention, generation and model memory—without assuming a machine-learning background
AI models are often explained from the middle. Someone introduces attention, embeddings or GPUs before explaining what calculation the model is trying to perform. Each definition may be correct, but the definitions do not join into a picture.
This article follows one incomplete sentence through a language model:
The bank approved the
We will introduce a term only when the model encounters the problem that term solves. The main explanation does not require mathematics. The exact calculations are available in expandable sections for readers who want them.
This is the technical companion to AI System Design, which applies these mechanics to product architecture, context design, latency and cost.
1. The model begins with one job
Given the incomplete sentence, the model calculates a score for every possible next token in its vocabulary. Those scores are converted into probabilities.
An illustrative result might be:
| Possible next token | Probability |
|---|---|
loan |
31% |
request |
14% |
payment |
8% |
application |
6% |
The model produces the distribution. It does not choose from it. Something outside the model applies a selection rule, which section 7 describes.
If the selection rule picks loan, the sequence becomes:
The bank approved the loan
The model then calculates a new distribution for the token after loan. Producing an answer means repeating this operation.
This creates four requirements:
- Text must become numbers because the model performs arithmetic.
- The numbers must preserve order because word order changes meaning.
- Each position must be able to use information from other positions.
- The final result must become scores for possible next tokens.
Tokens, embeddings, position information and attention exist to satisfy these requirements.
2. Text is divided into tokens
A model does not normally process one English word at a time. A tokenizer divides text into entries from a fixed vocabulary.
Our sentence might become:
["The", " bank", " approved", " the"]
Tokens are not always complete words. An uncommon word may become several tokens. Punctuation and spaces can also affect the division.
Every vocabulary entry has an integer identifier:
"The" → 791
" bank" → 3821
" approved" → 9342
" the" → 279
The model now sees the sequence of IDs:
[791, 3821, 9342, 279]
These numbers are labels, not measurements. Token 9342 is not greater or more meaningful than token 791. The IDs merely identify vocabulary entries.
That makes them useful for lookup, but not yet useful for the model’s learned arithmetic.
3. Each token ID selects a learned list of numbers
The model contains a large table with one row for every vocabulary token. Our second token, ` bank`, had ID 3821, so it selects row 3821.
A shortened version of that row might look like:
[0.18, -0.42, 0.07, 0.91, ...]
This ordered list of numbers is a vector. The learned starting vector for a token is its embedding.
The numbers were learned during training. The model repeatedly predicted tokens, measured its errors and adjusted its internal values to reduce those errors. The embedding table was adjusted along with the rest of the model.
Every occurrence of ` bank` starts from this identical row — the financial one and the river one alike. Nothing here distinguishes them yet; that only happens once the surrounding tokens are allowed to act on it, which is section 5.
An embedding dimension does not usually have a clean label such as “financial meaning.” Information is distributed across many numbers and interpreted by later model operations.
Embedding and activation are not the same term
The embedding is the token’s starting vector. As the input passes through the model, that vector is repeatedly updated. The vector at a particular point in the calculation is called an activation.
| Term | Meaning |
|---|---|
| Token | A discrete unit of input or output |
| Token ID | The vocabulary identifier for that token |
| Embedding | The token’s learned starting vector |
| Activation | The token’s current vector at one stage of processing |
The term “activation” therefore does not refer to another stored meaning. It refers to the numerical values currently moving through the model for this particular input.
For the extra curious: the embedding table's shape
If a vocabulary contains 100,000 tokens and the model represents each token using 4,096 numbers, the embedding table has:
100,000 rows × 4,096 columns
If E is that table, the starting vector for token ID i is the row E[i].
4. The model must preserve order and build context
These sentences contain many of the same tokens:
The dog chased the man.
The man chased the dog.
They do not mean the same thing. The model therefore incorporates information about each token’s position. Different model families implement position differently, but the requirement is constant: the calculation must preserve sequence order.
Order is not enough. Consider bank in these sentences:
The bank approved the loan.
They sat beside the river bank.
The token bank begins with the same embedding in both. Its later activation must become different because the surrounding tokens provide different information.
The model therefore needs an operation that lets each token position update itself using other permitted positions. That operation is attention.
5. Attention decides how positions contribute to one another
For each position, attention performs three steps:
- Calculate how strongly that position should use each available earlier position.
- Convert those strengths into weights.
- Combine information from the earlier positions using those weights.
The model first creates three numerical projections for every position:
- A query participates in deciding what other positions should contribute to the position being updated.
- A key participates in the matching calculation for each available position.
- A value carries the information that can be combined into the result.
Queries are not literal questions. Keys are not database keys. Values are not human-readable facts. They are different learned transformations of the current activations.
Queries and keys calculate the weights. The corresponding values are combined using those weights. This separation lets the model use one set of numerical features to determine relevance and another set to carry information.
The same operation, on our four tokens
Abstractly that is hard to hold. Run it on the sentence instead.
Our sequence is four tokens. Every position produces a query, a key and a value. To update position 4, the, the model takes position 4’s query and compares it against the key of every position it is allowed to see — positions 1 through 4. Four comparisons give four numbers; softmax turns them into weights that sum to 1.
An illustrative result for this sentence:
Read the bottom row. The position that will predict the next token puts 0.44 on approved and 0.31 on bank, and very little on The. Its updated activation is the values of those four positions, mixed in exactly those proportions. That mixture is why the distribution at the top of this article favours loan: the position carries bank and approved together, and in the training data that combination is followed by a small set of tokens.
Now the second sentence. In They sat beside the river bank, the position holding bank has river available to attend to, and puts weight there instead. Same token, same starting embedding, same weights in the model — a different mixture, and therefore a different activation. Nothing looked up a definition of bank. The disambiguation is a consequence of which positions were available and how strongly each was weighted.
This is also the honest answer to what the intermediate numbers are. They are not a hidden sentence, and they are not meaningless. Each position’s vector is a location in a space the model learned during training, where the operations in later layers can act on the distinctions that mattered for prediction. bank-after-approved and bank-after-river end up in different regions, because putting them in different regions is what made the model’s predictions less wrong.
Attention runs many times in parallel, and that is where heads come from
One round of the calculation above finds one kind of relationship. A model runs many in parallel, each with its own learned Wq, Wk and Wv. Each parallel copy is an attention head.
The model does not give every head the full vector. A representation of 4,096 numbers split across 32 heads gives each head 128 numbers to work with — that slice size is the head dimension. The heads run independently and their outputs are joined back together before the layer finishes. Different heads end up sensitive to different things; none of them was told what to specialise in.
Now the distinction that decides how much memory a conversation costs.
Queries are used once, in the step that produces the current token, and then discarded. Keys and values have to be kept, because every future token will attend back to them. So model designers noticed they could keep fewer key/value heads than query heads and let several query heads share one set. This is grouped-query attention, and it is the reason the two counts differ:
| Llama 3.1 8B | Count |
|---|---|
| Query heads | 32 |
| Key/value heads | 8 |
| Head dimension | 128 |
| Layers | 32 |
Four query heads share each key/value head. The saving is not a rounding detail: the cache stores keys and values, so the key/value head count is what multiplies into memory, and dropping it from 32 to 8 cuts the cost of every cached token by four.
This is the number that appears in the memory formula later in this article, and in the cost arithmetic in the companion guide. When a table says a token costs 128 KB, this is where three of its four factors come from.
A language model also applies a causal mask: a position may use earlier tokens, but it cannot use tokens that have not occurred yet. When predicting after approved, the model may use The bank approved; it cannot use the loan before those tokens exist.
For the extra curious: the attention calculation
Starting with the current activation matrix X, the model produces:
Q = XWq
K = XWk
V = XWv
Wq, Wk and Wv are learned parameter matrices. A common compact form of attention is:
Attention(Q, K, V) = softmax((QKᵀ / √d) + M)V
QKᵀ produces query-key compatibility scores. M applies the causal mask. Softmax converts the permitted scores into weights. Multiplication by V produces weighted combinations of the value vectors.
6. Many layers produce the next-token scores
Attention is only one part of a transformer layer. A layer also contains operations that transform each position’s values, normalise them and carry earlier information forward.
The model repeats these operations across many layers — 32 of them in the model tabulated above. At each layer, the activation for bank, approved and every other position changes as information is combined and transformed.
Each layer has its own attention, and therefore its own keys and values. Nothing is shared between layers. That is why the layer count multiplies into the memory a conversation occupies: the per-token cost is paid once per layer, every layer, for as long as the conversation is held.
After the final layer, the model takes the activation at the last position — position 4, the, the one whose row we read in the grid above — and converts it into one score for every token in the vocabulary. The other three positions were computed too, and during prefill their keys and values are kept, but only the last position’s activation is asked what comes next. Those raw scores are called logits. Softmax turns the logits into probabilities, returning us to the table at the beginning of the article.
The complete path is now visible:
text
→ token IDs
→ embedding vectors
→ position-aware activations
→ repeated transformer layers
→ vocabulary scores
→ next-token probabilities
→ selection rule
→ selected token
Everything up to the probabilities is the model. The step that follows is not, and it is the next section.
For the extra curious: what a learned matrix transformation does
A common neural-network operation is:
y = xW
x is an input vector, W is a matrix of learned parameters and y is the output vector. Each number in y is a weighted combination of numbers in x.
For an entire sequence, XW applies the transformation to the matrix of token activations. Modern accelerators are designed to perform enormous numbers of these operations efficiently.
7. A selection rule turns probabilities into one token
The model’s output is a probability distribution over the whole vocabulary. A distribution is not a token. Some rule has to reduce thousands of candidates to the single token that gets appended, and that rule lives in the serving layer, not in the model weights.
The simplest rule is greedy decoding: always take the highest-probability token. Given the distribution from section 1, greedy decoding selects loan every time.
Greedy decoding is not always what a product wants. It makes the model repeat itself on open-ended tasks and produces one fixed answer for one fixed prompt. The alternative is sampling: draw a token at random, in proportion to the probabilities. loan is then selected about 31% of the time and request about 14%.
Three controls shape that draw.
Temperature reshapes the distribution before the draw. The logits are divided by a number T before softmax. Below 1, the gaps between candidates widen and the leading token dominates. Above 1, the distribution flattens and unlikely tokens gain a real chance. At T = 0 the rule collapses back to greedy decoding.
| Temperature | Effect on our example | Typical use |
|---|---|---|
| 0 | loan every time |
Extraction, classification, structured output |
| ~0.7 | loan usually, other plausible tokens sometimes |
Assistants, general chat |
| ~1.3 | The tail becomes reachable | Brainstorming, variation |
Top-k limits the draw to the k highest-probability tokens. Top-p, or nucleus sampling, limits it to the smallest set of tokens whose probabilities sum to p. Both exist to cut off the long tail of near-zero candidates that temperature alone can make reachable — the tokens responsible for an answer that starts sensibly and then goes strange.
Temperature zero is not a determinism guarantee
Product teams routinely assume temperature = 0 means identical output for identical input. It removes the deliberate randomness, which is the largest source of variation, but it does not make the arithmetic reproducible.
Floating-point addition is not associative, so the order in which an accelerator sums values changes the last bits of a result. That order depends on how requests were grouped into a batch, which depends on the traffic arriving alongside yours. Mixture-of-experts routing and changes to the serving stack add further variation. Occasionally two candidate tokens sit close enough that the last-bit difference flips which one leads, and the outputs diverge from that token onward.
Design for it. If a downstream system requires an exact match, validate the output’s structure rather than comparing it byte for byte, and pin the behaviour you need in tests as a property, not a fixture.
For the extra curious: where temperature enters the calculation
Section 6 ended with softmax applied to the logits. Temperature is a division applied first:
P = softmax(logits / T)
Because softmax is exponential, dividing by a small T multiplies the ratio between any two candidates. If one logit exceeds another by 2.0, then at T = 1 their probability ratio is e² ≈ 7.4; at T = 0.5 it is e⁴ ≈ 54.6. T = 0 is not evaluated as a division — implementations special-case it to selecting the maximum.
8. Reading the input and writing the output behave differently
Everything to this point described producing one token. Our four input tokens were all known in advance, so the model can push all four through its layers together — position 4 does not have to wait for position 2, because position 2’s token was already there. Serving systems call this prefill.
Generating is different. Having selected loan, the model appends it and runs the calculation again on five tokens to get the sixth. It cannot start the sixth before the fifth exists, because the sixth attends to it. This phase is decode, and it is strictly one token at a time:
prefill The bank approved the ← 4 tokens, together
decode ... loan ← token 5
decode ... loan application ← token 6, only after 5
decode ... loan application yesterday ← token 7, only after 6
That asymmetry is the whole reason input and output are priced and timed differently. Input is a wide, parallel pass. Output is a queue.
Compare:
| Request | Input | Output | Dominant work |
|---|---|---|---|
| Contract extraction | 50,000 tokens | 200 tokens | Processing known input |
| Long-form writing | 2,000 tokens | 4,000 tokens | Sequential generation |
The same total token count does not imply the same latency or hardware behaviour — which is the observation the companion guide is built on.
Some of the generated tokens are not the answer
Nothing in the loop so far distinguishes a token meant for the reader from a token the model produces to work something out. Both are generated the same way, one at a time, each conditioned on everything before it.
Modern models exploit that. Given a hard problem, a model can generate a stretch of tokens that reason toward the answer — considering an approach, rejecting it, trying another — and only then generate the response. These are usually called reasoning tokens or thinking tokens. They are ordinary output tokens: same sequential generation, same per-token cost, same contribution to the KV cache. The only thing that makes them different is that the product usually does not show them.
This has three consequences a product team feels directly:
- They are on the bill as output. Output tokens are the expensive ones. A request whose visible answer is 200 tokens may have generated 4,000 to get there, and it is billed accordingly.
- They are slow in the same way any output is slow. Decode is sequential, so reasoning is time the user spends waiting before the answer begins.
- They are not always worth it. On a hard problem, more reasoning measurably improves the answer. On classification or extraction it is close to pure cost.
Because the tradeoff depends on the task rather than the prompt, providers expose it as a request parameter — a thinking budget, or an effort level — rather than something you phrase your way into. It is one of the few dials where the same prompt, unchanged, can differ several-fold in both cost and latency. The companion guide shows where that dial sits in a real request.
The KV cache prevents repeated work
Look again at the grid in section 5. To produce token 5, the model needed the keys and values of positions 1 to 4. To produce token 6, it needs positions 1 to 5 — the same four, plus one. Positions 1 to 4 have not changed: The cannot see anything after it, so its key and value are the same as they were.
Recomputing them anyway, for every generated token, in every layer, would be enormous waste. So the serving system keeps them. That retained state is the KV cache. Each new token adds one column to the grid and reuses every column already there.
This is also why the cache stores keys and values but not queries: a query is used in the row being computed and never referred to again, which is exactly the asymmetry that makes grouped-query attention worth doing.
The KV cache is not the conversation transcript. The transcript is text that an application can store in a database. The cache is model-specific numerical state created from a particular sequence.
It grows as the sequence grows and occupies fast accelerator memory. If it is discarded, the transcript still exists, but the model has to process that text again to rebuild usable state.
Where the cache can live
An active model uses several kinds of memory. They are not interchangeable.
| Location | What it can hold | What must happen before inference continues |
|---|---|---|
| Accelerator memory, usually HBM | Model weights and live KV-cache state | Nothing; the accelerator can use it directly |
| Host memory, usually DDR | Parked numerical cache state | Transfer the state back to accelerator memory |
| Application database or storage | Transcript as text | Process the text again to recreate model state |
HBM, or high-bandwidth memory, sits close to the accelerator’s compute units. It is fast and scarce. The model weights, current calculations and live KV caches compete for this capacity.
DDR, the system memory attached to the host machine, is larger and cheaper but slower. A serving system can move idle cache state there, but it must transfer that state back before the accelerator can use it again.
A database stores a different object. It can preserve the conversation text, but not the numerical state created inside the model. Returning from text requires computation, not merely a memory copy.
The exact capacity and transfer time depend on the hardware, interconnect, model and cache size. The product consequence is stable: keeping a conversation ready consumes scarce memory; parking it adds transfer delay; discarding it requires repeated input processing.
Why cached state disappears
There are three distinct events that are often described loosely as a cache miss.
- TTL expiry: Retained state is removed after it has gone unused for a configured period. The next request starts more slowly because state must be restored or rebuilt.
- LRU eviction: Memory fills before the timer expires, so the system removes the least recently used state to make room. Quick follow-ups may become slower during busy periods.
- Preemption: Active generation is paused or rescheduled so its resources can be used elsewhere. The user may see an answer stall after it has already begun.
These mechanisms have different symptoms and should be measured separately. Time to first token helps reveal cold or rebuilt state. Time between output tokens helps reveal interruptions during generation.
Why an unchanged beginning can be reused
The state at each position depends on the tokens before it — and, because of the causal mask, on nothing after it. That is the property prompt caching exploits. Consider two requests:
The bank approved the loan
The bank approved the mortgage
The keys and values for The bank approved the are identical in both. Not similar — identical, because none of those four positions was allowed to see the fifth token when its state was computed. A serving system can reuse all four and begin work at position 5.
Reverse it and the property disappears:
The bank approved the loan
On Tuesday the bank approved the loan
Every token has shifted position and each one now has different tokens before it. Nothing is reusable, despite the two sequences sharing almost every word.
If a request changes near the beginning, later cached state may no longer be valid because later positions were calculated using the changed earlier token. This is why prompt caching generally reuses an exact shared prefix rather than matching identical fragments anywhere in a request.
For the extra curious: why cache memory grows
A simplified KV-cache estimate is:
2 × layers × cached tokens × KV heads × head dimension × bytes per number
The factor of two represents keys and values. Exact requirements vary by architecture, precision, quantisation and serving technique. For a fixed setup, the important relationship is that KV-cache memory grows approximately linearly with sequence length.
9. Images and audio enter through different front ends
Text starts with a tokenizer because it contains discrete text symbols. Images and audio need different encoders.
A typical image path is:
pixels
→ image patches or vision-encoder inputs
→ visual feature vectors
→ model-compatible activations
A typical audio path is:
waveform
→ time segments or spectral features
→ audio feature vectors
→ model-compatible activations
Architectures differ, but the shared destination is an ordered collection of numerical representations the model can process.
What to keep in your head
You do not need to memorise every matrix to understand the path:
- The model calculates probabilities for possible next tokens.
- A tokenizer turns text into vocabulary IDs.
- Each ID selects a learned starting vector called an embedding.
- Activations are those numerical representations as they change through the model.
- Position information preserves order.
- Attention lets positions combine information from permitted earlier positions, running as many parallel heads — of which the key/value heads are the ones that cost memory.
- Repeated layers produce scores for the next token.
- A selection rule — greedy, or sampling shaped by temperature, top-k and top-p — reduces that distribution to one token.
- Some generated tokens are reasoning rather than answer; they cost and take time like any other output token.
- Generation repeats the calculation one selected token at a time.
- The KV cache retains earlier attention state so it does not have to be recreated at every step.
Run the sentence through all of it once. The bank approved the becomes four IDs; each ID pulls a learned row; position information keeps them in order; attention lets position 4 mix in 0.44 of approved and 0.31 of bank; 32 layers repeat that; the last position’s vector becomes 100,000-odd scores; softmax makes them probabilities; a selection rule picks loan; the four positions’ keys and values are kept so the next token costs one column instead of five. That is the whole machine. Everything else is scale.
These mechanics matter because they connect visible product behaviour to computation: long inputs affect the initial wait, long outputs take sequential time, growing conversations occupy more working memory, a changed prompt beginning can prevent reuse, and the selection rule — not the model — decides how much the same prompt varies.