What Actually Improved
Nine areas of AI progress, the metrics that define each, and how much room is left in every one before physics or information theory says stop
Two claims about AI get made constantly, and they fail for the same reason. One says the last nine years were “just scaling.” The other says nothing fundamental happened at all. Both treat capability as a single number that either went up a lot or didn’t.
Capability isn’t a single number. Progress happened in nine distinct areas, each with its own metrics, its own bottleneck, its own literature and largely its own people. They advanced in parallel, on different schedules, for different reasons. At any given moment one of them was the binding constraint: the thing everything else was waiting on. Much of the field’s history is the story of that constraint moving from one area to the next.
This article does three things. It names the nine areas and the metrics that define each. It places the work that moved those metrics on a shared timeline, so you can read across a year and see what was happening everywhere at once. And for every area it asks a question that gets asked far too rarely: how much room is actually left?
That last question is the useful one. Some areas are running into limits that can be proven: thermodynamics, information theory, computational complexity. Some are running into limits that are merely economic and will fall. Some have no known limit at all, and that absence is itself a finding, not a gap. Where the remaining distance is large, progress keeps going. Where it’s small, progress stalls no matter how much money arrives. Where nobody has drawn the wall, we genuinely don’t know — and those happen to be the areas everything now depends on.
This is the third of three connected pieces. AI System Design covers how workload shape decides architecture. How an AI Model Turns Input Into Output covers the mechanics inside a single request. This one covers where all of it came from and how much further it goes.
1. The nine areas
The areas below are ordered the way a capability actually travels: from the silicon it runs on, through the training that creates it, to the scoreboard that tells us whether anything improved. They group into four planes.
| Plane | Areas | The question it answers |
|---|---|---|
| The machine | Substrate · Architecture | What is it made of? |
| The learning | Pretraining · Post-training · Run-time reasoning | How does it become capable? |
| The delivery | Serving · Agents · Distribution | How does it reach a person? |
| The instrument | Evaluation | How would we know any of this? |
Evaluation sits across all of them deliberately. It tells the other eight whether they moved, and its failure modes — saturation, contamination, label noise — corrupt every claim made anywhere else. Treating it as a peer rather than a caveat is part of being honest about the rest.
Here is what each area actually is, in plain terms.
- Substrate — the physical machine: chips, memory, the wires between them, and the power they draw. Everything else runs on it.
- Architecture — the design of the model itself: how it is wired, how attention works, how much of it runs for any given token.
- Pretraining — the long, expensive run that sets the weights by predicting text. Where raw capability comes from.
- Post-training — turning a raw predictor into something that follows instructions and refuses sensibly. Small compute, enormous visible effect.
- Run-time reasoning — extra computation spent per question after training is over. The model works a problem through before answering, and you pay for that thinking.
- Serving — running the finished model for many users at once: how fast the first word arrives, how fast the rest follow, what a token costs.
- Agents — the system wrapped around the model: tools, memory, loops. Where a single answer becomes a multi-hour task.
- Distribution — who can obtain and run a given capability: open weights, price, and whether it fits on a laptop or a phone.
- Evaluation — how anyone knows whether the other eight moved. The instrument, and it has its own failure modes.
2. Five ways one area moves another
The areas are separate but not independent. Five patterns recur, and naming them makes the timeline readable as a mechanism rather than a list of events.
Unlock. A gain in one area removes a wall in another. FlashAttention changed how much memory traffic attention required, not the mathematics, just the traffic. That made long context economically serveable, which in turn made multi-hour agents plausible.
Handoff. One area stiffens, so effort migrates. As the returns from scaling pretraining data got harder to buy, run-time reasoning opened as an entirely new axis.
Substitution. A gain in one area buys what another can’t supply. Quantization and KV-cache compression substitute for memory capacity you can’t purchase.
Loop. A downstream gain refinances an upstream one. Post-training produced a usable product. The product produced revenue. The revenue bought substrate, which funded larger pretraining runs.
Constraint-shift. An external cap on one area redirects effort into others. Export controls on chips produced a disproportionate share of the world’s architecture and serving-efficiency research. More on that below.
3. Nine lanes on one timeline
Read a row to follow one area. Read a column to see what was happening in all nine at once. The parallelism is the point, and it’s what makes the progress feel sudden from outside. The highlighted staircase marks the binding constraint as it moves. The right-hand margin shows how much room each area has left.
4. The machine
Substrate
Metrics: FLOP/s per chip at a given precision · HBM capacity and bandwidth · joules per FLOP · dollars per FLOP · FLOPs per watt.
The line runs V100 to A100 to H100 to H200 to B200, and the important changes go beyond raw arithmetic. The H100 introduced practical 8-bit training, which halved the bytes moved for every number in a training run. Memory bandwidth rose from 2.0 TB/s on the A100 to 3.35 on the H100 to 7.7 aggregate on the B200. As section 6 shows, bandwidth rather than arithmetic is what actually limits generation.
This is also the only area in the chart with a substantial number of imposed caps: the October 2022 export controls, their widening in October 2023, restrictions reaching memory itself in 2024, a partial relaxation in December 2025, and the Chip Security Act in March 2026. A lane can be limited by policy long before it’s limited by physics.
Distance to the bound: roughly seven orders of magnitude. Landauer’s principle sets the energy cost of erasing one bit at kT ln2, about 2.9 × 10-21 joules at room temperature. Charged per 16-bit operation that’s around 5 × 10-20 J. Current accelerators spend about 5 × 10-13 J per FLOP.
That gap runs to roughly ten million, and reversible computing sits below even that floor. The conclusion cuts against most energy commentary: physics is not the constraint on compute efficiency, and won’t be for decades. Power delivery, fab capacity and capital are the real constraints. It’s exactly why the caps that bind this lane are drawn by governments rather than by nature.
Architecture
Metrics: loss at fixed compute · active parameters versus total · KV bytes per token · supported context length.
The 2017 transformer’s first contribution wasn’t accuracy but training parallelism: recurrence forced sequential processing, and attention didn’t. Everything downstream follows from that. The lane then splits into two long campaigns: making position information work at length (RoPE, and the interpolation methods that stretched it), and making attention cheaper per token. Grouped-query attention cut the number of key/value heads. Sliding windows bounded what each position could see. Multi-head latent attention compressed the KV cache into a smaller latent representation. Sparse mixture-of-experts made the number of parameters that run per token far smaller than the number that exist.
That last idea reshaped the economics. A model can hold 671 billion parameters and run 37 billion of them for any given token.
Distance to the bound: at it. Alman and Song proved a sharp transition in 2023: with head dimension of order log n, truly subquadratic attention is possible if and only if the matrix entries stay bounded below a threshold of order √log n. Above that threshold it’s impossible under the Strong Exponential Time Hypothesis.
This is the theoretical reason no linear-attention scheme is a free lunch. Every subquadratic method buys its speed by changing the problem: approximating, bounding entries, or imposing sparsity. Those are good trades worth making. They aren’t escapes.
5. The learning
Pretraining
Metrics: tokens · parameters · training FLOPs · the compute-optimal ratio between them · model FLOPs utilisation · dollars per run.
Two results define this lane, and the second corrects the first. The 2020 scaling laws established that loss falls predictably with compute. That sounds academic, but it was arguably the most consequential finding in the field’s financial history: it converted “will a bigger model be better” from a research gamble into an engineering forecast, and a forecast is something you can raise capital against.
Chinchilla then showed in 2022 that everyone had been undertraining. For a given compute budget, far more data and fewer parameters was the better trade. That single correction explains why a 7-billion-parameter model in 2024 outperformed a 175-billion-parameter model from 2020, and it redirected the whole industry from parameter counts toward data.
Distance to the bound: approaching, on the resource rather than the theory. Two different limits apply here. The scaling law’s irreducible term, around 1.69 nats per token in published fits, is the entropy of the data itself: a floor no compute budget crosses. Separately, Epoch AI estimates the effective stock of public human text at roughly 300 trillion tokens, with models on track to consume it somewhere between 2026 and 2032.
The first bound is still distant. The second is arriving now; we’re inside the front edge of that window this year. But a stock isn’t a law, and stocks get substituted. That’s precisely what synthetic data and the pivot to run-time reasoning are doing.
Post-training
Metrics: preference win-rate · instruction adherence · refusal correctness in both directions · jailbreak robustness · calibration.
This lane produced the most visible jump in the field’s history while using comparatively little compute. Reinforcement learning from human feedback turned a text predictor into something that answers the question asked. Constitutional AI replaced much of the human labelling with written principles. Direct preference optimisation later showed the same result was reachable without training a separate reward model at all.
The fact worth sitting with: GPT-3 existed in 2020. What arrived in late 2022 was largely a post-training and interface result, not a capability one — the clearest available proof that the visible jump and the underlying curve are different things.
Distance to the bound: nobody has drawn one. There is no impossibility theory for alignment. Adjacent results gesture at the difficulty (Arrow’s theorem on aggregating preferences, the formal characterisation of Goodhart’s law under proxy optimisation) but none of them bounds how well a model can be made to do what was meant. The honest answer is that we can’t say how much of this lane is left.
Run-time reasoning
Metrics: accuracy as a function of thinking-token budget · pass@1 versus pass@k · cost and latency per solved task, not per token.
Chain-of-thought prompting in 2022 showed that asking a model to work through a problem improved the answer. Process reward models in 2023 showed you could grade the reasoning steps rather than only the final answer. In 2024 this stopped being a prompting trick and became trained behaviour, and in January 2025 an open recipe using group-relative policy optimisation made the method reproducible outside the largest labs.
This is where the reasoning tokens you pay for but never see come from. The metric shift here is the real story: the meaningful unit stopped being cost per token and became cost per solved task.
Distance to the bound: bounded by the verifier, and unquantified. Reinforcement learning on reasoning works where an answer can be checked, and achievable accuracy can’t exceed what the checker can distinguish. Where verification is cheap and exact (mathematics, code, formal proof) the bound is high and the method works spectacularly. Outside those domains, which is most work of economic value, there’s no bound because there’s barely a method.
This lane didn’t meaningfully exist before 2024. Its arrival is the clearest case in the whole chart of a handoff: a new axis opening precisely as an old one stiffened.
6. The delivery
Serving
Metrics: dollars per million tokens · time to first token · time between tokens · concurrent sessions per card · cache hit rate.
Four years of work here, and nearly all of it attacks the same quantity. Continuous batching let new requests join a running batch. FlashAttention cut memory traffic without changing the mathematics. PagedAttention managed the KV cache the way an operating system manages virtual memory, ending the fragmentation that wasted most of a card. Speculative decoding drafted several tokens cheaply and verified them in one pass. Quantization to four bits put capable models on laptops. Prompt caching reused an identical prefix rather than recomputing it.
Why they all converge is visible in one calculation.
For the extra curious: the roofline that explains the whole lane
Generating one token requires reading the weights that token uses. So for a single stream:
tokens per second ≤ memory bandwidth ÷ bytes read per token
For a dense 70B model at 8-bit precision on one H100:
3.35 TB/s ÷ 70 GB ≈ 48 tokens/second
That’s a ceiling, not an estimate — real systems land below it. Every technique above is an attack on the same denominator. Batching amortises one read across many users. Mixture-of-experts reads only the active parameters. Quantization shrinks the bytes. Speculative decoding gets several tokens per read.
Decoding is bounded by reads, not by arithmetic. This is also why the KV cache occupies such expensive memory: it exists so that earlier positions don’t have to be read and recomputed at every step.
Distance to the bound: pressed against two of them. The bandwidth roofline above, and a second floor that no engineering touches at all: a round trip between continents can’t beat light in fibre, which comes to roughly 200 ms between Mumbai and Virginia. That latency floor is the entire argument for running models on the device.
Agents
Metrics: end-to-end task success · task horizon · per-step reliability · tool-call accuracy · cost per completed task · human interventions per task.
Retrieval, then interleaved reasoning and action, then tools with typed arguments, then a shared protocol for those tools, then the screen itself as an interface. The through-line is that a single answer became a long-running task — exactly the shift AI System Design is about.
The metric that matters most here is task horizon: the length of task a model completes reliably. METR’s January 2026 update puts the 50% time horizon at 320 minutes, and finds the doubling time has been accelerating: 89 days for models released since 2024, against roughly seven months measured across the longer run.
Distance to the bound: no ceiling has been drawn, but the arithmetic bites now. Per-step reliability r across n steps compounds as rn. Ninety percent success over a hundred steps demands 99.9% reliability per step. That isn’t a theorem, just multiplication, which is why it’s the practical constraint today rather than eventually.
The open question is whether verification and error correction can break the compounding the way fault-tolerance broke it for unreliable hardware: whether agency has a threshold theorem. Nobody has one. Everything being built right now is a bet on the answer.
Distribution
Metrics: open-weight lag in months behind the closed frontier · capability density · price at fixed capability · what class of device can run it.
BERT, then GPT-2’s staged release, then open replications of GPT-3, then a weights release in early 2023 that seeded most of what followed. Then permissive licensing, then frontier-scale open weights, then open reasoning models. By 2026 the lag is measured in months rather than years, and the models are cheap enough that an open model on commodity infrastructure is the default for most production work rather than the compromise.
Distance to the bound: real headroom. Allen-Zhu and Li found that language models store 2 bits of knowledge per parameter, and only 2, a figure that holds even under 8-bit quantization. That places a genuine floor under capability density: a 7B model tops out near 14 billion bits, which by their estimate exceeds English Wikipedia and textbooks combined. We’re not near it.
The unanswered question in this lane isn’t whether the open gap closes but what it converges to, and no theory predicts that number.
7. The instrument
Evaluation
Metrics: benchmark saturation half-life · contamination rate · construct validity · label-noise floor.
Every benchmark follows the same lifecycle: introduced, chased, saturated, contaminated, replaced. GLUE gave way to SuperGLUE within a year. MMLU held for longer. Human preference voting sidestepped static test sets. Then came benchmarks built from real work: actual repository bugs, questions hard enough to stump specialists, mathematics that stops research mathematicians.
Distance to the bound: past it, for the most-cited benchmark in the field. A benchmark’s ceiling isn’t 100%. It’s 100% minus the share of items that are mislabelled or ambiguous. MMLU carries documented label errors on the order of 6–9% of items, capping usable headroom near 95%. Frontier systems now report above 92%.
The remaining gap is smaller than the measurement error, so movement there is noise. SWE-bench Verified exists precisely because of this problem: ninety engineers hand-screened the task pool to remove broken tests, underspecified issues and unreliable environments.
This is why evaluation belongs beside the other eight areas rather than in a footnote. It’s the instrument every other claim in this article depends on, and when the instrument saturates, confident statements about progress quietly stop meaning anything.
8. What the export controls actually produced
The China thread isn’t a separate history. It’s the clearest available case of constraint-shift, and the nine-lane frame explains it without needing geopolitics.
Blocked at the substrate, the one area where a cap could be imposed from outside, Chinese labs directed effort into the areas where a cap couldn’t reach. The result is a body of work concentrated almost entirely in architecture, serving and distribution: multi-head latent attention compressing the KV cache, fine-grained expert routing so that a small fraction of a very large model runs per token, 8-bit training end to end, careful overlap of communication and computation, and a cheap reinforcement-learning recipe for reasoning. Then the weights were released openly, which made that efficiency work everybody’s baseline rather than one company’s advantage.
The honest accounting matters. Headline training-cost figures typically describe a final run and exclude the research, failed runs and infrastructure around it. A good deal of this work is excellent execution of ideas already published rather than invention from nothing. And the ecosystem still trails at frontier-scale pretraining and in evaluation depth.
But the structural lesson generalises well beyond one country, and it’s the most useful thing in this section: abundance optimises capability, scarcity optimises efficiency, and efficiency work compounds for everyone. A lab with unlimited chips has little reason to halve its KV cache. A lab without them has no other option. Once the technique is published, everyone’s serving costs fall.
9. Now do the same for things that touch the world
Everything above is necessary for a robot or a self-driving car and nowhere near sufficient. Five further areas gate anything embodied, and they behave completely differently, because their limits are physical and close, rather than theoretical and distant.
That contrast is the strongest evidence for this whole way of looking at the field.
- Sensing — measuring the world: lidar, cameras, radar, touch. Everything downstream is limited by what was perceived.
- Onboard energy — carrying the power. Every runtime, range and payload figure in robotics is downstream of one number.
- Actuation — moving against the world: motors, gearboxes, tendons. Turning decisions into force.
- Embodied policy — learning to act. The model that turns what was sensed into what the body does next.
- Safety validation — proving it is safe enough to deploy.
Set the two charts side by side and the divergence explains itself.
Compute sits roughly ten million times above its thermodynamic floor. Battery energy density sits within about 1.5× of what lithium-ion chemistry permits: 250 to 300 watt-hours per kilogram today against a theoretical ceiling near 400 to 460, while petrol carries around 12,500. Electric actuators are close to what the magnetic saturation of iron allows and have improved by only a small multiple in fifty years.
Language models could ride lanes with enormous headroom. Robots couldn’t. That difference in available headroom, more than any difference in algorithmic difficulty, is why one sprinted and the other crawled.
The exception proves the rule. One embodied lane genuinely did collapse: lidar fell from $75,000 a unit to under $200, a reduction of more than 99%. It moved because its binding limit was economic, not physical, and manufacturing scale defeats a price. It doesn’t defeat eye-safety limits on laser power, or the number of watt-hours in a kilogram. The same pattern holds in actuation, where cost and integration improved enormously while power density barely moved at all.
Two further bounds in this group deserve naming, because both get mistaken for engineering problems.
Embodied policy faces the data-stock problem in its worst form. Pretraining had an internet to read. Robotics doesn’t, and can’t: there’s no web-scale corpus of robot trajectories waiting to be found, because every hour of manipulation data has to be manufactured at the speed of physical reality, one robot-hour at a time. The escapes under way — simulation, pooling data across robot types, learning priors from human video — are all attempts to substitute a stock that can’t simply be collected.
Safety validation faces a statistical wall that better models don’t move. RAND’s result is that demonstrating with 95% confidence that a driverless fleet beats the human fatality rate would require roughly 275 million miles driven without a fatality, and under some assumptions, hundreds of billions. This is a property of how rare fatal crashes are, not of how good the driving is. No improvement in perception or planning shortens the distance needed to prove the improvement. It’s why the field leans on simulation, staged rollout and disengagement proxies, and why deployment has advanced one metropolitan area at a time.
10. What this predicts
The distance column turns the usual futurology into something closer to measurement. Three groups, three different expectations.
Where the gap is large, progress continues. Substrate has seven orders of magnitude of thermodynamic headroom; what binds it is capital, power and policy, all of which can change. Distribution has real room under the capability-density limit. Neither of these areas is going to stall for reasons of physics.
Where the gap is closed, progress stalls regardless of investment. Exact attention is at its complexity-theoretic threshold. Serving is pressed against the bandwidth roofline and the speed of light. The most-cited benchmark in the field has passed its own label-noise floor. In the embodied group, energy density and actuation are at the wall, and the safety-validation arithmetic doesn’t care how good the model gets. Money doesn’t buy past any of these; only a change of problem does.
Where no bound has been drawn, we genuinely don’t know. This is the uncomfortable part, because it’s precisely the set of areas everything now depends on. Post-training has no impossibility theory. Run-time reasoning is bounded by verifier quality in checkable domains and by nothing legible outside them. Agents have compounding arithmetic and no threshold theorem. These three carry most of the current expectations about the next few years, and they’re the three where no one can say what the ceiling is.
If you want one question to track, it’s the threshold question for agents: can verification break the compounding of per-step error the way error correction broke it for unreliable hardware? Every other open question in the delivery plane follows from that one.
What to keep in your head
- Progress was never one curve. It was nine areas moving in parallel on different schedules.
- At any moment one area was the binding constraint, and the history is that constraint migrating: architecture, then the training recipe, then alignment, then serving cost, then run-time reasoning, then reliability.
- The visible jump in late 2022 was mostly post-training and interface. The underlying capability had existed for two years.
- Scaling was predictable, which is what made it financeable. Chinchilla’s correction, not raw size, is why small models got good.
- Constraints determine which area a lab attacks. Abundance optimises capability; scarcity optimises efficiency; the efficiency work then compounds for everyone.
- Three areas are at a hard bound, two have vast headroom, and three have no bound anyone has drawn.
- Embodied AI is slow not because it’s intellectually harder but because its lanes sit close to physical limits that software lanes don’t.
- When a benchmark passes its label-noise floor, claims of progress on it stop carrying information.
The last nine years felt abrupt because several curves compounded while only one of them was visible from outside. The reason to watch metrics rather than announcements: the next jump will show up as a number in one of these lanes well before it shows up in a product.
A note on the figures. The Landauer comparison, the irreducible-loss term, the 2-bits-per-parameter result, the attention hardness threshold, the ~300 trillion token text stock, the MMLU label-noise ceiling, RAND’s 275 million miles, lithium-ion energy densities and the lidar price history are drawn from published work rather than recalled. The bandwidth roofline and the step-compounding figure are derived here from stated specifications and hold only for the configuration named. Assigning a milestone to a single lane is an editorial judgement; several belong in two at once, which is rather the point of drawing them in parallel.