A systems field guide for product managers
AI System Design
How workload shape—not just token count—determines the cost, speed and architecture of an AI product
Core premise: token count measures volume, not the shape of the work. How much information a product carries, how long it carries it, how often it pauses and how quickly it must respond decide the architecture.
Follow one requestThe premise
A model call is not a product.
A document pipeline and a coding agent might each process one million tokens. The document pipeline reads independent files overnight and returns short results. The coding agent carries one task for hours, repeatedly pausing for tools while its working information grows. The token count is the same. Almost every important system decision is different.
The difference is measurable. On one NVIDIA H100, the document pipeline runs ten jobs at a time through a queue that drains before morning. The coding agent gets two sessions on the same card, all afternoon. The arithmetic is worked out below, and it is the reason a token budget is not a capacity plan.
Two forces have made this question urgent rather than academic. Products stopped making single model calls and started running agents that loop through tools for hours. And the same products now have to run partly on a phone, a robot or a factory gateway, where the memory budget is not elastic and there is no larger machine to move to.
This guide follows one real request, decides where its information should live, then uses five questions to compare the systems behind chat, agent fleets, coding, on-device assistants, industrial edge, monitoring, documents and voice.
Act 01 · Draw the system
Draw the whole system.
Begin with the path from a user’s need to a useful result. The model sits in that path; it does not replace it.
“Where is order 4821, and what happens if it is late?”
Fetch the order, select the policy and carry forward relevant conversation.
Interpret the request, decide whether another lookup is needed and form an answer.
Show the status, explain the policy and offer the next action.
One support request
The application decides what the model gets to know.
The model does not automatically know the current location of order 4821 or the company’s delayed-delivery policy. The application has to retrieve those facts and decide which parts belong in this request.
A weak system sends the entire customer profile, every order, all previous tickets, the complete policy handbook and the full conversation. A better system sends the current order, the relevant policy section, a small amount of active context and clear instructions.
The second request is not merely cheaper. It is easier to keep current, easier to audit, less likely to expose unrelated data and less likely to distract the model with conflicting evidence.
Who decides what enters the request—and by what rule?
Five questions
Describe the work before choosing the model.
- How much information is needed? A single order lookup is small. A repository-wide coding task may carry files, logs, errors and decisions for hours.
- How long does the work continue? A one-shot extraction ends cleanly. An agent that loops through tools for two hours must control growth and decide what it can forget.
- How long are the gaps? A two-second tool call and a thirty-minute human pause should not keep expensive working state in the same way. An agent working autonomously has almost no gaps at all, which is why it is expensive to hold.
- How quickly must it respond? A batch result can wait. A voice assistant cannot leave a person in silence. A robot arm has a hard deadline.
- Where does it run? A datacentre lets you add memory. A phone, a robot or a plant-floor gateway does not, and its budget is shared with everything else the device is doing.
The fifth question used to be assumed away. It no longer can be: the same assistant increasingly answers on the device when it can and escalates to a server when it must, and those two paths are different systems wearing one product name.
Act 02 · Place the information
Put information in the right place.
Not everything the product knows should become conversation history.
The same request, decided line by line
“Where is order 4821, and what happens if it is late?”
Here is everything the product knows that could plausibly go into this one request, and the decision for each. The column that does the work is the last one — most of what is available should not be there.
Large, expensive on every turn, stale the moment the order moves, impossible to audit after a bad refund, and full of material that competes for the model’s attention.
Smaller, current at the moment of the decision, and explainable line by line when someone asks why the refund was approved.
Notice that “where it lives” and “how often it changes” decide the answer between them, and neither is a question about the model. A fact with an owner and an update rule belongs with its owner. The conversation carries the active problem — not a copy of the product’s database.
Conversation is not memory
Text, model state and durable facts are different objects.
The application can store a transcript as text. While the model processes that text, it also creates temporary numerical working state—commonly called the KV cache—that makes an immediate continuation efficient. Keeping that state ready consumes fast memory; losing it means processing the text again.
Neither object should be confused with durable product state. An order status belongs in the order system. A camera event belongs in an event store. A code change belongs in a file. The model can retrieve these facts when they matter instead of dragging them through every turn.
If this conversation disappeared, which facts would the product lose?
Long tasks
Compaction is a decision about forgetting.
A coding agent may accumulate file contents, test output, errors, diffs and failed approaches until the conversation becomes unwieldy. Shortening it into a summary creates room, but anything omitted is no longer available unless it was written somewhere durable.
The asymmetry is what matters: source files and test results can always be read again, so losing them costs a tool call. The reasoning — why this approach was abandoned, what the error actually meant — existed only in the conversation. Lose that and the agent repeats the work it already did.
What must survive when this task is shortened or restarted?
Approaches
Seven ways to keep the request small.
“Externalise state and compact strategically” is the right instinct and a useless instruction on its own. These are the mechanisms underneath it, each with the price it charges.
-
01
Give every fact a system of record
Order status lives in the order service, entitlements in billing, events in the event store. The conversation carries an identifier, not a copy. Nothing in the transcript can go stale, because the transcript no longer claims to know.
Cost: a lookup on the critical path. Budget it in your first-response time. -
02
Retrieve at decision time, not at session start
Preloading the customer’s profile “in case it is needed” pays for it on every turn and is wrong the moment it changes. Fetch the two fields this decision requires, when it requires them.
Cost: the model must be able to ask. This is what tool definitions are for. -
03
Keep tool output out of the transcript by default
A test run, a query result or a page of logs can be tens of thousands of tokens, most of it irrelevant. Write the full result somewhere addressable, put a short structured summary and a handle in the conversation, and let the model request the slice it needs.
Cost: an extra round trip when the model does need the detail. Usually far cheaper than carrying it every turn thereafter. -
04
Isolate work in sub-agents
A narrow task — search this codebase, check these three invoices — can run in its own context with its own brief and return only its conclusion. The exploration never enters the parent conversation, so the parent does not carry it for the next two hours.
Cost: the parent cannot see the reasoning it did not receive. Make the sub-agent return its evidence, not just its verdict. -
05
Compact to a schema, not to a summary
Free-text summarisation loses whatever the summariser judged unimportant, which is exactly the thing you will need. Compact into fixed fields instead: goal, constraints, decisions made and why, files changed, tests passing, tests failing, open problem, next action.
Cost: schema design. The upside is that a fixed shape can be tested, and a missing field is visible. -
06
Write progress into durable artifacts as you go
If the milestone exists in a file, a commit, a task record or a test result, compaction cannot destroy it. If it exists only in the conversation, compaction is a data-loss event.
Cost: the discipline to checkpoint before the context is nearly full rather than after. -
07
Order the request by rate of change
Stable instructions first, then slow-moving reference material, then session state, then the current turn. The shared opening stays byte-identical across requests, which is the only condition under which it can be reused.
Cost: none, and it is the most commonly skipped step in the list.
The trade-off nobody mentions
Dynamic retrieval and prefix reuse pull in opposite directions.
Fetching authoritative rules at request time keeps them current. Reusing a cached prefix requires the beginning of the request to be identical to last time. Do both carelessly and you get neither: a freshly retrieved policy paragraph, injected above the instructions, changes byte one of the request and invalidates every cached position after it. The system is now current and slow, and the metric that moved is first-response time.
The resolution is placement, not choice. Anything stable goes first and stays byte-identical. Anything freshly fetched goes after it, as late in the request as the task allows. A timestamp, a session ID or a personalised greeting at the top of a prompt is one of the most expensive lines a product can ship, and it is almost always there by accident.
In your prompt, what is the first thing that changes—and does it need to be there?
What happens between turns
Keeping a conversation ready is a memory decision.
The product sees one conversation. The serving system sees model state competing for a limited amount of fast memory.
Three places
Ready, parked or rebuilt.
Between two turns, a conversation’s working state is sitting in one of three places, and the choice is a pricing decision the product team rarely knows it is making. Keeping it ready costs scarce accelerator memory. Parking it in cheaper host memory costs a transfer before the next turn can start. Discarding it costs nothing until the user returns, then costs a full reprocessing of the transcript.
The serving team usually owns this setting. The product team owns the fact it optimises for: how long a user actually takes to reply. Those two groups often never speak.
Working state disappears for three different reasons—an idle timer expiring, a busy machine evicting the least recently used state, and generation being preempted mid-answer. They produce three different user complaints, and the symptom table at the end of this guide separates them. For why they exist, along with the attention, KV-cache and exact-prefix mechanics underneath, read How an AI Model Turns Input Into Output.
Act 03 · Compare the workloads
The same model can sit inside very different products.
Change the information, duration, gaps, latency requirement or location and the architecture changes with it.
Compare the workload
Pick a product. Watch the five constraints move.
The model may be similar. The system around it is not.
Short sessions, long gaps and modest context. This is the ordinary chat workload most serving systems expect.
Chat
- Information
- Small, grows slowly
- Duration
- Minutes
- Gap between steps
- Tens of seconds to minutes
- Latency pressure
- Fast first word
- Memory elasticity
- Server-side, can add capacity
Chat begins small, grows gradually and pauses unpredictably while people read and think. Design for quick follow-ups without assuming every user will return soon.
- Stable shared opening
- Relevant history only
- Measure return gaps
Agent fleet
- Information
- Very large, grows every step
- Duration
- Hours, unattended
- Gap between steps
- None — it never waits for a human
- Latency pressure
- Cost per task, not first word
- Memory elasticity
- Server-side, can add capacity
An autonomous agent holds a large context for hours with no idle gaps, and each step resends everything before it. Cache reuse and context curation decide whether the product is economically viable, long before model quality does.
- Protect the prefix
- Isolate in sub-agents
- Cost per completed task
Coding
- Information
- Very large, grows fast
- Duration
- Hours
- Gap between steps
- Seconds
- Latency pressure
- Throughput over speed
- Memory elasticity
- Server-side, can add capacity
A coding task carries files, tool results, errors and decisions for hours. Control what enters the conversation and preserve milestones somewhere durable.
- Filter tool output
- Checkpoint progress
- Compact at milestones
On-device
- Information
- Small, and hard-capped by hardware
- Duration
- While the app is in front
- Gap between steps
- Tens of seconds to minutes
- Latency pressure
- Immediate, or it feels broken
- Memory elasticity
- On the device — no larger machine exists
A phone shares a few gigabytes of unified memory with everything else running and hits a thermal limit before a memory limit. The engineering is in curating what gets sent and deciding when to escalate to a server.
- Hard context ceiling
- Define the offline set
- Design the escalation path
Industrial edge
- Information
- Bounded, nothing accumulates
- Duration
- Runs continuously
- Gap between steps
- Milliseconds
- Latency pressure
- Hard deadline
- Memory elasticity
- On the machine, often without a network
A robot cell, vehicle or line inspector must finish every decision inside a deadline, on hardware it cannot expand, sometimes with no connectivity. Worst-case latency matters more than average throughput, and a cloud fallback that occasionally takes four seconds is not a fallback.
- Bounded rolling state
- Worst-case deadline
- Degrade safely offline
Monitoring
- Information
- Fixed by design
- Duration
- Runs forever
- Gap between steps
- Fixed interval
- Latency pressure
- Seconds, and soft
- Memory elasticity
- Split: edge detectors, server judgement
Continuous video should not become a continuous conversation. Let inexpensive detectors create event records and call an expensive model only when a judgement is needed.
- Detect first
- Store events
- Escalate selectively
Documents
- Information
- Large, all at once
- Duration
- One call
- Gap between steps
- No turns at all
- Latency pressure
- Nobody waiting
- Memory elasticity
- Server-side, and time-shiftable
Independent documents with no person waiting can be queued and processed in batches. There is no reason to buy an interactive experience the product does not need.
- Independent jobs
- Queue the work
- Return structured results
Voice
- Information
- Small
- Duration
- Minutes
- Gap between steps
- Under a second
- Latency pressure
- Hard, conversational
- Memory elasticity
- Often split between device and server
Voice looks like chat but is governed by silence. The full path from speech detection through retrieval and speech generation must fit inside one conversational latency budget.
- Budget every stage
- Start speaking early
- Optimise the full chain
The arithmetic
One million tokens, three different capacity plans.
The figures below are for a specific, checkable setup: Llama 3.1 8B — 32 layers, eight key/value heads, 128 dimensions per head — running at two bytes per number on one NVIDIA H100, the 80 GB data-centre GPU that most current inference runs on. Your model and card will give different numbers. The ratios are the point, and the method is the part worth copying.
Where 128 KB comes from
One token to one invoice, in seven multiplications.
Every step below is one multiplication and the reason it is there. Nothing is rounded until the last line, so you can redo it with your own model and your own rate card. If head, key/value head and layer are not yet solid, the companion explains where they come from — the three of them supply three of the four factors here.
- 01One key vector, one attention head128 × 2 B256 B
Attention runs many times in parallel; each parallel copy is a head, and each works on a slice of 128 numbers. At two bytes per number — the usual 16-bit precision — that slice is a fixed cost per token, per head.
- 02Keys and values256 B × 2512 B
This is the factor people miss. Attention keeps two vectors per position, not one: the key it matches against and the value it carries forward. Hence the leading 2 in the formula.
- 03Every key/value head in the layer512 B × 84 KB
Llama 3.1 8B has 32 query heads but only eight key/value heads. That gap is deliberate: queries are used once and thrown away, keys and values are kept forever, so sharing them across query heads cuts this bill fourfold. This is the number that multiplies.
- 04Every layer4 KB × 32128 KB per token
Each of the 32 layers runs its own attention and keeps its own keys and values; nothing is shared between them. This is the line that produces the number the table above uses.
- 05A whole coding session180,000 × 128 KB22 GB
A long agent run carrying 180,000 tokens of files, tool output and history. The relationship is linear: double the context, double the memory.
- 06What one H100 has left80 − 16 GB64 GB for context
80 GB of high-bandwidth memory, about 16 GB of it holding the model’s weights. The remainder is what every concurrent session competes for.
- 07How many fit64 GB ÷ 22 GB2 sessions
Two coding sessions. Or, at 4,000 tokens a turn, about 130 chat conversations held ready at once on the same card.
And then the invoice
There are two ways to pay for that memory.
You either rent the hardware and do this arithmetic yourself, or you rent the tokens and let someone else do it. The price list in the second case is the arithmetic in the first, with a margin on top.
Path A · Rent the GPU
The memory budget is the bill.
An H100 runs roughly two to four dollars an hour. Take three. That hour costs the same whatever you run on it, so the cost per user is set entirely by how many users fit.
| Workload | Fits | Cost per session-hour |
|---|---|---|
| Chat, 4,000 tokens | ~130 | 2.3¢ |
| Coding agent, 180,000 tokens | 2 | $1.50 |
Same GPU, same hour, a 65× difference per user. Not because one workload is harder, but because context length decides how many people share the card. This is the number a pricing model has to survive.
Path B · Rent the tokens
The quadratic shows up as a line item.
Take the 50-step agent from above: each step adds about 2,000 tokens and resends everything before it, so the model processes 2,550,000 input tokens to produce a 100,000-token transcript.
| Per agent run | Input processed | Cost |
|---|---|---|
| No prefix reuse | 2,550,000 | $12.75 |
| With prefix reuse | 100,000 fresh + 2,450,000 cached | $1.85 |
At $5 per million input tokens, cache reads billed near a tenth of that and writes at a small premium. Roughly 7× on the same completed task, from one decision about the order of the request.
Figures are illustrative and the rate card moves; the shape does not. Two things are worth carrying away: memory scales linearly with context, so a context limit is a concurrency decision wearing different clothes. And an agent’s bill scales with the square of its steps unless the prefix is reused, which is why that one setting is worth more than most prompt engineering.
Every row above can be described as “about a million tokens a day.” One is about a hundred and thirty users, one is a queue that drains overnight, one is two engineers, and one never reaches your H100 at all. Token count told you the volume. It did not tell you how many machines to buy, and it is the number most roadmaps are still written in.
On the device
The budget stops being elastic.
A phone running a small model locally might have two to three gigabytes of unified memory available to it, shared with the operating system and every other app, and it is thermally limited before it is memory limited. At the 128 KB per token calculated above, working state alone would consume the entire budget within a few thousand tokens — which is why on-device models use aggressive quantisation, far smaller key/value dimensions and hard context ceilings rather than the generous windows their server counterparts advertise.
The design consequence is not “use a smaller model.” It is that context has to be curated before it is sent, that the device cannot hold a long conversation in working state between sessions, and that anything requiring a large context is a routing decision: answer here, or escalate to a server and accept the round trip, the privacy question and the offline failure mode.
In the loop
An agent’s cost grows with the square of its steps.
An agent step is not an incremental request. It is a new request whose input is the entire conversation so far. If each step adds about 2,000 tokens of tool result and reasoning, then step 50 sends 100,000 tokens of input to produce a few hundred.
Across the whole run, the model processes the sum of every intermediate length:
2,000 × (1 + 2 + … + 50) = 2,550,000 tokens processed
for a transcript that ends at only 100,000
Twenty-five times the transcript. That multiple is what prefix caching exists to remove: when the shared beginning is reused, a step reprocesses only what is new, and the run costs closer to the transcript length than to its square. This is why cache-hit rate is not an infrastructure statistic on an agentic product. It is the difference between a feature that ships and one that gets cancelled at the second invoice.
Agent fleet, in the datacentre
Memory is elastic. Attention is not.
An autonomous agent holds a large, growing context for hours with almost no idle gaps, because it is never waiting for a human to type. It is the most expensive thing to hold in memory and the least likely to be evicted safely. You can always add machines; what you cannot do is make a 200,000-token context produce a good decision when 180,000 of those tokens are stale tool output.
Primary constraint: context grows faster than its usefulness. Budget for curation, isolation and compaction as first-class work.Assistant, on the device
Attention is manageable. Memory is fixed.
A phone or a handheld industrial terminal has a memory ceiling set by hardware you do not control, shared with everything else running, and a thermal limit that arrives before the memory limit does. There is no larger instance. The context is small enough to reason about entirely, and the engineering goes into deciding what deserves to be in it and when to give up and call a server.
Primary constraint: a hard ceiling. Design the escalation path and the offline behaviour before the prompt.Act 04 · Configure it
Now configure it.
Everything above is a decision. Each one has a parameter attached to it, and most teams never touch them because nobody told them the parameters existed.
A product manager’s exposure to a model is usually one endpoint, a prompt and a model name. That is a fair description of a single call and a poor description of an agent: the controls that decide whether an autonomous agent is affordable are request parameters, not prompt wording. What follows builds one, in the Anthropic Messages API because its parameter names are public and stable enough to print — the same controls exist elsewhere under other names.
The use case
Overnight refund resolution.
Order 4821, the late delivery this guide opened with, was one of them. So were nine hundred others this week. Each claim needs the carrier record checked, the delayed-delivery policy applied, the customer’s history weighed, and a decision written: refund, partial credit, or escalate to a human. Nobody is waiting at 2am. The agent runs unattended until the queue is empty.
The mapping
Every decision in this guide is a parameter.
cache_controlMarks the end of the stable prefix so it is reused instead of reprocessed each step.toolsThe agent asks for the carrier record when it needs it, rather than carrying every order.context_managementClears old tool results from the context automatically, so claim 400 is not still carrying claim 1.task_budgetTells the agent its token ceiling so it paces itself and finishes, instead of being cut off mid-claim.effortBefore answering, the model can generate reasoning tokens — working out that the reader never sees but pays for as output. This sets how much of it to buy.modelA cheaper model reads the long carrier logs; the expensive one only sees the conclusion.The configuration
The same agent, written down.
from anthropic import Anthropic
client = Anthropic()
with client.beta.messages.stream(
model="claude-opus-5",
max_tokens=64000,
# Stable prefix. Refund rules, tone, escalation policy — the things that
# do not change between claim 1 and claim 900. The cache breakpoint goes
# at the END of this block, so everything above it is reused every step.
system=[{
"type": "text",
"text": REFUND_POLICY_AND_INSTRUCTIONS,
"cache_control": {"type": "ephemeral"},
}],
# How long the work continues, and how hard to think about each claim.
# "effort" buys reasoning tokens: output the reader never sees, billed
# and waited on like any other output. task_budget is advisory - the
# agent sees the countdown and wraps up gracefully. max_tokens is a
# hard cut, mid-sentence, with no warning.
output_config={
"effort": "medium",
"task_budget": {"type": "tokens", "total": 400000},
},
thinking={"type": "adaptive"},
# Keeping tool output out of the transcript, enforced by the server.
# Old tool results are cleared as the run proceeds, so a 900-claim night
# does not end with claim 1's carrier dump still resent on every request.
context_management={"edits": [{"type": "clear_tool_uses_20250919"}]},
# Fetching facts when needed. Nothing about order 4821 is preloaded —
# the agent asks for the authoritative record when it decides it needs it.
tools=[get_carrier_record, get_customer_history, issue_refund, escalate],
betas=["task-budgets-2026-03-13", "context-management-2025-06-27"],
messages=[{"role": "user", "content": "Work the delayed-delivery queue."}],
) as stream:
result = stream.get_final_message()
# The number that decides whether this is affordable.
print(result.usage.cache_read_input_tokens)Two lines deserve a note. thinking lets the model generate reasoning tokens before it answers — real output tokens, billed and waited on, that the reader never sees; effort is how much of that to buy, and it is the difference between a careful refund decision and an expensive one on a claim that needed no thought. And streaming is not stylistic: a run with max_tokens this large will hit an HTTP timeout without it.
The line that matters most
Watch cache_read_input_tokens, not the token total.
This is the quadratic cost calculated earlier, made observable. If that number is zero across repeated steps, the agent is reprocessing its entire history every time and you are paying the full square. It is the single most valuable number in the response object, and almost nobody reads it.
The usual cause is something small and volatile sitting above the cache breakpoint — a timestamp, a claim ID, a re-fetched policy paragraph, a greeting with the customer’s name. Any byte change in the prefix invalidates everything after it. The fix is placement: stable content first, volatile content after the breakpoint.
The escape hatch
Give the reading job to a cheaper model.
Carrier logs are long, mostly irrelevant and read once. Putting them through the expensive model is the most common avoidable cost in an agent like this. Run that sub-task separately on a smaller model and return only its conclusion to the main loop — the exploration never enters the expensive context, and never gets carried for the rest of the night.
summary = client.messages.create(
model="claude-haiku-4-5",
max_tokens=1024,
messages=[{"role": "user", "content": f"{CARRIER_LOG}\n\nDid it arrive late, and by how long?"}],
)
# Only `summary` goes back to the Opus loop. The log never does.One caution: caches are scoped to a model. Switching models inside a single conversation invalidates the cached prefix, which is why this runs as a separate call rather than a mid-conversation swap.
The same controls elsewhere
Every serving stack has these knobs. Only the spelling changes.
The names above are Anthropic’s. The reason to learn the underlying idea rather than the parameter is that the idea ports and the parameter does not. Here are the same four controls in three other places a team might actually be working.
cache_control on the last stable block
Implicit caching, on by default above 2,048 tokens; explicit caches for guaranteed reuse
--enable-prefix-caching, hashing the KV cache in fixed blocks
usage.cache_read_input_tokens
cachedContentTokenCount in the response metadata
Prefix cache hit rate on the metrics endpoint
effort, inside output_config
A thinking budget on the request
A property of the model you chose, not a dial the server gives you
--gpu-memory-utilization and --max-model-len: the arithmetic above, as a config flag
The discount is real
Prefix reuse is the largest single lever, on every platform.
Google documents a 90% discount on cached input tokens for its 2.5-generation models and later. Anthropic prices cache reads far below fresh input. vLLM’s saving is not a discount at all — it is compute you simply never perform.
Three different commercial models, one shared conclusion: the stable-prefix decision is worth more than almost any prompt change you could make, and it costs nothing but paying attention to the order of your request.
If you rent the GPU yourself
The arithmetic stops being theoretical.
An H100 rents for roughly two to four dollars an hour depending on provider and commitment. At that point the calculation earlier is not an analogy for your cost — it is your cost. The tokens you choose to hold in memory are the difference between two concurrent sessions and a hundred and twenty-eight on hardware you are paying for by the hour either way.
This is also where the abstraction stops protecting you. A managed API absorbs a bad context design into a slightly larger bill. A GPU you rented absorbs it by falling over.
Parameter names, beta identifiers and model availability move faster than any article. Treat the shapes above as the current form of durable ideas — a stable prefix, a bounded task, a filtered context and a cheap reader — and check the provider’s reference before shipping. The ideas outlive the spellings.
Act 05 · Measure it
Measure the experience, then find the system cause.
Infrastructure metrics become useful only when they explain something a user or product team can observe.
None of the rows above start with a model. They start with something a user noticed, and end with a decision about where information lives. That is the whole argument: the model is one component in a system you already know how to reason about, and the questions worth asking — what does this decision need, how long must we hold it, where does it run, what does that cost — are questions you were qualified to answer before any of this arrived.
What to carry out of here
Design the path, not just the prompt.
Draw one real request from need to result. Decide which information belongs in the conversation, which belongs elsewhere and what the system may forget. Then choose the model and infrastructure that fit the work.