Workload shape
The session you are designing — an agent loop, tool rounds, outputs.
Every tool result is prefill cost plus KV rent for the rest of the session.
Output tokens price at roughly 3–5× input — brevity is a cost lever.
Context architecture
The choices that set your prefix-cache hit rate.
Stable-prefix ordering, append-only
Stable content first; never mutate early tokens (no timestamps at the top). One mutated byte re-prefills everything after it.
Session affinity (KV pinning)
Turns land on the worker holding the session's blocks — or the provider's cache actually gets hit.
History compaction
Above 16K, compact history to 40% — superseded plans and stale tool output summarised away. A product decision about what the agent may forget; each compaction pays one fresh re-prefill of the rewritten context.
Decompose to subagents
Split the rounds across 4 isolated subagent contexts reporting 300-token summaries to a thin orchestrator — linear instead of context-times-turns.
Budgets
Effort and sampling — the dials that multiply decode.
Default low, escalate on a verifier or gate signal — accuracy improves much less than linearly with budget.
Applied to 20% of rounds; multiplies their thinking + output by k.
Routing & pricing
IInferenceRouter: right-size the model per round. Prices are illustrative defaults per MTok — edit to your provider.
Frontier in
cached in
out
Small in
cached in
out
$ / million tokens. Cached-input discounts exist because loading KVs beats recomputing them — the tmem vs tcompute split, on the price sheet.
Your design
$0.31 /session
$310 per 1,000 sessions
Naive build of the same solution
$2.87 /session
unordered context · no compaction · high effort · frontier-only · monolith
Design dividend
9.3×
$2,560 saved per 1,000 sessions
Where the tokens go
Per session. Grey is nearly free; the coloured and black segments are the expensive decode.
cached input (prefix hits)
fresh input (prefill)
thinking tokens
visible output
Peak context 22K tokens ≈ 7.0 GB KV at the ~320 KB/token GQA-70B figure (MLA-class models hold ~7% of that) — the HBM this one session occupies while it runs, or squats on during a slow tool call.
Cost as the session runs
Cumulative $ per session by round. Losing the cache turns the dashed naive curve quadratic; your design should stay near-linear.
Ranked: your next design move
Each lever priced from your current configuration — estimated saving per session if applied.