Every engineering team building autonomous coding agents or complex evaluation loops has spent the past eighteen months treating context length as an unalloyed good. Model providers marketed the expansion from 32k to 128k, 1M, and 2M tokens as a straight climb in capability. The standard pattern across agent frameworks became predictable: feed the full repository, append the entire test suite, include every terminal command output, and let attention sort out the dependencies.

That pattern rests on an implicit assumption: that generating a token carries a uniform marginal cost regardless of how deep it sits in the prompt.

On real physical hardware, it does not.

A research team at Nvidia led by Peter Belcak recently published empirical measurements on the hardware cost of long-context token generation (arXiv:2611.05306) across frontier open-weight models including Qwen 3 Coder Next, Kimi K2.6, and GLM 5.2. Running these models on dedicated H100 clusters at market rental rates ($3.85 per GPU-hour), the researchers measured peak decode throughput across varying context depths.

The data reveals a steep, convex cost escalation:

  • On GLM 5.2, generating 1M output tokens at a context depth of 256,000 tokens costs 3.6 times more than generating them at a 16,000 token depth.
  • On Kimi K2.6, the cost multiple rises to 5.6x.
  • On Qwen 3 Coder Next, generating tokens at the tail of a 256,000 token context costs roughly $10.00 per million tokens, compared to approximately $1.40 at 16,000 tokens. That is a sevenfold increase in unit production costs for the exact same model weights.

On local edge silicon, the penalty hits clock cycles rather than dollars. Inter-token generation latency approximately doubles as the prompt lengthens, halving throughput and doubling power draw per completed answer.

Why the Late Token Tax Happens

The physics behind this penalty stem directly from the key-value (KV) cache. Generating an output token requires evaluating attention against every previous token stored in memory. As an input prompt expands from ten thousand tokens to a quarter of a million, the physical footprint of that cache balloons from megabytes into tens of gigabytes per concurrent user stream.

Once accelerator high-bandwidth memory (HBM) fills with cached attention states, the serving engine faces an unyielding trade-off. It can slash batch sizes to prevent out-of-memory crashes, or it can swap cache lines to slower host memory. Either choice collapses memory bandwidth utilization. The compute cores spend their duty cycles waiting for memory transfers rather than performing matrix arithmetic. The late token tax is the cost of keeping dead memory warm.

The Retail Pricing Illusion

For developers using hosted model APIs, this hardware reality has historically been masked by blended pricing. Providers quoted flat rates per million tokens, creating an invisible cross-subsidy where short, simple prompts paid high margins to absorb the memory footprint of context-heavy enterprise workloads.

As multi-turn coding agents and terminal loops became the primary drivers of token volume, that subsidy broke down. Model providers responded by introducing tiered pricing (often charging double or triple rates for contexts beyond 128,000 tokens) and adding prompt-caching discounts to salvage their margins.

Developers adapted to these price signals by writing larger budget checks rather than re-evaluating their system designs. Long context became an expensive, unhedged substitute for state compression. Rather than deciding which intermediate variables, test results, and file paths actually mattered for the next step, frameworks dumped raw execution histories back into the prompt. Paying a compounding premium to avoid making an architectural decision is an expensive habit.

The Compaction Counterfactual

The practical core of the Nvidia paper is its second half. Evaluating long-horizon agent benchmarks including SWE-bench Verified, SWE-bench Multilingual, and Terminal-Bench 2.1, the researchers tested an explicit compaction harness called Ephemeral-only Context Compaction.

Instead of letting conversational histories accumulate into hundreds of thousands of tokens, the harness periodically condensed the execution state into a structured recap before resetting the active context window clean.

The findings dismantle the necessity of brute-force context windows:

Across hundreds of verified GitHub issue-resolution tasks, models operating with disciplined periodic compaction retained their baseline problem-solving accuracy. On long-horizon software engineering workloads, agents taking isolated actions against short, compacted contexts showed no degradation in resolution rates. The system achieved identical functional outcomes while operating at the lower, single-dollar tier of the hardware cost curve.

What This Means for Agent Design

The strategic takeaway extends beyond benchmarking. When evaluating agent architectures, inspect the state transition design before the context window specifications. A framework that boasts about processing two million tokens in a single shot is often paying a sevenfold premium to avoid building a state machine.

In production engineering, disciplined state compaction clears at a higher risk-adjusted return than brute-force memory allocation.