This article was created with the help of AI. The AXI architecture is by Rajnish Choubey. The tests were built and run by Shalendra Gaur.

An agent that keeps its whole chat has to hold that chat in memory for as long as the session is live. How many sessions fit on one chip is a memory question, so I measured it on a Google TPU chip.

The test

  • Chip: one Google TPU v6e in Colab. It reported a memory limit of 33,546,042,880 bytes (about 33.5 GB). JAX 0.11.2. Run on 31 October 2026.
  • What was put on the chip: real cache-shaped arrays (the shape of Qwen2.5-0.5B, bf16), session after session, until the chip reported out of memory. No model was run.
  • Lengths: the prompt of one AXI step (2,824 tokens), and the lengths of a kept chat from a 457-turn test on Gemini 3.8 Flash (mean 476,835 tokens, largest 949,591), plus 50,000, 100,000 and 250,000.
  • The plan and six yes/no lines were written and hash-locked before the run, with a probability on each.

Result

Prompt kept per session Tokens Sessions held on a TPU v6e-1 Sessions held on an NVIDIA T4 (earlier test)
AXI step 2,824 927 430
Kept chat at 50k 50,000 54 23
Kept chat at 100k 100,000 27 12
Kept chat at 250k 250,000 10 4
Kept chat, mean of the long run 476,835 5 2
Kept chat, largest of the long run 949,591 2 1
  • The ratio between the two ways is about 169 in memory (476,835 / 2,824). Memory grows in a straight line with tokens, so this is close to the token ratio by construction. What the test adds is the chip itself: its real memory limit and the real bytes a session takes.
  • A kept chat at the long run's mean length takes the memory of about 169 AXI-sized sessions, and at its largest length about 336.
  • Of the six lines, five read YES and one read NO. The NO: I guessed the real bytes would be within 2% of the plain count at every length. At the AXI length they were 4.5% over (36,277,760 against 34,701,312 bytes). The extra is about what rounding the token axis up to a tile size would give, but I did not confirm that cause.

What this does not show

  • This is memory only. Nothing here shows speed, quality, throughput or price. Real servers evict, offload, share prefixes and page memory, and the count means sessions in flight at once.
  • No model was run, the cache shape is a small model's, and there is one run per length.
  • The TPU holds more sessions than the T4 mainly because it has more memory (33.5 GB against 15.5 GB), not because of any speed.

AXI is a layer on the host side by Rajnish Choubey: the model is unchanged and the claims are about the state the host holds. The test page is docs/TPU_SESSIONS_TEST.md in the repository: https://github.com/gaurshalendra2412/axi-runtime

White paper, "AXI: The Cure for the Cost Crisis in AI" (Rajnish Choubey and Shalendra Gaur): https://doi.org/10.5281/zenodo.23958737