AI & ML
Where KV-Cache Error Lives in Looped Language Models: A Diagnostic Study of Ouro and Huginn
arXiv cs.DC0 views
arXiv:2610.41825v1 Announce Type: new Abstract: A looped language model applies one block of layers 4 times per token and keeps a key/value (KV) plane for each loop. We study where storage error changes the next-token distribution and how shared-weight error accumulates, using Ouro-1.4B (4 loops) on GSM8K and MATH and Huginn-0125 (32 loops) on GSM8K-derived chains, with dense, simulated quantization. Per unit of error, values carry 94% of the diagonal Fisher mass, but measured KL is dominated by keys with larger quantization errors. Local second-order prediction follows the measured KL of typical sequences, including 136 held-out chains, but misses a few confident rows, and a local second-order factor for shared weights at 32 loops ranges from 1.65 (INT8) to 4.09 (INT4), so it depends on the error direction. A cell-level allocator lowers one-step KL at 25-48% of BF16 bytes, while paired task-accuracy comparisons depend on budget and the 25% gap against K4V2 shrinks to near zero under a post hoc answer-extraction check. A later Ouro-2.6B test retains the key/value split but falsifies measured-cost allocation's advantage over Fisher at 31%; across all 21 ledger registrations, 60 of 176 criteria failed.
Source:https://arxiv.org/abs/2610.41825PreviousMemX: Memory-Aware Automatic Parallelism for Cross-Vendor Heterogeneous GPU TrainingNextA Deterministic Commitment Rule over Unstructured DAGs for Optimistic Replication
Related posts
I measured 528 sites' robots.txt: two thirds blocking GPTBot also block the crawler that cites them
0dev.to
My logo heals itself. Then I erased it into a stick.
0dev.to
Startup Founder Interview: RAI Is Building the Cognitive Architecture Behind Humanoid Robots
0HackerNoon

Your Prospecting Play Changed. Enrolled Contacts May Still Be Running the Old One.
0HackerNoon
Why AI-Native Organizations Don't Always Need Flatter Teams
0HackerNoon
I Used NYT Cooking’s New AI-Powered Recipe Modifier, and It’s Actually Pretty Good
0Wired