This digest covers LLM inference and serving developments from 2026-11-11 to 2026-11-18, scanning arXiv (cs.LG/cs.DC), the vLLM blog, Hugging Face blog, Together AI blog, Modal blog, and Latent Space.
🔥 Highlights
Last year we built megakernels. This year we deleted them: The road to Together Compiler — an agent-driven compiler replaces hand-tuned kernels, up to 1.7x faster.

DeepSeek-V4.1-Flash on Vera Rubin NVL72: 58x Throughput vs. GB300 NVL72 at 387 tokens/s — new hardware, rewritten kernels, a massive throughput jump.

Storing the Gemma 4 K=V Cache Once: Exact Global-KV Memory Savings in a Paged Serving Engine with Unchanged Attention Kernels — free KV-cache savings hiding in plain sight, no kernel changes needed.
Making Preemption Productive in Long-Output LLM Serving — stops reasoning models from thrashing their own KV cache under memory pressure.
D-Day (Decision Models Day) — a single recap packs disaggregated serving, quantization, and TPU benchmarks together.

arXiv
Storing the Gemma 4 K=V Cache Once: Exact Global-KV Memory Savings in a Paged Serving Engine with Unchanged Attention Kernels — 2026-11-17
In Gemma 4 31B's global attention layers, value is derived from a normalized projection of key, so storing both is redundant — Gemma 4's own report mentioned up to 37.5% savings, but no open serving engine exploited it. A vLLM prototype using two overlapping strided views (same paged-attention kernel, bit-identical output) cuts decode KV memory by 1.34–1.40x at 32K–64K tokens and lifts decode throughput 24–38% on A100. Free win for anyone serving Gemma 4 who wants more context or batch size from the same GPU.Making Preemption Productive in Long-Output LLM Serving — 2026-11-15
Targets "KV thrashing": under memory pressure, reasoning models with long outputs get preempted and rebuilt repeatedly with little real progress. Stint stashes preempted KV snapshots in a CPU pool, caps wait time, and uses decoding hidden state to estimate remaining work when choosing who to preempt. On Qwen3-32B/H200, p99 time-per-token drops 50.7% versus recomputation and throughput rises 12% — directly useful for anyone serving long-output agents or reasoning models that choke under memory pressure.MiSTiQ: MicroScaling Trellis Quantization for Extreme KV Cache Compression — 2026-11-16
A drop-in replacement for codebook-based vector quantizers: combines trellis coding with microscaling (MX) formats, beam-searching the cache into a large reconstruction space without needing huge codebooks, so attention runs directly in low precision. Beats baselines on Llama3/Qwen3 perplexity and GSM8K at ~1.25 effective bits per element — worth a look for teams who've already squeezed classic KV-cache quantization dry and want to go further without losing quality.SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding — 2026-11-17
A search engine that jointly tunes decoding strategy, draft size, and precision per workload/GPU combo, guided by an analytical cost model plus a small LoRA predictor that generalizes to unseen GPUs without recalibration. Reports 2.01x throughput over plain autoregressive decoding and 1.36x over standard speculative decoding (both via vLLM) across RTX 5090, H100, L40, and MI300X — valuable for anyone running a heterogeneous GPU fleet who wants speculative decoding tuned without manual per-hardware calibration.The Kernel Reality Check: Benchmarking and Distilling Efficient Attention at Scale — 2026-11-17
A NeurIPS 2026 paper stress-tests kernelized (linear-time) attention against FlashAttention across training, vision, and causal inference. Uncomfortable finding: many literature gains don't reproduce against modern baselines, and kernelized methods trail softmax by 2–10 points even at scale — except in inference memory for causal LLMs, where they still win. Their proposed LARA++ recovers up to 99% of softmax accuracy but with real compute overhead. Useful reality check before swapping FlashAttention for linear attention in production — short answer: not yet, in most cases.High-Throughput Serving of Nemotron 3.5 Transcribe with Regime-Aware Megakernels and Kernel Fusion — 2026-11-15
A serving engine for Nemotron 3.5 Transcribe (ASR) that dynamically picks, step by step, between persistent megakernels and fused kernel chains, with crossover points measured offline since continuous batching changes the work shape every iteration. On H100 SXM it hits 1.39x tuned-vLLM peak (up to 4.39x faster at low concurrency) and serves 8x more concurrent requests within a 400ms p95 budget at near-identical WER — a principle (pick the kernel strategy per load regime, not a fixed one) that generalizes well beyond ASR.SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding — 2026-11-13
For edge-cloud speculative decoding (draft on-device, verification in the cloud), proposes asynchronous verification (draft trees keep flying while earlier responses are still in transit, discarding stale requests) plus an RTT-aware router choosing between on-device generation, chain-of-draft, full reasoning, or cloud speculative decoding — all without added latency since it runs parallel to draft prefill. With 250ms extra RTT and equivalent accuracy, it delivers 1.95x throughput on SPROUT and 1.55x on OlympiadBench while sending only 39–47% of queries to the cloud — relevant for anyone serving hybrid edge+cloud models who doesn't want to be a prisoner of network latency.Broadcast, Don't Replicate: Eliminating Weight HBM in Datacenter LLM Serving — 2026-11-12
A speculative hardware idea: instead of replicating read-only weights in each accelerator's HBM, a central server broadcasts weights over fiber and each inference node passively listens via co-packaged receive-only optics, with dynamic slack for variable context and MoE support. In simulation at 80,000 accelerators, it eliminates petabytes of HBM while matching or beating throughput and energy per token of an HBM-based system. Early-stage and hardware-heavy, but the kind of idea that would reshape datacenter-scale capacity planning if it leaves the simulator.
vLLM Blog
DeepSeek-V4.1-Flash on Vera Rubin NVL72: 58x Throughput vs. GB300 NVL72 at 387 tokens/s — 2026-11-12
vLLM rewrote kernels for Vera Rubin's segmented memory domains and new FP4 instructions, cutting ~28% of prefill time (DEP2) and ~25% of decode (DEP), with a tensor-parallel decode path hitting 1,810 tok/s per user at batch size 1. It also pools KV cache across rack DRAM and spare HBM, keeping prefix-cache hit rate near 97% under high concurrency — relevant for anyone capacity-planning on Rubin-class hardware.Inkling 2 Small on vLLM: Day-0 Support — 2026-11-12
Day-0 support for Inkling 2 Small (594B-param MoE / 17B active, up to 1M context), with mixed NVFP4/MXFP8 experts and a native speculator (DSpark) that risks 7 tokens per step. Practical result: up to 630 tok/s per user at TP4 with acceptance length of 3.5 — good reference data for anyone evaluating speculative decoding in production.H3-Image Day-0 Support: 4K in 0.87s with Vera Rubin — 2026-11-12
Not a text LLM, but relevant to the serving stack: MiniMax's H3-Image diffusion model served via vLLM-Omni with MXFP8 in the diffusion transformer, 4-GPU sequence parallelism, conditioning-projection caching, and parallel VAE decode with CUDA graphs, reaching 0.87s median latency for 4K images and 6,115 images/hour across 4 single-GPU replicas.Reinforcement Learning: Optimizing 0.6B Dense Models on TPU v7 via vLLM — 2026-11-12
Co-design between vLLM/tpu-inference, Tunix, MaxText, and Pallas to accelerate GRPO rollouts for a 0.6B dense model on a 64-chip TPU v7 slice: a multi-step decode loop compiled into a single XLA loop, Pallas block-size tuning, fewer per-token syncs, and KV cache preserved across weight updates. That took the full rollout-to-update cycle from ~100 minutes to ~56 seconds. RL-focused rather than general serving, but the lesson — scheduling/sync overhead dominates FLOPs for small models — applies to anyone running vLLM on TPU.
Hugging Face Blog
- llama.cpp v1.0.0: A stronger foundation for the local AI ecosystem — 2026-11-13 Marks
libllama/libggml's v1.0.0, with formal semantic versioning to avoid API/ABI breakage, support for 150+ architecture variants, and nominal mentions of quantization and "all mainstream speculative decoding methods." Practical value: the largest local serving framework is gaining API stability — important for anyone who depends on llama.cpp in production and wants predictable upgrades, even though the post itself is more about release policy than performance numbers.
Together AI Blog
Last year we built megakernels. This year we deleted them: The road to Together Compiler — 2026-11-16
Together's account of moving from hand-written GPU "megakernels" to the Together Compiler, an agent-driven system that automatically rewrites and fuses forward-pass kernels. Reports up to 1.7x more tokens/s than the megakernel baseline (1,111 TPS on MiniMax M3.1-Flash, 1,205 TPS on DeepSeek V4.1-Flash), plus the finding that CUDA Graphs + Programmatic Dependent Launch reaches 98% of megakernel performance without writing any custom kernel. Practical value: real ammunition for deciding between investing in custom kernels versus accepting Torch Compile overhead — the dispatch-latency numbers (37µs dispatch for 15µs of GPU work) are a concrete example of where time actually goes.Making open models work for long-horizon agentic workloads — 2026-11-16
A technical post (co-authored with someone from Ollama) on why prefill dominates latency in coding agents that resend the entire session every turn, and how prefix caching keeps cost from growing quadratically with turn count. Covers both the harness side (don't mutate the system prompt, load tool definitions last, truncate large outputs) and the engine side (routing byprompt_cache_key, offloading idle sessions' KV cache to CPU, PagedAttention, continuous batching, disaggregated prefill/decode over RDMA). Practical value: essentially a cost/latency optimization checklist for serving long-running agents, and it explicitly notes that GPU memory, not compute, is the concurrency bottleneck in most cases.
Modal Blog
Nothing relevant this week — the only post in the window ("Scaling Intelligence: A landmark commission and exhibition," 2026-11-13) is about an AI-generated art exhibition, not inference or serving.
Latent Space
D-Day (Decision Models Day) — 2026-11-11
A data-dense recap on "decision models" (models that score choices in a single forward pass instead of generating token by token) alongside a pile of serving benchmarks: vLLM on NVIDIA Vera Rubin running MiniMax-M3.1-Flash at 10.2x more tokens/s/chip versus GB300; vLLM's encoder-prefill-decode (EPD) disaggregation cutting P90 TTFT by 44–47% on DeepSeek V4.1-Flash; TPU v7 with SGLang-TorchTPU giving 3x decode at small batch sizes; and Unsloth's GGUF Dynamic v3.1 quantization showing 5–10% accuracy gains. Practical value: a single post bundling real disaggregated-serving, quantization, and hardware benchmarks with comparable numbers across providers.GPT-6.2 Astra, the #2 Frontier Model is better and 20%-600% cheaper, but has mixed reviews; 6.2 Sol likely main workhorse — 2026-11-18
Beyond GPT-6.2 pricing (Astra vs. Sol), the post bundles several inference-economics items: vLLM reporting up to a 30% increase in speculative-decoding acceptance length using a sliding-window draft for Kimi K3.1 Pro; NVIDIA using a 4B draft model for Qwen3.8-27B; Zyphra reporting 9–60x speedups at 10M-token context after converting attention to sub-quadratic; and SemiAnalysis data showing 1.8% of uncached requests accounted for ~15% of total spend. Practical value: concrete numbers for deciding where to invest serving-engineering effort — speculative decoding tuning or sub-quadratic attention.
Through-line
Hardware-aware speculative decoding keeps showing up from completely independent angles this week — SpecSearch's per-GPU cost model, vLLM's sliding-window draft for Kimi K3.1 Pro, NVIDIA's 4B draft model, and SpecRelay's edge-cloud router all converge on the same idea: a single fixed speculative-decoding config is leaving throughput on the table, and the next wins come from tuning it per hardware and per workload. The other recurring thread is that custom kernels are losing ground to smarter scheduling and compilation — Together deleted its megakernels for an agent-driven compiler, the ASR serving paper picks kernel strategy per load regime instead of a fixed one, and the Kernel Reality Check paper is a timely reminder to verify "efficient attention" claims against current baselines before betting a production stack on them.
What's catching your eye this week — the KV-cache tricks, the speculative decoding tuning, or something else? Drop a comment below.
