AI & ML
Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell
arXiv cs.IR0 views
arXiv:2611.21974v1 Announce Type: new Abstract: Low-bit LLM quantization is useful only if a model preserves service-relevant accuracy, executes through supported GPU kernels, and meets measured memory and latency limits. We study Qwen3-8B on NVIDIA B200 with vLLM, including INT2/INT3, GPTQ, NVFP4, and mixed-precision configurations. We verify native packed execution and precision on all 144 fused linear modules and evaluate Korean, English, numerical, and reasoning quality under preregistered gates. Packed INT2 and INT3 execute, but all seven tested INT2 grids degenerate on generation. Isolated module-restoration effects are strongly non-additive; Korean and reasoning sensitivity rankings disagree, and a promising INT4/INT8 development candidate fails independent final evaluation. A configuration with 16-bit activations, 98 NVFP4 weight-only modules, 46 INT8 modules, and an INT8 output head passes all gates on the untouched FINAL-81 set (5,168 items): 5.917 GiB of weight-loading memory (limit 5.98), Korean teacher-correct retention 0.904 (gate 0.90), and TPOT ratio 0.939 relative to BF16. It improves paired accuracy over NVFP4 W4A4 by +1.4 percentage points (Bonferroni-adjusted interval [+0.2, +2.6]), compared with +3.4 points on the selection set. This comparison changes both activation precision and INT8 protection; exploratory analyses suggest that the activation change contributes substantially, but the isolated allocation effect is not established. Final checkpoints were not retained, and bitwise rebuild reproducibility remains unverified.
Source:https://arxiv.org/abs/2611.21974PreviousWhat RL on Expert-Built Tasks Teaches Nemotron 3.5 SuperNextSIMQ: A Single Graph-based Index for Multi-Vector Query
Related posts

iPhone 18 and iPhone Air 2 Display Production to Begin Soon Ahead of Expected March Launch
0MacRumors

iPhone Air 2 Reportedly Sticks With Same Display
0MacRumors
Hours later, Airbyte is still unavailable
0Hacker News New
Show HN: Qpilot – an AI agent that runs plain-text manual test cases in Chrome
0Hacker News Show
Show HN: Arbituro
0Hacker News Show
Show HN: Neopane. Run Neovim once, show its windows in your multiplexer's panes
0Hacker News Show