yolow64 0.2: a new decode kernel that beats Turbo-Base64 under full validation

yolow64 is the Base64 codec extracted from PageSift, published as 0.1 on Nov 9. Version 0.2 (plus a docs-only 0.2.1) is out, and the headline is decode.

Decode: new kernel

The bulk decoder was replaced with a four-block AVX2 hash loop using clean-room translation/validation tables, speculative validation, and per-quad constants held in registers. Full validation is retained for every block — nothing was traded away for speed. BMI2 pext tails handle padded 24- and 172-character inputs, and exact-length no-padding inputs of 16/32/64/128 characters bypass the general dispatch entirely.

The comparison that matters is against Turbo-Base64 built with FULLCHECK=1. Turbo's default build checks only the first of four blocks in its hot loop; comparing against that would be comparing against 25% validation. With validation equalized (GB/s, i7-10700, six-process medians):

Short inputs (ns/call): first at 12 B (7.00), 16 B (8.53), 24 B (10.14), 48 B (12.20), 96 B (15.68) and 128 B (17.18) in an eight-library run. Not first at 8, 32 and 64 B — those cells are still someone else's, and the raw tables in the repo say so.

Encode: streaming and cache-aware

• Adaptive non-temporal stores kick in above 3/7 of the detected LLC.
• A streaming encode API and a memory-bounded encode_to_writer join the one-shot calls.
• An optional parallel feature splits large inputs across cache-aware workers — and stays out of the way when the single-thread streaming path measures faster.
• Short 8/16/32-byte fast paths skip the generic tail.

Strictness went up, not down

Padding-bit checks now reject non-zero trailing bits at the short padded lengths and at the generic tail. AVX2 and BMI2 are detected independently: pext paths run only when both are present, AVX2-only and scalar CPUs fall back cleanly — a test-only switch forces the no-BMI2 path on a BMI2 machine to prove the fallback decodes identically. Exact-output canary tests cover caller-provided buffers from 1 B to 8 KiB, so the fast paths cannot write past your buffer.

Install

[dependencies] yolow64 = "0.2" 

Zero runtime dependencies, MIT, MSRV 1.89.

Caveats, as always: one shared machine (i7-10700, 16 GiB, AVX2+BMI2, no AVX-512), medians of repeated runs, snapshots rather than guarantees. Method, raw CSVs and the losing cells are in the repo: https://github.com/YOLOW92/yolow64 — the release record is research/release-0.2.0.md.

Its sibling extraction, yolowpdf, also landed on crates.io this week. If you push Base64 through a hot path, I'd like to hear where 0.2 lands on your hardware.