I wanted to remove personal data from survey answers before they left the user's browser. Names, phone numbers, addresses, ID numbers: found and masked on the device, so the server only ever sees [PERSON] and [PHONE].
That needs a model small enough to ship to a web page. This post is about how small it got, and about the part I did not expect to write: my own inference engine.
The result is newmean-pii: four models from 2.6 MB to 15.5 MB, 54 languages, Apache-2.0. You can try it at pii.newmean.ai without installing anything.
Step 1: shrink the model
All four models start from mMiniLMv2 (6 layers, hidden size 384) and are distilled from two much larger private teachers. Then they are cut down:
- Vocabulary pruning. The XLM-R vocabulary has about 250,000 pieces. Only 35,298 occur in my training data, so the rest go.
- Low-rank embeddings. After pruning, the embedding table is still most of the model. The smaller tiers factor it into a narrow table and a projection.
- 4-bit weights. Every weight matrix is stored in 4 bits. For the two larger tiers the embeddings are too, which changes the miss rate by at most 0.1 points and saves 2.7 to 5.2 MB.
- Layer sharing. The smallest tier has one transformer layer and applies it four times.
| tier | structure | parameters | download |
|---|---|---|---|
| large | 6 layers, full pruned embeddings | 24.4M | 15.5 MB |
| medium (default) | 6 layers, embeddings rank 192 | 17.7M | 11.3 MB |
| small | 4 layers, rank 64 | 9.6M | 7.1 MB |
| nano | one shared layer applied 4 times, rank 32 | 3.1M | 2.6 MB |
Step 2: notice the runtime
With the model at 2.6 MB, I looked at what else the page downloads. onnxruntime-web is about 2.4 MB after brotli. The runtime was as large as the model it ran.
I tried a minimal build of onnxruntime-web with only the operators my models use. That came to 0.34–0.43 MB, which is better but still a general runtime doing a specific job.
My models are all the same architecture: an XLM-R style encoder, a 4-bit matrix multiply, an embedding lookup, a token classifier. A program that runs only that does not need a graph interpreter, an operator registry or a memory planner.
Step 3: 483 lines of C
The engine is one file of freestanding C, compiled to WebAssembly with SIMD. No libc and no imports, so the module asks nothing of the host:
clang --target=wasm32-unknown-unknown -O3 -msimd128 \ -ffreestanding -nostdlib -Wl,--no-entry -Wl,--strip-all \ -o pii.wasm pii.c No libc means writing the few things I needed. The allocator is a bump pointer that never frees, because the model is loaded once and the buffers live as long as the page:
extern unsigned char __heap_base; static uintptr_t heap_top = (uintptr_t)&__heap_base; void *pii_alloc(uint32_t n) { uintptr_t p = (heap_top + 63) & ~(uintptr_t)63; uintptr_t end = p + n; uintptr_t have = __builtin_wasm_memory_size(0) * 65536u; if (end > have) { uintptr_t pages = (end - have + 65535) / 65536; if (__builtin_wasm_memory_grow(0, pages) == (uintptr_t)-1) return 0; } heap_top = end; return (void *)p; } The rest is the model itself: dequantize a panel of 4-bit weights, multiply, layer norm, attention, GELU. Each is written once, for the one shape it will ever see.
There are two builds. The second uses fused multiply-add through WebAssembly relaxed SIMD, and the loader picks it where the browser supports it (Chrome, Edge, Firefox).
JavaScript glue and WebAssembly together are about 13 KB after brotli.
What it bought
Desktop, headless Chrome, one thread, the 11.3 MB model:
| runtime | download | load | 40 tokens | 252 tokens |
|---|---|---|---|---|
| built-in engine | 13 KB | 63 ms | 23 ms | 147 ms |
| built-in engine, relaxed SIMD | 13 KB | — | 14 ms | 91 ms |
| onnxruntime-web 1.31 | 2.4 MB | 230 ms | 31 ms | 146 ms |
On four phones (one iPhone, three Android), the built-in engine is about 1.9× faster on short texts and about 24% faster on long ones. First load takes 0.77 s instead of 2.85 s. A 40-token message takes about 26 ms with the default model.
Where it does not win:
- Without relaxed SIMD, long texts are a tie (147 ms against 146 ms).
- onnxruntime-web can use several threads on cross-origin isolated pages. Mine is single-threaded.
- WebGPU runs long texts in 15 ms. Its runtime is 28.7 MB, so I left it out.
onnxruntime-web is still in the package as an option. Each engine is a separate chunk, so a page downloads only the one it picks.
How I know it is right
A hand-written engine is easy to get subtly wrong, so it is checked against the reference rather than trusted:
- Both engines return the same spans as the Python evaluator on a 412-text parity set (410 for one nano file, the same two texts for both engines).
- On every phone, model size and engine, the spans matched a desktop reference on 108 texts in 54 languages.
The Python, iOS, Android and Flutter packages run the same models through one C core, with token ids, offsets and logits equal to the npm package's.
What it costs
The engine runs exactly one architecture. If I change the model's structure, I change the C. That is a fair price for four models that share a design. It would be a bad one for a project that swaps architectures every month.
The model is not perfect either
Share of personal-data characters left unmasked, on datasets the models were not trained on: 4.9–6.3% on OpenPII (30 languages), 6.1–9.9% on Nemotron-PII (English), 9.5–11.5% on KDPII (Korean). Finance documents are the weak spot at about 21%. The two smallest models also mask too much in short table cells.
So it is a strong first pass, not a guarantee. The README has the full table against 8 other detectors and a regex baseline, and the technical report has the method.
Try it
npm install @newmean-ai/pii import { createRedactor } from "@newmean-ai/pii"; const redactor = await createRedactor({ size: "medium" }); const { text } = await redactor.redact("My email is kim@example.com"); // "My email is [EMAIL]" - Demo: pii.newmean.ai
- Code: github.com/newmean-ai/pii
- Engine source: js/engine/pii.c
I would like to know where it fails. If you speak one of the 54 languages, try names and addresses the way people really write them, and open an issue with what it missed or masked by mistake.