Autonomous coding agents, local developer tools, and edge robotics are bottlenecked by the traditional Python/PyTorch serving stack. On local workstations or 4 GB laptop GPUs, gigabytes of runtime overhead add steep cold-start latency and memory pressure.
We open-sourced Swizzle Runtime (swizzle-rt v0.1.0) under Apache-2.0: a single static Rust binary with custom x86-64 AVX2, NVIDIA CUDA (sm_75 to sm_120), and cross-vendor Vulkan compute kernels.
Core Architecture
- Bitwise XOR Shared-Memory GEMM Tiling (
GemmTile): Eliminates GPU SRAM bank conflicts and cuts shared-memory footprint by 9.1%. - RAD-1 Self-Speculative Decoding: Reuses the target model's own quantized weights to verify candidate branches in a single batched pass with 100% bit-identical greedy parity.
- Warm Start & Zero Wait: Two-tier KV/recurrent prefix cache plus double-buffered CUDA graph overlap.
- Ed25519 + BLAKE3 Execution Receipts: Every run can emit a cryptographic receipt (
swizzle-rt verify-receipt).
Verified Benchmarks vs. llama.cpp
Measured back-to-back on an AMD Ryzen 7 170 (8C/16T) + NVIDIA GeForce RTX 3050 Laptop GPU (4 GB VRAM, sm_86, CUDA 12.8), 5 repetitions with 5s cooldown, verified bit-identical greedy parity:
| Model | Device | Test | Baseline (tok/s) | Swizzle Runtime (tok/s) | Delta | Swizzle Latency |
|---|---|---|---|---|---|---|
Qwen3-0.6B-Q8_0.gguf | CUDA | pp512 | 11,550.31 | 11,804.23 | +2.2% | TTFT 43.37 ms |
Qwen3-0.6B-Q8_0.gguf | CUDA | tg128 | 196.84 | 209.70 | +6.5% | ITL p50: 4.84 ms |
Qwen3-0.6B-Q8_0.gguf | CUDA | edit256 | 737.47 | 759.75 | +3.0% | edit_identical: yes |
Qwen3-0.6B-Q8_0.gguf | CPU | edit256 | 65.80 | 90.61 | +37.7% | edit_identical: yes |
LFM3-1.7B-Q4_K_M.gguf | CUDA | tg128 | 134.82 | 144.26 | +7.0% | ITL p50: 7.04 ms |
LFM3-1.7B-Q4_K_M.gguf | CPU | tg128 | 16.15 | 18.06 | +11.8% | ITL p50: 55.85 ms |
Try It in One Command
Linux x86_64 (CUDA 12.8):
curl -fsSL https://raw.githubusercontent.com/Swizzle-AI/swizzle-runtime/main/install.sh | sh Windows x86_64 (PowerShell):
irm https://raw.githubusercontent.com/Swizzle-AI/swizzle-runtime/main/install.ps1 | iex Docker (GHCR):
docker run --rm -p 8080:8080 -v "$PWD/models:/models:ro" ghcr.io/swizzle-ai/swizzle-runtime:latest /models/model.gguf - GitHub Repo: https://github.com/Swizzle-AI/swizzle-runtime
- Submit your GPU numbers to the Hardware Leaderboard: https://github.com/Swizzle-AI/swizzle-runtime/issues/5
- Original Substack deep-dive: https://mlblueprints.substack.com/p/swizzle-runtime-v010-how-we-reached
