284B on 64GB RAM: We Read the Small Print Before Downloading 56 GiB

Everyone who has bought a car knows how to read a fuel-economy sticker. The number is real. It was also measured on a test cycle that isn't your commute, and the small print tells you which cycle. A benchmark line in a GitHub README works the same way. On Sunday, 8 November, a repository called llama-modchip reached the Hacker News front page with this sticker: "284B DeepSeek on 64GB RAM, no GPU." We spent Monday morning on the small print.

The model in question is a mixture-of-experts model. A mixture-of-experts (MoE) model is a network whose feed-forward weights are split into many separate blocks called experts, and it doesn't run all of them for each token; a small router picks a few experts per layer, token by token. So two numbers describe it: total parameters (everything in the file) and active parameters (the share that does work on any one token). For DeepSeek-V4-Flash the README gives 284.33B total and 13.17B active. Fitting a 284B MoE into 64GB is a problem about the first number. How fast it runs is mostly about the second.

The sticker, line by line

The repo describes itself as "a set of ggml/llama.cpp kernel patches, quantization transcoders, and runtime tools". In plain terms it's a modified build of llama.cpp (the open-source program most people use to run models locally) plus tools that re-pack model files into smaller formats. The headline claims below are from the README at v2.4.0, commit a9635a5 (pushed 8 November 2026, 21:01 UTC); the same commit's ARCHITECTURE.md, FAQ.md, RELEASE_NOTES.md, DeepSeek deep-dive in docs/ and benchmark JSON files supply the detail further down:

the DeepSeek-V4-Flash checkpoint shrinks from 76.87 GiB to 56.56 GiB, which is 1.71 bits per weight
peak memory at a 256K-token context is 57.61 GiB in one place and 58.9 GiB in another
on an AMD Ryzen 9 9955HX (Zen 5, 16 cores) with 64 GB and no GPU, prefill runs at 44.4 tok/s and decode at 9.1–14.7 tok/s (README self-reported, not re-tested by us)
a 30B Nemotron model packed to 14.35 GiB runs at 25–60.4 tok/s on the same CPU (README self-reported, not re-tested by us)
the files use a custom tensor type 40, which stock llama.cpp, Ollama and LM Studio "will reject"

Two terms before the arithmetic. A token is the unit a model reads and writes, roughly three-quarters of an English word. Prefill is the phase where the model reads your prompt; decode is the phase where it writes the answer one token at a time. They hit different limits in the hardware, which is why the README lists them separately.

The repo was created on 29 October, so it's 11 days old, and it had 534 stars when we checked the API this morning. Its history shows the README changing several times on Sunday alone, including a commit that added "an explicit quantization bpw breakdown table" and removed "emoji clutter". The HN thread had 382 points and 168 comments when we pulled it. One of the first comments, from akie, is the question this whole post is about: "At which quantization? That's the only thing I want to know."

Total parameters decide the RAM, active parameters decide the speed

An MoE router can choose any expert on the next token. You can't know in advance which 13B of the 284B you'll need, so all 284B have to be somewhere fast. On a GPU box that means video memory. On this setup it means system RAM. If part of the model sits on an SSD instead, every token that routes to a cold expert waits on a disk read, and the README's phrase for that is "page-cache thrashing".

Speed is the other number. During decode, each new token reads the active weights from RAM once. The profiling table in docs/DEEPSEEK_V4_OPTIMIZATIONS.md says that's 3.20 GB per token before the repo's pruning options and 2.85 GB with them. You can sanity-check that from outside: 13.17B active parameters at 1.71 bits each is about 2.8 GB, and the parts kept at higher precision would push it up a bit. The numbers agree.

That also gives a ceiling. The FAQ puts dual-channel DDR5 at about 75 GB/s. Divide 75 GB/s by 3.20 GB per token and you get about 23 tokens per second, the most this machine could decode if it did nothing but stream weights. The claimed decode with nothing pruned, 9.06–9.95 tok/s (README self-reported, not re-tested by us), would use roughly 29 GB/s of that. So the claim sits well under the physical limit. That doesn't prove it. It does mean nothing here breaks physics, which is worth knowing when a README reads the way this one does.

1.71 bits per weight, and where the 20 GiB went

The bits-per-weight figure is plain division. 56.56 GiB is 60.73 GB, or 485.8 billion bits; spread across 284.33 billion parameters, that's 1.709 bits each. It adds up. The same division on the 76.87 GiB starting file gives 2.32 bits per weight.

For scale, a normal 16-bit model stores 16 bits per weight, and most people who run models locally treat 4 bits as the point below which quality starts to fall off. One commenter, armcat, cited a paper saying the "peak size-to-performance ratio occurs at 4-bit". (We haven't read that paper; the summary is theirs.) And 1.71 is an average over very different tensors, which the README's breakdown table spells out:

embeddings, the first dense layer, the 43 shared experts and some attention projections stay at Q4_K, Q5_K or F32
the attention projections that were 8-bit get re-packed into the repo's "Fractal-64" format at 3.75 bits
the 256 routed experts per layer, which hold most of the weight, go mostly to IQ1_S at 1.5625 bits, with some "sensitive boundary layers" kept at IQ2_XXS (2.06 bits) or IQ1_M (1.75 bits)

ARCHITECTURE.md names those layers (0, 1, 26, 28, 41 and 42 in one set) and explains how the 1.56-bit experts are packed: an importance score computed from the weights themselves, without calibration data. It's a reasonable idea. It's also the part a reader can't verify by reading, because whether a 1.56-bit expert still knows what it knew is an empirical question.

One detail doesn't line up. The README says the starting point is "Unsloth UD-IQ1_S (76.87 GiB / 82.5 GB)". ARCHITECTURE.md describes the same 76.87 GiB file as "IQ2_XXS". HN user adrian17 spotted this first and later wrote: "only now noticed that 76.87GiB is 82.5GB, which happens to be size of the unsloth IQ1_S." Since the sizes match, the base is most likely the IQ1_S build. It's still the author's job to say so, and right now the two documents disagree.

Why 76.87 GiB doesn't fit in 64 GB

A "64 GB" machine has 64 GiB of RAM installed, and you never get all of it. The kernel, firmware reservations, the desktop and anything else running take a share. The README uses 62.5 GiB as the usable figure on a 64 GB Linux system. On that budget:

Sizes and the 62.5 GiB figure come from the README; bits per weight and room left are our arithmetic.

The bigger file wouldn't simply refuse to start. llama.cpp maps model files into memory, so it would load, and then the operating system would keep evicting pages and reading them back from disk. The README's own table lists the unpatched base at 5.91 tok/s decode, with "OOM / Swap on 64 GB" in the prefill column (README self-reported, not re-tested by us). Whatever that run looked like, nobody wants to do agent work inside it.

3.6 GiB of slack at peak is thin. A browser with a few tabs open will eat that. If you try this on a 64 GB box, run it headless, or plan on meeting the Linux out-of-memory killer.

How a 256K context fits in about 2 GiB

This was the number that surprised us most. The KV cache is the model's working memory for a conversation: for every token it has read, each layer keeps some vectors so it doesn't have to recompute them. It grows with context length, and on many models a 256K-token cache is bigger than the weights.

Here the gap between file size and peak memory is 58.9 minus 56.56, or 2.34 GiB, and that covers the cache plus compute buffers. So we worked out what a plain cache would cost. DeepSeek's attention stores one compressed "latent" vector per token per layer instead of full keys and values; the deep-dive gives it 512 dimensions, plus 64 for position. At 16 bits, across 43 layers and 262,144 tokens, that comes to about 11–12 GiB (our arithmetic, assuming one 512- or 576-wide FP16 vector per layer per token). It wouldn't fit.

The answer is in upstream llama.cpp. The deepseek4 model code at the commit the repo builds on keeps recent tokens in a sliding window and stores older history in compressed caches, at ratios of 4:1 (DSV4_CSA_RATIO = 4) and 128:1 (DSV4_HCA_RATIO = 128), chosen per layer from the model file's metadata. An 11 GiB cache compressed 4:1 is about 2.7 GiB, and at 128:1 it's under 0.1 GiB, so a mix of the two lands near what the README reports. The code doesn't show which layers use which ratio. That's stored in the GGUF header, and we didn't download the file.

Two practical consequences follow. Credit for the small cache goes to DeepSeek's design and the upstream implementation; the patch works around the cache (prefetching, and a fix for rolling it back after speculative decoding). And "256K context" here means the model can hold that much history in compressed form. It doesn't tell you how well the model pulls an exact detail from token 3,000 when it's at token 250,000. That's a quality question, and we found no long-context test for this model in the repo.

Why stock llama.cpp can't read the file

The README's compatibility note says Fractal-64 tensors are "GGML_TYPE_Q2_0, type ID 40" and that unpatched llama.cpp, Ollama or LM Studio "will reject them with unknown tensor type 40". GGUF files label every tensor with a numeric type ID, and a loader that doesn't know an ID refuses the file, so the story is plausible. We went looking for the definition.

The repo's build script pins upstream llama.cpp to commit d2f8305 (10 August 2026). Here's ggml/include/ggml.h at that commit, pulled through GitHub's API:

 GGML_TYPE_MXFP4 = 39, // MXFP4 (1 block) GGML_TYPE_NVFP4 = 40, // NVFP4 (4 blocks, E4M3 scale) GGML_TYPE_Q1_0 = 41, GGML_TYPE_Q2_0 = 42, GGML_TYPE_COUNT = 43,

Upstream already has a type called Q2_0, its ID is 42, and 40 belongs to NVFP4. Current master has the same values for those lines. The patch doesn't touch ggml.h at all. What it changes is the block layout in ggml-common.h:

 #define QK2_0 64 typedef struct { - ggml_half d; // delta (scale) - uint8_t qs[QK2_0 / 4]; // 2 bits per element + ggml_half d; // master FP16 super-block scale (2 bytes) + uint8_t scales[4]; // 8 x 4-bit shifted sub-scales ... + uint8_t qs[16]; // lower 2 bits for 64 3-bit fractal indices + uint8_t qh[8]; // upper 1 bit for 64 3-bit fractal indices } block_q2_0; -static_assert(sizeof(block_q2_0) == sizeof(ggml_half) + QK2_0 / 4, "wrong q2_0 block size/padding"); +static_assert(sizeof(block_q2_0) == 30, "wrong q2_0 block size/padding");

Upstream Q2_0 is 18 bytes per 64 weights, or 2.25 bits. The patched Q2_0 keeps the name and becomes 30 bytes per 64 weights, or 3.75 bits, with 3-bit indices. That's why a stock build can't read these files correctly: both builds know a type called Q2_0, and the bytes behind it differ. The struct comment also says FP16 scale, while the README calls it a BF16 scale.

What we couldn't reconcile is the number 40. In a build made from this patch on that commit, Q2_0 should still be 42. Maybe the files were written by an earlier fork with different numbering, or maybe the note is out of date. A stock binary won't load the model properly either way, so the practical advice stands. But if you hit an error that mentions 42, or a tensor size mismatch, instead of 40, this is probably why. There's a second effect worth knowing, as far as we can tell from the diff: since the patch redefines an existing upstream type in place instead of adding a new ID, a patched build would misread genuine upstream Q2_0 files. Keep that in mind before pointing it at other models in your folder.

HN user ar-nelson tried the whole path on a Haswell-era Xeon and hit two different failures. The repo's launcher stopped at "check_tensor_dims: tensor 'blk.0.ffn_gate_tid2eid.weight' has wrong shape", and the bundled binary at "Illegal instruction (core dumped)", which they traced to the prebuilt binaries needing AVX-512.

The one script we ran

The README offers a "Hardware Readiness Check (Zero Model Download Required)" as a one-liner that downloads the script with curl and pipes it straight into python3. We didn't, because piping unread code into an interpreter is how you find out what it does after it has done it. Our first attempt to get the repo, a plain git clone, timed out on port 443 from our network. GitHub's API and tarball endpoint worked, which is how we ended up with the tarball at commit a9635a5, and scripts/check_cpu_readiness.py was the first file in it we read.

It's 590 lines, standard library only. It reads CPU and memory facts (sysctl on macOS, /proc and lscpu on Linux), runs a short memory-bandwidth test by allocating up to 16 buffers of 128 MB and scanning them, and prints estimates. It makes no network calls and writes no files, and the shell commands it runs are fixed strings. That was enough for us to run it, in an empty directory, with Python's isolated mode so it couldn't import anything planted next to it:

mkdir -p /tmp/d09/work/devto_run && cd /tmp/d09/work/devto_run S=/tmp/d09/work/devto_repo/src/llama-modchip-a9635a566f/scripts/check_cpu_readiness.py shasum -a 256 "$S" # df09c236cc697ecb...c61d073aadae49c81227d9061ecfc40f1497 for i in 1 2 3; do python3 -I "$S" --json > run$i.json; done python3 -I "$S" # the human-readable report

The machine was an Apple M4 Mac mini with 24 GB, 4 performance and 6 efficiency cores, macOS 15.6.1 and Python 3.9.6. We ran it on Monday 9 November at 11:33 China Standard Time. Each run took about 0.4 seconds.

The four runs measured memory bandwidth at 81.2, 98.6, 95.1 and 96.0 GB/s, and estimated "Technical Prose + New Code" decode at 20.6, 24.6, 23.8 and 24.0 tok/s. All four gave the same verdict: Tier A, "production-ready interactive coding agent".

Reading the code explains all of it. The script is an estimator for the 30B Nemotron model, and its banner says so. It says nothing about the 284B DeepSeek build, which needs 56.56 GiB, on a machine that has 24. The only thing it measures is bandwidth. Everything else comes from a formula: roughly, a reference speed from the author's Zen 5 machine, scaled by measured bandwidth over 48 GB/s and some fixed factors for SIMD support and cache size. So when the bandwidth reading moved 21% between run 1 and run 2, the speed estimate moved 19%, and the verdict didn't move at all.

A few smaller things showed up on macOS. The script sets available RAM to 75% of the total by fiat (18.0 of 24.0 GiB). It reports /dev/shm as 0.0 GiB, because macOS has no /dev/shm, and still estimates a 9.5 ms restore from /dev/shm. Its recommended launch line uses taskset, a Linux tool, with -t 10, which counts the Mac mini's six efficiency cores as full threads.

None of that is sinister. It's an estimator that prints its output as a verdict. ar-nelson's Xeon also got Tier A before the binaries refused to run on it, which is the same lesson from the other side.

What the speed numbers were measured at

This is the sticker's test cycle. We didn't download the 56.56 GiB file or run inference, having no 64 GB x86 machine to run it on, so every speed figure in this section is README or docs self-reported and not re-tested by us.

The decode range of 9.1–14.7 tok/s mixes two modes. Plain decode with all six routed experts per layer is 9.06–9.95 tok/s. The 11.5–14.7 range needs "Stockfish-PVS", the repo's speculative drafter, which guesses several tokens ahead and has the big model check them in one batch. Speculative decoding speeds up text the drafter can predict, like repeated code or JSON, and helps less on fresh prose. A fair one-line summary would be "about 9–10 tok/s, more on repetitive output".

There are faster modes, Top-5 and Top-4, at 9.68–10.74 tok/s, which run fewer experts per token. The README is upfront that these change the output, and reports a KL divergence (a measure of how far the output probabilities drift) of 0.0042 to 0.0694 nats against the default. The readiness script's recommended launch line exports LLAMA_MOE_TOP_K=4 for the 30B model, so following its advice puts you in a pruned mode by default.

The prefill figure of 44.4 tok/s comes from a table where prompts ran from 63 to 580 tokens, plus six agent turns that grew the cache to 2,418 tokens. Prefill fell from 44.36 to 38.44 tok/s across those six turns. None of the speed tables runs at 256K. The 256K number is a memory figure: the size of the process when started with -c 262144. The README's own quickstart command for the 284B model uses -c 32768.

Here's what those rates mean in wall-clock time, assuming they held at depth (the trend says they'd drop):

reading a full 256K-token prompt at 44.36 tok/s takes 262,144 ÷ 44.36 ≈ 5,900 seconds, about 98 minutes; at 38.44 tok/s it's about 114 minutes
a 500-token answer at 9.06 tok/s takes about 55 seconds
the repo's slot restore, which reloads a saved cache in 13.1–32.6 ms, is the way around the first item for agent work: you pay for the long read once, save it, and reload it
Put together, it looks like this. It's a 284B model that writes at roughly human reading speed, after a prompt read that's fine for a few thousand tokens and slow for a long document. For a background job that may be perfectly acceptable. The repo also ships ./modchip claude, which points Claude Code at the local server, and for an interactive coding agent that re-reads large chunks of a repository, the prefill is the number to plan around.

What to check before trusting the numbers

Most of this is free, and the cheap checks come first.

The first row is the one to settle before anything else. The README's "bit-exact" and "1.00000 logit cosine similarity" compare the patched runtime with the unpatched runtime "on the same checkpoint". That shows the fast kernels compute the same thing as the slow ones, which is good engineering. It says nothing about how much the 1.71-bit file lost compared with the model it came from. The repo's 100-point quality gate results (benchmarks/results/*.json) are all for the 30B Nemotron model. Even there, 20 of the 100 points are for speed and RAM, so "100/100" means 80 quality points plus a full speed score, earned on five tool calls, three coding tasks and two needle tests.

With a 64 GB machine and a 128 GB one side by side for a day, the plan would look like this. It uses standard llama.cpp tools. We haven't run it, so check the flags against your llama.cpp version.

# 128 GB machine, stock llama.cpp: save reference logits from the 76.87 GiB base llama-perplexity -m DeepSeek-V4-Flash-0731-UD-IQ1_S.gguf -f wiki.test.raw \ --kl-divergence-base base_logits.bin # Same machine, patched binary: score the 56.56 GiB file against those logits llama-perplexity -m DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf \ --kl-divergence-base base_logits.bin --kl-divergence # 64 GB machine, patched binary, drafting off: speed at depth plus peak memory /usr/bin/time -v llama-bench -m DeepSeek-V4-Flash-0731-Fractal-00001-of-00003.gguf \ -p 512 -n 128 -d 0,32768

That KL test measures what the transcoding lost relative to the IQ1_S base. It won't tell you what the base had already lost relative to the full model; a full-precision reference for a 284B model needs far more memory than either machine has. Two smaller habits round it out. Before loading any GGUF from a fork, list its tensor types with the gguf Python package's gguf-dump tool. And read a readiness script before you run it; this one took us about ten minutes.

What reading a repo settles, and what it doesn't

Reading the repo deals with claim-scope risk: whether a number means what the headline makes you think it means. On that the result is mixed and specific. The sizes add up. The bits-per-weight figure adds up. The speed claims sit under the physical ceiling. The labels "lossless", "256K" and "9.1–14.7" each hold under narrower conditions than they suggest.

It doesn't deal with quality risk, whether a 1.71-bit DeepSeek-V4-Flash gives answers you'd accept. As far as we could find, nobody has published that test for this file, and the HN thread is split. walrus01, who said they had spent a lot of time with this model, wrote: "Even the 97GB Q2XS will show very significant divergence from the full precision thing." sparse-Matrix reported "Steady 12 to 14 TPS" on a Ryzen 7 5700U with 64 GB, without saying which model, and the 5700U row on the leaderboard is for the 30B one.

It doesn't deal with build risk either. ar-nelson's failures, the AVX-512 requirement in the prebuilt binaries and the type-ID mismatch are things you only find by trying. aappleby, who did try the 30B model on a Ryzen 7945HX with 96 GB, wrote that "it runs just about the same as plain old llama.cpp on the cpu (except prefill seems much faster)", and found no DeepSeek option in the setup.

And it doesn't deal with capacity risk. 3.6 GiB of headroom on a 64 GB machine is a setup that works until something else wants memory. The usual fix is more memory, which costs more than a second pair of RAM sticks and leaves quality risk where it was: a bigger machine can hold the 2.32-bit base unmodified, and that base still has its own quality questions.

Where more memory fits in

The other way past the 64 GB line is more memory. That's what we sell, so a 64 GB CPU box running a frontier-size model is in some sense our competition. Here's how the checks in the table above map onto a KarmaBox.

Start with fit, since that's what the whole repo is working around. Our product page lists two desktop configurations: KarmaBox, with one GB10 and 128 GB of unified memory, and KarmaBox Pro, with two linked GB10s and 256 GB. KarmaBox draws about 170 W. On the 128 GB unit the unmodified 76.87 GiB UD-IQ1_S build leaves about 50 GiB for the cache and everything else (our arithmetic from the published sizes, with no benchmark behind it). At that size the 62.5 GiB budget stops being the constraint, and so does the reason for a custom tensor format.

Then the last row of the checklist: watch peak memory with your usual software running. On a KarmaBox the usual software is the rest of the avatar team. The product page describes the same 128 GB holding office workflows, local video production, your knowledge base and the avatars together, and our local AI hardware page lists 3–4 concurrent tasks for KarmaBox and 6–8 for KarmaBox Pro. So the peak worth recording is the one taken while those tasks are running, with the model loaded and the cache filled to the depth you work at.

The quality row needs a side-by-side on your own prompts, whatever the hardware. In Karma the model selector at the top of a chat switches between models, as the quick start in the Karma docs shows, so the same prompt set can go to each candidate model and the answers can be compared before you settle which model each avatar uses.

Last, decide which jobs need the local model at all. Our local-first finance workflow page walks through a scenario at a 42-person tax firm. Anything touching a client's ledgers, original contracts or identity documents goes to avatars living on the box, and public research or website replies can go to cloud avatars for lower latency and stronger model capability. The firm writes that line into policy instead of judging it task by task, and the audit log records which machine ran each task, so the split can be checked afterwards. The page also recommends an outage drill: cut the network and confirm the on-box avatars finish their core work without it. Where that line falls decides whether a job needs a frontier-size model on your own hardware in the first place.

If the reason you want a local model is to give a coding agent to colleagues who aren't engineers, our guide to AI coding agents for non-engineers covers who should get one and with what access.

Which of these numbers would you re-measure first on your own machine?