A local model feels quick when one person is chatting. What happens when four requests arrive together?
Community project TensorFold, maintained by ashhart, has a native CUDA recipe for Spark-X2.6-4B. Its published tests follow that question from a single stream to concurrent serving, then add a longer input. The result is an engineering story about capacity, with clear limits.
When the queue grows
On an RTX 5090 running Linux, the author reports 191–204 tokens/s for one stream, 375–389 tokens/s combined for two, and 716–729 tokens/s combined for four, using an 8-bit conversion and a 32,768-token context setting. The rates are medians of three greedy 256-token replies per prompt. The four-stream number is aggregate throughput; each user does not receive 729 tokens/s.
Then give it a longer document
The same recipe records four prompt types before and after a 28k-token input:
| Prompt | Short input, tokens/s | After 28k input, tokens/s |
|---|---|---|
| Prose | 199 | 199 |
| Code | 191 | 198 |
| JSON | 203 | 197 |
| List | 204 | 200 |
Generation stayed in a similar range on these prompts. Reading the document still has a cost; these rates measure decode after the input, not total task latency or answer quality.
Speed after reading is only half the experience
A separate RTX 4090 / Windows / 8-bit test reports time to first token of 0.24 seconds at 2k input, 0.93 at 8k, 1.93 at 16k, and 4.65 at 29k. These measurements show the growing wait as the input expands. They use different hardware and conditions from the 5090 throughput test and do not establish a controlled comparison between GPUs.
The boundary belongs in the story
The 5090 server reports 9.47 GiB for the model plus 1.23 GiB per stream at the 32k setting. In consistency checks, 12/12 concurrent greedy replies matched their solo runs, and 12/12 replies with copy drafts off matched the drafted runs. Those are output-consistency checks, not correctness scores for user tasks.
The CUDA path in this recipe serves no drafter and no structured-output mode. The planned 30,720-token input point tokenized past the 32k limit and was refused. The recipe covers RTX 4090 and RTX 5090; other CUDA GPUs remain untested there. No claim of better answer quality follows from the throughput numbers.
Try the published recipe
Start with the pinned TensorFold CUDA recipe, which provides the exact Spark checkpoint revision, conversion and build commands, hardware details and limitations. The official Spark-X2.6 model repository is the model entry point.
If you try another card, share the GPU, quantization, prompt length, concurrency, decode rate and first-token wait together. That makes the next result useful to someone choosing a local serving setup.
Measurements above are the author's historical results, not an independent rerun by SparkLLM. The source labels the 4090 test October 25 and the 5090 test October 26, 2026, without specifying a timezone. We checked source revision 0fed2e4a5fcc03f109cfaaf8eda10734cce21fb7 on November 13, 2026 (UTC). The charts redraw its numerical report; they are not runtime screenshots. The small SPARK character is brand illustration, not test evidence.
