streamft 1.0.1: fine-tuning a 32B model on a 6 GB laptop GPU
I have one GPU: an RTX 3050 Laptop with 6 GB of VRAM, 34 GB of RAM and an NVMe SSD. It runs Windows. For the last months I have been building a small training engine for models that do not fit, and this post is the measured result for the 32B case.
streamft is now a trainer-only package. The local assistant I built around the same experiment is a separate project; it is not part of this wheel. The 1.0.1 release removes the last assistant modules and keeps only the training stack.
What works
A 32B NF4 model fine-tunes at 1024 tokens/step on this card.
| seq | step time | tokens/s | peak VRAM |
|---|---|---|---|
| 128 | 10.70 s | 12.0 | 3.18 GB |
| 512 | 17.95 s | 28.5 | 3.82 GB |
| 1024 | 27.91 s | 36.7 | 4.61 GB |
| 2048 | 64.53 s | 31.7 | 6.60 GB (CPU cache) / 7.93 GB |
The 32B weights are 19.2 GB. They never fully live in VRAM: they are mmap'd from disk and streamed block by block, with one transformer block resident at a time and recompute in the backward pass. The weight I/O is paid once per step, not once per token, so longer sequences amortise it.
At 1024 tokens/step the engine reaches 36.7 tok/s at 4.61 GB peak, 3.1× the seq-128 throughput. That is the useful operating point on this card. At 2048 the activation cache pushes the run over the 6 GB budget, the step spills into shared memory and throughput falls to 31.7 tok/s. Longer is not automatically better.
The 7B end-to-end demo
This is the concrete “it works†scenario: train an adapter on a 7B model, keep the 4-bit base on disk, apply the adapter at inference and call an OpenAI-compatible endpoint.
| metric | value |
|---|---|
| steps | 20 |
| median step time | 2.87 s |
| wall time | 88.6 s |
| peak VRAM | 5.65 GB |
| eval loss | 3.4411 → 2.3588 |
| adapter | 20.25 MB |
| serving | 32 tokens in 3.14 s |
The base stays frozen. Only the LoRA adapter is trained and saved. The same 4-bit base plus the adapter fits on the 6 GB card.
Why the streaming path exists
The engine is not a faster version of PEFT when the whole model fits. It is the version that still runs when the model does not fit.
Matched runs from the repository:
| model | PEFT | streamft |
|---|---|---|
| 3B | 0.58 s/step, 4.90 GB | 0.94 s/step, 3.85 GB |
| 7B | 8.96 s/step, 10.05 GB | 1.70 s/step, 5.51 GB |
| 14B | 18.56 s/step, 16.48 GB | 6.73 s/step, 3.86 GB |
| 32B | CUDA OOM before the first step | 12.5 s/step, 3.90 GB |
At 3B, PEFT is faster and uses a little more memory. At 7B it no longer fits and spills to shared memory. At 14B and 32B the streaming engine is the only one that stays on the card.
What changed in 1.0.1
- Removed the remaining assistant modules, CLI path and tests.
- Removed the assistant package, docs, examples and tests from the repository.
- Rebuilt the wheel from the clean trainer tree.
- 200 trainer/source tests pass.
- Package build,
twine checkand wheel install pass.
The wheel contains streamft only. There is no streamft.chat_server, no streamft.doc_memory and no atomzero dependency.
Commands
pip install streamft==1.0.1 streamft doctor streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512 streamft prepare --model Qwen/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4 streamft train --model-dir checkpoints/qwen0.5b_nf4 --sft --steps 20 --seq 512 streamft serve --model checkpoints/qwen7b_nf4 --adapter checkpoints/demo7b --port 8000 Honest limits
- 2k context does not fit on this 6 GB card; the run spills at 6.60–7.93 GB.
- The SFT path feeds one short Dolly row per step, so its
--seqis a truncation cap, not a long-context measurement. Use the LM path for throughput. - Training at 2048 context was not better than 512 at a matched token budget in the E4 experiment; the lever is data volume, not window length.
- The engine is not faster when the model fits. It is for the cases where the model does not.
- The numbers are measured on one laptop GPU, not a datacenter.
Links
- GitHub: https://github.com/byjameson/stream-finetune
- PyPI: https://pypi.org/project/streamft/1.0.1/
- ModelScope: https://modelscope.cn/models/byjameson/streamft
If you run a small GPU and a model that does not fit, the most useful feedback is a reproducible JSON row: model, seq, resident blocks, step time and peak VRAM. That is how the measured table stays honest.

