streamft 1.0.1: fine-tuning a 32B model on a 6 GB laptop GPU

I have one GPU: an RTX 3050 Laptop with 6 GB of VRAM, 34 GB of RAM and an NVMe SSD. It runs Windows. For the last months I have been building a small training engine for models that do not fit, and this post is the measured result for the 32B case.

streamft is now a trainer-only package. The local assistant I built around the same experiment is a separate project; it is not part of this wheel. The 1.0.1 release removes the last assistant modules and keeps only the training stack.

What works

A 32B NF4 model fine-tunes at 1024 tokens/step on this card.

seq step time tokens/s peak VRAM
128 10.70 s 12.0 3.18 GB
512 17.95 s 28.5 3.82 GB
1024 27.91 s 36.7 4.61 GB
2048 64.53 s 31.7 6.60 GB (CPU cache) / 7.93 GB

The 32B weights are 19.2 GB. They never fully live in VRAM: they are mmap'd from disk and streamed block by block, with one transformer block resident at a time and recompute in the backward pass. The weight I/O is paid once per step, not once per token, so longer sequences amortise it.

At 1024 tokens/step the engine reaches 36.7 tok/s at 4.61 GB peak, 3.1× the seq-128 throughput. That is the useful operating point on this card. At 2048 the activation cache pushes the run over the 6 GB budget, the step spills into shared memory and throughput falls to 31.7 tok/s. Longer is not automatically better.

The 7B end-to-end demo

This is the concrete “it works” scenario: train an adapter on a 7B model, keep the 4-bit base on disk, apply the adapter at inference and call an OpenAI-compatible endpoint.

metric value
steps 20
median step time 2.87 s
wall time 88.6 s
peak VRAM 5.65 GB
eval loss 3.4411 → 2.3588
adapter 20.25 MB
serving 32 tokens in 3.14 s

The base stays frozen. Only the LoRA adapter is trained and saved. The same 4-bit base plus the adapter fits on the 6 GB card.

Why the streaming path exists

The engine is not a faster version of PEFT when the whole model fits. It is the version that still runs when the model does not fit.

Matched runs from the repository:

model PEFT streamft
3B 0.58 s/step, 4.90 GB 0.94 s/step, 3.85 GB
7B 8.96 s/step, 10.05 GB 1.70 s/step, 5.51 GB
14B 18.56 s/step, 16.48 GB 6.73 s/step, 3.86 GB
32B CUDA OOM before the first step 12.5 s/step, 3.90 GB

At 3B, PEFT is faster and uses a little more memory. At 7B it no longer fits and spills to shared memory. At 14B and 32B the streaming engine is the only one that stays on the card.

What changed in 1.0.1

  • Removed the remaining assistant modules, CLI path and tests.
  • Removed the assistant package, docs, examples and tests from the repository.
  • Rebuilt the wheel from the clean trainer tree.
  • 200 trainer/source tests pass.
  • Package build, twine check and wheel install pass.

The wheel contains streamft only. There is no streamft.chat_server, no streamft.doc_memory and no atomzero dependency.

Commands

pip install streamft==1.0.1 streamft doctor streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512 streamft prepare --model Qwen/Qwen2.5-0.5B --save-quant checkpoints/qwen0.5b_nf4 streamft train --model-dir checkpoints/qwen0.5b_nf4 --sft --steps 20 --seq 512 streamft serve --model checkpoints/qwen7b_nf4 --adapter checkpoints/demo7b --port 8000 

Honest limits

  • 2k context does not fit on this 6 GB card; the run spills at 6.60–7.93 GB.
  • The SFT path feeds one short Dolly row per step, so its --seq is a truncation cap, not a long-context measurement. Use the LM path for throughput.
  • Training at 2048 context was not better than 512 at a matched token budget in the E4 experiment; the lever is data volume, not window length.
  • The engine is not faster when the model fits. It is for the cases where the model does not.
  • The numbers are measured on one laptop GPU, not a datacenter.

Links

If you run a small GPU and a model that does not fit, the most useful feedback is a reproducible JSON row: model, seq, resident blocks, step time and peak VRAM. That is how the measured table stays honest.