How do you fine-tune Qwen with LoRA in the cloud cheaply?

Fine-tune Qwen cheaply with one LoRA adapter on the smallest single GPU that holds twice the frozen weights: 16-bit LoRA up to about 9B, 4-bit QLoRA for 14B to 32B dense models. Nodus ships single-GPU Qwen3 presets from 0.6B to 8B in its managed nodus/sft runtime, on cards such as the 48 GB RTX A6000 listed at $0.333/hr on the Nodus pricing page.

Which Qwen models are current, and which are dense or MoE?

Two architecture lines are current, and they train differently. Qwen3 is text-only, with dense models from 0.6B to 32B plus mixture-of-experts (MoE) models such as 30B-A3B. Qwen3.5, Qwen3.6 and Qwen3.8 are vision-language models on a newer hybrid architecture, per the Qwen organization on Hugging Face.

The newest dense model is Qwen3.8-27B. Its model card describes 64 layers in a repeating block of three Gated DeltaNet (linear attention) layers and one gated full-attention layer, and a 262,144-token native context. Qwen3.5 ships dense 0.8B, 2B, 4B, 9B and 27B models, plus 35B-A3B, 122B-A10B and 397B-A17B MoE models. Every Qwen3, Qwen3.5, Qwen3.6 and Qwen3.8-27B repository in the table below is Apache 2.0 and ungated, so no Hugging Face token is needed.

An MoE model's memory follows its total parameters, not its active ones. Qwen3.6-35B-A3B runs about 3B parameters per token, but all 35.95B have to sit in GPU memory while you train.

Which Qwen size fits on which single GPU?

We sized each model from the parameter count Hugging Face reports for its weights, at 2 bytes per parameter for a 16-bit base. For 4-bit QLoRA we used 0.5 bytes per parameter and assumed the token embedding and output matrices stay in 16-bit. Our rule of thumb is that the frozen weights should take at most half the card, leaving the rest for activations, logits and the adapter's optimizer state.

Parameter counts and configs from each model's Hugging Face repository; vision models' counts include the vision encoder. GPU prices checked November 12, 2026 on the Nodus pricing page, where they are listed as hourly supply prices.

Under the half-the-card rule, only the 4-bit base keeps a dense Qwen above 9B on a 24 GB or 48 GB card.

Nodus's own Qwen3 presets are more conservative than our rule. The nodus/sft reference puts Qwen3-4B LoRA on an L40S, RTX A6000 or A100 80 GB, and Qwen3-8B QLoRA on the same 48 GB-and-up class. The 24 GB RTX 4090 saves only $0.043 an hour over the RTX A6000 at list price. At these sizes, take the preset rather than risk an out-of-memory restart.

When does QLoRA actually save money on Qwen?

QLoRA pays off from 14B up, where it drops you a whole price tier. Below that, the 16-bit and 4-bit options cost nearly the same per hour, and 16-bit LoRA skips the dequantization work QLoRA adds to every step.

The break-even is the ratio of the two hourly prices. For Qwen3.8-27B, LoRA needs an H200 at $3.593/hr and QLoRA fits on an RTX A6000 at $0.333/hr. QLoRA is cheaper unless it runs more than 3.593 / 0.333 = 10.79 times as long. Assume, for illustration, the LoRA run takes 5 hours on the H200: 5 x $3.593 = $17.97. QLoRA on the A6000 would still cost less unless it took more than about 54 hours (5 x 10.79).

At 27B to 32B, QLoRA moves a dense Qwen run from a $3.593/hr card to a $0.333/hr one.

We have not benchmarked QLoRA's slowdown on Qwen, so run 100 steps of each and compare. Our LoRA vs QLoRA guide explains where the slowdown comes from.

What does the Qwen chat template do to your training data?

It adds tokens you did not write, and the newest generation adds a system prompt. We rendered the same two-turn conversation through each model's own template with Hugging Face Transformers 5.22.2 and apply_chat_template. Qwen3-8B and Qwen3.5-9B both put an empty <think>\n\n</think>\n\n block before the last assistant answer and drop it from the earlier one.

Qwen3.8-27B differs twice. Its template preserves thinking from history by default (preserve_thinking), so every assistant turn keeps the empty think block. With no system message, it also inserts one that starts "Reasoning effort is set to xhigh."

With enable_thinking=False that system message disappears and the empty think blocks remain.

Your adapter learns the rendered text, so training with one setting and serving with another shows it a format it never saw. Render one example with the exact flags you will serve with, read it, and train on that.

Two more template details from the official docs:

• TRL's assistant_only_loss=True needs {% generation %} markers in the template, and the TRL SFT docs say TRL patches the template automatically for Qwen3.
• Qwen3's template writes tool calls as JSON inside <tool_call> tags, while the Qwen3.5 and Qwen3.8 templates use <function=name> and <parameter=...> tags. Re-render tool-calling data through the new template before you move it to a newer Qwen.

The Qwen3-8B model card documents the enable_thinking switch, and the Qwen3.8-27B model card documents reasoning_effort and preserve_thinking.

Which modules should a Qwen LoRA target?

Target all linear layers, and on the MoE models check what that actually covers. PEFT's LoRA guide recommends target_modules="all-linear", and that is the nodus/sft default. On dense Qwen3 that means q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj and down_proj in each layer.

The hybrid Qwen3.5 and newer models have extra names. Their weight index shows in_proj_qkv, in_proj_z, in_proj_a, in_proj_b and out_proj projection matrices in the linear-attention layers. A hand-written Qwen3 module list skips them, which leaves three of every four layers' attention untouched. Print the adapter's trainable module names once to confirm what you got.

The MoE models store their routed experts as fused tensors (mlp.experts.gate_up_proj and mlp.experts.down_proj in Qwen3.6-35B-A3B), not as linear layers. PEFT's docs say such parameters need target_parameters, which all-linear does not set, so a default LoRA leaves the routed experts unadapted. For the same reason we did not estimate a 4-bit footprint for the MoE row: check that your quantization library actually quantizes those fused tensors before you trust one.

How much memory do Qwen's logits take?

More than on most models, because Qwen's vocabulary is large. Qwen3 has 151,936 tokens and Qwen3.5 and newer have 248,320, per their configs. Assume the trainer computes the loss on full fp32 logits. Then a batch of 4 sequences of 2,048 tokens needs 4 x 2,048 x 248,320 x 4 bytes = 8.1 GB for one copy of the logits on Qwen3.5. Qwen3 needs 5.0 GB for the same batch.

That is enough to push a model that "fits" off a 24 GB card. Lower batchSize and raise gradientAccumulation by the same factor to keep the effective batch; the nodus/sft runtime defaults to 4 and 4.

How do you run a Qwen LoRA job on Nodus?

For Qwen3, use the managed nodus/sft runtime: one manifest, a dry-run price, then the run. The commands are copied from the Nodus training guide and Volumes guide.

$ pip install nodus-compute $ nodus login $ nodus create volume support-chats --size 10Gi $ nodus volume put support-chats ./chats.jsonl /chats.jsonl 

This manifest is the training guide's LoRA example with Qwen3-8B pinned to its current Hugging Face commit, a validation split, and a smaller per-GPU batch for the logits reason above. The presets have no Qwen3-8B LoRA entry, so the manifest names 48 GB-and-up GPUs. Without that block the run falls back to the runtime default, which includes the 24 GB RTX 4090. TrainingJobs are Beta, and we have not run this exact manifest.

apiVersion: nodus.dev/v1beta1 kind: TrainingJob metadata: name: qwen3-8b-support-lora spec: runtime: nodus/sft model: uri: hf://Qwen/Qwen3-8B@b968826d9c46dd6066d109eabc6255188de91218 data: volume: support-chats path: chats.jsonl format: JSONL columns: {prompt: question, completion: answer} validationFraction: "0.1" parameters: method: LoRA completionOnlyLoss: true steps: 2000 evalSteps: 100 batchSize: 2 gradientAccumulation: 8 lora: {r: 16, alpha: 32, targetModules: all-linear} validation: {patience: 3, requireImprovement: true} resources: gpu: {type: [A6000, L40S, A100-80GB], count: 1} maxCostUSD: "20.00" 

$ nodus apply -f sft.yaml --dry-run -o yaml # status.estimate: cost to completion, first hold, start time $ nodus apply -f sft.yaml $ nodus cp tj/qwen3-8b-support-lora:outputs/adapter/adapter_model.safetensors ./adapter_model.safetensors 

If spending reaches maxCostUSD, the run is suspended rather than failed, and raising the cap resumes it from its last checkpoint. The runtime saves one every 200 steps by default. The first run on a pinned commit imports the model once into a read-only Volume, so later runs do not pay GPU time to download weights.

The presets cover Qwen3 only, and the Qwen3.5 model card asks for the latest Transformers. For Qwen3.5, Qwen3.6 and Qwen3.8, run your own TRL script as a Job with nodus run, as in the quickstart. Pin the library versions you tested.

For a sense of scale on a small Qwen, our fine-tuning benchmark recorded a $0.048696 successful-run charge for a Qwen2.5-0.5B fine-tune on one NVIDIA L4, with 112.8 seconds in the trainer.

Which Qwen setup should you pick?

Apply this in order:

1. Qwen3 up to 8B: the nodus/sft preset, or 16-bit LoRA on a 48 GB card. The cheaper 24 GB option saves cents an hour.
2. Qwen3.5-9B: 16-bit LoRA on a 48 GB card, in your own TRL script, with a smaller per-GPU batch for the larger vocabulary.
3. Dense 14B to 32B, including Qwen3.8-27B: QLoRA on a 24 GB or 48 GB card. Move to 16-bit LoRA on a bigger GPU only if 100 measured steps show QLoRA running more than the price ratio slower.
4. MoE models: budget memory for total parameters, decide whether the routed experts need target_parameters, and check what your quantizer does with fused experts before planning a 4-bit run.
5. Every run: a dollar cap, a validation split, and one rendered training example checked against the template flags you will serve with.

FAQ

Should I fine-tune the Qwen base model or the instruct model?

Start from the post-trained model when your rows are instructions and answers. Use a -Base checkpoint, such as Qwen3.5-9B-Base, for raw domain text or your own chat format. The TRL SFT docs show the catch with a Qwen2.5 base: set eos_token to <|im_end|> so answers stop.

Can I fine-tune Qwen with thinking mode turned off?

Yes. Render your training rows with enable_thinking=False and serve with the same flag. On Qwen3, the /think and /no_think soft switches in user messages only work when enable_thinking is on, per the Qwen3 model card.

How big is a Qwen LoRA adapter?

Small. At rank 16 on all seven linear layers, Qwen3-8B gets 16 x (8,192 + 5,120 x 2 + 8,192 + 16,384 x 3) = 1,212,416 adapter parameters per layer. Over 36 layers that is 43.6M. That is about 87 MB in 16-bit, against 16.4 GB for a merged copy of the model, so keep the unmerged adapter even if you merge one for serving.

Does Nodus pick the GPU for a Qwen fine-tune?

For a Qwen3 model and method with a preset, yes: leave out spec.resources and the run takes that preset's GPUs. Anything else gets the nodus/sft default of one RTX 4090, L40S, RTX A6000 or A100 80 GB, so name the GPU yourself whenever your model needs more than 24 GB.

Originally published on the Nodus blog.