To fine-tune Gemma, load a Gemma 4 instruction checkpoint with Hugging Face TRL and PEFT or Unsloth, add LoRA adapters to its seven decoder projections, and train on one GPU sized to the model. On Nodus every Gemma 4 size fits a single card with QLoRA: Google measures 31B at 18 to 22 GB, which fits the 48 GB RTX A6000 listed at $0.333/hr on the Nodus pricing page.

Which Gemma models can you fine-tune today?

Gemma 4 comes in five sizes on four architectures, per Google's Gemma 4 model overview: E2B, E4B, 12B, 26B A4B and 31B. E2B and E4B are compact dense models with per-layer embeddings, 12B is an encoder-free multimodal model, 26B A4B is a mixture-of-experts (MoE) model, and 31B is the largest dense model. Gemma 3 and earlier remain downloadable.

The "E" means effective parameters. The real checkpoints are bigger: Hugging Face reports 5.12B parameters for gemma-4-E2B-it and 8.00B for gemma-4-E4B-it. The MoE model activates about 4B parameters per token, but all 25.81B have to sit in GPU memory while you train.

Start from the -it checkpoint for LoRA work. Google's fine-tuning best practices recommend it for LoRA and QLoRA SFT, with a learning rate of 1e-4 to 2e-4 and 500 to 5,000 examples.

Do you need to accept a license to fine-tune Gemma?

Not for Gemma 4. Google publishes it under Apache 2.0, and the five instruction-tuned Gemma 4 repositories on Hugging Face are ungated, so no access request or token is needed. Apache 2.0 still asks you to ship the license and mark files you changed when you redistribute your fine-tuned weights.

Gemma 3 and older are different. They use the Gemma Terms of Use, and repositories such as gemma-3-27b-it are gated. Those terms require you to pass the use restrictions on to anyone you distribute a fine-tuned model to and to give them a copy of the agreement. If you have a working Gemma 3 pipeline, the license is one more reason to move it to Gemma 4.

How much GPU memory does each Gemma size need for LoRA?

Google's own table shows the floor. Loading Gemma 4 at 16-bit takes 11.4 GB for E2B, 17.9 GB for E4B, 26.7 GB for 12B, 57.7 GB for 26B A4B and 69.9 GB for 31B. At 4-bit the same models take 2.9, 4.5, 6.7, 14.4 and 17.5 GB. Those overview figures cover weights plus 20% load overhead, not training.

Memory to load Gemma 4, 16-bit versus 4-bit: E2B 11.4 and 2.9 GB, E4B 17.9 and 4.5 GB, 12B 26.7 and 6.7 GB, 26B A4B 57.7 and 14.4 GB, 31B 69.9 and 17.5 GB.
From 26B up, a 16-bit Gemma needs an 80 GB-class card before training starts; at 4-bit, every size fits under 24 GB.

Training adds activations, adapters and optimizer state on top. Where Google or Unsloth publish a measured training peak, we used it. Our rule: the peak may use at most 80% of the card, and with no published peak, the 16-bit or 4-bit weights may use at most half. We then took the cheapest card on the Nodus pricing page that passes and supports bfloat16.

Model Type Parameters 16-bit weights Published training peak LoRA card QLoRA card
E2B dense, per-layer embeddings 5.12B 10.2 GB LoRA 8 to 10 GB; QLoRA under 10 GB RTX A4000 16 GB, $0.173/hr RTX A4000 16 GB, $0.173/hr
E4B dense, per-layer embeddings 8.00B 16.0 GB LoRA 17 GB RTX 3090 24 GB, $0.18/hr RTX A4000 16 GB, $0.173/hr
12B encoder-free multimodal 11.96B 23.9 GB none published RTX A6000 48 GB, $0.333/hr RTX A4000 16 GB, $0.173/hr
26B A4B MoE, 128 experts 25.81B 51.6 GB LoRA about 56 GB; QLoRA about 21 GB A100 80 GB, $1.09/hr RTX A6000 48 GB, $0.333/hr
31B dense 31.27B 62.5 GB QLoRA 18 to 22 GB H200 141 GB, $3.593/hr RTX A6000 48 GB, $0.333/hr

Parameter counts from each model's Hugging Face repository (they include any vision and audio encoders). Training peaks: E2B QLoRA from Google's (Q)LoRA guide; 26B A4B from Google's MoE fine-tuning guide; 31B QLoRA from Google's best practices; E2B and E4B LoRA from Unsloth's Gemma 4 training page. Prices checked November 19, 2026 on the Nodus pricing page, where they are listed as hourly supply prices.

Cheapest single GPU per hour for Gemma 4 fine-tuning, 16-bit LoRA versus QLoRA: E2B $0.173 and $0.173, E4B $0.18 and $0.173, 12B $0.333 and $0.173, 26B A4B $1.09 and $0.333, 31B $3.593 and $0.333.
Up to 12B, the choice between LoRA and QLoRA is worth pennies an hour; at 26B and 31B it decides the price tier.

Two rows need a footnote. The E2B LoRA peak is Unsloth's; plain Transformers holds 10.2 GB of 16-bit weights before training, so step up to the 24 GB RTX 3090 for $0.007/hr more. The 26B A4B QLoRA figure needs Unsloth's loader. Google notes that standard bitsandbytes 4-bit skips the fused expert tensors, so a plain Hugging Face setup still needs about 56 GB.

When does QLoRA actually save money on Gemma?

From 26B up, and less often than you would think on the MoE model. The break-even is the ratio of the two hourly prices. For 31B, compare 16-bit LoRA on the H200 at $3.593/hr with QLoRA on the RTX A6000 at $0.333/hr. That gives 3.593 / 0.333 = 10.79, so QLoRA wins unless it runs more than 10.79 times as long.

The MoE model is where we would still pay for 16-bit. Google's guides say 4-bit quantization of the 128 routed experts can shift which 8 experts the router picks, and prefer bfloat16 LoRA when memory allows.

On the A100 80 GB at $1.09/hr against the A6000 at $0.333/hr, the ratio is 1.09 / 0.333 = 3.27. As an illustration, a 6-hour 16-bit run costs 6 x $1.09 = $6.54, and QLoRA would save at most about $4.54 of that. We'd take the 16-bit run unless QLoRA scores as well on your held-out prompts.

The 31B 16-bit case has a middle option. The 96 GB RTX PRO 6000 is listed at $1.35/hr, 2.66 times cheaper per hour than the H200, but 62.5 GB of frozen weights leaves it only 33.5 GB. Try 100 steps at your real sequence length before you commit to it. Our LoRA vs QLoRA guide explains where QLoRA's slowdown comes from.

Which Gemma settings quietly break a fine-tune?

Four details from Google's own tuning docs are easy to miss, and none of them raises an error.

Attention kernel. Keep PyTorch's default scaled dot-product attention (SDPA). Gemma 4's global attention layers use a 512 head dimension, which standalone FlashAttention-2 does not support, per the best-practices checklist. If a recipe you adapt sets attn_implementation="flash_attention_2", remove it.

Precision. Train in bfloat16. Google warns that the E2B and E4B audio encoder can overflow float16, and suggests Unsloth if your card only does fp16.

Chat template. Gemma 4 wraps every turn in <|turn> and <turn|>, calls the assistant role model, and puts reasoning inside <|channel>thought and <channel|>, per Google's dataset formatting guide. In non-thinking mode, E2B and E4B emit no thought tokens, while 12B, 26B A4B and 31B emit an empty thought block. Train in the mode you will serve in.

If you want the model to keep reasoning, Google says at least 50% of examples should carry reasoning traces, and Unsloth's page says at least 75%.

Stop token. Gemma 4 ends an assistant turn with <turn|> (token ID 106), not only <eos> (ID 1). An evaluation that stops on <eos> alone will make a healthy adapter look like it rambles.

One more: a high starting loss on E2B and E4B is normal. Google attributes it to the smaller models spreading probability over the 262,144-token vocabulary. Unsloth reports losses of 13 to 15 on those models as expected. Don't stop the run or raise the learning rate because of it.

What does a Gemma LoRA script look like?

Use Google's own QLoRA recipe and change only what a cloud run needs. The script below is the code from Google's (Q)LoRA guide for its multilingual-reasoning example. We switched the model to 31B and marked our other edits # changed; they put checkpoints where Nodus can restore them. The checkpoint pattern comes from the Nodus checkpoints guide.

import os import torch import nodus from datasets import load_dataset from peft import LoraConfig from transformers import AutoProcessor, BitsAndBytesConfig from trl import SFTConfig, SFTTrainer STATE = os.environ.get("NODUS_STATE_DIR", "/nodus/state") # changed model_id = "google/gemma-4-31B-it" # changed def create_conversation(sample): system_message = ( f"reasoning language: {sample['reasoning_language']}\n\n{sample['developer']}" ).strip() return { "prompt": [ {"role": "system", "content": system_message}, {"role": "user", "content": sample["user"]}, ], "completion": [ { "role": "assistant", "reasoning_content": sample["analysis"], "content": sample["final"], }, ], "chat_template_kwargs": {"enable_thinking": True}, } dataset = load_dataset("HuggingFaceH4/Multilingual-Thinking", split="train") dataset = dataset.map(create_conversation, remove_columns=dataset.features) dataset = dataset.train_test_split(test_size=0.1, seed=42) quantization_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_storage=torch.bfloat16, ) processor = AutoProcessor.from_pretrained(model_id) tokenizer = processor.tokenizer peft_config = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.0, bias="none", target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], exclude_modules=r"model\.(vision_tower|audio_tower)\..*", task_type="CAUSAL_LM", ) args = SFTConfig( output_dir=STATE, # changed max_length=2048, num_train_epochs=1, per_device_train_batch_size=2, per_device_eval_batch_size=2, gradient_accumulation_steps=8, gradient_checkpointing=True, # changed optim="paged_adamw_8bit", logging_steps=10, save_strategy="steps", save_steps=20, # changed save_total_limit=2, # changed eval_strategy="epoch", learning_rate=2e-4, weight_decay=0.01, warmup_steps=0.05, lr_scheduler_type="cosine", max_grad_norm=1.0, bf16=True, model_init_kwargs={"dtype": torch.bfloat16}, report_to="none", # changed ) trainer = SFTTrainer( model=model_id, args=args, train_dataset=dataset["train"], eval_dataset=dataset["test"], processing_class=tokenizer, quantization_config=quantization_config, peft_config=peft_config, callbacks=[nodus.checkpoint.HFTrainerCallback()], # changed ) trainer.train(resume_from_checkpoint=os.environ.get("NODUS_RESTORED") == "1") # changed trainer.save_model("/nodus/outputs/adapter") # changed 

Because each row has prompt and completion keys, TRL masks the prompt and trains only on the reasoning and answer. Swap in your own data in the same shape. For Unsloth, Google's Unsloth page shows the only change: replace the BitsAndBytesConfig and LoraConfig setup with FastVisionModel.from_pretrained and FastVisionModel.get_peft_model. Our Unsloth on a cloud GPU guide covers that path in full.

For 26B A4B, use Google's MoE recipe instead. It adds target_parameters=["experts.gate_up_proj", "experts.down_proj"] so all 128 routed experts get adapters, and excludes .*router.* so the router stays frozen.

How do you run the Gemma fine-tune on Nodus?

Run the script as a Nodus Job on the card from the table, with a spend cap. The managed nodus/sft runtime has presets for Qwen3 models only, and we have not validated Gemma 4's multimodal checkpoints on it, so a script Job is the path we'd use today. The flags come from the nodus run reference.

$ pip install nodus-compute $ nodus login $ nodus run --name gemma4-31b-qlora --gpu A6000 --image nodus/pytorch \ --checkpoint /nodus/state --interruptible \ --max-cost 10 --timeout 24h -d \ -- sh -c 'pip install "transformers>=5.10.1" "trl>=0.16.0" "peft>=0.19.0" accelerate datasets bitsandbytes nodus-compute && python train.py' $ nodus logs -f job/gemma4-31b-qlora $ nodus cp job/gemma4-31b-qlora:outputs/adapter ./adapter 

The version pins are the ones Google's guides install. Add --dry-run first to see the estimate without starting anything. --max-cost 10 is a cap, not a quote: at the cap the Job saves its state and suspends until you raise it.

--interruptible lets Nodus use cheaper capacity that can be reclaimed. HFTrainerCallback makes the Trainer save at the end of the current step when Nodus asks, and NODUS_RESTORED resumes from that save on the next machine. With QLoRA each checkpoint holds the adapter and optimizer state, not 31B of weights. The script downloads the base model at start, so that time is billed; nodus get gpus shows the other card types if the A6000 is busy.

How do you check that the Gemma adapter worked?

Compare the base and tuned model on held-out prompts, generated with the same thinking setting you trained with, and stop generation on both token 106 and token 1. Google's example evaluates the base model before training for exactly this reason: on its multilingual-reasoning task, the untuned E2B reasons in the wrong language, which gives you a concrete failure to check.

Watch validation loss too. Google suggests holding out 5% to 10% of your data and keeping the checkpoint with the lowest validation loss. It also warns that 31B can memorize a small dataset quickly, so evaluate often on the larger models. The Qwen LoRA guide walks through the same before-and-after check for another model family.

FAQ

Can you fine-tune Gemma 4 31B on a 24 GB GPU?

Yes, with QLoRA. Google measures 18 to 22 GB with 4-bit weights and gradient checkpointing, so a 24 GB card works at modest sequence lengths. We pick a 48 GB card in the table because 2 GB of headroom leaves little room for longer examples.

Should you fine-tune Gemma E2B or E4B?

Unsloth recommends E4B with QLoRA over E2B with LoRA, because E4B is bigger and the accuracy lost to 4-bit is small. On Nodus both fit the 16 GB RTX A4000 at $0.173/hr with QLoRA, so the larger model costs nothing extra per hour.

Can you serve a Gemma LoRA adapter without merging it?

Yes. Google's guide notes that vLLM can load one base model and switch adapters per request with --enable-lora. Merge only when you need a single standalone checkpoint, for example to export GGUF for llama.cpp.

Does fine-tuning Gemma 4 on text remove its vision and audio abilities?

Not if you freeze the towers instead of deleting them. Google's best practices say to exclude vision_tower and audio_tower from LoRA rather than strip their weights, so the model keeps image and audio input at inference time. On 12B, text, vision and audio share the decoder layers, so a text LoRA still touches all three.

Originally published on the Nodus blog.