You rent a GPU on RunPod, download your model, start fine-tuning. Everything looks fine until the job crashes:
OSError: [Errno 28] No space left on device
Or your training completes but GGUF export fails:
RuntimeError: [Errno 28] No space left on device
You have spent money on GPU compute, potentially hours of training, and now you cannot finish the job.
Having a large storage volume attached does not mean every process is using it. Here is why it happens, how to diagnose it, and how to catch it before the next run starts billing.
Why RunPod pods run out of disk
RunPod pods have two storage layers:
Container disk (/): Fast, small (typically 10–20GB on standard templates), wiped when the pod stops.
Network volume (/workspace): Persistent, sized by you when you create it.
The problem is that several things write to container disk by default even when a network volume is attached:
- HuggingFace model cache (~/.cache/huggingface unless HF_HOME is set)
- Training checkpoints (most frameworks default to ./output)
- pip cache (~/.cache/pip)
- GGUF export artifacts
Any one of these is manageable. Combined on a 10–20GB container disk, they fill it fast.
Find out which filesystem is actually full first
Before changing any configuration, identify where space is being consumed:
df -h
Look at / and /workspace separately. If / is almost full but /workspace has space, you have a configuration problem not a capacity problem.
To find what is consuming space on the root filesystem:
du -xhd1 / 2>/dev/null | sort -h
The GGUF export case
GGUF export is the peak disk moment in a typical fine-tuning pipeline. For a 12B model you briefly hold simultaneously:
- The 16-bit merged model (~24GB)
- The f16 GGUF (~24GB)
- The quantised output (~6GB)
- Additional temporary working files
Combined peak can exceed 50GB. The exact amount depends on your conversion workflow, quantisation method, and whether intermediate files coexist. If your working directory only has 20GB free, the export fails partway through. Hours of training wasted.
One operator running a Gemma 12B fine-tune on Vast.ai hit exactly this training ran clean for hours, the GGUF export step failed because the working directory had ~20GB against a ~45–47GB peak requirement.
Three things to set before every job
- Point HF_HOME at the network volume
export HF_HOME=/workspace/.cache/huggingface
mkdir -p $HF_HOME
Add this to your pod startup script, not just your shell profile if training launches via SSH or an automated script it will not load .bashrc.
- Point your output directory at the network volume
Hugging Face Trainer
training_args = TrainingArguments(
output_dir="/workspace/outputs",
save_total_limit=2, # prevents old checkpoints filling the volume
)
Axolotl
output_dir: /workspace/outputs
- Size your volume for the full pipeline, not just training
Account for base model weights, checkpoints, and export artifacts simultaneously:
- 7B model: at least 60GB
- 12B model: at least 100GB
- 70B model: at least 350GB
Catch it before billing starts
uvx computefence doctor
Real output from a RunPod A40 pod that caught a disk warning before launch:
ComputeFence v0.3.2 — Pre-flight diagnostic
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
3 WARNINGS · 0 BLOCKERS · 5 PASSED
Storage
✓ HF_HOME is set to /workspace/.cache/huggingface
✓ Persistent volume(s) detected: /workspace
⚠ Volume /workspace — 9.8 GB free of 10.0 GB (below 20 GB)
Fix: Free up disk space or move checkpoints to a larger volume
9.8GB free on a 10GB volume. The HF_HOME was correctly set. The volume was mounted. But the available capacity was too low for a full training run. Caught before the job started.
ComputeFence warns below 20GB and blocks below 5GB.
Current limitation: ComputeFence does not yet calculate peak storage requirements from your model size. It checks available capacity and cache configuration. That is a narrower claim than “your pipeline will fit” but it catches the cases where you are obviously going to run out before you start paying.
Make the check automatic:
alias train="uvx computefence doctor && python train.py"
If disk is below 5GB, ComputeFence exits with a non-zero code and the training command does not run.
GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence
