You rent a 4-GPU pod. You launch your training job. Loss starts printing. Everything looks normal.

Then you run nvidia-smi and see one GPU pinned at 100% while the other three sit idle at 0%.

No exception. No warning. Accelerate is running and training is progressing. It is just executing on one GPU instead of four, at a quarter of the speed you are paying for.

What causes this

Accelerate uses a config file to decide how many processes to launch. If num_processes is set to 1, it launches one proces one GPU regardless of how many GPUs the pod has.

This is silent because Accelerate does not automatically use every available GPU. It uses exactly what the config specifies.

Common ways this happens on rented compute:

  • Config was generated on a single-GPU machine and copied to a multi-GPU pod unchanged
  • Config was created with accelerate config on a different instance type
  • num_processes was set to 1 during testing and never updated for the full run
  • No config file exists on a fresh container, so Accelerate falls back to single-process defaults

How to diagnose it

Check how many GPUs the pod has:

bash
nvidia-smi --query-gpu=name --format=csv,noheader | wc -l

Check what num_processes is set to in the Accelerate config:

bash
grep num_processes ~/.cache/huggingface/accelerate/default_config.yaml

If those two numbers don’t match, training is not using all available GPUs.

You can also check at runtime what Accelerate sees but only when invoked through accelerate launch, not plain python:

bash
accelerate launch -c "
from accelerate import Accelerator
acc = Accelerator()
print(f'num_processes: {acc.num_processes} | process_index: {acc.process_index}')
"

If num_processes prints 1 on a 4-GPU pod, that is the problem.

Note: num_processes and GPU count don’t have to match in every legitimate setup some workloads intentionally use fewer GPUs. The check is whether the config matches what your training job is supposed to do.

How to fix it

Regenerate the config for the current environment:

bash
accelerate config

Or override it at launch:

bash
accelerate launch --num_processes 4 train.py

For ephemeral pods where the GPU count varies, detect it dynamically:

bash
accelerate launch --num_processes $(nvidia-smi -L | wc -l) train.py

The --num_processes flag takes precedence over whatever is in the config file.

What ComputeFence checks today

computefence doctor warns if no Accelerate config file is found at all the missing-config case, where Accelerate silently defaults to single-process on a pod that may have multiple GPUs.

Comparing num_processes against the number of available GPUs is not a check ComputeFence runs yet. That is on the roadmap. Until it ships, the manual steps above are what catch a mismatched config.

bash
uvx computefence doctor

The pattern

This failure is expensive because it looks like success. Loss is printing and nothing has crashed. The only signal is slower-than-expected training and on a rented pod with no baseline, that is easy to miss until you check utilisation.

Check nvidia-smi right after launch. If GPUs are sitting at 0%, verify num_processes before the billing adds up. The hardware changed when you rented the new pod. The config did not.

pip install computefence · uvx computefence doctor