You launch a training job on RunPod. PyTorch detects your GPU, torch.cuda.is_available() returns True, and your environment checks appear to pass.

Then your script reaches its first real CUDA operation and crashes:

RuntimeError: CUDA error: no kernel image is available for execution on the device

If you are running on a Blackwell GPU with an older PyTorch build, you have hit a GPU architecture compatibility problem. The frustrating part is that your environment looks perfectly healthy right up until the first CUDA kernel executes.

What causes this error

CUDA kernels are compiled for specific GPU architectures, identified by compute capability for example sm_86 for an A40, sm_90 for an H100.

When you install PyTorch, the build you download contains kernels compiled for a fixed range of architectures. PyTorch 2.4.1+cu124 supports sm_50 through sm_90.

Blackwell GPUs: the RTX PRO 4000, RTX PRO 4500, RTX PRO 6000, and RTX 5090 have compute capability sm_120. They are outside that range.

With an incompatible PyTorch build, PyTorch does not fail immediately. torch.cuda.is_available() returns True. The GPU name appears in the output. Everything looks fine. Then the training script reaches the first real CUDA kernel call the first matrix multiplication, the first forward pass and it throws the error above.

By that point the pod is already allocated, your dependencies are installed, and billing has started.

Note on the B200: The B200 belongs to the sm_100 architecture family, not sm_120. Always check the actual compute capability reported by your GPU rather than assuming every Blackwell card uses the same architecture.

You can check this in two lines:

import torch
cap = torch.cuda.get_device_capability(0)
print(f"Device capability: sm_{cap[0]}{cap[1]}")
print(f"Supported architectures: {torch.cuda.get_arch_list()}")

If your device capability is not in the supported list, the run will fail when the first kernel executes.

Why torch.cuda.is_available() returning True is not enough

This is where the problem becomes deceptive.

torch.cuda.is_available() confirms that PyTorch can access CUDA and identify a device. It does not confirm that the installed build contains compatible kernels for every operation you intend to run. Architecture compatibility is a separate check.

GPU detection and GPU compatibility are two different things.

First-hand reproduction on RunPod

I reproduced this on a RunPod RTX PRO 4000 Blackwell pod using the default PyTorch 2.4.1+cu124 template.

The terminal output looked clean:

✓ PyTorch 2.4.1+cu124 detected (CUDA available — NVIDIA RTX PRO 4000 Blackwell)

Meanwhile PyTorch was printing a warning to stderr that sm_120 was not supported by this build. The pre-flight check passed. Training would have failed on the first real kernel call.

The warning that appeared in stderr:

UserWarning: CUDA device capability sm_120 is not supported by this PyTorch build.
Supported capabilities: sm_50 through sm_90.

If your training script runs in a tmux session, pipes output to a log file, or uses a framework like Axolotl or Unsloth, you will miss this warning entirely until the forward pass crashes.

The pod also reported /workspace - 466126.9 GB free of 2190892.5 GB unusually large volume figures that appear to be a virtualisation reporting artefact on early Blackwell deployments rather than real capacity.

This reproduction exposed a gap in the original pre-flight checks and led directly to the GPU architecture check added in ComputeFence v0.3.2.

Which GPUs are affected with standard PyTorch images

As of PyTorch 2.4.x, these cards are not supported by the builds shipped in standard RunPod PyTorch templates:

NVIDIA RTX PRO 4000 Blackwell: sm_120
NVIDIA RTX PRO 4500 Blackwell: sm_120
NVIDIA RTX PRO 6000 Blackwell: sm_120
NVIDIA RTX 5090: sm_120
NVIDIA B200: sm_100 (different architecture, same class of problem)

If RunPod assigns one of these GPUs and you are on the default template, you will hit this error on the first kernel call.

How ComputeFence catches it

Since v0.3.2, ComputeFence compares each GPU’s compute capability against the architectures supported by the installed PyTorch build using get_device_capability() and get_arch_list(). A mismatch is flagged as a blocker not a warning:

✗ BLOCKER: GPU architecture mismatch
NVIDIA RTX PRO 4000 Blackwell has compute capability sm_120
This PyTorch build supports sm_50 through sm_90
Training will fail on the first CUDA kernel call
Fix: Use a PyTorch build compiled for sm_120, or select a different GPU

This runs before billing starts.

uvx computefence doctor

Note: get_arch_list() reports architectures compiled into the PyTorch installation. It is a useful compatibility signal but does not guarantee that every third-party CUDA extension will work. Run a small representative CUDA operation before committing to a long training run on new hardware.

How to fix it

Option 1: Install a PyTorch build that supports your GPU

Use a release compiled for sm_120. Check pytorch.org/get-started/locally for current compatibility. After installing, verify with the two-line check above before launching your job.

Option 2: Select a different GPU

If you need to use an existing PyTorch image with pinned dependencies, select a GPU whose architecture is in the supported range. An A40 (sm_86) or H100 (sm_90) will work with PyTorch 2.4.x.

The broader lesson

GPU detection is not GPU compatibility. As providers add newer hardware generations, the gap between the latest cards and the PyTorch builds in standard templates will keep appearing. Passing torch.cuda.is_available() is not proof that your training workload can execute.

Check the architecture before billing starts.

GitHub: github.com/Francisco-Booth/ComputeFence
Install: pip install computefence