Run AI/ML containers on GPUs without vendor lock-in by keeping your model and serving code on open frameworks and APIs like PyTorch, ONNX, vLLM, or Triton Inference Server with an OpenAI-compatible API, packaging them as standard Open Container Initiative (OCI) images that get GPU access through the container runtime and Kubernetes device plugins, and storing weights and data in S3-compatible object storage. Then make sure you can get GPUs from more than one provider, because capacity is where most teams actually end up locked in.
GPU lock-in hides along two axes. The first is the GPU hardware vendor axis, where CUDA-only code ties you to NVIDIA silicon. The second is the cloud provider axis, where managed ML platforms, proprietary storage, and committed-spend agreements tie you to a single cloud.
Disclosure: Control Plane publishes this guide.
TL;DR
- Two lock-in axes. GPU hardware vendor lock-in (CUDA vs. ROCm, vendor-specific kernels) and cloud provider lock-in (managed endpoints, proprietary storage, committed quotas) are separate problems.
- Ten layers where lock-in hides. Model code, base images, container GPU access, Kubernetes scheduling, accelerator choice, serving APIs, training orchestration, storage, interconnects, and GPU capacity each carry their own lock-in risk.
- Capacity is the hidden lock-in. Quotas, reservations, and committed-spend contracts with a single provider are easy to miss because they rarely appear in architecture diagrams.
- Where to run. Control Plane, hyperscaler GPU instances, GPU neoclouds, and on-premises servers all support portable containers, with different trade-offs.
- Prove it with a portability test. Deploy the same inference image on a second GPU provider, load weights from external S3-compatible storage, and fail traffic over. If it works, your inference path is portable.
- Where Control Plane fits. Control Plane lets you deploy one workload definition across multiple GPU-equipped locations, including Control Plane-managed cloud regions, your own Kubernetes clusters joined through Control Plane’s Bring Your Own Kubernetes (BYOK), and your own GPU servers joined through Control Plane’s Managed Kubernetes (MK8s).
How do you run AI/ML containers on GPUs without vendor lock-in?
Avoiding GPU hardware vendor lock-in means writing model code in PyTorch or JAX, converting models to ONNX where cross-vendor inference matters, and avoiding hand-written CUDA kernels that cannot compile for other targets. Your container images should use minimal runtime base images that carry the CUDA or ROCm user-space libraries and let the host supply the GPU driver through the Container Device Interface (CDI) or the vendor’s container runtime hook. Serving should happen through vLLM, Triton Inference Server, or KServe exposing an OpenAI-compatible API so that clients never couple to a hardware backend.
Avoiding cloud provider lock-in means skipping managed ML endpoints like SageMaker, Vertex AI, and Azure Machine Learning when portability matters, packaging workloads as Open Container Initiative (OCI)-compliant container images, orchestrating them with Kubernetes and GPU device plugins, and storing weights, checkpoints, and datasets in S3-compatible object storage using open formats such as safetensors and Parquet. It also means keeping your GPU capacity sourced from more than one provider, because quotas and reservations are the practical lock-in that bites hardest when demand spikes and your single provider says no.
Where does GPU lock-in hide?
Ten layers carry lock-in risk, on one axis or both.
| Layer | Lock-in axis | Locked-in choice | Portable choice | How hard to undo later |
|---|---|---|---|---|
| Model code and kernels | Hardware vendor | CUDA-only custom kernels | PyTorch, OpenAI Triton kernels, ONNX Runtime | Hard. Rewriting kernels requires GPU programming expertise and regression testing. |
| Base images | Hardware vendor | Vendor-specific CUDA devel images baked into every build | Runtime images without build toolchains, with the kernel driver supplied by the host (CUDA or ROCm libraries stay in the image) | Medium. Rebuild images and CI pipelines, but application code stays the same. |
| GPU access in containers | Hardware vendor | Vendor-specific runtime hooks (e.g., nvidia-container-runtime) | Container Device Interface (CDI), now supported by containerd, CRI-O, Docker, and Podman | Medium. Requires container runtime configuration changes on every node. |
| Kubernetes GPU scheduling | Both | Vendor-specific node selectors hard-coded in manifests | Standard device plugins (NVIDIA GPU Operator, AMD GPU Operator) and Dynamic Resource Allocation (DRA), which became generally available in Kubernetes 1.34 | Medium. Update manifests and install the appropriate operator on each cluster. |
| Proprietary accelerators | Hardware vendor | Google TPU, AWS Trainium, AWS Inferentia | NVIDIA or AMD GPUs available from many providers | Very hard. Proprietary accelerators require framework-level changes (e.g., XLA for TPUs) and vendor-specific toolchains. |
| Model serving API | Cloud provider | SageMaker endpoints, Vertex AI endpoints, Azure ML managed endpoints | vLLM, Triton Inference Server, or KServe exposing an OpenAI-compatible API | Medium. Repackage the model into an open serving runtime and update client endpoints. |
| Training orchestration | Cloud provider | Managed training jobs (SageMaker Training, Vertex AI Training) | Ray Train, Kubeflow Trainer, or plain Kubernetes Jobs | Medium to hard. Rewrite job definitions and handle distributed coordination yourself. |
| Weights, datasets, and checkpoints | Cloud provider | Provider-only storage features (S3 Express One Zone directory buckets, S3 Object Lambda, GCS-specific IAM) | S3-compatible object storage with open formats (safetensors, Parquet) | Easy to medium. Copy data out, convert formats if needed, update storage paths. |
| High-speed interconnect for multi-node training | Both | Provider-specific fabrics (AWS Elastic Fabric Adapter, Google GPUDirect-TCPX and TCPXO on A3) | NCCL (NVIDIA) or RCCL (AMD) over standard InfiniBand or RoCE | Very hard. Multi-node training is the hardest workload to move because interconnect performance directly affects training throughput. |
| GPU capacity | Cloud provider | Quotas, reservations, and committed-spend agreements with one provider | Capacity you can source from several providers simultaneously | Hard. Committed-use agreements bill for the full term, and a second provider needs its own quota and validation. |
Inference workloads are far easier to make portable than multi-node training. An inference container typically needs one or a few GPUs, communicates with clients over HTTP, and loads weights from object storage. Multi-node training depends on high-speed GPU-to-GPU interconnects, collective libraries (NCCL on NVIDIA, RCCL on AMD) tuned to a specific network fabric, and checkpoint storage that must keep up with training speed. If you can only invest in portability for one workload type, start with inference.
Where can you run portable GPU containers?
Portable GPU containers run anywhere a host supplies a compatible GPU driver and exposes the device to the container runtime. The practical differences come down to what you manage, what stays portable, and what ties you in.
| Option | Examples | What stays portable | What ties you in |
|---|---|---|---|
| Hyperscaler GPU instances | AWS EC2 P5, Google Cloud A3, Azure ND H200 v5, Oracle Cloud Infrastructure (OCI) GPU instances | Your container images, Kubernetes manifests, model weights | Instance types, quotas, committed-use discounts, VPC networking |
| Hyperscaler managed ML platforms | SageMaker, Vertex AI, Azure Machine Learning | Model weights (if stored externally) | Proprietary training APIs, serving endpoints, SDK integrations, billing structure |
| GPU clouds and neoclouds | CoreWeave, Lambda, Nebius, Crusoe | Container images, Kubernetes workloads | Provider-specific Kubernetes extensions, capacity agreements, networking |
| Your own GPU servers | On-premises data centers, colocation facilities | Full stack control, container images, orchestration | Capital expenditure, hardware refresh cycles, physical infrastructure |
| A platform that runs one workload definition across GPU locations | Control Plane | One workload definition, Open Container Initiative (OCI) images, canonical DNS geo-routed endpoint, one identity for AWS, GCP, and Azure services | Platform dependency for orchestration and routing |
Hyperscaler GPU instances
Running containers on raw GPU instances from AWS, Google Cloud, Azure, or Oracle Cloud Infrastructure (OCI) gives you access to the latest GPU hardware. You install the GPU operator on a Kubernetes cluster or use the vendor’s container runtime on a VM. Portability stays high as long as you avoid provider-specific Kubernetes extensions and keep your images clean of cloud SDK dependencies.
Hyperscaler managed ML platforms
SageMaker, Vertex AI, and Azure Machine Learning offer convenience for training and inference but embed lock-in deeply. Their training job APIs, endpoint management, and monitoring are proprietary. If portability matters, treat these platforms as acceleration options for prototyping and keep production workloads on infrastructure you control.
GPU clouds and neoclouds
CoreWeave, Lambda, Nebius, and Crusoe each offer Kubernetes clusters or VMs with NVIDIA GPUs, with pricing and availability profiles that differ from hyperscalers for GPU-heavy workloads. A comparison of neoclouds and hyperscalers can help you evaluate which fits your workload profile. They offer fewer managed services than hyperscalers, which leaves fewer proprietary services to depend on.
Your own GPU servers
On-premises or colocated GPU servers remove cloud provider lock-in but introduce capital expenditure and operational burden. You own the drivers, the networking, and the cooling. This option makes sense for steady-state training workloads where utilization is consistently high.
A platform that runs one workload definition across GPU locations
With this option, one workload definition serves inference from a hyperscaler GPU region and a neocloud or on-premises GPU server at the same time, and the workload’s canonical endpoint uses DNS to geo-route clients to the nearest location that passes its readiness checks. It reduces cloud provider lock-in for the workload, and you take on a dependency on Control Plane for orchestration and routing.
How do you avoid GPU capacity lock-in?
You can have perfectly portable container images, open model formats, and vendor-neutral APIs, and still be stuck because your single GPU provider has no quota left when you need to scale.
Quotas and reservations are the mechanism. Cloud providers allocate GPU capacity through quota systems. Getting quota for high-demand GPUs like H100s or B200s often requires committed-spend agreements that lock you into one provider for one to three years. Committed-use discounts and savings plans bill for the full term whether or not you use the capacity.
Sourcing from more than one provider reduces that risk. Qualify at least two GPU providers for your inference workloads. Keep a warm deployment on a second provider that can absorb traffic if your primary provider runs out of capacity or has an outage. A guide on how to combine GPU capacity across providers walks through the mechanics of this approach.
Keeping a second provider warm requires periodic validation. Run the portability test on a regular schedule and after major changes. Load weights, send test traffic, and confirm that latency and output quality meet your thresholds.
Routing inference to whichever provider has capacity completes the setup. If your serving layer exposes an OpenAI-compatible API on both providers, a DNS-based or load-balancer-based routing layer can direct traffic to whichever location passes health checks. This pattern also helps you keep your AI product online during a model provider outage.
How do you prove your GPU workload is portable?
Test portability before you need it. Six tests cover the layers that most often break when a GPU workload moves.
| Test | How to run it | Pass condition |
|---|---|---|
| Run the same inference image on a second GPU provider | Push your Open Container Initiative (OCI) image to a registry accessible from both providers. Deploy it on a GPU node at the second provider with the same environment variables and model path. | The container starts, passes its readiness check, and returns correct inference results. |
| Switch the client to the same OpenAI-compatible endpoint on the second provider | Point your client SDK or HTTP calls to the second provider’s endpoint URL. | Responses are functionally equivalent. Client code requires only an endpoint URL change. |
| Load weights from S3-compatible storage outside the original cloud | Configure the serving runtime to pull weights from an S3-compatible bucket hosted outside your primary cloud (e.g., Cloudflare R2, MinIO, or a second provider’s object storage). | The model loads successfully and inference latency is within your acceptable range. |
| Schedule the workload on a second Kubernetes cluster with a different GPU operator setup | Apply the same Kubernetes manifests to a cluster running a different GPU operator (e.g., NVIDIA GPU Operator on one cluster, a manually configured driver on another). | The pod schedules, gets GPU access, and passes health checks. |
| Run a training job resumed from a checkpoint on different hardware | Copy a training checkpoint to the second provider’s storage. Launch the training job with the resume flag. | Training resumes without errors and loss curves continue from the checkpoint. |
| Fail inference traffic over to the second provider | Simulate a failure on the primary provider (stop the deployment or block traffic). Verify that DNS or your routing layer redirects traffic to the second provider. | Clients experience a brief interruption but resume receiving correct responses from the second provider. |
Automate what you can in CI. The tests that fail are the layers where you are still locked in.
How does Control Plane fit?
Control Plane is the unbreakable platform for running production workloads. It runs serverless, standard, stateful, and cron workloads natively on AWS, GCP, Azure, and Oracle Cloud Infrastructure (OCI), VM workloads on locations joined through Control Plane’s Managed Kubernetes, and workloads on Kubernetes clusters you already run or your own machines, including on premises, active-active across regions and clouds. For GPU workloads packaged as containers, it addresses the cloud provider axis of lock-in.
Workloads can attach NVIDIA GPUs, and the pricing page lists models from T4 to B200. A workload deployed to Control Plane’s Global Virtual Cloud (GVC), a named set of locations, runs in every location of that GVC from one workload definition. Locations can span Control Plane-managed regions on AWS, GCP, Azure, and Oracle Cloud Infrastructure (OCI), Kubernetes clusters you already run joined through Control Plane’s Bring Your Own Kubernetes (BYOK), or your own GPU servers joined through Control Plane’s Managed Kubernetes (MK8s), including on premises. GPU Ocean lets you pull GPU capacity from any provider, which you can do today with MK8s by joining GPU servers as locations.
Every workload has its own canonical endpoint that DNS geo-routes clients to the nearest location where that workload passes its readiness checks. If a GPU location stops passing readiness checks, new DNS lookups go to the nearest location that still passes them.
Control Plane’s Template Catalog, its library of ready-to-install apps, includes a vLLM template that serves models with an OpenAI-compatible API. Control Plane’s Universal Cloud Identity lets workloads reach AWS, GCP, and Azure services through one identity, with no long-lived cloud keys in your code. Control Plane handles Day 2 operations for you, including autoscaling, built-in observability, and security. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.
Control Plane does not make CUDA code run on other GPU vendors and does not make SageMaker or Vertex AI endpoints portable. On BYOK and MK8s locations, you keep responsibility for the underlying GPU servers and drivers.
The same principles apply beyond GPUs, as covered in how to avoid cloud vendor lock-in and Kubernetes AI infrastructure pain points.
FAQ
Short answers to common questions about portable GPU workloads.
How do I run AI/ML containers on GPUs without vendor lock-in?
Use open frameworks like PyTorch and ONNX, serve via vLLM or Triton with an OpenAI-compatible API, package as Open Container Initiative (OCI) images that use CDI or Kubernetes device plugins for GPU access, store weights in S3-compatible storage, and qualify at least two GPU providers so one provider’s quota limits do not block scaling.
Is CUDA vendor lock-in?
Yes, in practice, for code written directly in CUDA because CUDA targets NVIDIA GPUs; using frameworks like PyTorch and ONNX Runtime reduces that lock-in by allowing the same model code to run on multiple backends.
Can the same container run on NVIDIA and AMD GPUs?
Usually not with standard stacks. NVIDIA (CUDA) and AMD (ROCm) need different user-space libraries, so most teams build two image variants from shared application code and identical model weights. A few runtimes, such as Modular MAX, ship a single hardware-agnostic image that runs on both.
How do I avoid lock-in with SageMaker or Vertex AI?
Treat managed ML platforms as prototyping or acceleration options and run production workloads on open serving runtimes like vLLM or Triton on Kubernetes with S3-compatible storage and an OpenAI-compatible API.
What is the most portable way to serve an LLM?
Package vLLM or Triton as an Open Container Initiative (OCI) container image, expose an OpenAI-compatible API, store weights in safetensors on S3-compatible storage, and run on Kubernetes with a standard GPU device plugin.
Can I move GPU workloads between clouds?
You can move inference workloads easily if you follow portability practices; multi-node training is much harder because of interconnects and collective communication differences.
How do I avoid GPU quota and capacity lock-in?
Qualify at least two providers, keep a warm standby, run the portability test regularly, and avoid committing all GPU spend to a single provider.
How does Control Plane help run GPU workloads across providers?
Control Plane lets you deploy one workload definition to a GVC spanning multiple GPU-equipped locations, attaches NVIDIA GPUs (the pricing page lists models from T4 to B200), provides a canonical DNS-routed endpoint per workload, and lets you pull GPU capacity from any provider by joining GPU servers as MK8s locations, while handling autoscaling, observability, and security.
Does Red Hat’s RamaLama help with GPU portability?
RamaLama simplifies local LLM development by abstracting container runtime and GPU access, which helps base-image and GPU-access testing but does not solve cloud provider capacity sourcing or multi-provider routing.
To run GPU workloads across providers from one workload definition, talk to the Control Plane team or sign up free.
Originally published on the Control Plane blog.



