Choose an inference deployment option by model type (proprietary vs open-weight), traffic shape, latency target, model features, data control requirements, and cost model. Managed model APIs are the fastest way to start, managed ML endpoints and serverless GPU services trade some control for convenience, and self-hosted serving on containers gives the most control over cost, data residency, and portability. Control Plane runs containerized model servers on AWS, GCP, and Azure regions and on your own GPU clusters, with built-in NVIDIA GPUs on Control Plane-managed locations that offer them and no Kubernetes cluster to operate on those locations.
Disclosure: Control Plane publishes this guide.
TL;DR
Five deployment options:
- Managed model APIs: Amazon Bedrock, Google Gemini Enterprise Agent Platform (formerly Vertex AI), Microsoft Foundry (formerly Azure AI Foundry), and model providers’ own APIs
- Managed ML endpoints (Amazon SageMaker AI endpoints, Gemini Enterprise Agent Platform endpoints, Azure Machine Learning online endpoints)
- Serverless GPU inference services
- Self-hosted serving on containers or Kubernetes: vLLM, NVIDIA Dynamo-Triton (formerly Triton Inference Server), or KServe as the Kubernetes serving layer
- A platform layer that runs your model containers across clouds, such as Control Plane
Decision factors:
- Model type: a proprietary API model or an open-weight model
- Traffic shape: steady, bursty, or batch
- Latency target: interactive or offline
- Model features: structured output, tool calling, context length, and fine-tuned adapters
- Data control and residency: where prompts and completions are processed
- Cost model: per token, per GPU-hour, or reserved capacity
- Team skills: the operations experience you have to run what you choose
Top inference challenges: latency, GPU cost from low utilization, cold starts, autoscaling on the wrong signal, provider rate limits and outages, GPU capacity, data privacy, observability gaps, and multi-region availability.
Containers vs managed platforms: Running model servers in your own containers gives you full control over the runtime, weights, and optimizations. A platform layer removes the work of operating Kubernetes clusters, autoscaling, and multi-cloud networking while keeping that container-level control.
How Control Plane fits: Control Plane runs containerized model servers such as vLLM as workloads, on Control Plane-managed locations in AWS, GCP, and Azure regions or on GPU clusters you add as locations, and handles Day 2 operations for you, including autoscaling, built-in observability, and security.
How do I choose an AI model inference service for my application?
Start with these seven factors.
Model type
If you need a proprietary model from a provider such as OpenAI, Anthropic, or Google, you call it through a managed model API. If you run an open-weight model such as Llama, Mistral, or Qwen, you can call it per token through a managed API that hosts it, deploy it to a managed ML endpoint, or serve it yourself. Open-weight models give you portability and the ability to fine-tune, quantize, and change versions on your schedule.
Traffic shape
Steady, predictable traffic favors reserved or dedicated compute where GPU utilization stays high. Bursty traffic benefits from services that scale elastically and avoid paying for idle GPUs. Batch workloads, such as nightly embedding jobs, can run on spot or preemptible GPUs and tolerate queuing.
Latency target
Interactive applications such as chat, code completion, and retrieval-augmented search need a short time to first token and steady token output. Dedicated endpoints and self-hosted serving give the most predictable p99 latency. Offline workloads such as summarization pipelines or evaluation runs can tolerate cold starts and queuing.
Model features
List the features your application depends on before you pick a service. Structured output and tool calling vary by API and model, and self-hosted servers such as vLLM enforce JSON schemas through structured outputs (guided decoding). Context length sets KV cache memory, so long contexts need much more GPU memory per request than short ones. Serving many fine-tuned variants is cheapest as LoRA adapters on one base model, which vLLM can serve together on one GPU. Applications that call several models need a routing layer, such as an LLM gateway, for fallback and cost-based routing.
Data control and residency
Regulated industries and privacy-sensitive applications often require that prompts and completions stay in a specific region or in the organization’s own environment. Self-hosted serving and managed ML endpoints with private networking can meet these requirements. Managed model APIs process data on the provider’s infrastructure. Hyperscaler APIs such as Bedrock, Gemini Enterprise Agent Platform, and Microsoft Foundry offer regional processing and private connectivity, and retention terms vary by provider and contract. Check whether cross-region inference routing is on, because it can move requests out of your chosen region.
Cost model
Per-token pricing is simple to budget at low volume but grows with usage. Per-GPU-hour pricing rewards high utilization. Reserved capacity lowers the hourly rate but requires a volume forecast. Include idle GPU time, egress fees, and engineering time to get a true total cost of ownership. For the build-or-rent decision, see the build vs rent GPUs break-even model.
Team skills
Managed APIs need almost no MLOps experience. Managed ML endpoints need moderate DevOps and ML engineering effort. Self-hosted serving needs experience with GPU drivers, container runtimes, model optimization, autoscaling, and incident response. A platform layer needs container and configuration skills without cluster operations.
Decision table
| If you need | Start with |
|---|---|
| Fastest integration, proprietary models, minimal operations | Managed model API |
| Custom or fine-tuned models with cloud automation | Managed ML endpoint |
| Elastic GPU scaling for bursty open-weight workloads | Serverless GPU inference service |
| Full control over runtime, weights, data, and cost | Self-hosted serving on containers |
| Container control across several clouds without operating clusters | A platform layer such as Control Plane |
Can you compare different AI model deployment solutions?
This table compares five inference deployment options across six dimensions, based on each option’s published documentation.
| Option | What you manage | Model choice | Scaling and cold starts | Cost model | Data control | Best fit |
|---|---|---|---|---|---|---|
| Managed model APIs (Amazon Bedrock, Google Gemini Enterprise Agent Platform, Microsoft Foundry, provider APIs) | Prompts and application code. The provider runs serving and scaling. | A catalog of proprietary and open-weight models, with fine-tuning on some. | Automatic, with no cold starts for provider-hosted models; imported custom models can cold start. Rate limits can throttle bursts. | Per token, with provisioned throughput options for reserved capacity, billed hourly or on a fixed-term commitment depending on the provider. | The provider processes prompts and completions under its retention terms. Hyperscaler APIs offer regional processing and private connectivity. | Prototyping, low to moderate volume, and teams without ML infrastructure experience. |
| Managed ML endpoints (Amazon SageMaker AI endpoints, Gemini Enterprise Agent Platform endpoints, Azure Machine Learning online endpoints) | Model artifacts, container choice, endpoint configuration, and monitoring. | Your own custom, fine-tuned, or open-weight models. | Autoscaling on configured metrics. A minimum instance count avoids scale-up delay but adds cost; some services can scale to zero. | Per instance-hour while instances run. | Runs in your cloud account, with private networking options. | Custom models that need predictable latency on cloud-managed infrastructure. |
| Serverless GPU inference services (on-demand GPU platforms, such as Google Cloud Run with GPUs) | Container image and model weights. The provider schedules GPUs and scales. | Any model you package in a container, including custom and fine-tuned models. | Scales to zero by default, with optional warm instances. Cold starts include image pull, weight loading, and engine warmup. | Per second of instance time, including idle time before an instance scales down and any warm minimum instances. | Varies by provider. | Bursty or unpredictable traffic where cold starts are acceptable. |
| Self-hosted serving (vLLM, NVIDIA Dynamo-Triton, KServe on Kubernetes) | GPU provisioning, Kubernetes or VMs, networking, autoscaling configuration, monitoring, and patching. | Any model you can load, with full control over weights, quantization, and runtime. | You configure autoscaling. KServe uses Knative, HPA, or KEDA, and plain vLLM needs HPA or KEDA. Warm replicas avoid cold starts. | GPU instances plus engineering and operations time. | Full control; data stays in your environment. | High-volume, latency-sensitive, or compliance-driven workloads with an experienced platform team. |
| Control Plane (platform layer for containers across clouds) | Container image and workload configuration. Control Plane runs compute, networking, scaling, and Day 2 operations; on clusters you add as locations, you run the underlying servers. | Any model you package in a container, plus a vLLM template for Hugging Face models. | Serverless workloads scale on request concurrency and can scale to zero, and standard and stateful workloads, including GPU model servers, can scale on custom metrics through KEDA, which is enabled on Control Plane’s Global Virtual Cloud (GVC), a named set of locations across one or more clouds. The vLLM template runs as a stateful workload with a set replica count and caches weights on a volume to shorten restarts. | You pay for the CPU and memory a workload reserves, with no contracts or minimums; the pricing page lists GPU rates per GPU-hour. | Runs in the locations you select, on Control Plane-managed compute in AWS, GCP, and Azure regions or on your own clusters. | Teams that want container control, Day 2 operations handled, and active-active serving across regions and clouds, with their own GPU clusters as additional locations, without cluster operations. |
Many production architectures combine more than one option. A common pattern uses a managed model API for proprietary models and self-hosted serving for open-weight models where cost, data control, or customization matter.
What are common challenges in AI model inference and how can I overcome them?
This table maps each common problem to its cause and a fix.
| Challenge | Why it happens | How to fix it |
|---|---|---|
| Latency: time to first token and inter-token latency | Prefill on long prompts sets time to first token. Decode, which is bound by memory bandwidth, sets inter-token latency. Network hops add to both. | Enable prefix caching for repeated system prompts and retrieved context, use chunked prefill so long prompts do not stall other requests, quantize weights (FP8, AWQ, or GPTQ) to speed decoding, use speculative decoding to cut inter-token latency at low batch sizes, and serve from regions close to users. |
| GPU cost and low utilization | GPUs sit idle between requests or off-peak, and reserved capacity costs the same regardless of load. | Right-size GPU types, autoscale on request concurrency or queue depth, and scale idle low-traffic services down. See the best GPU cloud providers. |
| Cold starts from large model weights | Loading multi-gigabyte weights into GPU memory takes time when a new replica starts, on top of image pull and engine warmup. | Cache weights on persistent volumes, use quantized models, keep a warm minimum of replicas before expected traffic, and use a fast weight loader such as the Run:ai Model Streamer, which vLLM supports. |
| Autoscaling on the wrong signal | CPU utilization misses the real bottleneck, which is request concurrency, queue depth, or KV cache usage. | Scale on request concurrency, waiting requests (vLLM’s num_requests_waiting), or KV cache usage. GPU memory and GPU utilization are poor signals because vLLM preallocates most GPU memory and utilization reads high even at low load. |
| Throughput under concurrent load | Serving one request at a time wastes GPU cycles, and KV cache pressure forces recomputation. | Use a server with continuous batching (vLLM, SGLang, and TensorRT-LLM batch this way by default), size the KV cache for expected concurrency with max model length and an FP8 KV cache, use tensor parallelism within one node only when a model does not fit on one GPU, and add replicas for throughput. |
| Provider rate limits and outages | Managed APIs impose token and request limits, and a single provider is a single point of failure. | Retry with exponential backoff, keep a fallback model or provider, and route around unhealthy endpoints. See how to run AI inference across multiple GPU clouds. |
| GPU capacity and quotas | High-end GPUs are often unavailable in a given region or blocked by account quotas. | Request quotas early, qualify more than one GPU type per model, and keep capacity in more than one region or provider. |
| Data privacy and residency | Sending prompts to third-party APIs can conflict with residency rules or expose sensitive data. | Host open-weight models in your own environment or use managed endpoints with private networking, keep regional processing on, and review each provider’s retention terms. |
| Model and version management | Rolling out new model versions, rolling back bad ones, and running variants side by side adds risk. | Version model servers by container image tag and pin model weights to a fixed revision, roll out gradually, and keep the previous version ready for rollback. |
| Observability gaps | Standard monitoring misses tokens per second, time to first token, and KV cache pressure. | Export serving-framework metrics and track time to first token, inter-token latency, latency percentiles (p50, p95, p99), token throughput, queue depth, and KV cache usage. |
| Multi-region availability | One region adds latency for distant users and is a single point of failure. | Run replicas in at least two regions behind a global endpoint that routes to the nearest healthy one. |
For the infrastructure components behind these fixes, see what infrastructure you need to run an inference layer.
Where can I find a cloud friendly deployment platform for AI models?
A cloud friendly inference platform meets five criteria:
- Runs standard containers. You package your model server as a standard container image and deploy it without rewriting it for a proprietary runtime.
- Works on more than one cloud. You can run the same workload definition on AWS, GCP, and Azure regions and on your own GPU clusters.
- Supports GPUs. Your serving framework gets GPU access without manual driver work.
- Scales on demand signals. Autoscaling follows request concurrency or queue depth instead of generic CPU metrics.
- Keeps data where you choose. Inference traffic and model weights stay in the regions and clusters you select.
Control Plane is the unbreakable platform for running production workloads. It runs serverless, standard, stateful, and cron workloads natively on AWS, GCP, Azure, and Oracle Cloud Infrastructure (OCI), VM workloads on locations joined through Control Plane’s Managed Kubernetes, and workloads on Kubernetes clusters you already run or your own machines, including on premises, active-active across regions and clouds. For a wider comparison of hosting platforms, see the best PaaS providers in 2026.
How do containers compare with a managed PaaS for scaling AI infrastructure with predictable costs?
Running model servers in containers yourself gives you full control over the serving framework, weights, quantization, and GPU choice. The cost is operational work, including Kubernetes clusters, GPU drivers, networking, autoscaling policies, certificate rotation, and incident response.
A managed platform takes over that layer. You still bring your container image, and the platform runs compute, scaling, load balancing, and Day 2 operations. The cost shifts from engineering hours to platform fees.
| Dimension | Self-managed containers | Managed platform (Control Plane) |
|---|---|---|
| Control | Every layer | Container image and workload configuration, with infrastructure run by the platform |
| Cost predictability | Varies with utilization, spot pricing, and engineering time | Billing follows the CPU and memory a workload reserves, and the pricing page lists GPU rates per GPU-hour, so monthly cost tracks workload configuration |
| Operational work | Cluster management, patching, scaling, and monitoring | Control Plane runs clusters, patching, scaling, and monitoring on Control Plane-managed locations |
| Multi-cloud | A separate cluster and networking setup per cloud | One workload definition runs across AWS, GCP, Azure, and your own clusters. A GPU workload’s GPU setting applies to every location in its GVC, so run built-in GPUs (NVIDIA T4 or A10G) and GPUs on your own clusters in separate GVCs |
| Availability | You build redundancy | The 99.999% availability SLA applies to workloads running at least two replicas in at least two locations |
How Control Plane fits
Control Plane runs containerized model servers such as vLLM as workloads. On Control Plane-managed locations in AWS, GCP, and Azure regions there is no Kubernetes cluster to operate, and teams can also add GPU clusters they already run, including on a neocloud, as locations.
The vLLM template serves one Hugging Face model through vLLM’s OpenAI-compatible API on a built-in NVIDIA A10G or T4, or on your own GPU cluster registered as a location, and caches weights on a persistent volume so restarts skip the download. It runs in a GVC with exactly one location. For redundancy across regions, install it once per region, each in its own single-location GVC, and fail over between their endpoints in your client or gateway.
On Control Plane, serverless workloads scale on request concurrency and can scale to zero, and standard and stateful workloads can scale on custom metrics such as queue length through KEDA, enabled per GVC. A workload deployed to a GVC runs active-active in every location of that GVC from one definition, and the workload’s global endpoint uses DNS to route clients to the nearest location where the workload passes its readiness probes.
Control Plane’s Universal Cloud Identity (UCI) lets workloads call AWS, GCP, and Azure services, such as Amazon Bedrock or Google’s Gemini Enterprise Agent Platform, through an IAM role or service-account grant instead of long-lived keys in your code, with short-lived, least-privilege credentials at runtime, so an application can combine self-hosted open-weight models with managed model APIs.
Customers typically cut compute costs 30 to 50 percent compared with running directly on AWS, GCP, or Azure. Control Plane is PCI DSS Level 1, SOC 2 Type II, HIPAA, and GDPR compliant.
FAQ
These questions come up often when teams compare inference deployment options.
Should I use a model API or host my own model?
Use a model API when you need a proprietary model, want to start quickly, or have low to moderate traffic. Host your own model when you need control over data, want to fine-tune or quantize, or when per-token cost at your volume exceeds the cost of dedicated GPU compute.
What is the cheapest way to serve an open-weight model?
It depends on utilization. At high, steady traffic, self-hosted serving on reserved GPUs usually gives the lowest cost per token, and spot GPUs suit batch jobs that tolerate preemption. At low or bursty traffic, a per-token API that hosts the model, or a GPU service that scales down when idle, avoids paying for unused GPUs. Include engineering and operations time in the comparison.
How do I reduce inference latency?
Cut time to first token by reusing cached prompt prefixes and splitting long prompts into chunks. Cut the time between tokens with quantized weights and speculative decoding. Keep warm replicas near users so requests never wait on a cold start.
How do I keep inference running when a model provider has an outage?
Keep a fallback. For managed model APIs, route to an alternative provider or model when calls fail. For self-hosted models, run replicas in at least two regions. On Control Plane, a workload in a GVC with several locations runs active-active, and the workload’s global endpoint uses DNS to route clients to the nearest location where it passes its readiness probes. Every location in that GVC must offer the GPU model the workload requests.
What GPU should I use for LLM inference?
Match GPU memory to model size, context length, and concurrency. Weight memory is the parameter count times bytes per parameter: 2 at FP16 or BF16, 1 at FP8, and about 0.5 at 4-bit, plus headroom for the KV cache, which grows with context length and concurrent requests. A 7 or 8 billion parameter model needs about 14 to 16 GB at FP16 and fits on a 24 GB GPU such as the A10G with moderate context, or on a 16 GB T4 with 4-bit quantization. A 70 billion parameter model needs about 140 GB at FP16, so it runs on four 80 GB GPUs such as the H100 with tensor parallelism, on two 80 GB GPUs at FP8, on one 80 GB GPU only at 4-bit, or on one H200 or B200 at FP8. Control Plane’s pricing page lists NVIDIA GPUs from T4 to B200, subject to availability.
How do I autoscale AI inference workloads correctly?
Scale on request concurrency, waiting requests, or KV cache usage instead of CPU or GPU utilization, which do not reflect inference load. On Control Plane, a vLLM server running as a standard or stateful workload can scale on vLLM’s waiting-request count through KEDA.
Can I run inference across multiple clouds?
Yes. Package the model server as a container and run it in more than one cloud. On Control Plane, one workload definition runs across AWS, GCP, and Azure regions and your own clusters. A GPU workload uses one GPU source per GVC: built-in NVIDIA T4 or A10G on Control Plane-managed regions, or the GPUs on your own clusters added as locations.
How do I handle data privacy requirements for AI inference?
Host models in your own environment or use managed endpoints with private networking and regional processing. Send sensitive data to a third-party model API only when its retention and usage terms meet your requirements.
What infrastructure do I need to run an inference layer?
You need GPU compute, a model server such as vLLM or Dynamo-Triton, a container runtime or a serving layer such as KServe, autoscaling, load balancing, observability, and storage for model weights. For the full breakdown, see what infrastructure you need to run an inference layer.
Is a platform layer worth it compared to managing Kubernetes myself?
For most teams, yes, when engineering time is the constraint. A platform layer removes cluster operations and custom autoscaling setup while you keep control of the model server and weights. The tradeoff is a platform fee in place of engineering hours, and Control Plane customers typically cut compute costs 30 to 50 percent compared with running directly on AWS, GCP, or Azure.
To run inference workloads across clouds without operating clusters, talk to the Control Plane team or sign up free.
Originally published on the Control Plane blog.
