The Cloud Native Control Plane: Running AI Agents on Kubermetes

Deploying an AI agent script in a local container is easy. Running a fleet of autonomous, state-seeking, tool-executing agents in production without burning through your budget or creating a security nightmare is a distributed systems problem. The difference becomes apparent when an agent stops behaving like a conventional application.

A traditional microservice receives a request, executes a predictable sequence of operations, returns a response, and terminates. An autonomous agent might receive a task, reason about its objectives, invoke an external API, execute generated code, wait for another agent, retrieve historical state, retry a failed operation, and continue working long after the original HTTP request has disappeared.

Now imagine running a thousands of these agents simultaneously.

Some need GPU-backed inference. Others spend most of their time waiting for external tools. A few execute untrusted Python code. Some maintain conversations for hours, while others wake up in response to asynchronous events. One agent might trigger another, creating execution chains that are difficult to predict before they happen.

This is where the infrastructure conversation changes.

We are no longer discussing how to deploy an LLM-powered application. We are discussing how to operate a distributed system whose workloads are autonomous, resource-intensive, stateful, and capable of executing actions.

Kubernetes already has a vocabulary for many of these problems: scheduling, reconciliation, workload isolation, identity, resource quotas, event processing, and observability. The open-source ecosystem provides the remaining building blocks to connect agent frameworks, execution environments, and external tools into a common infrastructure platform.

The interesting engineering challenge is not putting an agent inside a Kubernetes Pod. It is building a platform where agents can be created, scheduled, authorized, executed, observed, suspended, and terminated without compromising the rest of the infrastructure.

The following architecture shows what that requires.

The Reality Check: Why Agentic AI Is a Distributed Systems Problem in Disguise

Consider a relatively ordinary recruitment automation workflow. A user submits a request to an HR agent to screen a collection of candidate resumes. The agent needs to:

1. Retrieve candidate information from a database.
2. Read and parse uploaded resumes.
3. Compare candidate experience against a job description.
4. Invoke an LLM for semantic evaluation.
5. Call an internal HR API to update candidate records.
6. Generate an evaluation report.
7. Notify the recruitment team.

A simple implementation might package the entire workflow into a Python application and deploy it as a container. Initially, everything works. Then the workload grows.

Recruiters submit hundreds of resumes simultaneously. LLM requests begin hitting rate limits. Some resumes contain malformed documents. A tool call times out after the agent has already completed three other operations. One agent retries a failed request repeatedly, while another loses its conversation state after a container restart.

Suddenly, the application is dealing with problems that have very little to do with prompt engineering.

The infrastructure is now responsible for managing autonomous execution.

Let's break down the operational problems.

1. Agent execution is not predictable.

A conventional application generally follows a known execution path. An agent can make decisions dynamically, invoke different tools, and change its execution plan based on intermediate results.

A workflow expected to finish in ten seconds might take several minutes because an external API is unavailable or the agent decides to investigate additional information. Traditional CPU and memory requests are not enough to describe every execution requirement.

2. Agent state outlives individual containers.

Agents frequently maintain conversation history, intermediate reasoning state, task checkpoints, tool results, and execution metadata. If a Pod disappears, the application should not necessarily lose the entire workflow. This introduces a familiar distributed systems requirement: separating application execution from durable state.

3. Tool execution introduces a security boundary.

An agent capable of executing generated Python or shell commands is fundamentally different from a service that only serves HTTP requests.

What happens when an agent executes:

cat /var/run/secrets/kubernetes.io/serviceaccount/token 

Or attempts to access internal network services that were never intended to be available to it? Running generated code inside an ordinary application container does not automatically make that execution safe.

4. Workloads have different compute requirements.

An agent invoking an external REST API might need only a few hundred megabytes of memory and minimal CPU. An agent running local inference might require an expensive GPU-backed node.

Running both workloads inside the same resource configuration creates unnecessary infrastructure costs and scheduling inefficiencies.

5. Autonomous execution can create runaway workloads.

Consider an agent stuck in a reasoning loop:

Agent | v Call LLM | v Evaluate response | v Tool invocation failed | v Retry | v Call LLM again | v Repeat indefinitely 

If every iteration consumes tokens, GPU time, and external API requests, one poorly controlled workflow can exhaust resources and affect unrelated workloads.

This is not an imaginary infrastructure concern. Any production system that permits autonomous execution needs explicit limits on retries, execution duration, token consumption, concurrency, and resource allocation.

The architectural shift

The important distinction is that an agent should not be treated as a long-running application container with unlimited authority.

It should be treated as a managed workload with:

• A declared execution specification.
• A controlled lifecycle.
• An isolated runtime.
• A defined identity.
• Explicit resource limits.
• Durable state.
• Observable execution history.
• A termination policy.

This is precisely where Kubernetes becomes interesting.

The Architectural Stack: Mapping Agent Needs to Cloud-Native Primitives

Kubernetes does not need to become an AI framework. LangGraph does not need to become a container scheduler. SPIFFE does not need to understand agent reasoning.

Each component should solve the problem it was designed to solve. The platform layer connects them.

Consider the following architecture:

 Developer / API / User | v Agent Gateway / API | v Agent Orchestration Layer LangGraph / AutoGen / CrewAI | v Kubernetes API Server | Agent Custom Resources | v Agent Operator | +----------------+----------------+ | | | v v v GPU Runtime Tool Runtime Event Workers Inference Wasm / gVisor CPU Workloads | | | +----------------+----------------+ | v Identity & Authorization SPIFFE / SPIRE / OIDC | v External Tool Services APIs / Databases / Cloud Services | v Observability Layer OpenTelemetry / Prometheus Grafana / Tempo 

This architecture separates responsibilities instead of forcing everything into one application.

There is one important architectural decision here.

The agent framework owns the workflow. Kubernetes owns the infrastructure lifecycle.

This distinction prevents the orchestration framework from becoming responsible for problems that already have mature solutions in the cloud-native ecosystem.

Now let's examine each layer.

Layer 1: Compute & Sandboxing (Scheduling GPUs, CPUs, and Isolating Tool Executions with gVisor/Wasm)

The first question when deploying an agent fleet is deceptively simple: Where should each execution happen?

Not every agent needs the same infrastructure. Treating all agent workloads as identical Pods is one of the quickest ways to create an expensive and inefficient platform.

1.1 Separate inference workloads from tool execution

An agent workflow usually contains multiple types of operations.

Inference workloads

These execute model inference, either through an external LLM provider or through locally hosted models.

They might require:

• GPU-backed compute.
• High memory capacity.
• Model weights and caching.
• High-throughput networking.
• GPU-aware scheduling.

Tool execution workloads

These execute individual actions requested by an agent.

Examples include:

• Running Python scripts.
• Executing shell commands.
• Processing uploaded documents.
• Calling internal REST APIs.
• Performing data transformations.

Most of these operations require CPU and memory rather than GPUs.

Agent coordination workloads

These manage workflow state, task scheduling, event processing, and communication between agents. They are generally lightweight compared with inference workloads. A reasonable infrastructure design separates these workloads into different Kubernetes node pools.

Kubernetes Cluster | +-- GPU Node Pool | | | +-- Local LLM Inference | +-- Embedding Models | +-- GPU-Accelerated Agents | +-- CPU Node Pool | | | +-- Agent Workers | +-- Workflow Execution | +-- API Integrations | +-- Isolated Execution Pool | +-- Python Tool Runtime +-- Shell Execution Runtime +-- Untrusted Code Workers 

Kubernetes node labels, taints, tolerations, affinity rules, and resource requests help direct workloads to appropriate infrastructure.

For example:

apiVersion: v1 kind: Pod metadata: name: local-llm-worker spec: nodeSelector: workload-type: gpu containers: - name: inference image: my-llm-runtime:v1 resources: limits: nvidia.com/gpu: 1 memory: 16Gi cpu: "4" 

This is a simplified example of GPU scheduling. A real deployment also needs the NVIDIA device plugin or GPU Operator, appropriate drivers, runtime configuration, and capacity planning.

The important point is that GPU allocation becomes an infrastructure decision rather than something buried inside an agent framework.

1.2 Scheduling heterogeneous workloads

GPU scheduling introduces a different set of challenges.

Imagine a cluster with four GPU nodes. Each node has a fixed amount of accelerator memory, and several agent workloads are competing for capacity.

A newly created inference workload might remain Pending because no suitable GPU is available, even though the cluster has plenty of ordinary CPU capacity. This is where Karpenter becomes useful. Karpenter provisions Kubernetes nodes based on unschedulable Pod requirements and available infrastructure capacity.

For an agent platform, it can help provision different instance types based on workload characteristics:

• GPU instances for local inference.
• Memory-optimized instances for model-heavy workloads.
• General-purpose instances for workflow workers.
• Smaller instances for lightweight tool execution.

The scheduling path becomes:

Agent Execution Requested | v Pod Created | v Kubernetes Scheduler | v Is Suitable Capacity Available? | +---+---+ | | Yes No | | v v Schedule Karpenter | | | v | Provision Node | | +-------+ | v Agent Execution 

Karpenter is not an agent-aware scheduler by itself. It responds to Kubernetes scheduling requirements. The platform must express those requirements correctly through resource requests, labels, affinity, and related scheduling constraints. That distinction matters when debugging capacity problems.

A workload that requests an unavailable GPU configuration will not magically become schedulable because Karpenter is installed.

1.3 The dangerous part: Executing generated code

Now consider an agent that receives this instruction:

"Analyze the uploaded CSV, calculate the average transaction value, and generate a chart."

The agent decides to execute Python. Where does that Python code run?

A naive implementation might execute it directly inside the agent's application container using Python's subprocess module. That is a bad security boundary.

Generated code is untrusted input, even when it originates from an agent running inside your own platform. A malicious document, prompt injection, or compromised tool response could influence what gets executed.

A container is a useful isolation mechanism, but an ordinary container shares the host kernel. Its security depends on namespaces, capabilities, seccomp, Linux security modules, runtime configuration, and the absence of exploitable kernel vulnerabilities.

For workloads that execute arbitrary code, stronger isolation deserves serious consideration. Two technologies are particularly interesting.

gVisor: A stronger application sandbox

gVisor provides an additional isolation boundary between an application and the host kernel.

Rather than allowing the application to interact directly with the host kernel through the usual system-call interface, gVisor implements much of that interface in its own application kernel, called the Sentry.

This reduces the host kernel surface exposed to potentially hostile application code.

A Kubernetes RuntimeClass can select the gVisor runtime:

apiVersion: node.k8s.io/v1 kind: RuntimeClass metadata: name: gvisor handler: runsc 

A tool execution Pod can then request that runtime:

apiVersion: v1 kind: Pod metadata: name: python-tool-execution spec: runtimeClassName: gvisor automountServiceAccountToken: false containers: - name: executor image: python:3.12-slim command: - python - /workspace/task.py resources: requests: cpu: "250m" memory: "256Mi" limits: cpu: "1" memory: "512Mi" 

This is a simplified abstraction. Production implementations should use controlled, preferably immutable execution images, read-only filesystems where practical, restricted capabilities, non-root execution, network policies, and ephemeral storage.

Also, gVisor requires compatible node runtime configuration. Adding runtimeClassName to a manifest does not install or configure gVisor automatically.

WebAssembly: Lightweight isolated execution

WebAssembly (Wasm) offers another interesting approach.

Instead of creating a conventional container for every small tool operation, the platform can execute compatible workloads inside a Wasm runtime.

Projects such as Wasmtime and Spin provide building blocks for running WebAssembly workloads.

Wasm is particularly useful for:

• Lightweight execution tasks.
• Deterministic data transformations.
• Portable tool implementations.
• Short-lived functions.
• Controlled access to host capabilities.

A Wasm runtime can expose only the filesystem, network, or host functions explicitly granted to a module.

That makes capability-based execution easier to reason about.

However, Wasm is not a universal replacement for containers. Native dependencies, arbitrary Python packages, system-level operations, and GPU workloads may require different execution environments.

A practical platform can support both.

The engineering rule: Never let the agent's reasoning process and its untrusted code execution share an unrestricted security boundary.

The agent decides what needs to happen. The execution runtime decides what is permitted to happen.

Layer 2: Identity & Security (SPIFFE/SPIRE, Workload Identity, and Zero-Trust Tool Access)

Once an agent can invoke tools, identity becomes a platform concern. Consider a recruitment agent that needs to read candidate records and update interview status.

The simplest implementation stores an API key in an environment variable:

env: - name: HR_API_KEY value: "some-static-api-key" 

It works, but it creates several problems.

What happens when the key leaks through application logs? How do you rotate it across hundreds of agents? How do you prevent a resume-screening agent from accessing payroll information? What happens when an agent's permissions need to be revoked immediately?

Static credentials do not provide a particularly good foundation for autonomous systems. The better approach is to give workloads identities and authorize individual actions.

2.1 SPIFFE: Giving every workload a verifiable identity

SPIFFE defines a standard for identifying software workloads independently of their physical location or infrastructure.

A SPIFFE identity is represented using a SPIFFE ID.

For example:

spiffe://agent-platform.example.com/ns/hr/sa/resume-agent 

This identity tells us:

• The workload belongs to the agent-platform.example.com trust domain.
• It operates in the hr namespace.
• It uses the resume-agent service account.

The identity does not depend on the Pod IP address, node name, or container hostname. That is particularly useful in Kubernetes, where Pods are frequently created, destroyed, and rescheduled.

SPIRE, the reference implementation of SPIFFE, provides the infrastructure for issuing and validating workload identities.

SPIRE typically involves:

• A SPIRE Server responsible for managing trust and identity issuance.
• SPIRE Agents running on infrastructure nodes.
• Workload attestation to establish which workload is requesting an identity.
• Short-lived X.509 certificates or JWT-SVIDs.

The resulting identity can be used for secure communication between services.

For example:

Resume Agent | | SPIFFE Identity v HR Tool Gateway | | Validate Identity v Authorization Policy | +-- Read Candidate Records | +-- Update Interview Status | +-- Deny Payroll Access 

The gateway does not simply trust a request because it originated from a Kubernetes Pod. It verifies the workload's identity and checks whether the requested operation is permitted.

2.2 OIDC and workload identity

Another important part of the architecture is OpenID Connect (OIDC). OIDC builds on OAuth 2.0 and provides a standardized way to represent authenticated identities using signed identity tokens.

In Kubernetes, projected service account tokens can be issued with specific audiences and limited lifetimes. An external service can validate these tokens against the cluster's configured issuer and signing keys.

Cloud workload identity systems can use this mechanism to exchange Kubernetes service account credentials for short-lived cloud credentials.

For example, an agent running in Kubernetes might need to read an object from Amazon S3.

Instead of storing permanent AWS access keys inside the container, the workload can use an appropriate AWS workload identity mechanism, such as EKS Pod Identity or IAM Roles for Service Accounts (IRSA).

The flow looks like this:

Agent Pod | v Kubernetes Service Account | v Projected OIDC Token | v AWS Identity Provider | v Temporary AWS Credentials | v Authorized S3 Access 

The agent receives temporary credentials associated with its assigned identity. When those credentials expire, they can be refreshed through the supported identity integration. This removes the need to distribute permanent cloud credentials across agent workloads.

2.3 The missing piece: Authorization at the tool level

Identity alone does not solve the problem.

Knowing that a workload is a legitimate resume-screening agent does not automatically mean it should be allowed to perform every operation in the HR system.

This is where authorization policies enter the architecture.

A tool gateway can enforce rules such as:

The platform can implement these policies through an authorization service, an API gateway, or a policy engine such as Open Policy Agent (OPA).

A simplified Rego-style policy might look like:

package agent.authorization default allow := false allow if { input.identity == "resume-agent" input.tool == "candidate-api" input.action == "read" } allow if { input.identity == "resume-agent" input.tool == "candidate-api" input.action == "update_interview_status" } 

This example illustrates the decision logic, not a complete production policy. A real policy must also validate the trustworthiness of the identity assertion, resource scope, tenant boundaries, and relevant request attributes.

2.4 Treat every tool invocation as an authorization decision

One mistake I would avoid when designing an agent platform is giving an agent a collection of credentials and trusting the framework to decide how to use them. The framework should not be the security authority.

Every tool invocation should pass through an authorization boundary.

For example:

Agent Framework | v Tool Invocation Request | v Identity Verification | v Authorization Policy | +---- Denied ----> Reject Request | v Credential Exchange | v External Tool API | v Audit Event 

There is another subtle issue worth addressing: an agent should not automatically inherit the permissions of the user who initially created its task.

A user might request a report, but the agent should only receive the narrowly scoped permissions needed to generate that report. Delegation should be explicit, bounded, and auditable.

Short-lived credentials, audience restrictions, narrowly scoped tokens, and centralized authorization help establish that boundary.

The security principle is straightforward: An agent should only be able to perform the actions explicitly granted to its workload identity, regardless of what its LLM-generated instructions request.

Layer 3: Orchestration & Scaling (CRDs, Event-Driven Agent Runtimes, and KEDA)

Now that compute and identity are handled, we reach the part that makes the architecture operationally interesting.

How do we manage the lifecycle of thousands of agent executions?

Creating Pods manually is not a platform strategy. Building a custom orchestration API inside a Python application is not particularly attractive either.

Kubernetes already provides an extensibility mechanism for representing application-specific resources. That mechanism is the Custom Resource Definition (CRD).

3.1 Representing agents as Kubernetes resources

A CRD allows us to introduce new resource types into the Kubernetes API.

Instead of managing agents as anonymous application processes, we can represent them as declarative resources. Imagine a custom resource named AgentRun. A developer could describe an agent execution like this:

apiVersion: agents.platform.io/v1alpha1 kind: AgentRun metadata: name: resume-screening-001 namespace: hr spec: framework: langgraph workflow: name: resume-screening version: "1.0" execution: runtime: gvisor timeout: 300s maxSteps: 20 resources: cpu: "1" memory: 1Gi identity: serviceAccountName: resume-agent state: backend: postgresql tools: - candidate-api - resume-parser retryPolicy: maxAttempts: 3 

This is an illustrative API design, not a built-in Kubernetes resource or an existing standard shared by all agent frameworks. Its purpose is to demonstrate how an agent execution can become a declarative infrastructure object.

The platform can now reason about:

• Which framework should execute the workflow.
• Which runtime should isolate its tools.
• Which service account should represent the workload.
• How long the execution may run.
• Which tools it can invoke.
• How many retries are permitted.
• Where its state should be stored.

The resource becomes the desired state of the execution.

3.2 Why an Operator is needed

A CRD defines a resource. It does not automatically implement the resource;s behavious. That is the responsibility of a Kubernetes Operator. An Operator watches custom resources and reconciles the actual cluster state with the desired state declared in those resources.

For our AgentRun example, the Operator might:

1. Watch for newly created AgentRun objects.
2. Validate the execution specification.
3. Resolve the agent framework and workflow version.
4. Create the required execution resources.
5. Attach the appropriate service account.
6. Configure runtime isolation.
7. Track execution progress.
8. Update the resource status.
9. Clean up temporary resources after completion.

The reconciliation loop might look like this:

 AgentRun CRD | v Agent Operator | v Read Desired State | v Validate Execution Policy | v Create Runtime Pods | v Observe Pod Status | v Update AgentRun Status | v Reconcile Again 

A simplified status section might look like:

status: phase: Running execution: currentStep: 4 completedSteps: 3 failedSteps: 0 conditions: - type: Ready status: "True" 

The Operator should be idempotent. If it receives the same reconciliation request repeatedly, it should not create duplicate executions or accidentally restart a completed workflow.

That is an important detail because Kubernetes controllers operate through repeated reconciliation, not one-time imperative commands.

For an actual implementation, I would also distinguish between an Agent resource, which describes a reusable agent definition, and an AgentRun resource, which represents one execution.

That separation makes lifecycle management significantly cleaner.

3.3 Mapping agent frameworks to Kubernetes

The platform should not force every agent framework into the same execution model.

Different frameworks have different strengths.

LangGraph

LangGraph is particularly interesting for workflows that require explicit state management, conditional transitions, checkpoints, and human intervention.

Its graph-based execution model maps naturally to long-running Kubernetes-managed workflows.

For example:

Resume Received | v Resume Parsing | v Candidate Evaluation | v Policy Check | +---- Needs Review ----> Human Approval | v Update Candidate Record | v Generate Report 

The workflow engine handles the execution graph. Kubernetes handles the compute and infrastructure lifecycle.

A production implementation should persist LangGraph checkpoints outside the worker Pod so that execution can resume after infrastructure failures.

AutoGen

AutoGen supports multi-agent collaboration and conversation-driven workflows.

A platform can run individual agent participants as separate workers or group them into a managed workflow execution, depending on the collaboration model.

For example, a research agent, validation agent, and reporting agent can communicate through controlled message channels.

However, separating every participant into a Pod is not automatically beneficial. It adds network overhead, scheduling activity, and operational complexity. The deployment boundary should reflect actual isolation and scaling requirements.

CrewAI

CrewAI provides a role-oriented approach to coordinating agents around tasks and responsibilities.

A crew can be packaged as a workflow deployment, while individual task execution can be delegated to isolated workers when the workload requires it.

The important design choice is to avoid coupling the infrastructure API too tightly to one framework.

A common Kubernetes platform should expose infrastructure-level concepts and allow framework-specific adapters to translate those concepts into execution behavior.

3.4 Event-driven orchestration: Agents should not spend their lives waiting

One of the biggest inefficiencies in naive agent deployments is keeping application workers alive while they wait for external operations. Consider an agent that calls a document-processing service. The service takes 45 seconds to process a large document.

Does the agent need to hold a worker thread, a container, and all its execution resources for the entire duration? Not necessarily. This is where event-driven architecture becomes useful. Projects such as NATS, Apache Kafka, and Knative Eventing provide mechanisms for asynchronous communication.

The workflow can be split into independent stages.

Agent Workflow | v Submit Document Job | v Publish Event | v Message Broker | v Document Processing Worker | v Publish Completion Event | v Agent Workflow Resumes 

The agent does not need to hold an execution thread while the document processor works.

Instead, it persists its checkpoint, waits for the completion event, and resumes when the result becomes available. NATS is useful for lightweight messaging and request/reply patterns, while JetStream provides persistence and replay capabilities. Kafka is appropriate when the platform needs durable event streams, replayable histories, partitioned consumption, and high-throughput event processing.

Knative Eventing provides Kubernetes-native event routing abstractions through resources such as Brokers, Triggers, and Sources.

These technologies are complementary in some architectures, but they are not interchangeable in every situation. The platform should select a messaging backbone based on delivery guarantees, retention, throughput, ordering, and operational requirements rather than deploying every available broker.

A critical detail is event delivery semantics.

Most distributed event systems provide at-least-once delivery under common configurations. Therefore, a completion event may be delivered more than once.

The workflow must use idempotency keys, durable checkpoints, and deduplication to avoid executing the same side effect repeatedly. An agent updating a candidate record should not accidentally update it three times because the completion event was redelivered.

3.5 KEDA: Scaling based on actual agent demand

Traditional Kubernetes Horizontal Pod Autoscaling primarily reacts to metrics such as CPU and memory utilization. That is useful, but agent workloads introduce other signals that matter more. Imagine a tool execution queue with 500 pending tasks. The worker Pods might be using almost no CPU because they are waiting for messages. A CPU-based autoscaler sees no reason to scale.

Meanwhile, the queue continues growing. This is exactly the type of problem KEDA (Kubernetes Event-driven Autoscaling) addresses. KEDA allows workloads to scale based on external and event-driven metrics.

Depending on the configured scaler, it can respond to:

• NATS message backlog.
• Kafka consumer lag.
• Queue depth.
• HTTP request rates.
• Prometheus metrics.
• Other supported external signals.

A simplified ScaledObject might look like:

apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: agent-tool-workers spec: scaleTargetRef: name: agent-tool-workers pollingInterval: 10 minReplicaCount: 0 maxReplicaCount: 50 triggers: - type: prometheus metadata: serverAddress: http://prometheus.monitoring:9090 metricName: pending_agent_tasks threshold: "20" query: | sum(agent_task_queue_depth) 

This example assumes the Prometheus metric is correctly exposed and the scaler is configured with appropriate authentication and permissions. The result is a worker pool that responds to actual pending work rather than waiting for CPU utilization to increase.

There is an important interaction between KEDA and Karpenter:

• KEDA decides how many workload replicas are needed.
• Kubernetes Scheduler determines whether those Pods can be placed.
• Karpenter can provision additional nodes when existing capacity is insufficient.

Together, they provide a useful scaling chain.

Agent Task Queue | v Queue Backlog Increases | v KEDA | v Increase Worker Replicas | v Kubernetes Scheduler | v Insufficient Node Capacity? | v Karpenter | v Provision Additional Nodes | v Execute Agent Tasks 

3.6 Preventing the infinite agent loop

Scaling an agent platform is not just about adding more workers. Sometimes the correct response to an increase in workload is to stop accepting additional execution.

Imagine an agent that continuously retries a failed tool call. KEDA sees a growing queue and creates more workers. Those workers encounter the same failure and publish more retry events.

The infrastructure scales successfully while the application becomes progressively less healthy. This is a classic positive feedback loop. The platform needs execution budgets.

For every AgentRun, I would enforce:

These are illustrative values, not universal production defaults.

The platform should also implement:

• Exponential backoff with jitter.
• Circuit breakers for failing external services.
• Dead-letter queues for repeatedly failing tasks.
• Per-tenant concurrency limits.
• Token and cost budgets.
• Explicit cancellation and timeout handling.

Autonomy without execution limits is not a production feature. It is an uncontrolled resource allocation problem.

Layer 4: Observability (Tracing Agent Reasoning Loops and Tool Latency with OpenTelemetry)

Here is a scenario that will feel familiar to anyone who has operated distributed applications. An agent takes 90 seconds to complete a task. The user reports that the system is slow.

You inspect the application logs.

You find:

Agent started LLM request completed Tool invoked Tool response received LLM request completed Workflow completed 

Nothing appears obviously wrong. But where did those 90 seconds go?

Was the LLM provider slow? Did the tool API take 60 seconds? Did the agent execute unnecessary reasoning steps? Was it waiting for an event? Did the workflow retry a request?

Ordinary application logs do not provide enough context to answer these questions. Agentic workloads need distributed tracing that follows execution across the entire workflow.

4.1 OpenTelemetry as the common telemetry layer

OpenTelemetry provides vendor-neutral APIs, SDKs, instrumentation, and collection protocols for generating and exporting telemetry.

It supports:

• Distributed traces.
• Metrics.
• Logs.
• Context propagation.

For agent infrastructure, OpenTelemetry provides a way to correlate events across frameworks, worker processes, tool gateways, inference services, and external integrations.

Consider the recruitment workflow again.

A single execution might generate the following trace:

AgentRun: resume-screening-001 | +-- Workflow Execution | | | +-- Resume Parsing | | | +-- LLM Inference | | | | | +-- Provider Request | | +-- Token Usage | | | +-- Candidate API | | | | | +-- Identity Verification | | +-- Authorization Check | | +-- HTTP Request | | | +-- Human Approval | | | +-- Report Generation | +-- Workflow Completion 

Every operation should carry trace context so that the platform can reconstruct the complete execution path. A tr...