An agent can produce a convincing explanation for an action. That explanation is not an authorization decision.
This distinction matters as soon as a model can call tools that change state: create a ticket, update a record, issue a refund, deploy a service, or send a message. A prompt can describe the intended workflow, and the model can select the right function, while the application still needs to prove that this caller may perform this exact operation on this exact resource.
A safe design puts that proof in ordinary application code at the tool boundary. The model proposes an action. A server-side policy decides whether the action is allowed, and the domain service performs it with a durable audit trail and retry semantics.
The model is a planner, not the policy engine
A tool schema limits the shape of a request. It does not establish who the caller is, which tenant owns a record, or whether a requested change is permitted. The same is true of system prompts and tool descriptions. They influence model behavior, but they are not enforceable access controls.
For example, an agent might receive a tool input like this:
const toolInput = { operation: "close_case", caseId: "case_781", reason: "resolved" }; The caller identity should come from the authenticated application request, not from a field the model can fill. If a tool accepts userId, tenantId, or role as model-supplied values and trusts them, the tool has delegated identity to generated text.
Instead, carry a trusted principal alongside the model-generated arguments:
type Principal = { subject: string; tenantId: string; permissions: ReadonlySet<string>; }; type CloseCaseInput = { caseId: string; reason: "resolved" | "duplicate" | "withdrawn"; }; async function closeCaseTool( principal: Principal, input: CloseCaseInput, ) { const record = await cases.getById(input.caseId); if (!record || record.tenantId !== principal.tenantId) { throw new Error("Case not found"); } if (!principal.permissions.has("case:close")) { throw new Error("Forbidden"); } return cases.close({ caseId: record.id, reason: input.reason, actorId: principal.subject, }); } The generic “not found” response can avoid revealing whether another tenant’s identifier exists. The important part is that authorization happens after resolving the resource and before performing the mutation. A permission check only at chat-session creation is not enough: permissions can change, and the model can take several steps between the first prompt and the final action.
Give every tool the smallest useful capability
Avoid a universal tool such as run_sql, call_url, or execute_command when the task can be represented by a narrower operation. A broad tool turns mistakes in planning into mistakes in the environment. A focused tool such as get_case_summary or add_case_note gives the policy layer a smaller surface to reason about.
Separate reads from writes. Read-only tools can often run automatically within the caller’s permissions. Mutating tools can require a stronger authorization check, an approval, or both. A preview operation is useful when it returns a bounded description of the intended change without applying it. But preview and commit must share the same server-side validation rules; otherwise the preview can approve one resource while the commit changes another.
Treat every model-produced argument as untrusted input. Validate it against a schema, enforce limits such as maximum string length and allowed enum values, and check resource ownership and destination policy in trusted code. Do not concatenate generated text into SQL, shell commands, URLs, or templates. If the function needs a destination, prefer a server-side identifier resolved through an allowlist over an arbitrary URL supplied by the model.
Make approval specific to the action
A human approval should bind to the concrete operation, not to a vague statement such as “the agent may manage cases.” Show the user the resource, fields that will change, the destination, and any consequential side effect. Then create an approval record that is tied to a digest of that exact operation.
A simple internal state model could look like this:
proposed -> validated -> awaiting_approval -> approved -> executed \\-> expired \\-> rejected The commit endpoint should re-check authorization and the operation digest. If the model changes the target case or amount after approval, the digest changes and the old approval cannot be reused. Approval tokens should expire, be single-use, and be bound to the authenticated principal and operation. This prevents an agent from treating approval for one step as permission to make a different change later in the conversation.
Human review is not a replacement for least privilege. If the agent can perform destructive actions without approval, a prompt-injection attempt or a planning error can still cause damage before anyone notices. Define which actions require approval in policy code, not in the model’s reasoning.
Retries must not repeat the side effect
Tool calls can time out after the downstream service has committed a change. The model or orchestrator then sees an ambiguous result and may retry. If the tool creates a second ticket, sends a second payment, or emits a duplicate notification, the system has turned a transport uncertainty into a business incident.
Require an operation identifier for state-changing calls. Store it with the caller, operation type, and a hash of the validated input. Enforce uniqueness in durable storage so concurrent retries cannot both execute. If the same identifier arrives with different input, reject it instead of silently reusing the original result.
async function executeOnce( principal: Principal, operationId: string, input: CloseCaseInput, ) { const inputHash = stableHash({ action: "close_case", tenantId: principal.tenantId, input, }); return operationStore.runOnce({ key: principal.subject + ":" + operationId, inputHash, execute: () => closeCaseTool(principal, input), }); } The storage layer needs to handle the race atomically, usually with a unique constraint or conditional write. An in-memory map is not enough when function instances scale horizontally or restart. For workflows that include a database update and an event, use a transaction plus an outbox or another durable handoff pattern; do not assume a successful model response means every downstream effect happened exactly once.
Keep an audit trail that explains the decision
For each tool invocation, record a correlation ID, authenticated subject, tool name, resource identifier, policy outcome, approval reference where relevant, idempotency key, and execution result. Record enough to investigate who did what and why, while avoiding raw prompts, credentials, or unnecessary personal data in logs.
A useful audit event separates the model’s proposal from the application’s decision:
{ "tool": "close_case", "subject": "user_42", "resource": "case_781", "policy": "allow", "approval": "approval_93", "operation_id": "op_6fe2", "result": "completed" } The log should answer whether the request was denied, whether approval was absent or expired, whether execution timed out, and whether a retry reused a prior result. Those signals make a useful incident timeline and help distinguish a model planning issue from a policy or downstream-service failure.
Test the boundary, not only the happy path
A strong test set should include cross-tenant identifiers, revoked permissions, malformed tool arguments, altered inputs after approval, duplicate operation IDs, concurrent retries, downstream timeouts, and prompt-injection strings in retrieved documents or tool output. Verify the server-side outcome in each case. A model saying “I will not do that” is not evidence that the authorization boundary works.
Also test the allowed path. Security controls that silently block legitimate work tend to be bypassed. Measure denial reasons, approval latency, duplicate suppression, and tool error rates, while keeping dashboards free of sensitive prompt content.
The design principle is simple: let the model suggest the next step, but make every tool prove that the caller is allowed to take it. Keep identity, authorization, approval, idempotency, and audit decisions in code that you can test independently of the model. That is where an agent stops being a prompt with credentials and becomes a system with an accountable security boundary.