As a Junior DevOps engineer holding the keys to the AWS kingdom, life was peak "unrestricted free will." I could spin up architecture, tear down stacks, and feel like a tech god. Then came the inevitable executive tap on the shoulder: "Show me the bill."

I pulled up AWS Cost Explorer. He saw the numbers, smiled, and signed off. Smooth sailing.

Then leadership dropped the big vision: "We are going full AI-first!"

Naturally, I handed him Amazon Bedrock. He was thrilled. And like every proper business executive, he eventually asked the classic follow-up: "Show me the bill."

I pulled up Cost Explorer again. Still green, still happy. Then came the twist that ruined my week: "Now break down the cost for EACH individual developer."

Here is where the corporate tragedy set in. Turns out we aren't running on a Payer Account. Which instantly meant:

  • Cost allocation tags on IAM users? Only the payer account can activate them. Not us.
  • CUR 2.0? Billing lives with the payer. And even with it, CUR refreshes a few times a day. That's an autopsy, not a guardrail.

Worse, leadership didn't just want visibility, they asked the million-dollar question: "Can we stop a developer before they set our budget on fire?"

AWS Budgets? Data lags 8–24 hours. By the time the alert lands, the agent loop has been burning tokens all night. A LiteLLM reverse proxy? Extra latency on every token, a shiny new single point of failure, and a pager that's now lowkey my personality.

So I did what any reasonable engineer does when backed into a corner: I built my own solution.

This is the engineering story of bedrock-cost-sentry: an event-driven, out-of-band IAM budget guard for Amazon Bedrock. Zero proxy, zero added latency, full /model freedom in Claude Code, and per-developer enforcement within 1–3 minutes, for about $1 a month.

1. The Context: What Are We Building?

At its core, bedrock-cost-sentry is a completely serverless, out-of-band control plane that monitors Bedrock consumption, calculates token-level dollar costs on the fly, and cuts off developers the second they cross their spending cap.

Developers do not connect to a custom domain or route their requests through a proxy. They talk directly to AWS Bedrock runtime endpoints in their chosen region using their standard IAM access keys:

export CLAUDE_CODE_USE_BEDROCK=1 export AWS_ACCESS_KEY_ID="AKIA..." export AWS_SECRET_ACCESS_KEY="..." export AWS_REGION="us-east-1"

The Architecture in 5 Steps:

  1. Direct Invocation: The developer's machine talks directly to Amazon Bedrock. Latency penalty = 0ms.
  2. Telemetry Ingestion: Bedrock writes invocation metadata (caller IAM ARN, model ID, input/output/cache token counts) to Amazon CloudWatch Logs.
  3. Event Filtering: CloudWatch subscription filters stream matching log records to an AWS Lambda function (enforcer).
  4. Accounting & Idempotency: The Lambda checks an idempotency ledger table in DynamoDB, calculates the call cost using an internal pricing engine, and atomically increments the user's spending tally.
  5. Enforcement (The Kill Switch): If the user exceeds their configured dollar or token cap, the Lambda attaches an inline IAM Deny policy (BedrockBudgetDeny) directly to the user's IAM entity. Their very next request fails with AccessDenied.

2. The Problem: Why Do We Need This?

Why go through the trouble of building an event-driven architecture when AWS already has billing tools? Because enterprise cloud governance is full of edge cases that official sales decks never mention.

Limitation 1: The "Next-Day Autopsy" of AWS Budgets

AWS Budgets and AWS Cost Explorer rely on billing pipeline data exports. That data lands on a delay anywhere between 8 and 24 hours. If an engineer accidentally kicks off an agentic script that burns $300 an hour at midnight, an AWS Budget alert arriving at 11:00 AM the next morning is not a guardrail—it's a funeral notice.

Limitation 2: The Enterprise Org Payer Account Lockout

AWS natively supports tagging Bedrock spend using Application Inference Profiles. In theory, you could give each developer an inference profile tagged User = alice, and group spend in Cost Explorer. But Cost Allocation Tags can only be activated in the AWS Organization Payer (Management) Account.

In any mid-to-large organization, engineering teams are relegated to member accounts or sandbox accounts. Access to the payer account is guarded by corporate finance and enterprise cloud teams behind a fortress of bureaucracy. If you cannot activate tags at the root payer level, Cost Explorer per-user cost allocation simply does not work.

Now I get why people rage-quit corporate for startups

Limitation 3: Why Proxies (LiteLLM / API Gateway) Weren't the Answer

The alternative was spinning up a proxy container. But proxies introduce heavy trade-offs:

  • Single Point of Failure: If your proxy crashes, the entire engineering department stops coding.
  • Latency & Streaming: Streaming completions (InvokeModelWithResponseStream, ConverseStream) require persistent HTTP chunking or WebSockets. Proxies frequently drop long-running connections.
  • Tooling Friction: You have to force developers and CLI tools to override their base URLs, bypass enterprise certificate authorities, and reconfigure their environments.

I'm not signing up to be the single point of failure for every coding agent in the company. We needed a system that observed reality out-of-band without getting in the developer's way and keep my job safe.

3. The Solution: How Does It Work?

Let's break down the mechanics of the stack into its key engineering components.

Component 1: Metadata-Only Ingestion (Zero Plaintext Prompts)

To count tokens, you must enable Bedrock Model Invocation Logging. But turning on logging often triggers an immediate veto from InfoSec: "Are we logging sensitive company code and user prompts to CloudWatch?!"

The answer is no. Tokens live in the metadata, so we turn off every payload flag. Compliance 101: follow the rules. Can't follow the rules? Don't get caught. Getting caught? I don't know you. (Author not liable. This is satire. Mostly.)

In modules/bedrock-logging/main.tf, we explicitly turn off payload delivery:

resource "aws_bedrock_model_invocation_logging_configuration" "this" { logging_config { # Token counts live in metadata — full prompt/response text is disabled. # Avoids logging proprietary source code in plaintext and cuts # CloudWatch ingest cost by orders of magnitude. embedding_data_delivery_enabled = false image_data_delivery_enabled = false text_data_delivery_enabled = false cloudwatch_config { log_group_name = aws_cloudwatch_log_group.bedrock.name role_arn = aws_iam_role.bedrock_logging.arn } } }

By setting text_data_delivery_enabled = false, Bedrock logs only the invocation metadata: caller identity, model identifier, and token numbers. Not a single line of code or prompt text enters CloudWatch. This also shrinks log payload sizes by ~99%, slashing ingestion and retention bills to pennies.

Component 2: Multi-Region Ingest, Centralized State

Developers don't stick to one region. Claude Sonnet might run in us-east-1, while regional quotas push someone to ap-southeast-1 (Singapore) or us-west-2 (Oregon).

Because Bedrock invocation logging is a regional singleton, each tracked region needs its own logging configuration and a local instance of the enforcer Lambda. However, spinning up separate DynamoDB tables across regions would fracture budgets.

Instead, regional enforcers stream events across AWS's backbone into a single, centralized DynamoDB table in us-east-1. Because DynamoDB rows are partitioned by IAM username (userId), Singapore usage and US usage fan into the exact same budget row:

# Primary region stores the central table; regional Lambdas point explicitly to it dynamodb = boto3.resource("dynamodb", region_name=os.environ.get("DYNAMODB_REGION")) TABLE_NAME = os.environ["DYNAMODB_TABLE"]

Why us-east-1? Honest answer: I picked it because it felt like the safest region. Oldest, biggest, the one AWS surely babysits the hardest. Then I remembered which region keeps showing up in outage post-mortems. Yes, that one. Bold choice, junior me.

It turned out to be the right call anyway, for reasons I only appreciated later:

  • The kill switch lives there. IAM is "global," but its control plane endpoint (iam.amazonaws.com) sits in us-east-1. So does Cost Explorer (ce:GetCostAndUsage). Keeping budget state next to the APIs that act on it means less cross-region plumbing.
  • Most traffic is already there. Claude calls land mostly in us-east-1, so the busiest enforcer writes locally. Singapore pays a ~200ms cross-region write on an async pipeline whose logs already arrive 1–3 minutes late. Nobody will ever notice.
  • Global Tables would lie to us. Multi-region DynamoDB Global Tables resolve conflicts with last-writer-wins. Two regions doing an atomic ADD costUsd on the same row at the same moment can silently drop one increment. A budget counter that undercounts is worse than no counter. One table means atomic stays atomic.

And if us-east-1 has one of its famous bad days? Enforcement pauses, but it fails open. Developers keep shipping; they just get a few unmetered minutes of free will. Which, to be fair, is exactly what they had before I showed up.

Yes, I know. One table is a single point of failure. I also know a fleet of replica tables, just to protect a few dozen budget counters, is the kind of architecture that looks great in a diagram and terrible on the bill I'm supposed to be shrinking. So the table gets cheap insurance instead: deletion protection and point-in-time recovery (PITR), both on. And if PITR ever lets me down, the real source of truth never lived in DynamoDB anyway. Every invocation's token counts sit in CloudWatch Logs (30-day retention), so I can replay them and rebuild every counter from scratch. We've already run a smaller version of this for real: after the Grok double-billing bug in Pain Point 4, we corrected the counters straight from the invocation log. Is it fun? Absolutely not. It's a long afternoon of Logs Insights queries and atomic ADDs. But "painful but possible" is a recovery plan, and it costs $0 a month.

Component 3: The Pricing Engine & Model Precedence

Log events don't report dollar amounts; they report token integers (inputTokenCount, outputTokenCount, cacheReadInputTokenCount, cacheWriteInputTokenCount).

Disclaimer: This enforcer speaks fluent Claude. Other providers have their own dialects of "token." Grok, for example, quietly stuffs cached tokens into inputTokenCount, and our enforcer happily billed them twice, about 3x the real cost (full story in Pain Point 4). New model family? Reconcile against Cost Explorer before you trust it.

The enforcer computes the cost in Python using an internal price table:

PRICE_PER_1K = { # modelId substring: {input, output, cacheWrite, cacheRead} (excerpt)  "claude-fable-5-1": {"input": Decimal("0.010"), "output": Decimal("0.050"), "cacheWrite": Decimal("0.0125"), "cacheRead": Decimal("0.00025")}, "claude-fable-5": {"input": Decimal("0.010"), "output": Decimal("0.050"), "cacheWrite": Decimal("0.0125"), "cacheRead": Decimal("0.001")}, "claude-opus-5-5": {"input": Decimal("0.004"), "output": Decimal("0.020"), "cacheWrite": Decimal("0.005"), "cacheRead": Decimal("0.0002")}, "claude-opus-4-1": {"input": Decimal("0.015"), "output": Decimal("0.075"), "cacheWrite": Decimal("0.01875"), "cacheRead": Decimal("0.0015")}, "claude-opus-4": {"input": Decimal("0.005"), "output": Decimal("0.025"), "cacheWrite": Decimal("0.00625"), "cacheRead": Decimal("0.0005")}, "claude-sonnet-5-5": {"input": Decimal("0.002"), "output": Decimal("0.010"), "cacheWrite": Decimal("0.0025"), "cacheRead": Decimal("0.0001")}, "claude-sonnet-5": {"input": Decimal("0.003"), "output": Decimal("0.015"), "cacheWrite": Decimal("0.00375"), "cacheRead": Decimal("0.0003")}, "claude-sonnet-4": {"input": Decimal("0.003"), "output": Decimal("0.015"), "cacheWrite": Decimal("0.00375"), "cacheRead": Decimal("0.0003")}, "claude-haiku-5-5": {"input": Decimal("0.0001"), "output": Decimal("0.0005"), "cacheWrite": Decimal("0.000125"),"cacheRead": Decimal("0.00001")}, # prompts > 100k tokens use a separate, pricier tier  "claude-haiku-4": {"input": Decimal("0.001"), "output": Decimal("0.005"), "cacheWrite": Decimal("0.00125"), "cacheRead": Decimal("0.0001")}, "grok-4.7": {"input": Decimal("0.002"), "output": Decimal("0.006"), "cacheWrite": Decimal("0.002"), "cacheRead": Decimal("0.0005")}, # ...plus Claude 3.x, Qwen, Nova Canvas (priced per image, so $0 per token) } # Fallback to the Opus 4.1 price so unlisted models never silently undercount DEFAULT_PRICE = PRICE_PER_1K["claude-opus-4-1"]

Important

Substrings and Dictionary Ordering: In Python 3.7+, dictionary iteration preserves insertion order. Notice how "claude-sonnet-5-5" precedes "claude-sonnet-5", "claude-fable-5-1" precedes "claude-fable-5", and "claude-opus-4-1" precedes "claude-opus-4". If you match shorter substrings first, your newer, cheaper models get billed at the legacy rate. Every model generation also needs its own key: "grok-4.6" does not match grok-4.7, and a miss silently bills a Grok call at the Opus rate (~7x over-count).

[!WARNING]
Token fields don't mean the same thing for every model family. For Claude, inputTokenCount excludes cached tokens, so input * price + cacheRead * price is correct. For Grok on bedrock-runtime, inputTokenCount includes the cached tokens, so the same formula bills every cached token twice (once at the full input rate, once at the cache-read rate). Grok needs (input - cacheRead) as billable input. See Pain Point 4 for how we found out.

If an unknown model arrives, it deliberately falls through to DEFAULT_PRICE (the Opus 4.1 rate). In financial engineering, an accidental over-count that triggers an alert is an annoyance; an accidental under-count that lets an unbudgeted model burn thousands of dollars is a disaster.

So yes, our numbers run a little hot. In our monthly audits, the sentry's total lands roughly 5–8% above Cost Explorer. The enforcer prices everything at public list rates, while the real invoice quietly applies discounts and credits it never tells us about. Add the occasional unknown model billed at Opus prices, and the report is always a bit pessimistic. And that's the point. A budget guard that's slightly paranoid blocks a developer one request early. A budget guard that's slightly optimistic sends finance a $4,000 surprise. I'll take paranoid every time. The rule we actually watch is direction, not size: if the sentry ever reads below Cost Explorer, something is being missed, and that's a bug, not a rounding error.

Component 4: The IAM Kill Switch

When a user crosses their budget ceiling, how do you sever their connection without tearing down their access keys?

You inject an inline policy directly onto their IAM user entity:

def attach_deny_policy(user_name): iam.put_user_policy( UserName=user_name, PolicyName="BedrockBudgetDeny", PolicyDocument=DENY_POLICY_DOCUMENT, )

In AWS IAM evaluation logic, an explicit Deny always supersedes any Allow. Even if the user belongs to an AdministratorAccess group or holds broad developer policies, the inline Deny halts further Bedrock calls on their very next API attempt. In IAM, an explicit Deny outranks everything. AdministratorAccess? Cute. Over budget means over budget, admin or not.

4. Implementation: Let's Get Our Hands Dirty

Let's dive into the core implementation details and patterns that make this system bulletproof.

Step 1: Defining Budgets in Terraform

Budgets and users are declared in terraform.tfvars:

users = [ { username = "dev-alice" team = "platform" cost_limit_usd = 250 # Monthly limit in USD input_token_limit = 0 # 0 = unlimited on this axis output_token_limit = 0 tags = { project = "backend-ai" } period = "month" }, { username = "dev-bob-agentic" team = "automation" cost_limit_usd = 20 # Daily sandbox allowance input_token_limit = 0 output_token_limit = 0 tags = { project = "rag-testing" } period = "day" } ] alert_emails = ["devops-alerts@example.com"] alert_threshold = 0.8 # Warn via SNS at 80% of limit

Step 2: The Terraform State vs. Runtime Mutation Trap

Here is a classic infrastructure-as-code conundrum:
You want to declare user budgets in Terraform (aws_dynamodb_table_item). But the enforcer Lambda continuously updates that exact same row at runtime (costUsd, inputTokensUsed, blocked).

If you run terraform apply next week to adjust an alert email, Terraform will see that costUsd = 142.50 differs from the declared costUsd = 0 in your code. By default, Terraform will happily overwrite the row, wiping out live usage counters and unblocking every blocked user!

To fix this, we split seeding from updating in modules/dynamodb/main.tf:

resource "aws_dynamodb_table_item" "user_budgets" { for_each = { for u in var.users : u.username => u } table_name = aws_dynamodb_table.budgets.name hash_key = aws_dynamodb_table.budgets.hash_key item = jsonencode({ userId = { S = each.value.username } costLimitUsd = { N = tostring(each.value.cost_limit_usd) } costUsd = { N = "0" } blocked = { BOOL = false } # ... other counters ... }) # Ignore changes to live operational data! lifecycle { ignore_changes = [item] } } # Replaced whenever a limit changes, so the provisioner re-runs; # updates ONLY limits via UpdateItem resource "terraform_data" "budget_limits" { for_each = { for u in var.users : u.username => u } triggers_replace = { cost_limit_usd = each.value.cost_limit_usd period = each.value.period } depends_on = [aws_dynamodb_table_item.user_budgets] provisioner "local-exec" { command = "python3 ${path.module}/sync_limits.py" environment = { TABLE_NAME = aws_dynamodb_table.budgets.name USER_ID = each.key COST_LIMIT = tostring(each.value.cost_limit_usd) PERIOD = each.value.period # ... } } }

The initial seed sets up the row with ignore_changes = [item]. Whenever you change a budget ceiling in terraform.tfvars, terraform_data fires a focused Python script (sync_limits.py) that executes an UpdateItem exclusively targeting costLimitUsd and resetPeriod. Live spend counters remain untouched.

Step 3: Marker-First Idempotency

CloudWatch log subscriptions deliver events with at-least-once semantics. If AWS retries an enforcer batch, or if Lambda encounters transient network issues, the same log event can be processed multiple times. Without deduplication, retried calls would double-count usage and prematurely block innocent developers.

In lambda-src/enforcer.py, we implement a dedicated idempotency ledger:

def claim_request(request_id): """Marker-first idempotency using DynamoDB condition expressions.""" if not PROCESSED_TABLE or not request_id: return True try: dynamodb.Table(PROCESSED_TABLE).put_item( Item={ "requestId": request_id, "ttl": int(time.time()) + PROCESSED_TTL_SECONDS }, ConditionExpression="attribute_not_exists(requestId)", ) return True except ClientError as e: if e.response["Error"]["Code"] == "ConditionalCheckFailedException": return False # Already counted! Skip it.  raise

Notice the critical design choice: The marker is claimed before incrementing the user's budget.
If the Lambda crashes after claiming the marker but before incrementing the budget, we under-count by a fraction of a cent. But if we incremented first and crashed before marking, a retry would double-count spend and trigger an unfair block. In billing systems, always fail in favor of the developer.

Step 4: The Inspection CLI (report.py)

No need to wade through the AWS Management Console to inspect usage. A standalone Python script scans the central DynamoDB table and generates rollups in seconds:

# Rollup total spend grouped by team python3 report.py --table bedrock-cost-sentry-budgets --region us-east-1 --group-by team

Sample output:

Cost by tag 'team': group cost($) users input output cacheRead cacheWrite platform 184.20 4 1,240,512 412,080 18,940,210 4,120,500 automation 18.50 2 180,400 84,200 1,450,000 210,000 Per-user: user cost($) limit($) % blocked input output cacheRead cacheWrite dev-alice 142.10 250.00 57 False 910,200 290,100 12,400,000 2,900,000 dev-bob 18.50 20.00 93 False 180,400 84,200 1,450,000 210,000 TOTAL 202.70 1,420,912 496,280

5. Pain Points & Hard-Won Lessons

Building systems on top of AWS telemetry sounds trivial until reality hits you in the face. Here are the scars we earned along the way:

1. The "Action: bedrock:InvokeModel*" IAM Trap

When writing the IAM deny policy, almost every engineer instinctively writes:

{ "Effect": "Deny", "Action": ["bedrock:InvokeModel*"], "Resource": "*" }

It looks comprehensive. It passes code review. And it completely fails to block Claude Code.

Claude Code and modern Bedrock clients do not invoke raw models with InvokeModel. They use the Bedrock Converse API (bedrock:Converse and bedrock:ConverseStream). In AWS IAM, InvokeModel* does not match Converse!

Your deny policy must be explicit:

DENY_POLICY_DOCUMENT = json.dumps({ "Version": "2012-10-17", "Statement": [{ "Sid": "DenyBedrockOverBudget", "Effect": "Deny", "Action": [ "bedrock:InvokeModel", "bedrock:InvokeModelWithResponseStream", "bedrock:Converse", "bedrock:ConverseStream", "bedrock-mantle:CreateInference", "bedrock-mantle:GetInference", "bedrock-mantle:CancelInference", "bedrock-mantle:CallWithBearerToken", ], "Source：https://dev.to/myuh250/stopping-the-10000-aws-bedrock-bill-how-we-built-a-zero-proxy-kill-switch-for-1month-278m
My favorites
PreviousOpenAI's own quickstart still calls a retired model. Your tests won't notice either.NextUnderstanding UUIDs? Try This Free Online UUID Parser

Related posts

Humanize Anthropic API Output in Kotlin: A Status-Page Blurb Linter + 5 Free Tools (2026)

Humanize Anthropic API Output in Kotlin: A Status-Page Blurb Linter + 5 Free Tools (2026)

dev.to AI
TokenCap 3.0: One Build, Every Tool Reads It

TokenCap 3.0: One Build, Every Tool Reads It

dev.to AI
Why a Search API Is Often the Wrong Abstraction for AI Agents

Why a Search API Is Often the Wrong Abstraction for AI Agents

dev.to AI
Command 250+ AI Workers via WhatsApp with Claw Agents AI

Command 250+ AI Workers via WhatsApp with Claw Agents AI

dev.to AI
Amazon Is Back in the Takeout Business — This Time, Uber Is Doing the Driving

Amazon Is Back in the Takeout Business — This Time, Uber Is Doing the Driving

dev.to AI
Tutorial: Show the Serving Tier Beside Every AI Draft

Tutorial: Show the Serving Tier Beside Every AI Draft

dev.to AI