Your LLM Invoice Has No Stack Trace: Building Cost Observability Into a Marketplace

Here is a pattern that shows up in almost every marketplace that adds AI features. Week one, the feature launches. Week four, usage is up and everyone is happy. Week five, the provider invoice lands at roughly double what anyone projected, and the only explanation available is a single number on a billing dashboard.

That dashboard tells you how much you spent. It does not tell you why: which feature, which tenant, which workflow, or which retry loop burned through the budget. For that you need cost monitoring built into your own stack, the same way you built logging and APM into it.

This post is a practical guide to wiring that up, based on the Journeyhorizon article on LLM cost monitoring for marketplaces.

What "LLM cost monitoring" actually means

It's the continuous measurement, attribution and optimization of spend from model API calls in production. Providers bill per token, with separate input and output prices and a different rate card for each model. So the cost of any call comes down to three things you control: the model, the tokens you send, and the tokens you get back.

Observability vendors already compute this. As the source article describes, Datadog multiplies the token counts on each LLM span by public prices (with an option for negotiated rates), splits input into non-cached, cache-read and cache-write buckets, and treats reasoning tokens as a subset of output. That breakdown matters because cached input is usually billed at a fraction of normal rates, while reasoning tokens can inflate output quietly.

Granularity is the whole point. A retrieval pipeline that loads full product listings can use many times the tokens of a simple search. An agent retrying a failed tool call can burn budget in minutes. A bulk import running on a frontier model where a small one would do wastes money without anyone noticing.

Why marketplace AI costs are hard to forecast

Two sides, many models

Seller tools (listing enrichment), buyer features (search, recommendations) and platform features (moderation, support) often run on different models. The price spread is large: as of October 2026, Claude Sonnet 5.5 and gpt-6.1-sol list at $2 input / $10 output per million tokens, while Claude Haiku 5.5 and gpt-6-luna list at $0.10 / $0.50. That's a 20x difference for the same token volume.

Event-driven traffic

A seller onboarding push can trigger thousands of enrichment calls in one day. A marketing campaign can multiply search volume overnight. Your cost curve follows your growth curve, spikes included.

Agents multiply calls

One user request can fan out into retrieval, routing, tool calls, retries and guardrail checks. The article cites a Gartner prediction from August 2026 that inference cost per agentic workflow will rise more than fivefold through 2028, even as per-token prices fall.

Hidden line items

Embeddings, re-ranking, vector database queries and timeout retries all add up. A feature that looks cheap in staging can be unaffordable under production load. That's why cost visibility belongs in the build phase, not the post-mortem.

The metrics to track

Start with this list and resist adding vanity metrics:

• Tokens per request (mean and p95)
• Cost per marketplace operation: one listing enrichment, one search, one support reply, one moderation check
• Cost per user, tenant and market side for showback
• Output-to-input token ratio
• Cache hit rate
• Share of traffic on premium models
• Budget burn rate and anomalies
• Quality score alongside cost, so a cost cut can't silently degrade results

Cost per operation is the bridge to the business model. If enriching a listing costs more than the commission that listing typically earns in its first month, the feature needs a cheaper model or a different trigger.

Worked example: listing enrichment

Assume about 1,500 input tokens and 300 output tokens per enrichment call at October 2026 list prices.

Same feature, same volume, an order-of-magnitude difference driven purely by model choice and processing mode. You can only make that call if you can see the numbers per operation.

Building the foundation

1. One path for all inference traffic

Route every LLM call through a single gateway or a shared client library. Tag every request consistently from day one:

await llm.call({ model: "claude-haiku-5.5", feature: "listing_enrichment", tenantId, userId, env: process.env.NODE_ENV, input, }); // the wrapper logs input/output/cached tokens and computed cost 

Implementation options include plain middleware logging, OpenTelemetry spans shipped to Grafana or Datadog, or AI gateways and observability tools such as Helicone, Portkey, LiteLLM, Langfuse and Comet Opik.

2. Governance that fires before the invoice

Set daily or monthly quotas per user, feature, tenant or environment. The article suggests a simple alert ladder:

alerts: - name: budget_warning threshold: 80% # notify - name: budget_critical threshold: 100% # critical alert, optional throttling - name: spend_anomaly condition: hourly_spend > N x normal_hourly_rate 

Add per-user rate limits on expensive features. Decide whether a cap hard-blocks requests or posts to a channel like Slack for a human decision. Either is fine. A report that arrives after the invoice is not.

3. A continuous optimization loop

Find the largest cost drivers, reduce them without hurting quality, repeat.

Optimization levers, in rough order of payoff

• Model routing. Send classification, tagging and short summaries to small models. Keep frontier models for complex reasoning. Gartner recommends multimodel tiering, and mid-tier options like Gemini 3.8 Flash ($0.75 / $3.75 per million) sit in between.
• Prompt caching. Keep stable content such as system instructions, taxonomies and policy text at the start of the prompt. Cached reads on Sonnet 5.5 are $0.10 per million tokens versus $2 for regular input.
• Semantic caching. Reuse responses for near-identical queries like popular searches or common support questions, and track the hit rate.
• Batch processing. Move bulk imports, re-categorization and nightly enrichment to batch APIs for 50% off.
• Lean prompts and structured outputs. Trim instructions, limit history and use schemas to cut both tokens and retries.
• Agent loop limits. Cap tool calls and retries per request, and log every time the cap is hit.
• Self-hosting at stable volume. Open-weight models can undercut API rates at high, predictable usage, once you account for GPUs, engineering and maintenance.

If you're using off-the-shelf components, such as an AI search box for Sharetribe, tag their calls separately so you can measure cost per session and per converted buyer.

A 30-day rollout plan

1. Week 1: Wrap every LLM call in a shared client that logs tokens, model, cost, feature and user or tenant ID.
2. Week 1: Reconcile your logged totals against the provider invoice to validate your pricing table.
3. Week 2: Build a dashboard with total spend, spend by feature and model, and cost per operation.
4. Week 2: Configure budget warnings, hard caps and an anomaly alert.
5. Week 3: Rank features by cost and pick the top three to five drivers.
6. Weeks 3–4: Test one change per driver (cheaper model, shorter prompt, caching or batching), measuring cost and quality together.
7. Week 4: Ship what holds quality, revert the rest, and schedule a monthly review.

A note on expectations

Monitoring by itself saves nothing. What it does is make savings visible and safe. The article notes that moving a high-volume task to a small model can cut its cost by more than 90% at current list prices, and batch APIs halve eligible jobs, but you only find those opportunities once every call is attributed.

Review alerts continuously, dashboards weekly during launches or fast growth, and at least monthly once things are stable or whenever a provider changes pricing.

Takeaway

Treat AI cost like latency: a first-class metric, instrumented from the first sprint and reviewed in technical design. If you'd like help setting that up, Journeyhorizon runs AI cost optimization engagements that cover instrumentation, benchmarking and reducing spend for marketplace teams.