Your Node.js AI App Needs a Budget, Not Just an API Key

Connecting an application to an LLM is relatively straightforward.

You send a prompt, receive a response, and integrate the result into your application. A basic AI feature can be built in a few hours.

But what happens when that feature starts getting real traffic?

One user asks a simple question. Another uploads a 100-page document. Someone else maintains a conversation containing 50 previous messages. Meanwhile, background jobs are generating summaries, processing files, and calling AI models.

Your application might be working perfectly from a functional perspective, but your API consumption keeps increasing.

And the problem isn't always the model.

It is often the way your application manages AI requests.

As AI features become part of production applications, managing token consumption, request frequency, and model costs becomes an architectural responsibility.

For applications using multiple models and providers, a centralized approach can make these controls easier to maintain. This is one reason AI gateway architecture becomes increasingly useful as an application grows.

In this article, let's look at how to introduce practical AI cost controls in a Node.js application without building unnecessarily complicated infrastructure.

1. Understand What Actually Drives AI Costs

Before implementing cost controls, understand where the consumption comes from.

Most LLM providers calculate usage based on input and output tokens, although pricing can also vary by model, caching, and supported modalities.

Input tokens include:

• System instructions
• User prompts
• Previous conversation messages
• Retrieved documents and knowledge
• Tool definitions and structured inputs

Output tokens include generated content and, depending on the model, other generated tokens that may not appear in the visible response.

Consider a customer support chatbot.

A user sends:

"How can I reset my password?"

The actual request might contain much more than that sentence:

System instructions + Customer support guidelines + Previous conversation history + Retrieved knowledge base documents + Current user question 

The application pays for the complete request, not just the latest user message.

This is particularly important in applications that maintain long conversations or process large documents.

The first architectural decision should be to measure the complete AI request rather than looking only at the user's input.

2. Put a Maximum Output Limit on Every Request

Never allow your application to generate unlimited output when a feature has a clearly defined purpose.

A document summary does not need 5,000 generated tokens. A classification operation might need only a few.

Set limits based on what the feature actually requires.

Here's an example using the OpenAI Node.js SDK:

import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY }); export async function generateAnswer(prompt) { const response = await openai.chat.completions.create({ model: process.env.OPENAI_MODEL, messages: [ { role: "system", content: "Provide clear and concise answers." }, { role: "user", content: prompt } ], max_completion_tokens: 500 }); return { content: response.choices[0]?.message?.content ?? "", usage: response.usage }; } 

Here, max_completion_tokens establishes an upper limit on generated tokens.

One important detail: this limit can include reasoning tokens that are not visible in the final response. It is not necessarily a limit on the number of words the user receives.

Also, this example uses the Chat Completions API. Other providers and API endpoints may expose different parameter names.

The objective isn't to make every response short. It is to prevent features from generating significantly more content than their purpose requires.

3. Control Input Size Before Calling the Model

Output limits solve only half the problem.

Your application must also control input consumption.

Consider a document summarization feature that accepts uploaded files.

A user uploads a small PDF. Another uploads a document containing several hundred pages.

Without input controls, your application could attempt to send a huge amount of content to the model.

Start with basic request validation:

function validatePrompt(prompt) { if (typeof prompt !== "string") { throw new Error("Prompt must be a string"); } if (!prompt.trim()) { throw new Error("Prompt cannot be empty"); } const MAX_CHARACTERS = 12000; if (prompt.length > MAX_CHARACTERS) { throw new Error("Input exceeds the allowed size"); } return prompt.trim(); } 

This is an initial payload-size check, not an actual token limit.

Why does that distinction matter?

The number of tokens represented by a string depends on the content and tokenizer. Images, files, and tool definitions introduce additional complexity.

For more predictable control:

1. Use a tokenizer compatible with the selected model for text-only estimates.
2. Use a provider's token-counting endpoint when available.
3. Account for system instructions, conversation history, and retrieved context.
4. Set separate limits for different AI features.
5. Reconcile estimated usage with the actual usage returned by the provider.

A chatbot, translation feature, and document summarizer should not necessarily share the same input limits.

A single global limit rarely works well across different AI workloads.

4. Introduce Distributed Rate Limiting

One user should not be able to generate unlimited AI requests.

A common implementation is request-based rate limiting.

For example:

• Free users: 10 requests per minute.
• Standard users: 30 requests per minute.
• Enterprise users: Custom limits.

These are illustrative values. Actual limits should depend on your application's traffic and infrastructure.

In a distributed Node.js application, Redis is a practical choice because multiple application instances can share the same counters.

However, there is an implementation detail worth considering.

The following approach looks simple:

const count = await redis.incr(key); if (count === 1) { await redis.expire(key, 60); } 

But INCR and EXPIRE are separate operations. If the application fails between them, the counter might remain without an expiry.

A better approach is to execute both operations atomically using a Redis Lua script.

The following example uses the Redis Node.js client v4 interface:

const RATE_LIMIT_SCRIPT = ` local current = tonumber( redis.call('GET', KEYS[1]) or '0' ) local limit = tonumber(ARGV[1]) local window = tonumber(ARGV[2]) if current >= limit then return { 0, current, redis.call('TTL', KEYS[1]) } end current = redis.call('INCR', KEYS[1]) if current == 1 then redis.call('EXPIRE', KEYS[1], window) end return { 1, current, redis.call('TTL', KEYS[1]) } `; async function checkRateLimit(redis, userId) { const key = `ai:requests:${userId}`; const result = await redis.eval(RATE_LIMIT_SCRIPT, { keys: [key], arguments: ["10", "60"] }); const [allowed, count, ttl] = result; return { allowed: Number(allowed) === 1, count: Number(count), retryAfter: Math.max(0, Number(ttl)) }; } 

The Lua script executes the counter update and expiry logic as one atomic operation.

This is a fixed-window rate limiter. It is simple and useful for many applications, although it can allow bursts around window boundaries. For smoother traffic control, consider a sliding-window or token-bucket algorithm.

Also, use the authenticated user or tenant identity when constructing the Redis key. Do not trust an arbitrary user ID supplied by the request.

One more important distinction: request rate limiting controls how frequently users call your application. It does not guarantee control over how much those requests cost.

For that, you need usage-based budgets.

5. Track Actual Token Consumption

Do not rely entirely on your provider's billing dashboard.

Your application should know how much each feature and user consumes.

LLM APIs generally return usage information after a successful request.

For Chat Completions, the response includes fields such as:

• prompt_tokens
• completion_tokens
• total_tokens

For the Responses API, the corresponding fields include:

• input_tokens
• output_tokens
• total_tokens

You should normalize these fields inside your application so that the rest of your system doesn't depend on a particular provider's response format.

For example:

function normalizeUsage(response) { const usage = response.usage ?? {}; return { inputTokens: usage.prompt_tokens ?? usage.input_tokens ?? 0, outputTokens: usage.completion_tokens ?? usage.output_tokens ?? 0, totalTokens: usage.total_tokens ?? 0 }; } 

Now you can maintain a common usage format irrespective of the provider adapter.

A simple usage record could look like this:

{ "userId": "user_123", "feature": "document_summary", "provider": "openai", "model": "configured-model", "inputTokens": 2400, "outputTokens": 450, "totalTokens": 2850, "status": "success" } 

Avoid storing sensitive prompts or personal information unnecessarily. Usage metadata is often sufficient for cost monitoring.

Over time, this information helps answer important questions:

• Which AI feature consumes the most tokens?
• Which customers generate the highest usage?
• Which models are being used most frequently?
• Are costs increasing because of longer prompts or larger responses?
• Is a particular workflow generating unnecessary requests?

You cannot optimize what you cannot measure.

6. Introduce Per-User or Per-Tenant Spending Budgets

Rate limiting controls request frequency.

Usage tracking tells you what happened.

A spending budget controls how much a user or tenant is allowed to consume.

Imagine a SaaS application where every customer receives a monthly AI allowance.

Instead of relying only on request counts, calculate the estimated cost of each operation.

A simplified cost calculation looks like this:

function estimateCost({ inputTokens, outputTokens, inputPricePerMillion, outputPricePerMillion }) { return ( inputTokens * inputPricePerMillion + outputTokens * outputPricePerMillion ) / 1_000_000; } 

The pricing values must come from configuration based on the selected provider and model.

For more accurate estimates, account separately for cached input, image or audio usage, and other billable categories where applicable.

For financial accounting, use integer units such as currency micros or an appropriate decimal type rather than accumulating floating-point currency values.

Now comes an important architectural problem.

Suppose a customer has a monthly budget of $5.

Two requests arrive at almost exactly the same time.

Both requests check the current balance and see that enough budget remains. Both proceed to the model.

The combined cost exceeds the allowance.

This is a concurrency problem.

The solution is to reserve the estimated cost before sending the request.

For example, in MySQL:

UPDATE ai_user_budgets SET reserved_micros = reserved_micros + ? WHERE user_id = ? AND spent_micros + reserved_micros + ? <= monthly_limit_micros; 

The estimated cost is supplied for both placeholders. Check the number of affected rows.

If no row was updated, the reservation could not be made, so reject the request.

This makes the budget check and reservation a single atomic database operation.

After the LLM responds:

1. Calculate the actual cost using the returned usage.
2. Convert the reservation into actual spending.
3. Release any unused reserved amount.
4. Record the final usage against the request.

If the request fails, release the reservation according to your failure-handling policy.

For a production implementation, give each reservation a unique request ID and expiration time. This makes it possible to recover reservations left behind by crashed application instances.

Also account for cases where actual consumption exceeds the estimate. A reservation should be based on a conservative estimate, ideally including the maximum permitted output, and the final reconciliation must have an explicit overrun policy.

The distinction between checking a budget and reserving a budget becomes especially important when multiple application instances process requests simultaneously.

7. Stop Sending Entire Conversation Histories

Conversation history is another source of unnecessary token consumption.

Consider a chatbot that has been running for 50 messages.

Does every new request really need all 50 previous messages?

In many cases, it doesn't.

A more practical architecture combines:

• A rolling summary of older conversation messages.
• A limited window of recent messages.
• The current user message.

For example:

function buildMessages( summary, recentMessages, userMessage ) { return [ { role: "system", content: "You are a helpful support assistant." }, ...(summary ? [{ role: "system", content: `Conversation summary: ${summary}` }] : []), ...recentMessages, { role: "user", content: userMessage } ]; } 

This reduces the amount of repeated context sent to the model.

However, summarization itself has a cost.

It should not run after every message. Choose an appropriate threshold and preserve important information such as customer preferences, decisions, unresolved issues, and previous actions.

The objective is to maintain useful context without repeatedly transmitting unnecessary information.

8. Use the Right Model for the Right Task

Not every AI feature needs the same model.

Consider an application with these features:

Using an expensive model for every operation can increase costs without providing meaningful improvements.

A model-selection layer can help:

function selectModel(task) { const models = { classification: process.env.CLASSIFICATION_MODEL, translation: process.env.TRANSLATION_MODEL, summarization: process.env.SUMMARY_MODEL, reasoning: process.env.REASONING_MODEL }; const model = models[task]; if (!model) { throw new Error(`Unsupported AI task: ${task}`); } return model; } 

The application can choose a model based on the task rather than hardcoding one model throughout the codebase.

Of course, model selection should be validated against output quality, latency, reliability, and actual cost. Cheaper does not automatically mean better.

9. Add Caching Where It Makes Sense

Another opportunity is avoiding repeated calls for identical or safely reusable requests.

For example, your application might receive the same request multiple times:

"Explain what a database index is."

Generating the same answer repeatedly may be unnecessary.

An application-level cache can help with suitable workloads.

async function getCachedResponse( redis, cacheKey, generate ) { const cached = await redis.get(cacheKey); if (cached) { return JSON.parse(cached); } const result = await generate(); await redis.set( cacheKey, JSON.stringify(result), { EX: 3600 } ); return result; } 

This is a simplified example.

A production cache needs a carefully designed key containing the relevant model, prompt version, configuration, and tenant or access scope. It also needs an invalidation strategy.

Do not share cached answers across users or tenants when the underlying context contains private information.

Caching is particularly useful for repeated reference questions, stable knowledge, and deterministic operations.

It is less suitable for requests that depend on changing data, real-time information, or user-specific context.

Track cache hits separately. A cache hit avoids a provider call, so your application-level request count and provider-level token consumption should not be treated as the same metric.

10. Pay Special Attention to AI Agents

Traditional chatbots usually respond to individual requests.

AI agents are different.

An agent might make several model calls, invoke external tools, retrieve information, evaluate its results, and repeat the process before completing a single task.

A user might submit one request, but the application could generate ten or twenty LLM calls behind the scenes.

This makes cost estimation much more complicated.

For agent-based applications, consider enforcing:

• Maximum model calls per task.
• Maximum total tokens per execution.
• Maximum execution time.
• Maximum spending per task.
• Maximum tool invocations.
• Explicit retry limits.

A runaway agent loop should never be able to consume unlimited resources.

These limits should apply to the entire task, not just individual model requests.

This is particularly important when designing AI agent architectures, execution workflows, and guardrails.

11. What Should You Monitor in Production?

A production AI application needs more than an API usage counter.

I would start with these metrics:

A basic dashboard can provide enough information to identify problems early.

As the application grows, introduce alerts for unexpected consumption spikes, unusual request patterns, and approaching customer budgets.

One important operational consideration is deciding what happens when Redis, the usage database, or the provider becomes unavailable.

For example, a system that cannot verify a customer's spending allowance should not silently allow unlimited AI requests. Define fail-open and fail-closed behavior according to the risk of each feature.

Cost controls are not simply monitoring features. They are part of your application's operational reliability.

12. Bring Everything Together

These controls become much easier to manage when they are not scattered throughout your application.

A practical architecture could look like this:

 User Request | v Authentication | v Rate Limiter | v Input Validation | v Context Manager | v Usage Estimator | v Budget Reservation | v AI Gateway | v LLM Provider | v Response Processing | v Actual Usage Capture | v Budget Reconciliation | v Response 

Each component has a specific responsibility.

Application layer: Handles user permissions, subscription plans, and business rules.

Rate limiter: Controls request frequency.

Context manager: Controls conversation history and retrieved information.

Usage estimator: Estimates the potential cost before execution.

Budget manager: Reserves, tracks, and reconciles spending.

AI gateway: Handles provider communication, timeouts, retries, and model selection.

Monitoring layer: Provides visibility into application-wide AI consumption.

If you haven't already considered the gateway layer, our article on what an AI gateway is and how it works explores the architecture, provider abstraction, routing, rate limiting, and cost tracking in more detail.

This separation also makes it easier to introduce additional providers without rewriting your business logic.

Final Thoughts

Building an AI feature is no longer the most difficult part of integrating AI into an application.

The real engineering challenge begins when that feature becomes part of a product used by hundreds or thousands of people.

An application that works perfectly with ten test requests might behave very differently under real production traffic.

You need to think beyond API integration.

Token limits, distributed rate limiting, usage tracking, budget reservations, context management, caching, and model selection should be treated as architectural responsibilities rather than features added after an unexpectedly large API bill.

My recommendation is simple: build these controls into your AI integration layer from the beginning.

It is much easier to design an application with predictable AI consumption than to retrofit spending controls after customers are already using it.

How are you controlling AI costs in your applications? Are you tracking usage at the API level, per user, or per tenant?

I'd be interested to hear how other developers are approaching this in production.

Further Reading

If you're exploring production AI architecture, these articles may also be useful:

• What Is an AI Gateway? How It Works and Why It Matters
• AI Agents Explained: How They Work, With Real Examples
• AI Governance Consulting: What It Is and When You Need It

You can also read my previous DEV.to article: Stop Calling Your LLM Directly: Build a Small AI Gateway in Node.js.

Explore more AI and software architecture insights at VoidSyntax.