RAG Embeddings Admission — 3 Token Gates Before Batch Document Indexing

Provider portability makes a cheap RAG estimate harder: a replacement backend needs its own token counts and a separately identified embedding index, so carrying yesterday's estimate across the change can hide the work you are about to authorize. TL;DR: for a B2B SaaS code reviewer, admit document indexing, retrieval, and answer generation against explicit token budgets, then promote a compatible index generation only after it can return useful evidence for structured findings. A Node.js application can own that workflow even when the budget admission component is written in Go.

The accounting boundary is the point.

Consider a hypothetical production change: the review service answers questions about repository documentation, retrieves relevant passages through semantic search, and returns structured findings on a proposed code change. A deployment selects a replacement provider while a documentation update is waiting to be indexed. There is no observed incident or measured saving in this example; the postmortem question is what would let us reconstruct the decision if reviews subsequently stopped producing usable evidence.

What page fired? A dashboard showing successful requests would leave that question unanswered. The useful signal is whether the service can produce an accepted review for the intended repository revision, with evidence from the intended document generation, inside its admitted work budget.

How should a RAG token estimate bound embeddings and batch indexing?

Treat the estimate as an admission record for a particular operation. For this design, the record binds the tenant, repository revision, index generation, backend configuration, counter version, and budget policy to the proposed work. Those fields are application choices, not a promise that a generic API supplies them. Without that binding, an estimate can describe one set of documents while the worker processes another, and the resulting ledger can look internally consistent while answering the wrong operational question.

Count the actual text that will cross each boundary. For embeddings, that means the final chunks after parsing and overlap have been applied. For a generated review, it means the assembled request: review instructions, the code change, the user's question, retrieved passages, and whatever structured-output instructions the selected interface requires. Don't silently treat a character count as an exact token count. Make the tokenizer and its configuration an explicit dependency, and treat an unavailable exact counter as an unresolved estimate rather than manufacturing precision.

Here is a deliberately small arithmetic example. These are chosen planning inputs, not benchmark results: 8 documents, 3 chunks per document, and 200 tokens per chunk under the selected embedding counter. A full indexing pass admits 4,800 embedding tokens. If 2 documents change and chunking remains the same, the changed content accounts for 1,200 tokens before any retries or additional work.

The batch is a scheduling unit.

A batch of 6 such chunks also contains 1,200 tokens; splitting the full pass into four batches does not, by itself, change the example's 4,800-token total. Use batching to bound pending work and recovery scope. Any throughput benefit, request limit, or billing difference needs evidence for the selected backend and configuration.

The generation row uses a separate counter from the embedding rows. Its component counts are also chosen inputs; in an implementation, count the complete serialized request under the applicable rules instead of assuming separately counted pieces add up exactly. The 400-token output ceiling is a policy for this example, so it still needs a check that a useful structured finding fits.

Cost follows this ledger. Keep embedding input, generation input, generation output, and any separately billed retrieval or storage work distinct, then apply the applicable billing terms to each category. A token estimate alone is not a complete bill. This separation also makes provider changes reviewable without making a volatile unit price the architecture's central assumption.

Reconstruct the change before blaming semantic search

The hypothetical failure has several plausible branches: the new request could use an old budget estimate, retrieval could address the wrong document generation, or the output allowance could be too small for the expected findings. A good postmortem should be able to distinguish them from records, rather than infer the answer from a smooth latency graph. Preserve the admitted input identity and configuration, the selected index generation, attempt outcomes, validation results, and usage when the backend supplies it.

Count attempts separately from accepted results.

For this service, define an accepted result as a finding object that passes local validation and refers to evidence in the authorized repository revision. A request that returns content but fails that contract must stay visible as unsuccessful review work. Decide whether a sustained absence of accepted results should page the on-call engineer; put ordinary fluctuations in indexing volume on a different signal, because they answer a different question.

The design invariant is that retrieval and estimation refer to explicitly compatible configurations. Give the embedding model and document-processing configuration a generation identity; create a candidate generation when that identity changes, and route a query only to a generation whose query embedding configuration is declared compatible. This is a conservative application contract. Do not infer compatibility merely because vectors have the same length.

For deployment, retain the previous generation while checking the candidate against a fixed set of repository questions and expected evidence. Include the revision in those checks. A plausible passage from an obsolete document can be worse than an empty retrieval result if it leads the reviewer to report a defect that the code change already resolves.

There is a trade-off here: separate generations require storage and promotion logic, while an in-place replacement reduces temporary duplication but makes a clean rollback harder to reason about. The right choice depends on whether reviews may pause during reindexing and whether the previous generation remains usable. Record that decision before the worker starts.

Put 3 admission gates ahead of the indexing worker

The first gate fixes document identity and processing configuration. The second verifies a token count against the selected embedding configuration. The third checks the resulting work against a reservation before dispatch. Keep provider-specific tokenization behind an adapter, but keep the meaning of the admission record under application control.

The following Go component illustrates the second gate and the token portion of the third. It checks one batch against a caller-supplied allowance. It neither chunks documents nor implements a tokenizer; the injected counter must match the actual embedding request. Document identity checks and durable reservation belong around this function.

package admission import ( "context" "errors" ) type Counter interface { Count(context.Context, string) (int64, error) } type Plan struct { Generation string ChunkCount int InputTokens int64 } func Prepare( ctx context.Context, generation string, chunks []string, allowance int64, counter Counter, ) (Plan, error) { if generation == "" || len(chunks) == 0 { return Plan{}, errors.New("generation and chunks are required") } if allowance <= 0 || counter == nil { return Plan{}, errors.New("positive allowance and counter are required") } var total int64 for _, chunk := range chunks { if err := ctx.Err(); err != nil { return Plan{}, err } n, err := counter.Count(ctx, chunk) if err != nil { return Plan{}, err } if n < 0 { return Plan{}, errors.New("counter returned a negative token count") } if n > allowance-total { return Plan{}, errors.New("batch exceeds token allowance") } total += n } return Plan{ Generation: generation, ChunkCount: len(chunks), InputTokens: total, }, nil } 

The subtraction check keeps the running total within the positive allowance before adding the next count. That matters even in illustrative code: a budget check should not introduce an overflow path while trying to bound work. Errors return no admitted plan, leaving the caller responsible for a visible rejection or a smaller batch.

This isn't a shared quota implementation. Two workers can each pass a local check against the same allowance, so the production dispatch path needs an atomic reservation tied to an operation identity before either worker sends work. Give a retry its own attempt record, and reconcile the reservation when the outcome is known. If a timeout leaves the remote outcome unknown, retain that uncertainty explicitly; returning the entire allowance immediately would permit another attempt against capacity that might already have been consumed.

In a Node.js service, this contract can sit behind a worker boundary or be implemented in the application's own runtime. An additional service creates deployment and diagnostic work, so introducing Go solely to perform this arithmetic would be a poor trade-off. The useful boundary is the admission record, not the programming language.

Structured findings need a separate acceptance path. The function-calling guide listed in References describes schema-based tool interfaces; use that as an interface reference, then define the code review service's own validation rules for file paths, evidence identifiers, and repository revisions. A structurally valid object still needs those application checks before it becomes an accepted finding. Never let a model-produced tenant or repository identifier select the authorization scope.

When is a smaller design enough?

A full rebuild can be reasonable for a small documentation corpus when its admitted work fits the maintenance window and a review pause is acceptable. Changed-document indexing becomes useful when avoiding repeated work justifies tracking document hashes, deletions, and generation membership. Neither decision follows from the word “batch.”

Likewise, a single backend can be the better operational choice when there is no concrete migration requirement. A generic adapter does not establish equivalent retrieval quality or identical token accounting; each replacement still needs its own counter, request mapping, and evidence checks. Provider portability earns its complexity when there is a defined change to rehearse.

To reduce LLM work in this scenario, start with the evidence admitted to each review: remove repeated passages, bound the retrieved context, and check that the remaining evidence still supports the expected findings. For indexing, avoid resubmitting unchanged content only when its processing and embedding configuration are also unchanged. These are proposed controls, not claimed savings, and their acceptance criterion is a useful review at a known work boundary.

Keep the deployment decision tied to that criterion. Promote a candidate document generation when representative questions retrieve the intended evidence and the resulting findings pass the service's contract within its admitted budget. If they do not, retain the failure record and investigate the boundary that rejected the work.

What page fired should have an answer.

Sources

References:

• Function calling guide, including tool schemas and structured interface behavior: https://platform.openai.com/docs/guides/function-calling
• Go language specification, for the integer arithmetic and control flow used in the admission example: https://go.dev/ref/spec