Short answer: use a freshness window based on document change rate and incident risk, then route only changed pages through incremental indexing; this keeps retrieval answers current without paying to re-embed a quiet corpus.

For a property-management team, the concrete case is a watcher that diffs a leasing or maintenance web page and alerts an engineer. The same page often has an internal runbook explaining what the alert means. If retrieval serves yesterday's runbook after a policy change, the alert is technically correct but operationally dangerous. I have been paged for missed jobs and duplicate deliveries, so I treat freshness as a delivery guarantee with an audit trail, not as a slider hidden in a vector database.

The first decision is not a vendor. It is a service-level statement: “a changed source appears in search within N minutes.” Pick N separately for pages that can trigger an on-call action and pages that are background reference. A seven-day window for an archived glossary may be fine; a 15-minute window for an emergency access procedure may be mandatory.

That distinction matters.

How should a retrieval architecture set freshness windows for internal engineering documentation?

Start with a source inventory. Record the canonical URL, owner, expected update interval, last successful fetch, content hash, and the maximum acceptable staleness. A scheduler can then choose the next fetch from policy instead of polling every page at the same rate. Use conditional HTTP requests (ETag and Last-Modified) so an unchanged page costs a metadata round trip rather than a new embedding job; the HTTP caching model is documented in RFC 9110.

Separate three clocks: fetch freshness, index freshness, and answer freshness. A crawler may fetch a new page immediately, while a queue waits for an embedding worker, and a response cache may still hold an older answer. Put all three timestamps in the retrieval trace, along with the scheduler attempt, queue age, revision hash, and cache key. During a postmortem, compare them in order: if fetch time is current but index time is old, inspect worker capacity; if both are current but answer time is old, invalidate or version the response cache; if fetch time is old, the source policy or validator is the bottleneck. For the property-management watcher, this prevents an engineer from widening every crawl window when the real issue is a saturated embedding queue. The trace should make that diagnosis possible from one alert, without reconstructing state from scattered logs.

The useful unit is a revision, not a URL. Hash normalized main content, keep the source revision ID in every chunk, and delete or tombstone chunks from superseded revisions. This prevents a changed heading from leaving an old chunk with a deceptively high score. It also makes rollback a pointer change: restore the previous revision in the manifest, then reprocess only the affected documents.

Where does index cost grow, and what should be measured?

Embedding cost is driven by changed tokens, chunk overlap, and fan-out into replicas. A simple daily estimate is changed_tokens / tokens_per_chunk * embedding_price, but that is only a planning approximation; queue retries, compaction, and storage replicas add work. Track those counters by source owner so a noisy generated page cannot quietly consume the budget for every team.

I once assumed that shortening the polling interval would improve reliability. It did not. It created duplicate deliveries when two scheduler workers observed the same change. Give each revision an idempotency key such as source_id:content_hash, and make the index writer upsert on that key. The alert path should likewise deduplicate on source_id:revision_id, with a visible reason when an event is suppressed.

Use a small control table to make the trade-off explicit:

Source class Example Freshness target Cost control
Action-critical Emergency access runbook 15 minutes Conditional fetch, priority queue
Frequently edited Leasing policy page 1 hour Hash gate before embedding
Reference Historical glossary 7 days Batch crawl, long cache TTL

Measure p50 and p95 source-to-index delay, stale-hit rate, duplicate-job rate, and tokens embedded per changed page. A green average can hide a bad tail: one 12-hour delay on an emergency procedure matters more than thousands of fast glossary updates.

A safe incremental indexing loop in Go

The loop below keeps scheduling, fetching, and indexing separate. The conditional request details belong in the HTTP client; this coordinator only advances the revision after the index confirms an upsert.

package freshness import ( "context" "crypto/sha256" "encoding/hex" ) type Revision struct { SourceID string Body []byte Hash string } type Store interface { LatestHash(ctx context.Context, sourceID string) (string, error) Upsert(ctx context.Context, revision Revision) error MarkSeen(ctx context.Context, sourceID, hash string) error } func Process(ctx context.Context, store Store, revision Revision) error { sum := sha256.Sum256(revision.Body) revision.Hash = hex.EncodeToString(sum[:]) previous, err := store.LatestHash(ctx, revision.SourceID) if err != nil { return err } if previous == revision.Hash { return nil } if err := store.Upsert(ctx, revision); err != nil { return err } return store.MarkSeen(ctx, revision.SourceID, revision.Hash) } 

A failed upsert must leave the source marked as pending, so the next run retries the same hash. Never mark a revision current before the index confirms it. Add a deadline to each fetch and a bounded queue; unbounded retries turn a temporary outage into a cost incident.

Verification, rollback, and the catch

Verification should replay a fixed set of queries against a pinned corpus. Include a page changed in the last window, a page just outside it, and a deleted page. Assert both relevance and revision identity. For the property-management watcher, one test asks for the current maintenance escalation path and checks that the returned chunk carries the latest content hash, not merely similar wording.

The catch is that short windows are not suitable when the source cannot provide stable validators, has strict rate limits, or changes mostly cosmetic markup. In those cases, stick with a longer crawl interval plus a manual reindex trigger for high-risk pages. Conversely, a long window is a poor fit for runbooks tied to active incidents; route those documents to a priority queue and require an owner acknowledgement when indexing misses its target.

I'm not sure a single global freshness number is useful for any organization. Your mileage will vary with edit patterns and queue capacity. The defensible choice is the one you can explain from observed delay, stale-hit, and index-cost metrics, then exercise in a rollback drill before an incident.

References