Short answer: keep the crawl boundary at canonical product records and their current availability, then make freshness explicit in retrieval architecture instead of hoping chunking will repair stale data.
That sounds obvious until a crawler starts treating every marketing page, review, PDF, and out-of-stock variant as searchable inventory. In an e-commerce product search system, a retrieval result can be semantically close and still be operationally wrong. A customer asks for a controller that works on a given console; the index returns a discontinued bundle because its description shares the right words. The miss is not a model problem. It is a boundary problem. When this happens, teams often tune the embedding model first, because that dashboard is close at hand, but the better investigation follows one product revision from source event to chunk, filter, ranker, and final stock lookup; if any step loses the revision or locale, a more accurate embedding only makes the wrong answer easier to retrieve.
Scope first.
How should retrieval architecture set crawl boundaries for e-commerce product search?
Start with an allowlist, not a spider. Define which URL or feed records are authoritative, which fields are customer-visible, and which events invalidate them. A useful boundary is the product identity plus variant identity, price and currency, stock state, category path, compatible platforms, and a timestamp. Editorial guides can enrich a result, but they should not create a product entity by accident.
The crawl should exclude cart URLs, account pages, faceted combinations that have no stable canonical URL, internal search results, and tracking parameters. Respect robots.txt, canonical links, and sitemaps as signals; none of them replaces an explicit product-feed contract. If a feed says a SKU is gone, mark its chunks inactive immediately. If the source only exposes HTML, retain the last known document with a tombstone and a short expiry rather than silently serving it forever.
This is the first operational rule: every chunk needs a parent record and a validity window. Without those two fields, a vector index is an archive pretending to be a catalog.
The failure mode is usually freshness, not recall
RAG research describes a generator backed by retrieved passages, which is useful for product questions because the answer can be grounded in current source text rather than only model parameters. The paper does not define a retail crawl policy, though. That policy is ours to own; the paper is listed in References.
I have been paged for missed jobs and duplicate deliveries, so I put the same suspicion on ingestion pipelines: a green crawl metric can hide a bad catalog. A worker may process the same product twice, miss a deletion event, or finish after a newer update and overwrite it. The index then looks healthy while answers age in place.
Use an immutable source revision and an idempotency key such as store_id:sku:revision. Store the source timestamp, fetch timestamp, and expiry separately. A 304 response means the representation did not change; it does not mean the product is still sellable. A 410 or an explicit deletion event should remove the document from the retrieval set, even if its embedding remains for audit.
Chunking should follow meaning and update behavior. Keep a product's compatibility and safety claims together, but split long manuals by heading so one changed section does not force a full re-embed. Put the SKU, variant, locale, and revision in metadata on every chunk. Never rely on the chunk text alone to carry stock or price; those fields change too often and should be joined from a transactional source at answer time.
Here is a deliberately small ingestion shape. It is not a vendor SDK; it shows where the boundary and freshness checks belong.
package catalog import "time" type Chunk struct { ID string SKU string Revision string Text string SourceTime time.Time ExpiresAt time.Time Active bool } func Accept(c Chunk, now time.Time) bool { if c.ID == "" || c.SKU == "" || c.Revision == "" || c.Text == "" { return false } if !c.Active || !now.Before(c.ExpiresAt) { return false } return !c.SourceTime.After(now.Add(5 * time.Minute)) } The five-minute skew here is a policy example, not a universal SLA. Your mileage may vary: a flash-sale catalog may need seconds, while a long-lived accessory manual can tolerate days. Make the value configuration, emit it with every decision, and test the boundary with a clock you can control.
A crawl boundary needs a retrieval contract
Search quality improves when retrieval has a contract that downstream ranking and generation can enforce. I use four checks before a passage is eligible:
| Check | Why it exists | Failure action |
|---|---|---|
| Identity | Ties text to a SKU and variant | Reject orphan chunks |
| Scope | Confirms locale, category, and channel | Filter before ranking |
| Freshness | Prevents expired availability claims | Exclude or refresh |
| Evidence | Retains source revision and URL | Cite or abstain |
The contract should also state what happens when no chunk qualifies. Returning a confident answer from an expired passage is worse than returning “I can’t verify that item.” Add an abstention path, and log the reason: outside crawl boundary, stale revision, missing compatibility field, or low score after filters.
Filtering before vector similarity matters. A high-similarity chunk for the wrong region can outrank a lower-similarity chunk that is actually purchasable. Apply hard predicates first, then lexical and semantic ranking, then a freshness-aware reranker. Keep the final product lookup separate so price and stock are read at response time.
What should chunking and freshness tests verify before release?
Treat the index like a deployable service. Build a fixture catalog with an active SKU, a changed SKU, a deleted SKU, two locales, and one duplicate delivery. Run the same fixture through a full crawl, a replay, and an out-of-order replay. The expected counts should be stable. A second run must not double the chunks, and a late revision must not replace a newer one.
Then test the user-visible contract. Ask a compatibility question after changing a platform field. Ask for an unavailable item after a tombstone. Ask in a second locale. The answer should either cite the matching current revision or abstain; it should never borrow a stale chunk merely because its embedding score is high.
Observability belongs beside the tests. Track boundary rejects, active chunks by revision, age of the oldest eligible chunk, deletion lag, duplicate rate, and the percentage of answers that abstain. Alert on a rising deletion lag, not only on crawler HTTP errors. A crawler returning 200 for a login page is still a failed ingestion.
Rollback is simpler when source revisions are immutable. Keep the previous active revision, switch an index alias or filter to it, and replay the affected feed. Do not mutate a chunk in place while a rollback is in progress; that makes it impossible to explain which answer came from which source.
The trade-offs: where this architecture is not a fit
The catch is operational overhead. Per-revision metadata, tombstones, and a transactional product join cost storage and engineering time. For a tiny catalog that changes once a month, a scheduled rebuild with a short document set may be more sensible. If your source cannot provide stable product identity or deletion signals, a crawl-only design cannot promise reliable availability answers; use a curated feed or narrow the questions the system accepts.
Deep chunking also has a downside. Splitting every paragraph can improve update granularity while weakening context for compatibility rules that span sections. Keep related claims together and measure answer accuracy on real question pairs. I’m not sure any fixed chunk size generalizes across stores; the evidence is in your change log and evaluation set, not in a magic token count.
Stick with a simpler lexical index when filters, exact SKU matching, and freshness dominate the workload. Add semantic retrieval when customers use descriptions, misspellings, or natural-language constraints that lexical matching consistently misses. The decision should follow failure data, not a preference for a particular database.
The practical sequence is boring, which is good: define the allowlist, capture identity and revisions, filter by validity, test replay and deletion, then tune chunk boundaries. That order keeps a retrieval architecture honest when the catalog changes under it.

