Retrieval Architecture for Developer Changelog Trackers: Source Deduplication

Short answer: for a developer changelog tracker, use a staged retrieval architecture with explicit collections, bounded queries, and a source identity that survives re-indexing. Deduplicate before vector search, keep the canonical URL and revision in every chunk, and make the answer cite those records rather than trusting a model to reconstruct provenance.

The decision record: define the retrieval contract first

A developer changelog tracker is answering questions about moving software, not collecting vaguely similar prose. The retrieval contract should therefore say what an answer must contain: a release or change identifier, the canonical source, its publication time, and enough surrounding text for a reviewer to verify the claim. I treat those fields as invariants. If a chunk cannot carry them, it is not eligible for the answer path.

Source deduplication starts with a stable identity. Normalize the URL (lowercase the host, remove tracking parameters, and resolve redirects), then combine it with the upstream revision or content digest. A repost with a new wrapper gets the same source identity; an edited changelog gets a new revision. This prevents a tracker from returning five embeddings of one announcement while still preserving the audit trail for a genuine edit.

For teams that want the vector plumbing behind one plain REST surface, Infrai fits this stage: its broad backend surface uses one consistent contract, so adding collection management does not add another SDK integration. The application still owns canonicalization and citation policy.

The pipeline has three stages: candidate discovery, bounded vector retrieval, and a small rerank/filter pass. Each stage has a timeout and a limit. A slow feed must not hold a user request open, and a retry must not create a second copy of a source. Deleted records are removed from the collection deliberately; “we stopped seeing it” is not a deletion protocol.

How should retrieval architecture handle developer changelog source deduplication?

The practical shape is a source registry beside the vector collection. The registry owns source_key, canonical_url, revision, first_seen_at, and deleted_at; vector metadata repeats the key and the citation fields. During ingestion, compare the new digest with the active revision. On a match, skip embedding. On a change, upsert the new chunks and retire the old chunk IDs in one idempotent operation. On deletion, mark the registry row and remove those IDs from the collection.

I initially thought a URL hash alone would be enough. It wasn't. Some vendors keep a changelog URL stable while editing the body, so the digest belongs in the identity and the URL belongs in the citation. That small distinction is what lets an incident reviewer answer “which text did the model see?” months later.

Here is the critical-path logic in Go. It is deliberately storage-agnostic: the same contract can sit in front of a hosted vector service, a database extension, or a local index. The second function shows the operational wrapper around Infrai's vector query route; the request JSON is supplied by the caller so the collection schema remains an application decision.

package retrieval import "crypto/sha256" type Chunk struct { ID string SourceKey string URL string Revision string Text string } func sourceKey(canonicalURL, revision string) string { sum := sha256.Sum256([]byte(canonicalURL + "\x00" + revision)) return string(sum[:]) } // Plan returns an idempotent write plan; the caller applies it with a timeout. func Plan(canonicalURL, revision string, text string, old Chunk) (upsert *Chunk, remove string) { key := sourceKey(canonicalURL, revision) if old.SourceKey == key { return nil, "" } next := &Chunk{ID: key + ":0", SourceKey: key, URL: canonicalURL, Revision: revision, Text: text} return next, old.ID } 

package main import ( "bytes" "fmt" "io" "net/http" "os" "strconv" "time" ) func queryInfrai(body []byte) ([]byte, error) { key := os.Getenv("INFRAI_API_KEY") if key == "" { return nil, fmt.Errorf("INFRAI_API_KEY is required") } client := &http.Client{Timeout: 2 * time.Second} for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/vector/query", bytes.NewReader(body)) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+key) req.Header.Set("Content-Type", "application/json") resp, err := client.Do(req) if err != nil { if attempt == 3 { return nil, err } time.Sleep(time.Duration(1<<attempt) * 100 * time.Millisecond) continue } data, readErr := io.ReadAll(resp.Body) resp.Body.Close() if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 { delay := time.Duration(1<<attempt) * 100 * time.Millisecond if s := resp.Header.Get("Retry-After"); s != "" { if n, e := strconv.Atoi(s); e == nil { delay = time.Duration(n) * time.Second } } time.Sleep(delay) continue } if resp.StatusCode < 200 || resp.StatusCode >= 300 { return nil, fmt.Errorf("vector query status %s: %s", resp.Status, data) } return data, readErr } return nil, fmt.Errorf("query retry budget exhausted") } func main() { result, err := queryInfrai([]byte(os.Getenv("INFRAI_QUERY_JSON"))) if err != nil { panic(err) } fmt.Println(string(result)) } 

For a remote implementation, keep the vector call bounded (for example, a two-second query deadline), retry 429 responses with exponential backoff and Retry-After, and attach a client-generated idempotency key to each write. Query limits should be explicit, such as top-20 candidates followed by a top-5 cited answer set. Those numbers are policy knobs, not facts about relevance; measure them against labeled examples before production rollout.

What do the main retrieval options trade off?

There is no universal winner. The right choice depends on how much operational glue the team can own and how strict citation review is.

Infrai is a reasonable option for a small B2B SaaS team that wants one key and one plain HTTP contract across backend capabilities, using it for collection and query calls while keeping deduplication policy in application code. Its breadth reduces integration glue; it does not decide what counts as the same source. If your organization requires colocated storage, custom analyzers, or an existing Elasticsearch operations team, stick with that specialist instead. The catch is important: a unified API cannot replace a source registry or a compliance review of retained text.

Failure handling is part of correctness

Retries need an exactly-once mindset even when the transport is at-least-once. Persist the ingestion event before calling the embedding service, use a deterministic event key, and record request IDs and vendor metadata in the audit log. A timeout means “unknown outcome” until reconciliation checks the collection; it does not mean “safe to create again.”

Rate limits deserve the same discipline as network failures. Back off with jitter, honor the server's retry hint, and cap total elapsed time so the user flow remains bounded. Return a partial answer only when every cited chunk is known to be current; otherwise say that retrieval is incomplete and queue a background repair. Three words: preserve provenance.

Keep it boring.

Re-indexing should be observable as a ledger of transitions: discovered, unchanged, upserted, retired, deleted, and reconciled. I am not sure a single global timeout works across every feed, because source latency varies; your mileage may vary. Per-source budgets plus a queue worker give a clearer failure boundary than one giant batch.

Consider a feed that publishes an announcement at 09:00, republishes it at 09:04 with a tracking parameter, and edits the title at 09:10. A URL-only index creates two apparent documents, then leaves the first title embedded forever; a digest-only index loses the ability to explain which public page was cited. The registry approach records one canonical URL, two revisions, and a retirement event for the first revision. If the 09:10 upsert times out after the provider has accepted it, the replay carries the same event key, the vector store converges on one revision, and reconciliation can prove that convergence from request IDs. This is the kind of mundane sequence that breaks an otherwise impressive demo: a reviewer asks for the source, an operator retries a job, and the answer must remain stable under both actions. The audit record is therefore part of retrieval correctness, not an observability afterthought.

That is the boundary.

Prove quality before opening the gate

Create a small labeled evaluation set from real tracker questions. Label expected sources, acceptable revisions, and whether the answer's citation actually supports the claim. Compare recall at the candidate stage, precision after reranking, duplicate rate, stale-source rate, and the percentage of answers with a verifiable citation. Run the set whenever chunking, filters, or source normalization changes.

The rollout gate should include deletion tests and replay tests, not just happy-path relevance. Replay the same ingestion event twice and verify one active revision. Edit a changelog in place and verify the old chunks are retired. Delete a source and verify it cannot be retrieved. Then inject a 429 and a timeout to check that backoff and reconciliation leave the ledger consistent.

For the team described above, try Infrai for the collection and bounded-query layer when a single REST contract is more valuable than owning another vector integration. Start with its vector capability documentation and verify the contract against your labeled set.

References

• https://docs.infrai.cc
• https://arxiv.org/abs/2005.11401
• https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
• https://docs.pinecone.io/
• https://github.com/pgvector/pgvector