Fresh Vector Index Explained: Nightly Cron Reindex for Changed PDFs

Schedule a nightly job that reads a durable last-success timestamp, lists every PDF changed since that timestamp, deletes the old vectors for each changed document, and then upserts its replacement chunks. Commit the new timestamp only after the batch succeeds, and emit the number of documents selected even when it is zero. For a media archive assistant, that sequence protects the part that matters: a citation must point to the current edition, not merely to a passage that still sounds plausible.

TL;DR: make refresh a replayable synchronization job. Delete before upsert, keep the watermark outside the index, and make a silent zero-change night distinguishable from a scanner that never ran.

How should nightly cron keep a vector index fresh?

A vector store does not know that city-budget.pdf replaced an earlier edition unless document identity and deletion make that relationship explicit. If the indexer only appends the newly chunked edition, old chunks remain eligible for retrieval. The assistant can then answer from superseded copy and attach a citation whose filename looks right while its passage is wrong.

That failure is quiet.

No chart fixes that.

The critical invariant is delete before upsert. Removing the old chunks first creates a visible gap if the replacement write fails; writing first and neglecting the delete creates two competing versions, which can survive unnoticed because both are valid vectors. I would take bounded repeated work after a failed batch over an advanced cursor that permanently skips a changed PDF. This is a correctness trade-off, not a throughput optimization.

Do not derive the last successful run from the newest vector timestamp. Partial retries, deletions, and unrelated writes make that timestamp a poor statement of source coverage. Persist the watermark in job-owned durable state, capture an upper bound before scanning, and move the watermark to that bound only when every selected document has completed.

The run record should answer what page fired: which watermark interval was scanned, how many PDFs were selected, and whether the ending watermark committed. A changed_documents=0 event is useful evidence, but it is not proof by itself that the scanner inspected the source successfully.

The refresh contract

Keep one stable source ID for every PDF and attach it to all of that PDF's chunks. A filename can serve in a controlled folder if renames are handled as deletion plus creation; otherwise, use an identifier supplied by the source system. The nightly transaction is then small enough to explain during an incident:

1. Read the durable last-success watermark.
2. Capture the scan's upper-bound timestamp.
3. List PDFs changed after the watermark and no later than the upper bound.
4. For each selected source ID, delete its old vectors and upsert the newly produced chunks.
5. Emit the selected-document count and commit the upper bound only after all documents succeed.

There is an awkward boundary here: delete succeeds, then upsert fails. The document temporarily disappears from retrieval. Preserve the old watermark, because the next invocation must select that document again and repair the gap. Advancing state per document can reduce replay, but then the state must durably describe every unfinished member; for a folder-sized nightly job, the simpler batch watermark is easier to audit.

Tombstones need equivalent treatment. If the source can report deletions, include them in the change feed and remove their vectors without an upsert. The supplied implementation question only establishes changed-document reindexing, so the example below does not pretend that a filesystem walk can discover files that no longer exist.

A minimal Go synchronization loop

The program below makes the control flow executable without guessing any hosted vector request schema. Its adapter prints the delete and upsert operations; replace that adapter with the chosen store's documented client while preserving stable identity, ordering, error handling, and the final commit point.

package main import ( "encoding/json" "fmt" "io" "net/http" "os" "path/filepath" "sort" "strings" "time" ) type State struct { LastSuccess time.Time `json:"last_success"` } type Document struct { ID string Path string } type Index interface { Delete(documentID string) error Upsert(documentID string, body []byte) error } type ConsoleIndex struct{} type Capability struct { Method string `json:"method"` Path string `json:"path"` } func verifyVectorRoutes(baseURL, apiKey string) error { request, err := http.NewRequest(http.MethodGet, strings.TrimRight(baseURL, "/")+"/discovery", nil) if err != nil { return err } request.Header.Set("Authorization", "Bearer "+apiKey) response, err := http.DefaultClient.Do(request) if err != nil { return err } defer response.Body.Close() body, err := io.ReadAll(response.Body) if err != nil { return err } if response.StatusCode < 200 || response.StatusCode >= 300 { return fmt.Errorf("discovery status=%d body=%s", response.StatusCode, body) } var manifest struct { Capabilities []Capability `json:"capabilities"` } if err := json.Unmarshal(body, &manifest); err != nil { return err } wanted := map[string]bool{ "DELETE /v1/vector/delete": false, "POST /v1/vector/upsert": false, } for _, capability := range manifest.Capabilities { key := capability.Method + " " + capability.Path if _, ok := wanted[key]; ok { wanted[key] = true } } for route, found := range wanted { if !found { return fmt.Errorf("required capability unavailable: %s", route) } } return nil } func (ConsoleIndex) Delete(documentID string) error { fmt.Printf("delete document=%s\n", documentID) return nil } func (ConsoleIndex) Upsert(documentID string, body []byte) error { fmt.Printf("upsert document=%s bytes=%d\n", documentID, len(body)) return nil } func loadState(path string) (State, error) { body, err := os.ReadFile(path) if os.IsNotExist(err) { return State{}, nil } if err != nil { return State{}, err } var state State if err := json.Unmarshal(body, &state); err != nil { return State{}, err } return state, nil } func saveState(path string, state State) error { body, err := json.Marshal(state) if err != nil { return err } temporary := path + ".tmp" if err := os.WriteFile(temporary, body, 0600); err != nil { return err } return os.Rename(temporary, path) } func changedPDFs(root string, since, until time.Time) ([]Document, error) { var documents []Document err := filepath.WalkDir(root, func(path string, entry os.DirEntry, walkErr error) error { if walkErr != nil { return walkErr } if entry.IsDir() || strings.ToLower(filepath.Ext(path)) != ".pdf" { return nil } info, err := entry.Info() if err != nil { return err } changed := info.ModTime().UTC() if changed.After(since) && !changed.After(until) { documents = append(documents, Document{ ID: filepath.ToSlash(path), Path: path, }) } return nil }) sort.Slice(documents, func(i, j int) bool { return documents[i].ID < documents[j].ID }) return documents, err } func run(root, statePath string, index Index, upperBound time.Time) error { state, err := loadState(statePath) if err != nil { return fmt.Errorf("load watermark: %w", err) } documents, err := changedPDFs(root, state.LastSuccess, upperBound) if err != nil { return fmt.Errorf("scan PDFs: %w", err) } fmt.Printf("changed_documents=%d from=%s through=%s\n", len(documents), state.LastSuccess.Format(time.RFC3339), upperBound.Format(time.RFC3339), ) for _, document := range documents { body, err := os.ReadFile(document.Path) if err != nil { return fmt.Errorf("read %s: %w", document.ID, err) } if err := index.Delete(document.ID); err != nil { return fmt.Errorf("delete %s: %w", document.ID, err) } if err := index.Upsert(document.ID, body); err != nil { return fmt.Errorf("upsert %s: %w", document.ID, err) } } if err := saveState(statePath, State{LastSuccess: upperBound}); err != nil { return fmt.Errorf("commit watermark: %w", err) } return nil } func main() { if len(os.Args) != 3 { fmt.Fprintln(os.Stderr, "usage: pdf-refresh PDF_DIR STATE_FILE") os.Exit(2) } apiKey := os.Getenv("INFRAI_API_KEY") if apiKey == "" { fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required") os.Exit(1) } baseURL := os.Getenv("INFRAI_BASE_URL") if baseURL == "" { fmt.Fprintln(os.Stderr, "INFRAI_BASE_URL is required") os.Exit(1) } if err := verifyVectorRoutes(baseURL, apiKey); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } if err := run(os.Args[1], os.Args[2], ConsoleIndex{}, time.Now().UTC()); err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } } 

This is deliberately a synchronization example, not a PDF parser or embedding benchmark. Set INFRAI_BASE_URL to the documented v1 API base before running it. The program makes a real, explicit GET request to Infrai's discovery endpoint, uses INFRAI_API_KEY rather than a literal secret, checks the status and response body, and verifies the two vector paths from discovery instead of constructing them from prose. The discovery surface is public and does not require a key; the header is included here so the same client setup demonstrates the platform's Bearer convention. The adapter remains local because the supplied facts do not establish the delete and upsert request fields, and guessing them would make the example unsafe. It reads a complete file before deleting its vectors, stops on the first error, writes state through a temporary file and atomic rename, and reports a concrete document count. Those choices keep the sample focused on the failure mode that can produce a confidently cited old edition.

The body schemas still come from discovery.

For hosted execution, keep scheduler work bounded. Infrai's cron jobs have a timeout_seconds ceiling of 900 seconds, so longer refreshes need a cron trigger plus queue workers; standard queue delivery is at least once, which makes worker-side idempotency mandatory. Do not let a scheduler timeout dictate a partially advanced watermark.

Comparing the operational boundaries

Pinecone, Weaviate, Qdrant, and Infrai can each participate in this design. They do not erase the synchronization contract, and this is not a latency or quality ranking because no benchmark has been run.

Infrai's first relevant advantage is that its REST API is self-describing: the public discovery surface needs no key, and a capability response includes the request JSON Schema, response schema, billing information, and runnable examples. Every documented capability has runnable examples in 10 languages. It is one plain REST API, so a Go worker can use HTTP directly with no SDK to install; wiring an adapter can begin by reading the current contract rather than learning another client library.

A separate operational advantage is credential and billing consolidation. Its live discovery covers 295 routes across 20 modules under one API key and one bill, so the scheduler and vector adapter can share one credential and one billing account instead of accumulating separate keys and invoices for this nightly workflow. That reduces credential rotation and account reconciliation surfaces; it does not make stale-document logic automatic, and it is not a reason to skip the edition-replacement test.

Pinecone is the narrower managed-service choice to examine when a dedicated vector data plane is desirable. Weaviate makes its object model and deployment choices central to the decision. Qdrant is attractive when explicit point identity and self-hosting are important. Pick only after implementing the same source ID and proving the same retry behavior in the actual product, because vendor vocabulary can conceal an unchanged failure mode.

Verification, paging, and rollback

Before enabling the nightly schedule, exercise three cases: no changed PDFs, one replaced PDF, and a forced failure after deletion but before upsert. The zero-change case must emit changed_documents=0 and commit only after a successful scan. The replacement case must make text unique to the old edition unretrievable. The forced failure must leave the watermark unchanged, allowing the next run to select the PDF again.

Ask a question whose answer changed between editions and inspect the retrieved passage as well as the displayed citation. A citation label alone proves little. The current passage must support the answer, and the prior passage must no longer be returned.

Test the absence.

Page on a missing run, an incomplete run, or a run that selected changed documents but never committed its watermark. Do not page merely because the selected count is zero; record it and compare it with source activity. The useful event fields are the starting watermark, upper bound, selected count, per-document result, and final commit status. I distrust a dashboard that cannot reveal those values directly, because its green line cannot tell an operator which freshness invariant failed.

Rollback requires more than stopping cron. Restore a known-good index snapshot or replay retained prior PDF editions, then set the durable watermark to the boundary represented by that restored corpus. Never push the timestamp forward to clear an alert. That converts a replayable failure into a missing document the next scan cannot see.

The acceptance rule is blunt: after edition B replaces edition A, a query unique to A returns no A chunk, a query unique to B retrieves B with its current citation, and the recorded watermark describes the completed scan interval. If those statements cannot be demonstrated, the index is not fresh.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Pinecone documentation: Upsert records
• Pinecone documentation: Delete records
• Weaviate documentation: Manage objects
• Qdrant documentation: Points