The standard way to feed an LLM context today goes like this: scrape a few pages, truncate them, concatenate the pieces into one big string, and stuff it into the prompt.
That approach has three problems. First, no provenance — the model cites a number, and you have no idea which page, which paragraph, or which crawl it came from. Second, no sense of time — the page changed its price last week, you're still holding the old version, and when the old and new copies disagree, nobody tells you. Third, no incrementality — one digit changes and the whole pipeline reruns; and when the token budget runs out, you chop the tail off, which may well be the table header.
PageSift 1.6, codename "Context", is built at exactly those three problems. The one-sentence version: context shouldn't be a concatenated string; it should be a compiled artifact — with identity, lineage, a budget, and a manifest of sources.
Repo: https://github.com/YOLOW92/pagesift (single Rust binary, Apache-2.0)
First, give knowledge a stable identity
In 1.6, content is broken into KnowledgeUnits. Each unit records four separate things: who I am (unitId), what state I'm in (revisionId and a content hash), where I live (document, byte span, heading path), and where I came from (lineage: created, unchanged, merely relocated, rewritten, split from, merged from).
Take a product page. After a redesign, untouched paragraphs keep their unitId. A paragraph that only got pushed further down the page is marked relocated — it doesn't get to impersonate a content change. A price that actually changed gets a new revisionId. Downstream systems cite "this fact", not "paragraph 7 of Tuesday's crawl". Everything else in 1.6 stands on that.
When a page changes, only redo what the change touches
Above the units sits Delta, a persistent incremental log. When a document is re-crawled, only the derived assets affected by the change are recomputed (the lexical index, the semantic sketch, the structure index and the graph index each carry their own dependency stamps); the rest is reused. In the local benchmark — 60 resources, 720 derived assets, one document edited — incremental maintenance is 4.37× faster than a rebuild, with 98.6% of assets reused. A document that comes back 304 Not Modified reuses 100%.
The delta log has consumer checkpoints: it can be replayed and acknowledged, so external systems consume at their own pace and resume after a crash without losing or duplicating work. And the index reports exactly which store version it currently reflects. No hand-waving.
In search, the two kinds of time must not be mixed
Search runs several strategies together: keywords, exact symbol matching, heading boosts, plus a disableable semantic sketch. It deliberately separates two notions of time: the observation cutoff (what you knew at that moment) and the validity window (the period during which a fact actually held). "What did this cost last month?" and "What does it cost now?" travel along two different timelines — most freshness bugs come from collapsing the two into one.
When two sources disagree on a price, search doesn't crown a winner. It retrieves both sides of the conflict together and counts independent sources honestly (syndicated mirrors don't inflate the count). Scores are reported as six decoupled dimensions — relevance, support, freshness, source independence, coverage, calibrated confidence — so the caller does the weighing.
The output is compiled, not truncated
The final step is called Pack. Give it a question and a token budget, and it compiles a context for you: every segment carries a provenance header (which unit, which document, what status, under which heading), with a full provenance manifest attached.
Some things are MustKeep: a table's header and caption, the negations and qualifiers inside a sentence, both sides of an active conflict. They go in as a group or not at all. If even the MustKeep closure doesn't fit the budget, Pack returns budgetInsufficient: true. If it doesn't fit, it says so — it won't quietly amputate half the evidence and pretend the result is complete.
One layer up sits Action, the entry point meant for agents: answer straight from local context when it's still fresh, with zero network calls (context mode); demand at least two independent sources when verifying (verify mode); and before spending money on a new crawl, estimate the value of information and check whether coverage can be estimated at all. On a small or heavily skewed corpus it returns estimable: false — not a made-up number.
How to use it
Context ships on every surface at once: the CLI (pagesift context), HTTP (/v2/context), an MCP tool (pagesift_context), Python, Node.js, and a C ABI. Installation is the usual:
- Python:
pip install pagesift - Node:
npm install pagesift - Rust:
cargo add pagesift, or grab the single binary from GitHub Releases
This release also bumps the HTML extraction library, yolowhtml, to 0.1.2. The reason is a very typical bug: article mode was treating academic figures and their captions as page furniture and deleting them — the text still said "see Figure 1", but the figure was gone. Figures, captions, and table captions are all preserved now.
Closing
PageSift's through-line hasn't changed: turn expensive, repeated AI information work into engineering capability that accumulates, can be verified, and can be reused. 1.5 turned observations into claims with a history. 1.6 turns claims into compilable context.
What comes next stays on the same line: from context toward conclusions, from a single node toward collaboration, from retrieval toward computation. Details when it ships.

