PageSift started as a Rust crawler chasing extreme performance. It has since grown into a web content engine — web scraping, document parsing, structured extraction, and an AI-agent surface — and 1.2 adds one more engine to it.

GitHub: https://github.com/YOLOW92/pagesift

1.2: ITNA, an access-barrier recovery engine

  • When an HTTP fetch gets blocked, it decides intelligently whether to escalate to a browser retry
  • TLS/HTTP2 browser fingerprinting over HTTP, HTTPS, and SOCKS5 proxies
  • A session identity pool: cookies, egress, and fingerprint managed together, isolated per user
  • Safe HTTP → browser handoff of session state
  • Tuned browser automation markers, with waiting for non-interactive JS challenges
  • A shared request budget that caps retries, timeouts, and resource use

ITNA is off by default and enabled on demand. It does not solve interactive CAPTCHAs, and it does not promise to bypass every anti-bot system.

Catching up on 1.1: documents and languages, massively expanded

I never got around to posting about the 1.1 series, so here it is in one go:

  • A long list of new file formats: legacy Office, ODF, WordPerfect, e-books, email, archives, databases, and more
  • 373 source-code languages recognized, with function/class-level structure extracted for mainstream languages
  • New bindings for Go, Java, C#, Ruby, PHP, Lua, Swift, Zig and more, plus LangChain and LlamaIndex integrations

Measured on 1.1:

  • PDF → Markdown: 2.4 ms/page
  • HTML → Markdown: 0.24 ms/page
  • Main-content extraction: 1.0 ms/page
  • Office parsing: 91.7 files/sec, about 3.40× xberg

1.1.1–1.1.3: still digging into the Rust layer

I implemented Zstandard compression/decompression in pure Rust — FSE, Huffman, LZ matching, cache-aware hash chains, all written from scratch.

Where it stands

PageSift has passed 200 stars on GitHub, and it is still developed and maintained by one person.

The goal was never just a fast crawler. I want it to be genuinely high-performance, low-overhead, self-hostable AI data infrastructure.

Fully open source — search YOLOW92 / PageSift on GitHub. Tests, issues, and technical discussion are all welcome.