⚡ Why Another Search Tool?

I love ripgrep and grep — they are legendary for one-off manual terminal lookups. But while designing local AI coding agents and automated developer loops, I hit a fundamental bottleneck: traditional search tools have zero memory.

When an autonomous coding agent or CI test runner performs dozens of searches across the exact same 100,000-file codebase within a few minutes, re-traversing the directory tree and re-scanning gigabytes of files from disk on every single turn wastes seconds of user latency.

To solve this, I built poler-harvest in Rust.

🎯 How It Works

poler-harvest combines high-throughput multi-threaded search with content-addressed L1–L3 persistent caching:

  1. Cold Run (Zero Cache):

    • Multi-threaded traversal powered by rayon, aho-corasick, and regex.
    • Automatically segments matching anchor maps using FastCDC (Fast Content-Defined Chunking) with BLAKE3 deduplication.
    • Compiles anchors into an ultra-compact binary crystal file (.t5a / .tar.poler).
  2. Warm Run (Instant Lookups):

    • Returns exact matches in 2 milliseconds — streaming results directly through memory-mapped L1–L3 cache.
    • Up to 300×–400× faster than cold searches on large codebases.
  3. Honest Invalidation (7/7 Regression Tests):

    • The cache never serves stale data.
    • Automatically validates files by size, mtime, and content hash, selectively re-scanning only modified, added, or deleted files in sub-milliseconds.

📊 Benchmarks (Debian 12, 100,000 Files / 12 MB)

Tool Cold Search Warm Search (Repeated) Speedup
GNU grep 3.8 (grep -r) 407–451 ms 407–451 ms 1×
ripgrep 13.0 (rg) 728–760 ms 728–760 ms 1×
poler-harvest 633–760 ms 2 ms 364× faster 🚀

(Measurements taken with hyperfine on Linux 6.x, x86_64, NVMe).

🚀 Quick Install (Linux x86_64)

One-line installer:

curl -sSL https://raw.githubusercontent.com/poler-engine-org/poler-harvest/main/scripts/install.sh | bash 

Arch Linux / CachyOS (Pacman):

sudo pacman -U https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest-0.96.1-1-x86_64.pkg.tar.zst 

Debian / Ubuntu / Mint (.deb):

curl -LO https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler-harvest_0.96.1_amd64.deb sudo dpkg -i poler-harvest_0.96.1_amd64.deb 

Python Wheels:

pip install https://github.com/poler-engine-org/poler-harvest/releases/download/v0.96.1/poler_harvest-0.96.1-py3-none-manylinux_2_34_x86_64.whl 

🛠 Usage Examples

# Exact search with 2ms warm lookups: poler-harvest . "TODO: fix this bug" address # Contextual markdown search for documentation: poler-harvest src/ "network timeout" markdown -c 2 # Regex search with machine-readable JSONL stream: poler-harvest src/ "struct.*Error" -r -f json # One-off scan without touching persistent cache: poler-harvest /var/log "kernel error" --no-cache 

💎 Source-Available & Open Tooling

The project is published under the POLER Custom Source-Available License v1.0 — completely free for personal use, security audits, academic research, and community developer tooling:

⭐ GitHub Repository: https://github.com/poler-engine-org/poler-harvest

I would love to hear feedback, architectural suggestions, or questions about the FastCDC chunking pipeline from the community!