logjar packs log files 20-165x smaller (usually about half the size of xz -9), keeps them byte-for-byte lossless, and searches them without decompressing them whole.
- Repo: GitHub - zaydmulani09/logjar: Searchable compressed log archives: 20-165x smaller than raw, grep without decompressing. Rust library + single binary. · GitHub
- Crate: crates.io: Rust Package Registry
- Write-up: I built a log compressor you can grep without decompressing - DEV Community
It splits each line into a template plus typed variables (the idea from CLP, OSDI '21, and LogGrep, EuroSys '23) and stores the variables in columnar zstd segments. To search, it compiles the query (glob or regex) to a Glushkov automaton and runs it symbolically over each template: an integer slot is "one or more of [-0-9]", a dictionary slot is "any of this segment's real values". Templates that can't reach an accepting state are never decoded.
On a 1.5 GiB HDFS log, a single block-id search takes 0.9 s, against 7.9 s for zcat | grep. Plain grep on the uncompressed file from a warm cache is still faster (0.2 s), and the README says so.
let archive = Archive::open("app.ljar")?; let q = Query::regex(r"after \d+ retries")?; archive.search(&q, &[], |m| { println!("{}", String::from_utf8_lossy(&m.line)); true })?; Library + CLI, MIT, MSRV 1.85. I'd welcome feedback on the API, and especially log formats it compresses badly.
1 post - 1 participant
