JsonFuse: filter and project a huge NDJSON file in PHP with one call

I just released JsonFuse 1.0.0, a native PHP extension for one narrow job: read NDJSON from a stream, validate each record, filter it, keep some fields, and write NDJSON to another stream. All in one call. 🚀

The loop we all write

while (($line = fgets($in)) !== false) { $row = json_decode($line, true); // ...hand-written checks for field presence and types... if (($row['event'] ?? null) !== 'purchase') continue; fwrite($out, json_encode(['user_id' => $row['user_id'], 'amount' => $row['amount']]) . "\n"); }

Every line becomes a full PHP array, even when you drop it a moment later.

The one-call version

use JsonFuse\Plan; use JsonFuse\Runner; $plan = Plan::compile([ 'fields' => ['user_id' => 'int64', 'amount' => 'float64'], 'where' => ['event' => 'purchase'], ]); $in = fopen('php://memory', 'w+b'); fwrite($in, <<<'NDJSON' {"event":"view","user_id":1,"amount":0} {"event":"purchase","user_id":2,"amount":19.99,"currency":"EUR"} {"event":"purchase","user_id":3,"amount":5} NDJSON); rewind($in); $out = fopen('php://output', 'wb'); $stats = Runner::process($in, $out, $plan); printf("read %d, matched %d, emitted %d, wrote %d bytes\n", $stats->recordsRead, $stats->recordsMatched, $stats->recordsEmitted, $stats->bytesWritten);

{"user_id":2,"amount":19.99} {"user_id":3,"amount":5.0} read 3, matched 2, emitted 2, wrote 56 bytes

Filters: ranges, lists, nested keys

$plan = Plan::compile([ 'fields' => ['order' => 'int64'], 'where' => [ 'event' => 'purchase', 'amount' => ['>=' => 10, '<' => 1000.0], 'day' => ['>=' => '2026-11-01', '<' => '2026-12-01'], 'country' => ['path' => ['address', 'country'], 'in' => ['DE', 'FR', 'NL']], ], ]);

All entries must match (AND). Numbers compare by exact value. Strings compare as bytes, with no collation.

Nullable fields, nested paths and defaults

$plan = Plan::compile(['fields' => [ 'id' => 'int64', 'city' => ['type' => 'string?', 'path' => ['address', 'city'], 'default' => null], 'tier' => ['type' => 'string', 'default' => 'free'], 'discount' => ['type' => 'float64', 'default' => 0], ]]);

{"id":1,"city":"Lyon","tier":"pro","discount":2.5} {"id":2,"city":null,"tier":"free","discount":0.0} {"id":3,"city":null,"tier":"team","discount":0.0}

A missing key writes the default. A JSON null is never replaced by a default. The output stays flat.

Compressed input works (any PHP stream does)

$in = fopen("compress.zlib://$dir/logs.ndjson.gz", 'rb'); $out = fopen('php://stdout', 'wb'); Runner::process($in, $out, $plan);

Errors you can handle

try { Runner::process($in, $out, $plan); } catch (RecordException $e) { printf("record: %s, field %s, line %d, record %d, byte offset %d\n", $e->getReason(), $e->getField(), $e->getInputLine(), $e->getRecordNumber(), $e->getByteOffset()); }

record: type_mismatch, field id, line 3, record 3, byte offset 18 emitted before the error: 2

Every exception has a stable getReason() code. Messages never contain record bytes, so they are safe to log. Records written before the bad one stay in the output. JsonFuse does not roll back, so for all-or-nothing, write to a temp file and rename() after success.

Benchmarks (with the conditions)

Intel Xeon Platinum 8358, x86-64, PHP 8.3.36 NTS, opcache and JIT on, shared host, pre-release build. Full pipeline, byte-identical output checked. Median of 5 to 10 runs.

1 million narrow records, 10% match (isolated re-run):

Implementation	Median	JsonFuse is
JsonFuse	245.2 ms
`fastjson` loop	522.8 ms	2.13x faster
PECL simdjson loop	613.8 ms	2.50x faster
`json_decode` loop	973.5 ms	3.97x faster

vs json_decode: 3.55x to 5.06x at 100k records and above.
vs fastjson (fastest baseline everywhere): 2.08x to 2.63x on narrow records at 100k+. Wide records (100 fields): only 1.53x to 1.71x. Short inputs (10k): 1.49x to 2.15x.
vs JSON Machine, 100k records: 21.5x to 23.8x.
Memory: every implementation sits near 25 to 28 MiB. No memory win is claimed.

Not measured: cold cache, parallel workers, opcache off, non-x86-64. Method and raw data: benchmarks/RESULTS.md.

What it is NOT

Not a general JSON library: no decode, no encode, no iterator, no JSONPath. NDJSON in, flat NDJSON out. For general use, take json_decode or iliaal/fastjson.

What it gives you is a hard memory bound: memory depends on maxRecordBytes (default 1 MiB), not on file size. In my tests, 2 GiB of 1 MiB records peaked at 27.2 to 27.4 MiB RSS, and one 64 MiB line was refused at the cap after reading 0 records.

Quality checks

PHP 8.3 to 8.6, NTS and ZTS. Linux x86-64 and aarch64, macOS arm64. CI on every platform. ASan and UBSan, a libFuzzer harness, and ThreadSanitizer on a multi-thread harness. 🧪