Why one boolean is not enough
A common first draft for a local annotation or review queue adds a done boolean to every record. It becomes true when someone looks at the record and false before that. The problem appears at export time: the same field cannot say whether the record has a label. A record can be processed yet unlabelled, or unprocessed yet read downstream as finished. Separate two predicates first. reviewed means the record has been handled, including giving a label or explicitly abstaining. label_present means the record actually carries a label from the taxonomy.
local-label-queue is a small public tool I made; the source and samples are downloadable. It uses three states instead of two booleans: unlabelled, labelled, and abstained. The label field holds a taxonomy string only when status is labelled, and is null in the other two states. The tool has no done or has_label boolean field. An external system that defines done as reviewed makes labelled and abstained both true. One that defines done as label_present makes abstained and unlabelled both false. The three states keep the gap between the two predicates.
What the 12 synthetic records show
The bundled demo-annotations.jsonl contains 12 records: 2 labelled, 1 abstained, and 9 unlabelled. Every non-labelled row has label: null. Reviewed therefore counts 3, label_present counts 2, and the difference is the abstained row. Abstaining on every record gives a reviewed fraction of 1 while labelled stays 0. That shows abstention counts toward workflow progress but creates no label and implies nothing about label correctness. If a single boolean is defined as reviewed, it merges labelled with abstained. If it is defined as has_label, it merges abstained with unlabelled. Splitting the two questions keeps the legitimate state of 100% processed with zero labels from disappearing.
The two lines below fetch the public sample and count it. The result should match the description above.
curl -sL https://huggingface.co/spaces/WideChaos/local-label-queue/raw/d8068775077cd4149a39ec9a2c6cb0ab52045d54/demo-annotations.jsonl -o demo-annotations.jsonl node --input-type=module -e 'const rows=(await import("node:fs")).readFileSync("demo-annotations.jsonl","utf8").trim().split("\n").map(JSON.parse);const c=rows.reduce((a,r)=>(a[r.status]=(a[r.status]||0)+1,a),{});console.log(c, rows.filter(r=>r.status!=="labelled"&&r.label!==null).length)' The output should include { labelled: 2, abstained: 1, unlabelled: 9 }, with zero non-labelled rows whose label is not null. This checks state modeling and export fidelity. It does not check label correctness because the sample is original synthetic material, not human annotation or a gold standard.
This screenshot shows the public Space I deployed, displaying the same 12 synthetic records: 2 labelled, 1 abstained, and 9 unlabelled, with reviewed near 25% including the abstention. That share measures queue progress only; it says nothing about label correctness. The page also has no autosave.
revision and checkpoint serve different purposes
demo-checkpoint.json uses the same inventory and still shows 2/1/9, but its revision field is 7. revision counts changes, including undo operations. It is neither a processed count nor a labelled count. Reading it as completion progress creates false numbers.
The checkpoint is for resuming. The tool keeps decisions only in page memory; it has no autosave, browser storage, or cloud sync. Download the checkpoint explicitly before a refresh. On restore, it verifies the inventory hash and content hash, shows a preview, and replaces the current state explicitly rather than merging. JSONL export keeps every original group and text, not only labelled rows, so review all records before sharing the file with someone else.
SHA-256 is a recomputable consistency check. It detects a changed or truncated file but proves nothing about origin, label truth, or encryption, and it cannot stop someone from repacking a file after verification.
Use and source: Live app · Pinned source
Author: Renchao Wu
