Teams evaluating document extraction usually start by asking which APIs are the most accurate available today. Every vendor answers with a published accuracy number, and every one of those numbers comes from a dataset the vendor picked. Your documents have your scan quality, your vendors' invoice layouts, your handwritten fields, and your edge cases, so the accuracy figure that predicts how a tool behaves in your pipeline is the one you measure on your own files.

This guide walks through the benchmark harness we would build to compare document extraction accuracy across vendors. It covers the test set, the answer key, typed normalization, field and table scoring, paired significance testing, consistency, uncertainty signals, and audit traceability. The code is plain Python with no vendor dependency, so you can point it at the output of any tool, including LandingAI's Agentic Document Extraction (ADE).

What you are actually measuring

A useful extraction benchmark measures four separate things, and folding them into one accuracy number hides the failures that matter most. Keep them apart from the start:

  • Field accuracy: whether each extracted value matches the answer key after normalization
  • Table fidelity: whether every cell lands in the right row and column, with nothing added or dropped
  • Consistency: whether the same document produces the same output across repeated runs
  • Traceability: whether each value points back to the place in the document it came from

The first three tell you whether the output is correct. The fourth tells you whether a reviewer or an auditor can verify it, which is a different property and deserves its own column in your results.

Step 1: Build a test set from production documents

The test set should come from the documents your pipeline will actually receive, sampled across the variation you see in production. Pull files directly from your intake, then sort them by the dimensions that change extraction difficulty:

  • Document type: invoices, statements, forms, contracts, and anything else in scope
  • Source quality: born digital PDFs, clean scans, faxes, and phone photos
  • Layout: single column, multi column, dense tables, and pages that mix print with handwriting
  • Length: single page files alongside long documents where tables cross page breaks

Record each document's stratum and how much of your production volume that stratum represents, since the report uses those shares later. Make sure the hard cases are well represented, because a benchmark built mostly on clean files will rank every tool close together and tell you little.

Step 2: Write the answer key, including the empty fields

The answer key is a JSON file per document that records the correct value for every field in your schema, and it has to record absence as carefully as presence. When a field genuinely has no value on the page, write null. That single choice is what lets your harness catch a tool that invents values.

{ "doc_id": "inv_0042", "invoice_number": "INV-20931", "invoice_date": "2026-03-14", "total_amount": "18450.00", "po_number": null, "vendor_name": "Acme Industrial Supply" } 

Next to the answer key, declare a type for every field. The type decides how the harness compares values in Step 3:

FIELD_KINDS = { "invoice_number": "id", "invoice_date": "date", "total_amount": "amount", "po_number": "id", "vendor_name": "text", } 

The answer key has its own error rate. Have two people label the same subset independently, compare their keys field by field, and resolve every disagreement before scoring any tool. Fields where labelers disagree often are usually ambiguous in the schema, and tightening their definitions improves the whole benchmark.

Freeze the schema before you run any tool, and use the same schema for every tool you compare. If you need a starting point, Build Schema drafts an extraction schema from sample documents or a written description of the fields, and it keeps a version history, which helps when the schema changes partway through an evaluation.

Step 3: Normalize values by field type

Normalization removes formatting differences that carry no meaning, so the harness compares values instead of strings. One function that guesses the type of every value causes more errors than it removes: it turns PO number 0042 into the number 42, and it reads a European amount like 1.234,56 as 1.23456. Declaring the type per field avoids both.

import re from datetime import datetime from decimal import Decimal, InvalidOperation def parse_amount(text, decimal_comma=False): negative = text.startswith("(") and text.endswith(")") cleaned = re.sub(r"[^\d,.\-]", "", text) if decimal_comma: cleaned = cleaned.replace(".", "").replace(",", ".") else: cleaned = cleaned.replace(",", "") try: amount = Decimal(cleaned) except InvalidOperation: return text return -amount if negative else amount def normalize(value, kind="text", date_formats=("%Y-%m-%d",), decimal_comma=False): if value is None: return None text = " ".join(str(value).split()) if text == "": return None if kind == "text": return text.casefold() if kind == "id": return text.replace(" ", "").upper() if kind == "amount": return parse_amount(text, decimal_comma) if kind == "date": for fmt in date_formats: try: return datetime.strptime(text, fmt).date().isoformat() except ValueError: continue return text raise ValueError(f"unknown field kind: {kind}") 

A few choices in this code are deliberate. Amounts use Decimal, so $18,450.00, USD 18450, and 18450 compare as equal without floating point surprises, and accounting negatives in parentheses keep their sign. IDs keep leading zeros and ignore spacing.

The id branch uppercases values, so drop .upper() if case carries meaning in your identifiers. Set decimal_comma per document based on its locale. For dates, list only the formats your documents use, since 03/04/2026 parses as March 4 under %m/%d/%Y and April 3 under %d/%m/%Y, and that choice belongs in the answer key rather than in a parser's guess.

Step 4: Score fields with an error taxonomy

Each field comparison should land in one of four outcomes, since "wrong" covers failures with very different consequences. The taxonomy separates a missed value from an invented one:

from collections import Counter def score_field(expected, predicted, kind="text", **opts): e = normalize(expected, kind, **opts) p = normalize(predicted, kind, **opts) if e is None and p is None: return "correct" if e is None: return "invented" # value returned for a field the page leaves empty if p is None: return "missing" return "correct" if e == p else "wrong" def score_document(answer_key, output, field_kinds): return {field: score_field(answer_key.get(field), output.get(field), kind) for field, kind in field_kinds.items()} def summarize(scored_docs): per_field = {} for doc in scored_docs: for field, outcome in doc.items(): per_field.setdefault(field, Counter())[outcome] += 1 return per_field 

Watch the invented count closely. A missing value usually fails loudly downstream, while an invented value that looks plausible passes validation and lands in your system of record.

Report every per field accuracy with a confidence interval, because the sample size behind each field varies. The Wilson score interval handles small samples well:

from math import sqrt def wilson_interval(successes, n, z=1.96): if n == 0: return (0.0, 1.0) p = successes / n denom = 1 + z**2 / n center = (p + z**2 / (2 * n)) / denom margin = z * sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom return (center - margin, center + margin) 

With 38 correct out of 40, the 95 percent interval runs from roughly 83.5 to 98.6 percent. With 380 out of 400, it narrows to roughly 92.4 to 96.7 percent. The interval tells you how precisely you have measured one tool on one field, and it answers a different question from whether one tool beats another.

Step 5: Compare tools with a paired test

Every tool runs on the same documents, so the right comparison looks at the cases where the tools disagree. McNemar's exact test counts the fields one tool got right and the other got wrong, in each direction, and checks whether that split is larger than chance would produce:

from math import comb def mcnemar_exact(a_correct, b_correct): """Paired outcomes per (document, field): True when that tool was correct.""" only_a = sum(1 for a, b in zip(a_correct, b_correct) if a and not b) only_b = sum(1 for a, b in zip(a_correct, b_correct) if b and not a) n = only_a + only_b if n == 0: return only_a, only_b, 1.0 k = min(only_a, only_b) p = 2 * sum(comb(n, i) for i in range(k + 1)) / 2**n return only_a, only_b, min(1.0, p) 

If tool A is right on 10 fields where tool B is wrong, and B is right on 2 where A is wrong, the two sided p value is about 0.039. Comparing two overlapping confidence intervals by eye is a weaker test, and it can miss a real difference between tools evaluated on the same documents.

Step 6: Score tables at the cell level

Tables need their own score, because a table can contain every correct value and still be wrong if the values sit in the wrong cells. Merged cells come first: expand every row span and column span into a full grid, repeating the merged value in each position it covers, for both the answer key and the tool's output.

def expand_spans(cells): """cells: dicts with row, col, text, and optional rowspan and colspan""" if not cells: return [] n_rows = max(c["row"] + c.get("rowspan", 1) for c in cells) n_cols = max(c["col"] + c.get("colspan", 1) for c in cells) grid = [[None] * n_cols for _ in range(n_rows)] for c in cells: for r in range(c["row"], c["row"] + c.get("rowspan", 1)): for k in range(c["col"], c["col"] + c.get("colspan", 1)): grid[r][k] = c["text"] return grid 

Then compare the grids cell by cell and score both directions. Recall catches dropped cells, and precision catches cells the tool added:

def table_cell_scores(expected_rows, predicted_rows, column_kinds=()): def cells(rows): out = {} for i, row in enumerate(rows): for j, value in enumerate(row): kind = column_kinds[j] if j < len(column_kinds) else "text" out[(i, j)] = normalize(value, kind) return out expected, predicted = cells(expected_rows), cells(predicted_rows) matched = sum(1 for pos, v in expected.items() if pos in predicted and predicted[pos] == v) precision = matched / len(predicted) if predicted else float(not expected) recall = matched / len(expected) if expected else 1.0 return precision, recall 

Both scores are strict on purpose. If a tool drops one row of a multi page table, every row after it shifts and recall falls sharply. If a tool appends rows that are absent from the page, precision falls. For a structural metric with a published definition, TEDS, the tree edit distance based similarity introduced in Image based table recognition: data, model, and evaluation, was designed to capture cell misalignment and OCR errors in table recognition.

Step 7: Run every document more than once

Consistency is easy to skip and expensive to discover in production. Run each document three to five times through each tool and measure how many fields return the same normalized value every time:

def consistency(runs, field_kinds): stable = 0 for field, kind in field_kinds.items(): values = [normalize(run.get(field), kind) for run in runs] if all(v == values[0] for v in values): stable += 1 return stable / len(field_kinds) if field_kinds else 1.0 

Comparing normalized values matters here too, since a tool that returns 1,200.00 on one run and 1200 on the next has produced the same value twice. A tool that is right on four runs out of five will look strong in a single pass benchmark and still produce a steady trickle of errors at production volume.

Step 8: Check whether the uncertainty signal means anything

If a tool returns a confidence signal, test whether low scores actually line up with errors before you build review routing on top of it. Pair each field's confidence with its outcome from Step 4 and compute the error rate per bucket:

def error_rate_by_confidence(rows, edges=(0.0, 0.5, 0.8, 0.95, 1.01)): """rows: (confidence, outcome) pairs""" report = {} for low, high in zip(edges, edges[1:]): outcomes = [o for c, o in rows if low <= c < high] if outcomes: errors = sum(o != "correct" for o in outcomes) report[f"[{low}, {high})"] = (len(outcomes), errors / len(outcomes)) return report 

A useful signal shows error rates that fall as confidence rises. Treat the scores as a ranking for triage rather than a probability of correctness, which is also how LandingAI describes the confidence scores in its first generation parse output.

Signals arrive at different levels across tools, so align them to fields before bucketing. In ADE's second generation, the DPT-3 Pro parsing model returns no confidence scores, and the DPT-3 Verity model, currently in Preview, returns a confidence value per word. To get a field level score from per word values, find the words that a field's source ranges cover and take the lowest confidence among them.

Step 9: Verify that extracted data is traceable for audits

Verifying that extracted data will hold up in an audit means checking that every value can be traced to the exact place it came from. For each extracted field, record whether the tool returns a source location, such as a page and a bounding box or an offset into the parsed text, and whether that location actually contains the value. Spot check a sample by hand, since a location that points at the wrong region misleads the reviewer.

Score it as pass or fail per field and keep it next to accuracy rather than inside it. A value with a valid location can still be the wrong value, such as a number taken from the row above, so traceability makes errors checkable and leaves accuracy to the other steps.

Setting an accuracy target for regulated document extraction

The accuracy rate a regulated workflow needs comes from the cost of an error on each field, so it differs from field to field. A wrong account number in a KYC file and a wrong page count in a cover letter carry very different consequences, and one blended target hides that difference.

Set the target per field with your compliance team, then hold the benchmark to it with the interval from Step 4:

  • Accuracy: a field meets its target when the lower bound of its Wilson interval sits at or above the target, rather than the point estimate
  • Invented values: set a separate ceiling for the invented rate on the fields that drive decisions, and compare it against the upper bound of that rate's interval
  • Review coverage: decide which fields always go to a person regardless of score, such as identifiers, amounts, and signatures, and measure the time a reviewer needs per field using each tool's source locations

Reporting the results

The report should keep every dimension separate, so the tradeoffs stay visible to whoever makes the decision:

Metric How it is computed What it tells you
Field accuracy Correct outcomes divided by documents containing the field, with a Wilson interval Which fields each tool handles reliably
Paired difference McNemar's exact test on fields where the tools disagree Whether one tool is reliably better on a field
Invented rate Invented outcomes divided by fields the key marks empty How often a tool fills gaps with plausible values
Table precision and recall Cells matched by position after expanding merged cells Whether table structure survives extraction
Consistency Fields returning the same normalized value across runs How much variance to expect at volume
Traceability Fields with a source location that contains the value How quickly reviewers and auditors can verify output

Break each metric down by document type and source quality. When you report a single overall number, weight each stratum by its share of production volume, since the test set deliberately oversamples hard cases and a plain average misstates what production will see.

Questions to ask vendors before you run the benchmark

The harness also gives a CTO a concrete list of questions for vendors evaluated for regulated deployments, because each step depends on something the vendor's output has to support:

  • Absence: what does the API return when a field has no value on the page, and can the schema mark fields as nullable
  • Source locations: does every extracted value come with a page, a bounding box, or an offset into the parsed text, and what happens to values the system computes or reformats
  • Tables: how are merged cells and tables that continue across pages represented in the output
  • Uncertainty: is there a confidence signal, at what level does it apply, and which models or features return it
  • Stability: how are model versions pinned, so a benchmark result stays valid after the vendor ships an update
  • Data handling: which retention and deployment options apply when the benchmark runs on production documents

Where ADE fits in this harness

ADE maps onto several of the steps above. Build Schema covers Step 2, generating the extraction schema from sample documents and keeping a version history, so every tool runs against the same frozen definition. On the parse side, ADE returns tables as HTML by default so merged cells survive, and each table cell block carries its grid position and span counts, which map onto the inputs expand_spans expects in Step 6.

For Step 9, ADE's Extract metadata pairs every returned value with character offsets into the parsed Markdown. When Extract derives a value instead of lifting it verbatim, for example a computed total or a document level summary, those offsets come back as null.

Treat a populated field with null offsets as a value with no verbatim source and send it to review, then check how your own reformatted fields, such as dates converted to a standard format, behave before you count them against the tool.

For page level checks, the Ground API joins the extraction metadata with the parse output and returns the blocks, pages, and bounding boxes behind each field. Extraction can also run with the grounding stage skipped for speed, so keep it enabled for benchmark runs, since skipping it returns null ranges for every value.

For a reference on how to publish a benchmark that others can reproduce, LandingAI's DocVQA benchmark write up is a useful model.

DocVQA is a question answering benchmark rather than a field extraction one, and the post reports 99.16 percent accuracy on its validation split, with answers generated from parsed output and no access to the page images. The part worth copying is the method: the evaluation code is public on GitHub, and the post includes a gallery of all 45 errors with a classification for each, so readers can judge the failures themselves.

If you need a version of this process for a procurement decision rather than an engineering harness, the vendor bake off guide for testing extraction accuracy covers the same ground as a scorecard, with no code required.

FAQ

What are the most accurate document extraction APIs available today?

The answer depends on your documents, since published accuracy figures come from datasets each vendor selected. Run the candidates through a benchmark built from your own production files, score fields, tables, consistency, and traceability separately, and compare tools with a paired test.

How do you compare document intelligence accuracy across leading vendors?

Run every vendor on the same test set with the same frozen schema and answer key, normalize values by field type, and compare tools field by field with McNemar's exact test on the cases where they disagree.

What accuracy rate do you need for regulated document extraction?

Set the target per field based on the cost of an error on that field, then require the lower bound of the field's confidence interval to meet it. Add a separate ceiling for invented values on fields that drive decisions.

How do you verify extracted data is audit compliant?

Check that every extracted value comes with a source location, like page coordinates or a text offset, and that the location actually contains the value. Score this per field alongside accuracy, since a value can be traceable and still wrong.

What questions should a CTO ask when evaluating document AI vendors for regulated industries?

Ask how the API represents missing values, whether every value carries a source location, how merged and multi page tables come back, what confidence signal exists and at what level, how model versions are pinned, and which data retention and deployment options apply.

Why score table precision as well as recall?

Recall alone ignores cells the tool added. A tool that returns every real row plus three rows absent from the page would score perfectly on recall, and precision is what exposes it.