TL;DR: Fluent AI output is not necessarily correct. In this project, I built a lightweight Python workflow that uses explicit scoring rubrics, structured human annotations, Pydantic validation, evaluator-agreement checks, and an adjudication queue. On the included synthetic example dataset, the pipeline compared 15 shared cases, reported 86.7% exact error-label agreement and 93.3% exact severity agreement, routed three cases for review, generated reports, and passed 11 tests. These figures demonstrate the workflow; they are not a general measure of model performance.
Project:*
AI Data Quality & LLM Response Evaluation Benchmark on GitHub
— -
The illusion of competence
Imagine asking an AI assistant a straightforward question. The response arrives instantly. It is polished, well structured, and delivered with complete confidence.
But it names the wrong city as the capital of Australia.
Or it invents a source, misstates a product specification, or gives an answer that ignores a critical instruction. The problem is not that the response looks unreliable. The problem is that it looks convincing.
Large language models are designed to generate plausible continuations from context. Fluency and confidence in the wording should not be treated as proof that a claim is true. A response can be relevant, readable, and still be factually wrong.
For engineering teams, product owners, and people deploying AI in real workflows, the practical question is therefore:
How do we detect and measure poor AI output before people or downstream systems rely on it?
The answer begins with structured evaluation — not a quick glance at a few outputs or a general impression that a model “feels good.”
What LLM evaluation actually means
LLM evaluation is the systematic assessment of model outputs against defined criteria. Depending on the use case, those criteria may include:
Accuracy: Are the claims factually and logically correct?
Relevance: Does the response address the actual request?
Completeness: Does it include the information needed to satisfy the task?
Instruction following: Does it respect explicit constraints, required content, and requested formats?
Safety and policy compliance: Does the response avoid prohibited or harmful content where those requirements apply?
A useful evaluation process starts with representative test cases and clear expectations. It defines a rubric, records decisions in a consistent format, checks the quality of those records, and investigates meaningful disagreement.
A leaderboard score can be useful context, but it does not automatically tell a team how a model will behave on its own data, domain, instructions, or production workflow.
A small open-source benchmark
To make these ideas concrete, I built the AI Data Quality & LLM Response Evaluation Benchmark: a lightweight Python project demonstrating rubric-based evaluation, annotation quality assurance, structured validation, evaluator agreement analysis, disagreement detection, and adjudication.
The project is deliberately designed as an educational and portfolio-scale implementation. It uses a small set of synthetic evaluation cases and annotations so that the workflow can be inspected and run locally.
It does not call a paid model API, require a GPU, or depend on a hosted evaluation service. The included pipeline is built with Python, pandas, Pydantic, pytest, and JSON/JSONL files.
Repository: github.com/Enterprise-AI-Solutions/AI-Data-Quality-LLM-Evaluation-Benchmark
The four-dimension scoring rubric
The project includes a configurable rubric with four dimensions, scored from 1 to 5:
TABLE 01 — THE FOUR-DIMENSION SCORING RUBRIC
Table 1. THE FOUR-DIMENSION SCORING RUBRIC.
The weights are project defaults, not universal industry standards. A legal-document assistant, a customer-support bot, and a creative-writing assistant may need different priorities. The weights are stored in rubrics/evaluation_rubric.json so they can be adjusted for a particular evaluation context.
IMAGE 01 — The four-dimension scoring rubric*
Figure 1. The project’s default rubric weights. These are configurable project settings, not universal evaluation standards.
The configured pass threshold is 3.5 out of 5. A weighted score is calculated as:
(accuracy × 0.35) + (relevance × 0.25) + (completeness × 0.20) + (instruction_following × 0.20)
Consider an illustrative annotation with Accuracy = 1, Relevance = 5, Completeness = 3, and Instruction Following = 5:
(1 × 0.35) + (5 × 0.25) + (3 × 0.20) + (5 × 0.20) = 3.10
The resulting score is 3.10/5, below this project’s 3.5 pass threshold. This example shows why a fluent and relevant answer can still receive a low quality score when its accuracy is poor. The score is a rubric result, not a calibrated probability that the answer is correct.
The evaluation data also needs quality checks
There is an important principle behind this project: the data used to evaluate AI output must itself be checked.
Human annotators can omit required fields, enter scores outside the permitted range, select inconsistent labels, or provide rationales that do not explain their decisions. If those records enter a scoring process unchecked, the resulting report can look precise while resting on poor-quality input.
The benchmark uses Pydantic schema validation to check annotation records before scoring. The model includes fields for the case ID, evaluator ID, four dimension scores, primary error category, severity, confidence, and rationale. Numeric scores are constrained to the permitted range, and the rationale has a minimum length.
Here is an example of the annotation format used by the project:
{ “case_id”: “LLM-001”, “evaluator_id”: “EVAL-001”, “accuracy”: 1, “relevance”: 5, “completeness”: 3, “instruction_following”: 5, “primary_error”: “factual_error”, “severity”: “major”, “confidence”: 5, “rationale”: “The response directly answers the question but identifies the wrong city as the capital of Australia.” } Schema validation cannot prove that an annotation is substantively correct. It can, however, catch structural problems and enforce agreed data constraints before those records are used in downstream calculations.
A shared error taxonomy
Consistent error labels make evaluation records easier to compare, review, and audit. The initial benchmark taxonomy includes:
TABLE 02 — A SHARED ERROR TAXONOMY
Table 2. A SHARED ERROR TAXONOMY.*
The definitions matter. For example, a factual error and an unsupported claim are related but not identical: a claim may be unsupported by the supplied context without being demonstrably false. Evaluation guidelines should explain how annotators are expected to distinguish categories in the particular task being assessed.
Why two evaluators are useful
One evaluator may classify an answer as a factual error while another may label it as an unsupported claim. They may agree on the type of problem but differ on severity. Or they may assign slightly different scores for completeness.
Disagreement is not automatically evidence that an evaluator has failed. It can reveal ambiguous task instructions, unclear rubric boundaries, or genuinely difficult cases that deserve review.
The benchmark compares two annotation files using shared case IDs. Its current agreement script calculates:
exact agreement on the primary error label;
exact agreement on severity;
the number and rate of cases with a label or severity disagreement;
mean absolute score differences for each rubric dimension.
Important methodological note: the current implementation reports simple exact-match agreement. It does not calculate Cohen’s kappa or another chance-corrected statistic. Exact agreement is easy to interpret, but it should not be described as chance-corrected inter-rater reliability. A larger study may need additional agreement measures selected for the type and distribution of its labels and scores.
What the sample run showed
I ran the pipeline against the repository’s included synthetic data. The agreement output reported 15 shared cases:
TABLE 03 — Evaluator agreement results from the sample run
Table 3. Evaluator agreement results from the sample run.
The mean absolute score differences reported by the script were:
TABLE 04 — Mean absolute score differences by dimension*
Table 4. Mean absolute score differences by dimension.
These results describe only this small set of example annotations. With 15 shared cases, one case represents about 6.7 percentage points, so the percentages are sensitive to individual decisions. They should not be used to claim that a particular model, evaluator team, or production workflow has a specific reliability level.
The value of the run is that it demonstrates the mechanics: compare two sets of annotations, surface differences, and create a review list rather than silently treating every decision as settled.
IMAGE 02 — Evaluator agreement summary
Figure 2. Terminal output showing the agreement summary for 15 shared cases
From disagreement to adjudication
The benchmark creates an adjudication queue when it detects one or more of these conditions:
the primary error labels differ;
the severity labels differ;
a score in any rubric dimension differs by two or more points.
In the sample run, three cases were routed to the queue:
TABLE 05 — Cases routed to the adjudication queue
Table 5. Cases routed to the adjudication queue.
The generated CSV includes both evaluators’ labels, severities, and written rationales, alongside fields for adjudication status, final error label, final severity, and the adjudicator’s rationale. This gives a reviewer the information needed to examine the disagreement and record a decision.
The script creates the queue; it does not automatically decide which evaluator is right. Human review remains part of the workflow.
IMAGE 04 — Human adjudication flow
Figure 4. Disagreement detection leads to a review queue; the final decision remains a documented human judgement
IMAGE 03 — Adjudication queue
Figure 3. Three example cases routed to human review
The workflow and generated artefacts
IMAGE 05 — End-to-end benchmark workflow
Figure 5. High-level view of the benchmark workflow, from input validation through reporting and tests
The project brings the steps together in a linear pipeline:
Validate the evaluation cases.
Validate each evaluator’s annotation file.
Calculate weighted quality scores and pass/fail status.
Analyze evaluator agreement and score differences.
Create a CSV queue for meaningful disagreements.
Generate a human-readable evaluation report.
Run the automated tests.
The pipeline produces files including scored evaluations in JSONL, a quality-summary CSV, an agreement-summary JSON file, a disagreement-queue CSV, and a Markdown evaluation report.
The attached run completed successfully and reported 11 passing tests. That is useful evidence that the included test suite passed in this run; it does not prove the project is free of defects or that the benchmark is production-ready.
IMAGE 06 — Pipeline completion and passing tests
Figure 6. The sample pipeline completed and all 11 tests in the included suite passed
Running the project locally
The repository provides a shell script for running the pipeline on macOS/Linux and compatible VS Code environments. The README also includes a PowerShell script for Windows.
Clone the repository:
git clone https://github.com/Enterprise-AI-Solutions/AI-Data-Quality-LLM-Evaluation-Benchmark.git cd AI-Data-Quality-LLM-Evaluation-Benchmark Create and activate a virtual environment, then install the dependencies:
python -m venv .venv source .venv/bin/activate python -m pip install — upgrade pip pip install -r requirements.txt Run the pipeline:
chmod +x scripts/run_pipeline.sh ./scripts/run_pipeline.sh On Windows PowerShell, use the repository’s scripts/run_pipeline.ps1 script instead. The README also documents how to execute the validation, scoring, agreement, adjudication, reporting, and test steps individually.
Why this matters to AI teams
For an AI engineer or ML practitioner, the project is a reference implementation of an evaluation workflow that is often discussed conceptually but not always demonstrated end to end.
For a team building an AI product, the ideas map to practical quality controls:
a rubric defines what acceptable output means for a task;
annotation QA checks the structure and completeness of evaluation records;
agreement analysis helps identify inconsistent or ambiguous judgements;
adjudication provides a documented escalation path for difficult cases;
reports make quality issues visible for follow-up.
These practices can support model evaluation, human-feedback workflows, and the preparation or review of labelled data. They do not, by themselves, guarantee model safety, factual correctness, or production performance.
Limitations and next steps
This repository is a small, synthetic, portfolio-scale demonstration. It is not a model leaderboard, a comprehensive AI safety benchmark, or a production evaluation service. Its sample size is intentionally limited, and its results should not be interpreted as claims about any commercial or open-source model.
Potential extensions include a larger and more diverse evaluation set, additional annotators, a chance-corrected agreement analysis where appropriate, a Hugging Face dataset, Label Studio integration, local candidate-response generation, RAG factuality checks, citation correctness checks, and multilingual evaluation.
For real deployments, teams should also document dataset provenance, rubric versions, annotator training, reviewer qualifications, disagreement policies, and known limitations. Sensitive or proprietary data should only be used with appropriate authorization and safeguards.
The bottom line
The AI industry has invested heavily in building more capable models. But capability alone does not tell us whether an output is dependable for a particular task.
Trustworthy AI also requires a way to evaluate outputs repeatedly, inspect the quality of evaluation data, understand where reviewers disagree, and resolve difficult cases transparently.
This benchmark is a small demonstration of that workflow. It is a starting point for experimentation — not a claim that evaluation is solved.
Explore the project on GitHub
The complete open-source implementation, including the evaluation pipeline, scoring rubric, annotation validation, evaluator agreement analysis, and adjudication workflow, is available here:
View the AI Data Quality & LLM Evaluation Benchmark on GitHub
Contributions, Questions, feedback, ideas for extensions and suggestions are welcome!
You can open an issue or pull request in the repository, or leave a comment below.
— -
This article was drafted with AI assistance and reviewed and edited by the author.
