Here is a CI check I see in a lot of repos that use AI coding agents with a spec-driven workflow: count the - [x] lines in tasks.md, fail if any are unticked. It's cheap, it's green most of the time, and it answers the wrong question.
It tells you the agent finished its list. It doesn't tell you whether that list adds up to the spec you wrote.
I rebuilt that check in .NET 10 as a proper gate that reads spec.md, plan.md, tasks.md and the team constitution, and fails the build on real gaps. Below are the six kinds of gaps it catches that a checkbox count never will, with the skeleton of how each check works.
Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. Join Tech Skill Builder. The complete, tested gate from this post is in there.
New to the constitution → specify → plan → tasks flow? My Spec Driven AI Development guide walks through GitHub Spec Kit step by step.
Gap 1: A requirement that nobody was asked to build
Picture a password-reset feature. The spec says:
FR-003: System MUST expire reset links after 30 minutes. The agent wrote tasks for sending the email, the reset form and the token table. All ticked. None of them mention expiry. A checkbox check is happy, because it only looks at tasks that exist.
The fix is to check from the spec's side: every buildable requirement needs at least one task that cites its id.
IEnumerable<Finding> CheckCoverage(SpecDocument spec, IReadOnlyList<TaskItem> tasks) { foreach (var req in spec.Requirements.Where(r => r.NeedsBuildableWork)) { if (tasks.Any(t => t.Cites(req.Id))) continue; // functional requirement with no task => CRITICAL finding // ... severity rules, location and suggested fix in the complete project } } I make tasks cite ids explicitly, like (FR-003) at the end of the line. Keyword matching can help as a hint, but I'd never let it pass a requirement on its own.
Gap 2: A question that was never answered
Spec Kit lets you leave [NEEDS CLARIFICATION: ...] markers in a requirement. That's healthy during drafting. It's a problem when the feature is "done" and the marker is still there, because it means the agent quietly picked an answer and nobody knows which one.
This check is almost embarrassingly simple: an open marker on a requirement is a HIGH finding. Simple checks that run every time beat clever ones that run when someone remembers.
Gap 3: A word no test can fail
"Reset emails MUST be sent quickly." Quickly is not a requirement, it's a mood. An agent will implement it, pick a number in its head, and never tell you.
The rule I use is blunt on purpose: a vague word (fast, quickly, secure, scalable, intuitive...) with no number anywhere in the requirement is a finding. Add "(p95 < 2 s)" and it passes.
if (ContainsNumber(req.Text)) yield break; // measurable, fine var hits = VagueWords.Where(words.Contains); // whole-word match only // HIGH for words like "fast"/"secure", MEDIUM for "simple"/"easy" // ... word lists and severity mapping in the complete project Gap 4: A team rule that lives only in a prompt
Most constitutions have lines like "Tests MUST be written before implementation for every user story." If that rule is only in the agent's prompt, it gets followed most of the time.
The gate parses the constitution's upper-case MUST / MUST NOT sentences and recognises a few shapes it can actually check: test-first per story (compare the position of the first test task with the first implementation task), tasks must cite ids, forbidden libraries in plan.md. Anything it can't check gets listed as "not checked", so nobody mistakes silence for a pass.
Gap 5: References that point at nothing
Agents are good at producing plausible ids. A task that says (FR-014) when the spec stops at FR-010, or two tasks marked [P] (parallel) that both edit the same file, will both sail through a checkbox count. They're cheap to detect once you've parsed the three files into records, and they're exactly the kind of thing a human reviewer skims past.
Gap 6: Findings that change name every time you edit
This one isn't a gap in the spec, it's a gap in most gates. If a finding's id includes a line number, adding one sentence at the top of spec.md renames every finding below it, and every waiver and PR comment that referred to them breaks.
So ids come from what the finding is about, never where it is:
public static string StableId(FindingCategory category, string key) { // key examples: "requirement|FR-003", "vague|FR-005", "parallel|T014|T015" // hash category + key, keep a short prefix + 6 hex chars, e.g. COV-xxxxxx // ... full implementation in the complete project } Where does the LLM fit?
It advises, it doesn't decide. The gate computes pass/fail from deterministic findings first. Only after that does a model (through Microsoft.Extensions.AI) look at the requirements the code didn't flag and ask "correctly by what rule?". Its answers are stored as advisory notes, and if it's down, the build result is exactly the same.
On my sample feature (my own test data, not real projects), the "all tasks ticked" check passed with 23 of 23 done. The spec gate on the same files failed with 11 findings, 5 of them critical, and caught all 10 defects I'd planted. It returned the same ids in 100 of 100 runs and took under a millisecond and a half per analysis.
A quick checklist for your own pipeline
- [ ] Does CI check coverage from the spec's side, not just ticked boxes?
- [ ] Do open clarification markers fail the build?
- [ ] Are vague words without numbers flagged?
- [ ] Are constitution MUST rules enforced in code, with unchecked ones listed?
- [ ] Do task references point at ids that actually exist?
- [ ] Are finding ids stable across edits, so waivers survive?
- [ ] If the model is unavailable, is the gate's decision unchanged?
If you answered "no" to two or more, your green build is mostly measuring how diligent the agent was at ticking boxes.
Want the complete working version? The full .NET 10 Spec Traceability Gate (Clean Architecture, ASP.NET Core API plus CLI, 177 passing tests, all offline, plus an in-depth PDF guide) is in Tech Skill Builder, $5/month for a very limited time only, normally $10. Grab it now before it goes back to $10.
And if you want the workflow this gate checks, from constitution to implementation, start with the Spec Driven AI Development guide.
When your agent's task list is all ticked but a requirement has no task, does your CI fail or just warn? I'd like to hear how other teams handle it.