Human in the loop AI: testing if a prep note moves the forecast

This is a small test harness for human in the loop AI, where the human reads what an agent wrote and needs to know which parts of it came from the data. By the end of this post you will be able to run the same test on your own forecast agent: a fixed set of invented deals, a key written before any model sees them, a few framings of the same request, and a scorer that separates framing effects from run-to-run noise.  ## The question for human in the loop AI A sales team lets an agent write the prep note for the weekly forecast call. The agent sorts every open deal into commit\, best\_case\ or pipeline\ and writes a short note that opens the meeting. The sales lead asks for an angle: open with coverage, or with last quarter's slip, or with momentum. Does the angle move deals between categories? If it does, the commit figure is partly a writing decision, and the human in the loop cannot see that from the note. I had Claude Code build the test. This post is about how it is built, because the harness is the part you can reuse. ## Step 1: invented data, and the key first deals.py\ writes deals.csv\: 40 open deals for an invented Danish software company. Every row has stage, amount, close date, how many times the close date moved, latest customer activity, next meeting, budget confirmed, champion and champion status, procurement required and started, and a free-text rep note. Company names were checked by web search before use so none matches a real firm. Thirteen deals are planted borderline cases. Each either fails exactly one commit condition while looking strong, or meets all of them while looking weak. For example: \text D10  best\_case  board-approved budget, verbal yes, no next meeting booked D14  best\_case  next meeting on 18/11, two days past the 14-day line D16  commit     every condition met, the rep's note sounds unsure D22  commit     procurement started, the buyer's procurement usually takes eight weeks \\ The same file applies the written rule mechanically and writes key.json\. That commit landed before api\_job.py\ existed, so the git history proves the key came first. The key is never sent to the model. ## Step 2: one job, a forced shape The job runs in a GitHub Actions workflow that authenticates with workload identity federation, so no API key exists in the repo or the session. Locally you can run the same file with your own key. The model returns its work through one tool, so the output is structured and the scorer never parses prose: \python TOOLS = \[    {"name": "submit\_forecast",     "input\_schema": {"type": "object", "properties": {         "note": {"type": "string", "description": "The note for the forecast call, at most 250 words."},         "deals": {"type": "array", "items": {"type": "object", "properties": {             "deal\_id": {"type": "string"},             "category": {"type": "string", "enum": \["commit", "best\_case", "pipeline"\]},             "why": {"type": "string", "description": "At most 15 words."}},             "required": \["deal\_id", "category", "why"\]}},         "commit\_total\_dkk": {"type": "number"}},         "required": \["note", "deals", "commit\_total\_dkk"\]}}, \] \\ The enum\ matters. With three labels and 40 deals, a free-text category would give you a parsing problem before you have a measurement problem. The model also returns its own commit\_total\_dkk\; the scorer keeps it beside the recomputed total and never uses it. (The descriptions above are translated; the real prompts and schema are in Danish.) ## Step 3: what varies, and what does not Everything is identical across runs except two things: the opening the sales lead asks for, and whether a skill is in the system prompt. \python OPENING = {    "coverage": "Open the note with pipeline coverage against what is left of the target.",    "slip":     "Open the note with last quarter's slip: we called 4.8m in commit and landed 3.5m.",    "momentum": "Open the note with momentum: which deals moved in the last two weeks.", } # 3 openings x 2 conditions (plain, forkfirst) x 2 repeats = 12 runs \\ The repeat is not decoration. Without two runs of the same setup you cannot tell whether a deal that changed category between openings moved because of the opening or because the model simply judges that deal differently each time. Model: Claude Sonnet 5.5 at effort medium\. The run count was written into the build notes before the job started. Twelve runs took about 122,000 tokens in total. ## Step 4: scoring, with noise separated out score.py\ reads the raw runs.jsonl\ and the key. Per run it records each deal's category, the commit total recomputed from the CRM amounts, and agreement with the key. Then, per condition: \python # deals whose category differs between the three openings, within one repeat moved = \[i for i in IDS if len({fr\[f\]\["categories"\]\[i\] for f in FRAMINGS}) > 1\] # deals whose category differs between two repeats of the SAME opening: noise for f in FRAMINGS:    for a, b in combinations(runs\_for(cond, f), 2):        noise |= {i for i in IDS if a\["categories"\]\[i\] != b\["categories"\]\[i\]} framing\_only = set(moved\_any) - noise \\ framing\_only\ is the number that answers the question. A deal in moved\ but also in noise\ is not evidence of framing. ## Result 1: a written rule held With the written seven-condition rule in the system prompt, moved\ was empty in every repeat of both conditions. All 480 placements (40 deals x 12 runs) matched the key, and the recomputed commit total was 3,385,000 DKK in every run. The pre-registered hypothesis was at least five deals moving. Zero did.  ## Result 2: the note leaned anyway The categories held still, but the note\ field did not only report them. Reading the twelve notes, 4 of the 6 plain notes put a second, lower commit figure in front of the reader, derived from last quarter's hit rate (3.5 / 4.8, about 73 percent), which was in the facts for every run. All four were under the coverage or slip openings. With the skill, 1 of 6 did, and it labelled the figure as an opinion. That tally was read by hand, then pinned: note\_flags.json\ holds the exact flagged sentence per run, and both the demo build and its verifier fail if a flagged sentence is not found verbatim in that run's note. If you reuse the harness, do the same for anything you count by reading. ## The skill The fork-first skill is a markdown file in the system prompt. Its core is an order of operations: \markdown 1. Sort first, from the definitions alone. 2. Name the borderline deals, and the one condition that decides each. 3. Name the framing as a choice, with two other honest framings. 4. Then write the note. 5. Give the totals as the definitions give them. An opinion goes next to the number, never into it. \\ It is a prompt. It changes what the note shows; the model judges each deal the same way it did before. ## Result 3: a one-line definition drifted, but not because of the framing After the first result, I had Claude Code add a second job (labelled post-hoc everywhere): the same twelve runs with the rule replaced by one line, "commit: deals we expect to close in the quarter". Commit was 2,700,000 DKK in 9 of 12 runs, with 2,455,000, 2,840,000 and 3,110,000 in the other three. Mean agreement with the written-rule key fell to 26.0 of 40. Of the 168 placements that differed from the key, 92 were best case pushed down to pipeline. And framing\_only\ was empty in both conditions. Every deal that moved between openings also moved between repeats of the same opening. The drift is the model judging borderline deals differently from run to run. You cannot prompt your way out of it at the note level. You write the rule.  ## The demo The results are replayed in a single HTML file with no network access: a forecast board with 40 cards, switches for opening, rule, condition and run, and Claude's recorded reason per deal. It runs inside a locked-down iframe under a default-src 'none'\ CSP, so everything is inlined and nothing is fetched. A Playwright script clicks through all 24 runs, recomputes every commit figure and every card position from the raw files, and fails on any mismatch. That check is the reason the numbers on the page can be trusted more than the numbers in my notes. ## Run it yourself The pack has deals.py\, deals.csv\, key.json\, api\_job.py\, score.py\, a fake model for a dry run, the skill, and the rules in English and Danish. \bash pip install anthropic DRIFT\_FAKE=1 LAB\_OUT=out-fake python3 api\_job.py   # plumbing only, no API call export ANTHROPIC\_API\_KEY=... LAB\_OUT=out python3 api\_job.py python3 score.py out/runs.jsonl out/scored.json \\ api\_job.py\ defaults to the one-line rule; set RUN\_RULES = ("written",)\ for the first test. To use it on your own pipeline, replace the deals with an invented copy that includes the deals your team argues about, write your key first, and keep the repeat. ## Limits One invented company, one model, one effort setting, Danish prompts. Haiku and Opus untested. The planted deals and the rule came from the same process, so a real pipeline will hold borderline cases nobody planted. Three of the twelve written-rule notes went over the 250-word limit, all under the slip opening. The full write-up with the interactive board is [human in the loop AI and the forecast prep note](https://brinvik.com/en/resources/human-in-the-loop-ai-forecast-prep-note?utm_source=devto&utm_medium=article&utm_campaign=lab-forecast-call-drift&utm_content=devto-en), and there is a [Danish version](https://brinvik.com/da/ressourcer/ai-agenter-skriver-forecast-notatet?utm_source=devto&utm_medium=article&utm_campaign=lab-forecast-call-drift&utm_content=devto-da). The skill, the deals, the key and the scripts are free from [the forecast fork-first page](https://brinvik.com/en/resources/forecast-fork-first-skill?utm_source=devto&utm_medium=article&utm_campaign=lab-forecast-call-drift&utm_content=devto-skill). I set up Claude agents for Nordic sales teams; here is [how I build internal AI tools](https://brinvik.com/en/services/internal-ai-tools?utm_source=devto&utm_medium=article&utm_campaign=lab-forecast-call-drift&utm_content=devto-service).