When I hand a task to an AI, the bill I get back is in tokens. But the question I actually have is: how long did one finished piece of work take, including the retries?
So I built a small tool to measure it, and published it.
sunnydachs / cost-per-task
Measure what a task actually costs to complete with an LLM: attempts, tokens, wall-clock. Deterministic verifiers, stdlib only.
cost-per-task
Measure what a task actually costs to complete with an LLM — attempts, tokens and wall-clock — instead of the price per request.
Token price tells you the price of one request. It does not tell you the price of one finished piece of work: the failed attempts, the retries, and the time you spent waiting for the machine to answer. This tool measures that, with a deterministic verifier so the PASS/FAIL decision never depends on a human.
Why
Most teams already log tokens per call. What nobody logs is the denominator: how many successful tasks those tokens produced. That ratio ("cost per successful task") is the number that decides whether a cheap model with retries beats an expensive model that gets it right first try.
The point of this repository is the procedure, not a verdict about any model:
- pick tasks whose output…
No observability tool prices a finished task
First, the gap. I read the official docs of 14 LLM observability products and of the OpenTelemetry GenAI conventions (checked 2026-11-05).
- None of them ships "cost per successful task" as a first-class metric. The ingredients are there (per-model cost, session rollups, pass/fail verdicts), but the division has to be assembled by the user.
- None of them aggregates retry cost natively. A retry is just another trace.
- None of them records the time a human spent reviewing or fixing the output. They store the correction, not the minutes.
The standards side is the same. The GenAI conventions have no cost attribute, no task-success attribute and no retry attribute; all three exist only as unmerged proposals.
Add convention for operation costs #287
Propose new conventions
Cost tracking seems to be an important use case in observability, especially for GenAI / LLMs where costs can be unpredictable (depend on input and output token counts).
But cost tracking in general is something that's always been useful, it's very common for calling external APIs to have a cost associated with it.
I'd like to propose we add an operation.cost: int and operation.cost_units: str attributes (open to suggestions on naming, units, etc.) to generalize this.
With these attributes in place it would be easy to make generic queries such as:
- Most expensive operations
- Total cost of LLM calls
- Spend per service
The metric name itself is not new. The OpenAI scorecard has a section on what a successful task actually costs, and Arize and Fireworks published a benchmark with the same name and the same formula (total spend divided by successful runs, plus a "retry tax").
- OpenAI scorecard: https://openai.com/index/a-scorecard-for-the-ai-age/
- Arize benchmark: https://arize.com/blog/cost-per-successful-task-ai-model-benchmark
So the part worth doing is not the metric. It is the procedure.
Four tasks, machine-checked
Every task is graded by code, so no part of the result depends on my judgement of "looks good enough".
- Task 1: normalize a messy CSV (rename columns, drop a row)
- Task 2: fix a Python bug (the returned code is actually executed against a test)
- Task 3: build a JSON document from order details (keys and values compared)
- Task 4: extract fields from a noisy invoice (numbers and dates compared)
The procedure is four steps: run the same prompt up to three times, log every attempt, then aggregate.
tasks (4) ──same prompt──> model (2 sizes) ──output──> verifier (code) │ verdict + attempts + tokens + seconds │ ▼ tokens / wait time per solved task The smallest model won on every axis
I ran two model sizes: A (mid-sized) and B (the smallest class available). The cash cost is unknown in my setup, so the only things I can assert are token counts and wait time.
- CSV normalization: A 743 tok / 62s, B 226 tok / 1.0s
- Bug fix: A 281 tok / 62s, B 248 tok / 1.0s
- JSON build: A 676 tok / 73s, B 260 tok / 1.8s
- Invoice extraction: A 433 tok / 94s, B 302 tok / 1.2s
- Total: A 2133 tok / 291.5s, B 1036 tok / 5.0s
- Per solved task: A 533 tok, B 259 tok
Both models solved all four tasks on the first attempt, so the retry tax was 1.0 and no retry ever happened.
The gap that matters was not the token count. Tokens differed by 2x; wait time differed by 58x. If you run this in a nightly batch, 58x slower is the difference between a batch that finishes tonight and one that does not.
Three failures I hit while measuring
The measurement turned up three bugs that had nothing to do with the metric.
- A fallback model had quietly died. It answered HTTP 410 (end of life). Nothing notices while the primary model works, and my first run burned three attempts on it.
- A large model was unmeasurable. One attempt waited 1050 seconds and never came back; a 20B-class model timed out at 120 seconds.
- My own weekly pipeline was stuck on the same symptom. Extracting 272 items did not finish in three hours (a healthy week takes about 84 minutes), and the log held 7 HTTP 410s and 8 HTTP 429s.
Retries against a dead fallback are a cost that never shows up on an invoice. They show up in logs and on a clock.
How to reproduce it, and what I did not measure
Four steps: pick tasks whose output code can judge, write the verifier, run the same prompt up to three times per model, and append one JSON line per attempt. The repository is a single file with no dependencies and works against any OpenAI-compatible endpoint.
What I did not measure, stated plainly:
- Four tasks only, Japanese only, two model sizes.
- The cash cost is unknown.
- Harder tasks should expose the retry tax; this run did not.
- The retries here are naive: the verifier's error is not fed back to the model.
Human review minutes were out of scope. No observability product records them either, so there is no published number to compare against. The way to get one is simple: judge the output yourself, stopwatch in hand, and keep the seconds.
Measuring is cheap. A stopwatch and a small piece of code that decides whether an output is correct are enough.
The code and the procedure are public: https://github.com/sunnydachs/cost-per-task
This is a personal OSS project — no warranty. Please use it at your own risk; issues and ideas are welcome.

