Editorial illustration of a first patch and a revision, not a screenshot of model output.
The first patch is where a coding agent starts costing a team time. If that patch passes the visible suite, a reviewer can move on to the diff and private regression tests. If it fails, the agent may still finish, but the route from first patch to accepted change now includes another test run, another read of the failure, and another edit. Collapsing both paths into “all tests passed” erases a practical difference.
On November 20, 2026, I tested two Crazyrouter routes, deepseek-v4.1-flash and deepseek-v4-pro, against the same small Node.js repository tasks. There were three independent runs per route per task, 12 runs in total. The prompts were in Chinese, and the models were allowed to inspect the tests. I am reporting the behavior of those route IDs on that date, not an independently verified comparison of specific upstream DeepSeek weight snapshots.
AI disclosure: This article was drafted with AI assistance from saved API responses, code diffs, and test logs for Crazyrouter. The account owner should check the technical claims and DEV's AI disclosure setting before publication.
A scorecard that does not discard the first attempt
One task began with broken text, number, and date utilities and a node --test baseline of 0/8. The other began at 1/23 and asked the agent to implement an LRU cache, asynchronous memoization, retry logic, and a duration parser from a README. Each run had its own repository copy. Both routes received the same Chinese task text, system instructions, file tools, max_tokens=8192, and disabled thinking. They could read the README, source, and tests, but write only to src/; test-file hashes did not change. Node was v24.21.0.
I counted the first test run after the first code edit as the first pass. A test run that merely inspected the broken baseline did not count. When a model saw failures and edited again in the same session, I recorded that separately.
| Task | V4.1 Flash | V4 Pro |
|---|---|---|
| Repair existing utilities, eight tests | Two first-pass 8/8 runs; one run interrupted before any code was written | Three first-pass 8/8 runs |
| Implement four modules, 23 tests | Two first-pass 23/23 runs; one 20/23 followed by 23/23 | Two first-pass 23/23 runs; one 21/23 followed by 23/23 |
That interrupted Flash run is easy to mishandle. It cannot be called a failed code patch because there was no patch. It also cannot be erased from a delivery report: a task still went unfinished. Both routes encountered connection interruptions in the batch, so elapsed time here would include waiting and continuation, not just model execution.
The denominator changes with the question. Among runs that produced code, Flash passed the whole visible suite on the first patch in 4/5 and Pro in 5/6. After feedback, those figures became 5/5 and 6/6. Counting every run that was started instead gives completed delivery in 5/6 for Flash and 6/6 for Pro. The unfinished Flash run belongs in the last figure, but cannot diagnose its code quality. None of these small counts estimates a general success rate.
Across the runs that produced code, first-pass full-suite results were Flash 4/5 and Pro 5/6; post-feedback visible-test results were Flash 5/5 and Pro 6/6. Human edits and complete-task billing were not measured.
Failure one: a parser that understood units but not the grammar
The README accepted a single duration such as 2d, compound strings such as 1h30m and 1m500ms, decimal values, spaces between segments, and units in different orders. One Flash first pass parsed the entire input as one number-and-unit pair. It could handle the simple case but rejected compound strings, leaving three of the 23 visible tests failing. After the agent read the failures, it changed the parser to consume successive segments and reached 23/23.
This is a useful example of a spec that looks smaller than it is. A reviewer who tests only one happy-path duration may see a working function. The acceptance cases need to cover the grammar: single versus compound input, ms versus m followed by s, decimals, whitespace, duplicate units, and invalid units. Those cases came from the task's README and visible tests. A project with a different duration contract would need different assertions.
Failure two: a cache that returned the right answer twice
One Pro first pass reached 21/23. It already shared work between concurrent calls. But when no TTL was supplied, it did not retain the successful result for a later call. The caller received the correct value both times while the underlying function ran again.
The two failures appeared under different key cases: object properties in a different order, and a custom keyFn. Their shared condition was the absence of a TTL. Counting calls to the wrapped function showed the behavioral defect; reading only the returned values would have missed it. Once the agent stored successful results without an explicit TTL, the same visible suite passed 23/23. The evidence does not show that its original key normalization was faulty.
When I evaluate an agent-generated cache, I want assertions on both output and work performed. A useful local acceptance set covers simultaneous callers, a later repeat caller, rejection followed by retry, TTL expiration, and custom keys. The count of underlying calls is especially important when the function wraps a paid API or a slow database query.
What I would measure before switching a production route
For each task, I would keep a row with these fields:
- First patch: Did the agent write code, and what passed before any failure feedback?
- After feedback: Which failed behaviors were fixed, and how many additional cycles were needed?
- Human review: What did a reviewer still have to change, and did private regression cases pass?
- End-to-end operation: What were the actual elapsed time, connection failures, and billed usage for an accepted task?
This study fills the first two fields and identifies one unfinished run. It does not fill the human-edit or whole-task billing fields, so I would not use it to announce a cheaper or faster default route. The tests were visible to the models, and the sample contains only two small tasks. A large codebase, hidden tests, image inputs, or a different prompt language could change the decision.
The API record supports that narrower reading. Calls went to https://api.crazyrouter.com/v1/chat/completions. One saved first response from the four-module task for each route returned HTTP 200 and finish_reason=tool_calls. Flash returned model ID deepseek-v4.1-flash with 502 input, 59 output, and 561 total tokens. Pro returned deepseek-v4-pro with 511 input, 37 output, and 548 total tokens. Thirteen tokens separate those individual responses; they do not account for the later tool calls, tests, revisions, and retries in a complete repository run.
DeepSeek's official update history is useful release context, but I could not verify the exact upstream snapshots behind these two Crazyrouter route IDs. The comparison is therefore dated and route-specific. If you already have a route that passes your team's regression suite, keep it as the baseline and run the other against the same real failures before changing the default. The Crazyrouter integration directory lists possible coding-tool connections; the runs here used a separate file-tool harness and do not validate every integration on that page.
An agent that reaches green after feedback may be useful. The first patch, the review work, and the delivery failure still deserve their own rows in the scorecard.
👉 Try it with a single API key: https://crazyrouter.com/en/pricing?utm_source=devto&utm_medium=article&utm_campaign=offsite_article&utm_content=cr_ea5cfc1ea667_bottom_cta

