Choose an AI code reviewer by running it on changes your team understands. Record which findings deserve a fix, which claims are wrong, and how much time the team spends checking them. A comment count alone cannot tell you whether a reviewer helped.
Download the evaluation worksheet (CSV)
1. Pick changes that represent your work
Start with a small set of recent pull requests: a routine feature, a bug fix, a refactor, and a change to a sensitive path such as authorization or billing. Include changes that your team believes are correct. Those help reveal unsupported warnings.
Ten PRs can be a manageable starting point for a pilot. It is not enough to establish a universal accuracy ranking. Expand the sample before making claims beyond your own team’s experience.
For every PR, save the base and head commit hashes. If you compare tools, give them the same revision. A review of the original change and a review after fixes answer different questions.
2. Record the context and settings
Keep the repository instructions, review depth, included files, and available context alongside the results. Record the date and product configuration too. If one reviewer receives an explanation of an important business rule, give the other reviewer equivalent information where its configuration allows it. Document any difference you cannot remove.
Evaluate the workflow you would use after the trial. If your normal process includes a follow-up review after a fix, include that run in the time and cost records.
3. Judge each claim against the code
Have a developer familiar with the change classify each distinct finding. When a claim is disputed, record the evidence and ask another developer to check it.
How to classify review findings
A comment that the team ignores is not automatically false. It may be correct but low priority, already known, or hard to understand. Keep those reasons separate so that the configuration change addresses the actual problem.
Count repeated descriptions of one bug as one distinct issue. Retain the duplicate count separately to measure how much extra reading the reviewer creates.
4. Measure the time needed to use the feedback
For each PR, record the tool’s elapsed review time, the developer’s time checking findings, and the actual charge. Keep code-fixing time separate: fixing a useful discovery is different from spending ten minutes disproving a warning.
Open the CSV worksheet in your spreadsheet app. Use one row per tool and review run; keep follow-up runs on separate rows. Record costs in the same currency when comparing tools.
5. Score the results
Precision is the share of assessed findings judged correct: valid findings divided by valid findings plus false positives. Report the counts for each classification first, then calculate it per tool from the same set of pull requests.
Precision (valid and actionable + valid, deferred) ÷ (valid + false positives) Define which findings enter the denominator before you start counting. Duplicates and unresolved findings stay out of it, but keep both counts visible next to the result: duplicates measure extra reading, and a large unresolved count means the score rests on fewer claims than the reviewer made. If a reviewer reports nothing, precision is undefined rather than 100%.
For example, a reviewer with 12 valid findings and 3 false positives across the trial scores 12 ÷ 15 = 80%. A second reviewer with 4 valid findings and no false positives scores 100% on far fewer useful findings, so read precision together with the valid count and the triage time from step 4.
Recall, the share of real defects a reviewer finds, needs a known set of defects. Reviewing your ordinary PRs does not establish that every bug has been found. You can add a separate exercise using known historical defects, but label those results separately and withhold the later fix from the reviewer.
6. Compare seat fees and usage-based pricing
No seat fees means adding a developer does not add a per-person charge. A product can still have a subscription, usage charges, or both. Compare the total bill for the same workload, including follow-up reviews after a new push.
Shared credits can suit a team with many occasional contributors. A seat-based plan may cost less for a smaller team running frequent reviews, depending on its limits. Neither model is always cheaper. For example, 100 PRs with two review runs each means budgeting for 200 runs, not 100. This is an illustrative workload.
Use the actual charges in your trial worksheet to estimate a normal month and a busier month. Compare recurring prices over the same billing period, separate introductory discounts from renewal prices, and check whether tax is included. Keep developer time saved separate from the amount you pay for the tool.
7. Choose a rollout threshold before expanding
Decide what would make the trial useful for your team. That might be finding previously missed issues in a critical path while keeping triage time acceptable. A threshold should reflect your repository and reviewers, not a number borrowed from another company’s benchmark.
Start with manual reviews on one repository. Use the same worksheet to check whether a change to instructions or feedback scope improves the results. Once the team trusts the feedback, expand automatic reviews gradually.
How HeyDeer fits
HeyDeer reviews GitHub pull requests with a workspace plan and shared credits. Every plan includes unlimited developers and repositories, with no per-developer seat fees. Charges depend on the review work performed.
Each private repository review, including the first, uses credits for the work performed, even when it finds no issues. Reviewing another push or running another review on the same PR uses more credits. Free includes 1,000 credits in the first month from workspace creation, then 300 each month. Public repository reviews are free within fair-use limits. Monthly credits expire at the next refresh; add-on credits are valid for 12 months. Additional credits are purchased automatically only when a workspace admin turns on auto-reload.
The cost calculator estimates the number of review runs with a few repository checks. Count each new push and repeated review separately; more checks can increase usage.
Check the current plans and cost calculator, credit rules, and spending controls against your observed workload before choosing a plan. The rollout guide covers manual and automatic review settings, and reducing noise maps common problems to specific adjustments. Our internal evaluation shows how we scored reviewer contributions during development.
