I'm building CiteWise, a tool that checks whether an academic reference points to a real paper. You paste a citation, it looks the work up in Crossref, DataCite, OpenAlex, Semantic Scholar, PubMed and arXiv, and shows you which fields match and which don't.
The obvious way to build this in 2026 is "send the reference to an LLM and ask if it's real". I tried a version of that early on and it's a bad idea for the same reason people need this tool in the first place: LLMs are very good at producing things that look right.
So the LLM is in the pipeline, but it isn't allowed to decide anything. This post is about where that line sits, and about the bug in my first version that crossed it without me noticing.
The pipeline
Roughly:
Parse the raw string into fields (title, authors, year, journal, DOI...) with a rule-based parser.
Query the scholarly sources in parallel.
Score every candidate against the user's input, field by field.
Pick a status: verified, ambiguous, conflicting metadata, not found, and so on.
Step 3 is deterministic. Title similarity, author overlap, year, DOI equality, a penalty for contradictions. No model involved. If you run the same reference twice against the same source responses, you get the same score.
Where the LLM comes in
Some references are just messy. A student copies a citation out of a PDF and gets Smith J Lee K Deep learning for something 2019 Nature Methods 16 4 pp 1-9 with no punctuation. The rule parser gets half the fields, the search queries are bad, and nothing comes back.
That's the only case where the LLM runs: there's no DOI, and the first pass ended as not found or insufficient evidence. The model gets the raw string and tries to fill in the missing fields. Then we search again with those fields.
The bug
Here's a simplified version of what my first implementation did after the second search:
java
var enriched = llm.completeMissingFields(candidate);
var retry = resolver.resolve(enriched);
if (retry.score() >= first.score()) {
candidate = enriched; // <- this line
resolution = retry;
}
Looks reasonable. If the LLM-assisted search scores at least as well, use it.
The problem is what candidate means. It's the thing every candidate paper gets scored against. After this swap, the scorer was no longer comparing source records to what the user typed. It was comparing them to what the LLM thought the user meant.
So if the model guessed a plausible title, found a real paper with that title, and then the scorer compared the real paper to the model's guess, you'd get a high score. The user's original reference could have been wrong in exactly the way we're supposed to catch, and the result would say "verified".
The >= made it worse. A tie meant the LLM's version won.
The fix
The LLM output is now treated as search hints only. It can help us find candidates. It can't become the thing we compare against.
java
// Enrichment supplies retrieval hints only.
// It never becomes evidence about the user's input.
ResolutionResult resolveWithHints(Candidate original, Candidate hints) {
return resolve(original, retrieval.retrieve(hints));
}
Candidates are fetched using the enriched fields, then scored against the original parsed input. And the second pass only replaces the first if its score is strictly higher:
java
var selected = retry.score() > first.score() ? retry : first;
If the LLM found the right paper, it'll match the user's original fields better than whatever the first search returned, and the score goes up. If it found a plausible-but-wrong paper, the original fields won't match it, and the first result stands.
A smaller related change: the old code had a status called POSSIBLE_HALLUCINATION, assigned when the LLM had touched the input and nothing was found. That's a strong claim to make from "we didn't find it". New results never produce that status. Old records that have it are shown as "not found under the old rules".
The other thing I got wrong: treating silence as success
The second lesson wasn't about the LLM at all.
When a search returned no candidates, the old code decided between "not found" and "insufficient evidence" like this:
java
boolean searchCompleted = diagnostics.isEmpty()
|| diagnostics.stream().anyMatch(d -> d.ok());
diagnostics.isEmpty() counted as a completed search. So if something went wrong before any source reported back, we'd confidently tell the user their reference doesn't exist.
Now every check records which sources actually responded, and the result carries a separate completeness field: COMPLETE, PARTIAL or INCOMPLETE. For a reference with a DOI, Crossref and DataCite both have to answer. Without a DOI, at least two of the three general-purpose sources have to. If that doesn't happen, the result says so and suggests a retry, instead of saying "not found".
This also changed billing. A complete search that finds nothing still costs a check, because the search actually ran. A search where the sources were down doesn't.
What I'd tell someone building something similar
Decide up front which parts of your pipeline are evidence and which are search. Keep the LLM on the search side.
Be suspicious of >= when one side came from a model.
"No results" and "no response" are different outcomes. Make your types say so, or someone (me) will merge them by accident.
A tool that says "I couldn't check this properly" is more useful than one that guesses.
If you want to see it on a real reference, it's at citewise.dev. There are 10 free checks a month. I'm especially interested in references it gets wrong, so if you have a weird one, throw it at it and let me know in the comments.
