We run two duplicate finders over our TypeScript repos. One is static: it parses every function, throws away the names and compares what is left. The other asks a local embedding model what each function means and compares the vectors. They disagree a lot, and most of what we learned came from the cases where they disagree.
For this post I rebuilt both as small standalone scripts, with fourteen sample functions that reproduce the cases we kept running into. Everything below is real output from those scripts. The code is here: https://github.com/CosmicDriftGameStudio/duplicate-finder-samples. You need Bun, and for the embedding part Ollama with nomic-embed-text.
Why we bother
We had the static check for months, but nothing ran it. When we finally ran it over all our repos at the end of October, it reported 118 groups that span files or modules. Typically the same logic had been written a second time by someone who didn't find the first version. At one point we had six copies of an HTML escape function.
The static finder: a fingerprint per function
The script uses the TypeScript compiler to find every function, method and arrow function with a body of at least 120 characters. Shorter bodies match each other all the time and are never worth extracting. Each body is then normalized: every identifier becomes _, every string and number becomes #, whitespace goes away.
s = s.replace(/"[^"]*"/g, "#"); s = s.replace(/'[^']*'/g, "#"); s = s.replace(/`[^`]*`/g, "#"); s = s.replace(/\b\d+(\.\d+)?n?\b/g, "#"); s = s.replace(/\b(true|false|null|undefined)\b/g, "#"); s = s.replace(/\b[a-zA-Z_$][\w$]*\b/g, (m) => (KEYWORDS.has(m) ? m : "_")); The normalized body plus the parameter count is the fingerprint. Two functions with the same fingerprint are an exact group. An optional second pass compares the rest pairwise: 70% edit distance on the normalized body, 30% overlap of the identifiers.
$ bun ast-duplicates.ts samples --fuzzy 0.8 exact 1.00 accounts/disable-account.ts:disableAccount + accounts/enable-account.ts:enableAccount exact 1.00 forms/is-field-row.ts:isFieldRow + scheduling/is-instant-like.ts:isInstantLike similar 0.87 feed/escape-xml.ts:escapeXml + mail/escape-html.ts:escapeHtml 3 group(s) What it's good at: copies that only differ in names and values
enableAccount and disableAccount are the same handler with true swapped for false and one audit string changed:
export async function disableAccount(accountId: string, actorId: string) { const account = await db.accounts.find(accountId); if (!account) throw new Error(`account ${accountId} not found`); if (account.enabled === false) return account; const updated = await db.accounts.update(accountId, { enabled: false }); await db.audit.insert({ accountId, actorId, action: "account.disabled" }); return updated; } After normalization the two bodies are identical, so the fingerprint matches with a score of 1.0. There's no threshold and nothing to tune, and the same input always gives the same answer. That's why this is the check that runs automatically for us. We had this exact pair in our own code and replaced both with one factory that takes the flag.
Where it's wrong: same shape, different meaning
The second exact group is a false positive:
export function isInstantLike(value: unknown): value is InstantLike { return ( typeof value === "object" && value !== null && "epochMilliseconds" in value && typeof value.epochMilliseconds === "number" && "timeZone" in value && typeof value.timeZone === "string" ); } isFieldRow in the forms code checks fieldName and fieldType in the same pattern. Every type guard over an object with two fields looks like this. Once names and strings are gone, nothing is left that tells them apart, and merging them would make no sense. We hit the same thing in our own code: a type guard for date values and one for form field rows, flagged as one group.
Where it misses: same logic, different shape
export function orderTotal(lines: OrderLine[]): number { let total = 0; for (const line of lines) { const gross = line.unitPrice * line.quantity; const discount = line.discountPercent ? (gross * line.discountPercent) / 100 : 0; total += gross - discount; } return Math.round(total * 100) / 100; } export const invoiceTotal = (items: InvoiceItem[]): number => { const sum = items.reduce( (acc, item) => acc + item.price * item.count * (1 - (item.rebate ?? 0) / 100), 0, ); return Math.round(sum * 100) / 100; }; Both compute the same total. One uses a loop, the other reduce, and the static finder doesn't see a group even in the fuzzy pass. A fingerprint only catches copies, and a second person solving the same problem rarely writes a copy.
A gap I found while writing this
escapeHtml and escapeXml should be an exact group: the only difference is ' against '. They only show up in the fuzzy pass, at 0.87. Here is what the normalizer makes of them:
escapeXml {return _._(/&/_,#)._(/</_,#)._(/>/_,#)._(/#&_;#&_;");} escapeHtml {return _._(/&/_,#)._(/</_,#)._(/>/_,#)._(/#&_;#&##;");} The " inside the regex literal /"/g pairs with the quote of the next string, so the string regexes cut across the code. The 39 in ' then ends up outside a string and becomes #. Our production check had the same normalizer and the same gap. While writing this post we switched it to the tokens of the already parsed syntax tree: string, template and JSX text tokens become #, regex literals stay as they are, comments are gone. The first run with the new normalizer found 12 more groups across files, real duplicates that had been hidden all along, most of them because the bodies differed only in their comments. The sample here keeps the regex version, because it's shorter to read and it shows the gap.
The embedding finder: what does the function mean?
The second script sends each function body to nomic-embed-text through a local Ollama, with comments stripped and whitespace collapsed, and compares every pair by cosine similarity.
body: JSON.stringify({ model: MODEL, input: `search_document: ${text}` }), The search_document: prefix is what the model expects for documents.
$ bun embedding-duplicates.ts samples --threshold 0.8 0.9881 accounts/disable-account.ts:disableAccount + accounts/enable-account.ts:enableAccount 0.9312 feed/escape-xml.ts:escapeXml + mail/escape-html.ts:escapeHtml 0.9304 reports/correlation.ts:correlation + stats/correlation.ts:pearsonCorrelation 0.8261 invoices/invoice-total.ts:invoiceTotal + orders/order-total.ts:orderTotal 0.8084 cache/redis-client.ts:createCacheClient + db/client.ts:createDbConnection 5 pair(s) at or above 0.8 What it's good at: the pair the fingerprint missed
orderTotal and invoiceTotal are in the list, at 0.83. The model doesn't care about loop against reduce or discountPercent against rebate. The two type guards that fooled the fingerprint score 0.78, just under the cut.
That margin is the catch. With --threshold 0 the scale gets visible: slugify and parseDuration, which have nothing in common, still score 0.67. The real invoice pair is 0.05 above the type guards. At the script's default of 0.9 it drops out, and only the three pairs at 0.93 and above remain. The ranking is mostly sensible, but no single threshold separates real from false, and the scores depend on the model and its version.
Where it's wrong: similar words, different code
createDbConnection and createCacheClient sit at 0.81, almost as high as the real duplicate. Both read a URL from the environment, throw when it's missing, parse a number and build a client. One returns a Postgres pool with a listener connection, the other a Redis client with an error handler. There is nothing to extract.
We saw this at a larger scale. Three database connection factories in our code scored 0.9961 and 1.0 against each other, and they return different shapes on different layers. Seven queries on the same JSON column clustered together because they share identifiers like the column and table name, not logic.
Where it's dangerous: almost the same
pearsonCorrelation and correlation score 0.93, higher than the invoice pair. They are the same formula, with one difference: pearsonCorrelation returns 0 when a series has no variance, correlation returns NaN. An embedding can't see a missing early return. Merge them, and one caller changes behavior.
We had this one for real. The correlation helper in a test suite stayed as it was, because the caller's tolerance check depends on getting NaN. Another pair stayed because one of them sorts its input and the other doesn't, and a later step relies on the order.
How we put the two together
For each pair, the question is which finders saw it. combine.ts runs both and sorts the pairs by that:
$ bun combine.ts samples both ast 1.00 emb 0.99 accounts/disable-account.ts:disableAccount + accounts/enable-account.ts:enableAccount -> extract one helper both ast 0.87 emb 0.93 feed/escape-xml.ts:escapeXml + mail/escape-html.ts:escapeHtml -> extract one helper ast-only ast 1.00 emb - forms/is-field-row.ts:isFieldRow + scheduling/is-instant-like.ts:isInstantLike -> same shape: check that both mean the same thing embedding-only ast - emb 0.93 reports/correlation.ts:correlation + stats/correlation.ts:pearsonCorrelation -> same intent: run both on the same inputs before merging embedding-only ast - emb 0.83 invoices/invoice-total.ts:invoiceTotal + orders/order-total.ts:orderTotal -> same intent: run both on the same inputs before merging embedding-only ast - emb 0.81 cache/redis-client.ts:createCacheClient + db/client.ts:createDbConnection -> same intent: run both on the same inputs before merging A pair both finders report is the easy case. A pair only the static finder reports has the same shape, and someone has to check whether both functions are for the same thing. A pair only the embedding reports looks like the same idea, and someone has to check that the two functions behave the same. For that last group, check-equivalence.ts feeds both functions the same inputs:
$ bun check-equivalence.ts correlation [1,2,3,4] [2,4,6,8]: stats 1, reports 1 correlation [1,2,3,4] [4,3,2,1]: stats -1, reports -1 correlation [5,5,5,5] [1,2,3,4]: stats 0, reports NaN total [{"price":19.99,"count":3}]: orders 59.97, invoices 59.97 total [{"price":19.99,"count":3,"rebate":15},{"price":4.5,"count":3}]: orders 64.47, invoices 64.47 total []: orders 0, invoices 0 The totals agree on these three inputs, which makes the merge worth trying. The correlation differs on the third input. The connection factories don't get this far, because they don't even return the same type.
In our repos the combining is less automatic than this script. Both finders read one shared ignore list for deliberate parallels, such as plugins that implement the same interface. For the static check, a function can also opt out with a // guard:dup-ok comment. The static check runs after every review pass over merged pull requests and compares against a baseline: known groups pass, a new group becomes a finding in our review queue, and the baseline only ever goes down. Over three cleanup steps it went from 49 to 22, 3 and finally 0. The sample has a simpler version of the same idea behind --baseline and exits with 1 on a new group, so you can also put it in CI.
The embedding finder doesn't run automatically, because it needs Ollama. We run it by hand before a cleanup round and judge the pairs in bands. In one round, the band between 0.90 and 0.95 had 77 clusters across repos: 9 real duplicates, 23 partly, 13 already on our list, and 32 that weren't duplicates at all. A pair that both finders reported became one checklist item, which is what combine.ts does automatically here.
What we haven't solved
The embedding band between about 0.8 and 0.95 mixes real duplicates with false positives, and the sample shows it on a small scale: the real invoice pair at 0.83 sits right next to the connection factories at 0.81. A threshold can't separate them. A few things we're considering:
- Embed the signature and return type along with the body, so a function returning a Redis client and one returning a pool drift apart.
- Have a language model judge the pairs in the band instead of a person. That costs money for every pair, and we'd need to measure how often it's wrong.
- Make the equivalence check automatic: generate inputs from the parameter types and compare the outputs, instead of writing
check-equivalence.tsby hand for each pair.
If you run duplicate detection on a real codebase: what did you add to get the false positives down? And has anyone automated the "do these two actually behave the same" step in a way you'd trust before a merge?
