Exact-match duplicate checks feel reassuring right up until they miss almost everything that matters.
Same vendor, same invoice number, same amount, same date — that duplicate is easy to catch, and it is also the rarest kind. The payments that actually drain a business arrive disguised: invoice 10482 comes back as 010482, a hyphen moves, the vendor name is abbreviated, or the second payment is the invoice amount minus a small credit. Every field is close. No field is identical. A check built on equality waves both payments through.
What worked: scoring similarity instead of demanding equality
The approach that held up was treating duplication as a similarity problem, not a matching problem. Normalise first — strip leading zeros and punctuation from invoice numbers, fold vendor name variants toward a canonical supplier record, and compare amounts within a tolerance that accounts for credits and partial payments. Then score the combination: a near-identical number plus a close amount plus the same supplier family is worth a human look even when no single field matches exactly.
Vendor master data turned out to be half the battle. The same supplier often exists twice, once under a full legal name and once under a short trading name, with different vendor IDs. An invoice entered against each record can be paid twice without any single screen ever showing a conflict. Deduplicating the vendor list was less glamorous than tuning the matching rules, and it removed more false negatives.
What broke: the second payment rail
The case that broke early versions came through the payment rail itself. An urgent invoice gets settled on a company card to keep a supplier relationship intact. The original paperwork keeps moving through accounts payable and is paid again by bank transfer weeks later. Neither payment is wrong on its own. Together they are one purchase paid for twice — and no invoice-level check sees it, because the card payment never produced an invoice record in the same system. Cross-rail reconciliation had to become a first-class input, not an afterthought.
False positives were the other tax. Score too loosely and the review queue fills with legitimate repeat purchases — monthly services that genuinely do cost the same amount every month. The fix was boring and effective: recurring charges with a stable history get a lower starting score, and every flagged pair has to show its reasoning (which fields were close, and by how much) so a reviewer can dismiss it in seconds.
The lesson
Duplicate detection is not a report, it is a queue design problem. The goal is not a perfect matcher; it is a short, explainable exception list that a busy finance team will actually clear. That is the design centre of ERPSav, a cost recovery product from CloudVerg that runs rules-based recovery engines across duplicate supplier payments and other leak categories such as lease overcharges, utility and telecom billing errors, and card processing fees.
Builders working on similar problems: how are you handling the vendor-variant problem — canonical supplier records up front, or similarity scoring at match time? The trade-offs cut both ways, and the discussion is genuinely open.
