The same paper can arrive from one database as a bare DOI and from another as a resolver URL. Normalize the DOI before grouping those rows, and preserve the search source in the resulting record. For citations whose DOI is missing, keep the original rows for inspection.
This synthetic Python example handles DOI URLs, bare DOI strings, and empty values. It removes an initial resolver prefix and uses case-insensitive comparison.
import re records = [ ("Index A", "https://doi.org/10.5555/example.12"), ("Index B", "10.5555/EXAMPLE.12"), ("Index B", ""), ] by_doi = {} manual = [] for source, raw in records: value = re.sub(r"^https?://(?:dx\.)?doi\.org/", "", raw.strip(), flags=re.I) if re.fullmatch(r"10\.\d{4,9}/\S+", value): by_doi.setdefault(value.casefold(), set()).add(source) else: manual.append((source, raw)) print({doi: sorted(sources) for doi, sources in by_doi.items()}) print(manual) The first output has one normalized key, 10.5555/example.12, with both sources. The second retains one record for manual review. The regular expression is a first-pass shape check. A valid-looking identifier can fail to resolve, and preprints and published versions can have different DOIs. Before merging around a DOI, resolve it and compare title and author metadata. Keep the retrieved date and original identifiers in the export.
I work on RigorMed, an AI research workspace for evidence checks and research deliverables. A useful output for this step is an evidence table with the normalized DOI, original record sources, and a manual review flag.
