At Landvex we need a reference holding of what genuine document families look like: bank statements, employment certificates, invoices, and some ID/travel documents used in Europe. Not customer files. Layout, issuer markers, typical field structure. Receiving country and issuing country are often not the same, which already makes "just scrape one national portal" pretty useless!?
What's actually worked when the material has to be lawful and auditable?
A few points I'm turning over:
- Public official specimens versus anything that looks like a customer document
- Is scraping those public pages even the right architecture, or do you start with data partnerships and treat crawlers as a last resort?
- How do you store provenance so a later reviewer can see where a reference came from, when it was fetched, and under what licence? Some official libraries are explicitly not for reuse in a product.
- Layout fingerprints versus storing full page images. Full images are easier to train on, and a lot more legally awkward.
- Rate limits, robots.txt, and portals that redesign their HTML every quarter — is a scraper maintainable compared to a small set of official download endpoints you actually monitor?
I'm specifically not looking for tips on grabbing customer bank files or bypassing logins. That's a hard no. The question is how people have built a clean, traceable reference corpus for document review without walking into a copyright or ToS problem.
What would you do first if you had to stand this up properly?
