Audience note: this is for developers who maintain a self-written scraper for one specific site and are deciding whether to fix it themselves or adopt an open-source, site-specific project.
The problem. Anti-bot systems keep escalating. Recent practitioner discussions describe Playwright bots being flagged within a week of polling every 1–5 seconds, and — per one commenter — blocks based on TLS fingerprinting rather than IP address (HN thread, observed 2026-11-14: https://news.ycombinator.com/item?id=50371246). A separate Ask HN thread records one author's estimate that Cloudflare challenges appear on roughly 50–60% of the sites they visit (https://news.ycombinator.com/item?id=50355959). Against this background, adopting a site-specific scraper is not a "clone and run" decision: sites rebuild their pages, parsers silently return zero rows, and READMEs age.
What follows is a pre-adoption checklist distilled from three open-source scraper READMEs (all read 2026-11-14; the claims below are the authors' own, with their stated measurement dates — not independently reproduced in this article).
1. Look for dated verification, not badges.
A green CI badge tells you the code compiled. A dated verification table with a commit hash tells you someone ran it against the live site. Example: the rakuten-scraper README documents a verification run dated 2026-11-09 pinned to a specific commit, with row counts and exit status (https://raw.githubusercontent.com/2scraper/rakuten-scraper/main/README.md). If a README has no dated run against the live target, treat the project as unverified, whatever its star count.
2. Check whether the failure mode is documented.
The best READMEs describe how the site rejects clients. Two instructive cases:
- rakuten-scraper reports that its target sits behind Akamai and that rejection is driven by User-Agent/TLS fingerprint mismatch, not address type — a plain HTTP client with a Chrome User-Agent receives a 42-byte refusal under HTTP 200, which is why status codes alone are not a signal [same README].
- potterybarn-scraper documents product pages failing along three independent axes (non-US exit IP, minimal headers, and the client library itself), and — notably — admits openly that it does not know which part of the handshake is being scored (https://raw.githubusercontent.com/2scraper/potterybarn-scraper/main/README.md). That honesty is a positive signal: it means the author measured instead of guessing.
If a README says nothing about the site's anti-bot behavior, expect to rediscover it yourself at runtime.
3. Look for breakage history, not just green states.
Sites change. flippa-scraper records that its target rebuilt the search page on 28–29 September 2026, after which the old parser returned zero listings and the canary went red; the Playwright engine was re-verified on 15 October 2026 — but the README explicitly states the other three engines were not re-verified after the rebuild (https://raw.githubusercontent.com/2scraper/flippa-scraper/main/README.md). Two lessons: (a) a canary/verification log with visible failure periods is more trustworthy than an always-green state, and (b) "multiple engines" does not mean "multiple verified engines" — check each engine's last-verified date.
4. Read the declared limits before the features.
Every mature scraper README lists hard caps and known gaps. rakuten declares a site-side cap of 6,750 results per query and routes it cannot read; flippa declares schema columns that went null after the rebuild. If those limits conflict with your use case, no amount of engine flexibility will fix it — this is the cheapest thing to discover, so discover it first.
5. Check dependency hygiene and isolation requirements.
All three READMEs warn that engines must run in separate virtualenvs due to conflicting pyee and urllib3 pins, and the flippa README flags a known-vulnerable urllib3<2 in its pyppeteer engine. If you plan to embed the scraper in a larger service rather than run it standalone, these conflicts become your problem — verify pins against your own lockfile before adopting.
6. Beware the "captcha-solver" topic label.
Notably, the rakuten README states that captcha is not configured on its readable routes at all — the observed failure mode there is fingerprint mismatch, not a captcha wall. A repo topic or tag is not a verified capability. This matters because the current discourse overweights captchas: the Ask HN thread above reflects real fatigue, but several of the hardest failures documented in these three projects had nothing to do with captchas.
A note on sampling. Everything above comes from three repositories in one organization (7–9 GitHub stars each), each with a single site as its target, and each README's claims carry their own authors' dates — none of the runs were reproduced for this article. One green verification on one date is not a reliability statement; the flippa rebuild shows why. If you adopt any scraper, budget for periodic re-verification of your own, and prefer projects that publish canary results you can compare against.
The pattern to internalize: evaluate a site-specific scraper the way you would evaluate a dependency with an expiry date. Dated live-site verification > badges; documented failure modes > claimed features; breakage history > star count. And for the decision itself: if a project's README already describes your target site's exact rejection behavior, you are buying back the debugging weeks someone else already spent.
References (all read 2026-11-14):
- https://raw.githubusercontent.com/2scraper/rakuten-scraper/main/README.md
- https://raw.githubusercontent.com/2scraper/flippa-scraper/main/README.md
- https://raw.githubusercontent.com/2scraper/potterybarn-scraper/main/README.md
- https://news.ycombinator.com/item?id=50371246
- https://news.ycombinator.com/item?id=50355959