AI & ML
ProdCrawl: A Replayable Test Collection for Focused Crawling
arXiv cs.IR0 views
arXiv:2610.40855v1 Announce Type: new Abstract: A focused crawler searches for web pages on a specific topic under a constrained download budget. Since it can only fetch a limited number of pages, its success depends on choosing which links to follow. Focused crawlers are usually evaluated on live websites by harvest rate, the fraction of downloaded pages that are relevant. This has two problems: on the one hand, websites change over time, so a crawl cannot be repeated under the same conditions; on the other hand, there is no ground truth for which pages are relevant. As a result, comparisons between crawlers are unreliable. We present ProdCrawl, a replayable test collection for focused crawling. It comprises static mirrors of 621 company and organization websites, about 1.1 million pages, with each page labeled as a product page or non-product page based on a majority vote among three large language models. Any crawler can be run on exactly the same sites and scored against the same labels. We use ProdCrawl to compare simple baselines (breadth-first, depth-first and text classifiers) with an approach that scores each link by searching for similar link texts whose target pages have already been labeled. Our results show that text classifiers and keyword matching do not significantly outperform breadth-first crawling, whereas neural embeddings, which compare link texts by semantic meaning, outperform all simple strategies. Because every run evaluates over identical pages and labels, these results are exactly reproducible. The collection, codebase, and models are publicly released to support experimental replication and fair crawler comparison.
Source:https://arxiv.org/abs/2610.40855PreviousRetrieve-then-Disambiguate: Scalable Reviewer DiscoveryNextContext as Infrastructure: Knowledge Compilation for Retrieval in Agentic Systems with Harnesses
Related posts
How to change the Flutter app icon on iOS and Android
0dev.to
GPT-6 Astra covertly underperforms on safety research
0Hacker News
Giving passengers a bit more time to board helps flights leave on time
0Phys.org Tech
Mir 2.31 Released With Expanded Wayland Protocol Support
0Phoronix

AI promised big productivity benefits, but workers aren't happy with what they've seen so far
0TechRadar

Motorola Signature 27 goes official with true flagship specs and a $1,599 price tag
09to5Google