AI & ML
ProdCrawl: A Replayable Test Collection for Focused Crawling
arXiv cs.IR0 views
arXiv:2610.40855v1 Announce Type: new Abstract: A focused crawler searches for web pages on a specific topic under a constrained download budget. Since it can only fetch a limited number of pages, its success depends on choosing which links to follow. Focused crawlers are usually evaluated on live websites by harvest rate, the fraction of downloaded pages that are relevant. This has two problems: on the one hand, websites change over time, so a crawl cannot be repeated under the same conditions; on the other hand, there is no ground truth for which pages are relevant. As a result, comparisons between crawlers are unreliable. We present ProdCrawl, a replayable test collection for focused crawling. It comprises static mirrors of 621 company and organization websites, about 1.1 million pages, with each page labeled as a product page or non-product page based on a majority vote among three large language models. Any crawler can be run on exactly the same sites and scored against the same labels. We use ProdCrawl to compare simple baselines (breadth-first, depth-first and text classifiers) with an approach that scores each link by searching for similar link texts whose target pages have already been labeled. Our results show that text classifiers and keyword matching do not significantly outperform breadth-first crawling, whereas neural embeddings, which compare link texts by semantic meaning, outperform all simple strategies. Because every run evaluates over identical pages and labels, these results are exactly reproducible. The collection, codebase, and models are publicly released to support experimental replication and fair crawler comparison.
Source:https://arxiv.org/abs/2610.40855PreviousRetrieve-then-Disambiguate: Scalable Reviewer DiscoveryNextContext as Infrastructure: Knowledge Compilation for Retrieval in Agentic Systems with Harnesses
Related posts

Motorola Signature 27 goes official with true flagship specs and a $1,599 price tag
09to5Google
Running AI Models Locally in 2026: Ollama, vLLM & Quantization
0dev.to

Most companies don't think they could leave their main cloud provider quickly should the worst happen
0TechRadar

'Nothing will ever replace the emotional depth and creative inspiration of human beings as artists and creators' — Call of Duty: Modern Warfare 4 composer speaks out against AI in video games, says 'm
0TechRadar

Two dead, five injured in Taiwan PCB factory explosion
0Tom's Hardware

HomeView will support third-party AI chatbots, suggests code
09to5Mac