AI & ML
ProSearch: Benchmarking Multi-Constraint Protocol Retrieval in Experimental Science
arXiv cs.IR0 views
arXiv:2610.41396v1 Announce Type: new Abstract: In AI for Science scenarios, experimental design and method reproduction require retrieving protocols that satisfy multiple interdependent constraints, yet existing benchmarks lack systematic evaluation of this capability. To address this gap, we introduce ProSearch, a benchmark for multi-constraint protocol retrieval in experimental science. ProSearch has three key features: (1) realism, with queries grounded in real experimental records and expert-defined templates; (2) complexity, with 60 experimental fields and their dependencies; and (3) diagnosticity, with 3-5 explicit constraints per query, including numerical and negation constraints. We construct ProSearch through LLM-based extraction, data-driven constraint selection, controlled query generation, and evidence-based relevance validation. The resulting benchmark contains 1,515 queries across six experimental science domains. We conduct extensive experiments on ProSearch with 19 representative retrievers and complementary analyses. The best-performing retriever achieves an average nDCG@10 of only 0.290, which improves to 0.474 with reranking. Structured diagnostic analysis shows that explicit constraint matching provides a strong ranking signal, while manual error analysis identifies joint constraint binding as a key bottleneck. ProSearch provides a testbed for systematically evaluating scientific retrieval systems.
Source:https://arxiv.org/abs/2610.41396PreviousBrowseFlow: Sparse Event Flow Matching for Generative User SimulationNextHow BERT-based Cross-Encoders Model Relevance in Information Retrieval?
Related posts

HomeView will support third-party AI chatbots, suggests code
09to5Mac

Google’s AI search fumbles leave schools in the lurch
09to5Google
AI Agents Keep Failing. Are You Fixing the Wrong Layer?
0HackerNoon
What Remains of Me When AI Can Remember, Think, and Act for Me?
0HackerNoon
Why Churn Prediction Models Fail at Customer Retention
0HackerNoon
The Strange Future of AI-Written Software: Do Companies Really Own Their Code?
0HackerNoon