Researchers introduce SWE-CoWork, a benchmark of 101 development assignments across 101 real repositories, 11 languages, and 12 domains. Agents work in a simulated organisation with 26 colleagues, recover requirements from people and records, and must collaborate and verify their work.

Why it matters

Real coding work is collaborative and underspecified; benchmarks that test only isolated code generation may overstate workplace readiness.

Analysis

SWE-CoWork addresses a gap between isolated coding tests and workplace practice by adding distributed requirements and colleagues. That should expose planning, communication, and verification failures. Main uncertainty: whether simulated colleagues behave realistically enough for results to predict performance in real organisations.

Source: arXiv

AI disclosure: this article was produced with AI assistance and reviewed for accuracy.