DevOps
SWEOps-Bench: Evaluating Agents for End-to-End Resolution of Software-Induced Cloud Incidents
arXiv cs.SE0 views
arXiv:2611.06099v1 Announce Type: new Abstract: As autonomous agents advance, the traditional boundary between software development and cloud operations gradually blurs, increasing the need for integrated incident detection, localization, and remediation. Existing benchmarks evaluate isolated diagnostic queries or repository-level program repair, failing to test continuous incident resolution across operational clusters and source repositories. We introduce SWEOps-Bench, an interactive benchmark comprising 55 reproducible incident tasks derived from confirmed historical defects across 16 Kubernetes operator projects. Each task preserves the original software repository and initializes in a healthy cluster before a controlled workload triggers the historical fault. Agents investigate runtime evidence and submit source or configuration repairs through a terminal interface, while the harness builds and deploys the changes within a shared time budget. Evaluation covers stage-wise accuracy, verified recovery, penalized lifecycle latency, and token consumption. Across six agent-model configurations, faulty-state detection ranges from 72.7% to 90.9%, whereas overall recovery ranges from only 14.5% to 52.7%, exposing a substantial gap between recognizing incidents and resolving them end to end. Trajectory analysis further shows that reported diagnosis and final recovery only partially coincide: some tasks achieve verified recovery despite an incorrect reported RCA, while runtime symptom clearance can still fail fix-specific functional verification. SWEOps-Bench establishes a rigorous foundation for evaluating autonomous agents in integrated development and operational environments.
Source:https://arxiv.org/abs/2611.06099PreviousSynMOE: A Syntax-Aware MOE-Guided Approach for PEFT-Based Code Intelligence LearningNextSBVUL: Statement-Based Vulnerability Detection
Related posts

Dux Modular Shell 2.1 Adds Video Wallpapers, New Interface Styles
0Linuxiac
PIMROP: A Feasibility Study of Return-Oriented Programming on Commercial Processing-in-Memory Architectures
0arXiv cs.CR
Continuous Integration and Continuous Delivery: Are CI/CD Pipelines DevOps?
0dev.to DevOps
Distribution Release: NebiOS 10.3
0DistroWatch
Linux Mint 27 arrives next month, and I like it already: New features and other upgrades to expect
0ZDNet
omaforge 0.2.9: star addons to find them later
0dev.to Linux