ORCA-bench: How Ready Are Language Model Agents for Oncall?
Orca-Bench: How Ready Are Language Model Agents for Oncall?
Five frontier coding agents were tested on 1,079 production-style RCA tasks in a live microservice testbed with Prometheus, Jaeger, and OpenSearch. The best agent hit 25.3% root-cause accuracy on medium-difficulty tasks and 10% on hard ones. The weakest model hallucinated an implausible root cause in 40% of reports. Removing source-code access hurt every metric. The authors stress that real production systems are orders of magnitude larger, so these numbers are a lower bound—agents aren't ready for unsupervised oncall.
Why it matters: Five coding agents chased 1,079 root causes in simulated microservice data; the best hit 25.3% on medium cases, and one fabricated causes in 40% of reports.