Skip to content
AI HOT (Curated Pool)

Justin Cormack on AI Agent Evaluation: Start With Evidence, Not Coverage

Justin Cormack 用 35 万行 Rust 复盘 AI Agent 评估:从证据开始

Justin Cormack built an S3-compatible storage system with AI, reaching 350k lines of Rust. He ran 1,500 tests against real S3 as an oracle, which caught real S3 500 errors. Chasing 100% coverage backfired—agents wrote trivial tests. Docs were often wrong, and AI was bad at finding edge cases from them. His hard rule: fix flaky tests immediately, or the agent learns to ignore failures.

Why it matters: Justin Cormack built an S3-compatible object store in about 350,000 lines of Rust and pinned its behavior with roughly 1,500 S3 tests, a reusable evidence-based method for evaluating AI agents.

Read the original ↗Export Markdown