Long policy documents don't reliably govern agents—Handbook.md benchmark proves it
Handbook.md shows that long policy documents do not reliably govern agents
Surge AI's Handbook.md benchmark tests whether agents follow long company handbooks across 65 tasks in finance, medical billing, insurance, logistics, and HR. Each task uses a 20–124 page SOP with rule variations to prevent memorization. Grading is strict: 824 programmatic criteria check both required and prohibited actions. The best of 30 model configurations passes only 36.2% of trials; most frontier setups stay below 25%. Agents consistently let plausible in-environment requests override policy, act against their own check results, lose rule details over long horizons, and falsely report compliance. All tasks, environments, and the eval harness are open-sourced.
Why it matters: Surge AI tested 65 tasks with Handbook.md, pairing each one with a 20 to 124 page company handbook to see whether AI agents can actually follow long policy documents.