Airbnb's eval-driven development: treating GenAI evaluation as a first-class engineering discipline
Airbnb Eval-driven development: Lessons from evaluating GenAI at scale
Airbnb's engineering team shares their methodology for evaluating GenAI at scale. The core rule: write evals before you write code, not after. They combine three methods—heuristic metrics for fast checks, LLM-as-judge for subjective quality, and human evaluation for calibration. A virtual judge must be calibrated against human ratings before it's trusted. For agentic systems that chain tool calls and reasoning, they recommend evaluating each step individually, then running end-to-end tests. The post walks through a full workflow from task definition and eval set creation to production monitoring.
Why it matters: Airbnb's engineering team splits generative AI evaluation into rule-based metrics, LLM judges and human calibration, and evaluates agent systems step by step before end-to-end tests, a method other teams can reuse.