Computing Life · Share · YageAug 2, 2026, 08:00 UTC+8

When your product is used by AI, not just humans: how to evaluate AI-friendliness

当产品不仅给人用，也给 AI 用：如何评估你产品的 AI 亲和度？

This piece argues that as AI coding tools become primary users of dev products, a product's AI-friendliness is a hard requirement. Supabase open-sourced supabase/evals and found that Agent failures often stem from unfriendly docs, CLI hints, or error messages—not model intelligence. Stripe, Convex, and Vercel are all building vertical regression evals instead of chasing generic leaderboards. Vercel's data is striking: default Agent Skills went unused in 56% of cases, yielding the same 53% pass rate as no docs; embedding an 8KB AGENTS.md index directly in context hit 100%. The post recommends pulling 20–50 real pain points from support tickets and GitHub Issues, then running a two-layer setup of static linting in CI plus dynamic sandbox evals. Fix the product side first on every failure before swapping models.

Why it matters: Fresh angle backed by concrete cases (Supabase, Vercel), not just theory. But it's a personal blog without primary data or exclusive interviews, so authority is limited—capped at 78, the featured threshold.

Read the original ↗Export Markdown

Take

This piece is worth opening because it turns the vague complaint of "our product isn't AI-friendly" into an engineering problem you can measure. Supabase, Stripe, and Convex have all stopped chasing generic model leaderboards. Instead, they're building vertical regression evals for their own products—every time a test fails, they fix docs, error messages, or CLI hints before even thinking about swapping models.

The sharpest data comes from Vercel: baseline pass rate with no docs was 53%. Adding default Agent Skills? Still 53%—the Agent simply didn't read them in 56% of cases. Forcing Skill invocation via prompt instructions bumped it to 79%. Embedding an 8KB AGENTS.md index directly in context hit 100%. The takeaway isn't that Skills are useless; it's that you need eval data to confirm the Agent is actually consuming what you provide.

I'd discount the 100% figure a bit—the post doesn't disclose Vercel's test set size or task scope, so it may be a narrow scenario. But the direction is solid. Static scanners like Fern or Mintlify check formatting hygiene; they can't replace end-to-end sandbox evals that verify whether the task actually got done. If you're setting this up for your own product, the article's advice is practical: pull 20–50 real pain points from support tickets and GitHub Issues, run a two-layer setup with static linting in CI plus dynamic sandbox evals, and fix the product side first on every failure.

Source: Computing Life · Share · Yage · yage.ai
Published: Aug 2, 2026, 08:00 UTC+8 · Aug 2, 2026