Hacker News front pageSep 28, 2026, 01:24 UTC+8

Adding "Do not guess" cut made-up fields from 71% to 20% in web extraction

Calling the AI bluff: Adding "Do not guess" cut made-up fields from 71% to 20%

Earn an Honest Dollar tested 16 models and 3 paid APIs on web extraction with missing fields. Without "Do not guess," models invented 405 of 573 missing fields (70.7%). Adding the instruction dropped that to 116 of 574 (20.2%). Gemini 3.8 Flash made up only 1 of 36 missing fields at $0.16. Firecrawl, a paid API, made up 24/36—worse than 13 models. A cheap checker using GPT-6 Luna caught 38 of 49 made-up values with zero false rejections, costing $0.0049. The post notes these are synthetic pages; real-site results may differ.

Why it matters: A controlled twin-page experiment that quantifies how one prompt sentence suppresses model fabrication. Solid data, reproducible method, directly useful for anyone doing web extraction. Not a product launch or industry event, so it doesn't hit the 85+ band, but HKR all three p...

Read the original ↗Export Markdown

Take

The numbers here are hard to ignore. Without the instruction, 16 models invented 405 of 573 missing fields — 70.7%. Add 'Do not guess, use null for missing fields,' and that dropped to 116 of 574, or 20.2%. Gemini 3.8 Flash led the pack: 1 made-up field out of 36, full run cost $0.16.

I'd discount the ranking a bit. The test uses 42 synthetic page pairs across 7 hand-crafted page types — not real websites. Real pages have partial data, ambiguous labels, and messy structure that trip models up more. Also, only one run per model, so don't squint at the exact order.

The real sting is Firecrawl: a paid API that made up 24 of 36 missing fields, worse than 13 models with the instruction, and every fabrication copied the decoy value. Meanwhile, a plain fetch plus GPT-6 Luna cost $0.0049 for the full run and made up only 5 fields.

The cheap-checker idea is worth stealing. GPT-6 Luna caught 38 of 49 made-up values with zero false rejections, checking 126 page-value pairs for $0.0049. For anyone building agent marketplaces or scraping pipelines: pick a service from measured results, then run a dirt-cheap model to verify each answer. That's the useful pattern here.

Source: Hacker News front page · earnanhonestdollar.com
Published: Sep 28, 2026, 01:24 UTC+8 · 2 days ago