Hacker News front pageApr 30, 2026, 00:01 UTC+8

Show HN: A New Benchmark for Testing LLMs for Deterministic Outputs

Show HN: A new benchmark for testing LLMs for deterministic outputs

Interfaze released Structured Output Benchmark, scoring schema pass rate, types, and value accuracy across text, image, and audio. Each record has a JSON Schema and human plus LLM-checked ground truth; GLM-4.7 ranks No. 2 overall. The key bug is field-level value error: GPT-5.4 ranks 3rd on text and 9th on images.

Why it matters: HKR-H/K/R all pass: the ranking has a hook, the methodology is concrete, and structured-output reliability matters to builders. Single-source Show HN launch with no adoption signal keeps it in the 72–77 band.

Read the original ↗Export Markdown

Take

SOB is aimed at the right failure mode: structured-output bugs rarely come from broken braces now; they come from valid JSON carrying wrong leaf values. The benchmark splits seven metrics, makes Value Accuracy primary, and uses a parse-fail zeroing gate plus a coverage gate to stop schema-only wins. The dataset is concrete enough to be useful: 5,000 text records, 209 image records, and 115 audio records, each paired with a JSON Schema and human-authored, LLM-cross-checked ground truth.

My caveat is the modality framing. Image and audio inputs are converted into text-normalized context before scoring, so this is not measuring end-to-end vision or ASR extraction. It is measuring schema handling and value grounding across different content distributions. GPT-5.4 ranking No. 1 overall, No. 3 on text, and No. 9 on images is the uncomfortable part for anyone shipping document pipelines.

Source: Hacker News front page · interfaze.ai
Published: Apr 30, 2026, 00:01 UTC+8 · Apr 30, 2026