r/LocalLLaMAMay 5, 2026, 23:40 UTC+8

ProgramBench: Can We Really Rebuild Huge Binaries from Scratch?

ProgramBench: Can we really rebuild huge binaries from scratch? (doesn't look like it)

ProgramBench released 200 tasks for agents rebuilding programs from target executables and usage files. The team spent about $50k generating 6M lines of black-box behavioral tests, with no internet or decompilation. GitHub, Hugging Face, and Docker images are open-sourced, with pip-based evaluation available.

Why it matters: HKR-H/K/R all pass: a provocative coding-agent failure angle plus concrete benchmark scale and rules. Reddit sourcing and no cross-source cluster keep it in the 78–84 band, not P1.

Read the original ↗Export Markdown

Take

ProgramBench lands in the right sore spot: it tests whole-program reconstruction, not patch repair. The setup gives agents a target executable plus README-style usage files, then blocks internet access, decompilation, and cheating. The benchmark has 200 tasks and roughly $50k of generated black-box behavioral tests, filtered from 6M lines. That is a cleaner stress test than another curated “agent built an app” thread.

I buy the mechanism more than the headline pessimism. A model must choose a language, design abstractions, and build architecture from observed behavior. That breaks a lot of SWE-bench-shaped muscle memory. The authors also say open models have behaved worse so far, partly from overfitting to SWE-bench-like tasks. Harsh, but plausible: train on patch leaderboards long enough, and you get patch machines.

Source: r/LocalLLaMA · reddit.com
Published: May 5, 2026, 23:40 UTC+8 · May 5, 2026