ProgramBench: Can We Really Rebuild Huge Binaries from Scratch?
ProgramBench: Can we really rebuild huge binaries from scratch? (doesn't look like it)
ProgramBench released 200 tasks for agents rebuilding programs from target executables and usage files. The team spent about $50k generating 6M lines of black-box behavioral tests, with no internet or decompilation. GitHub, Hugging Face, and Docker images are open-sourced, with pip-based evaluation available.
Why it matters: ProgramBench gives agents 200 tasks to rebuild programs from nothing but a binary and its docs, with no network or decompiling, and the results show agents mostly cannot do it.