This piece is worth reading because it connects two seemingly opposite evaluations to make one sharp point. Princeton's shadow peer review had AI-written papers scored 2/6 and 1/6 by human reviewers—clear rejects. Prime Intellect's long-horizon benchmark had Fable 5 closing 81.7% of the gap to the human record in 8.7 days. Looks like a blowout versus a breakthrough, but dig into the run logs and the gap isn't in ideation. All 18 models proposed similar optimization ideas. Grok 4.5 twice landed on the right direction and twice killed it by mistaking its own scaling bug for a hypothesis failure. The real divider is what happens after an experiment says no: attribution, scoping, archiving, resurrection, and combination of negative results.
The post walks through five gates with concrete examples. On attribution: stronger systems measure hardware noise baselines first, use 3 seeds instead of 1 for borderline results, and separate code bugs from hypothesis failures in their accounting. On scoping: 62 runs wrote their own scripts to measure environment noise rather than trusting official estimates; 42 discovered GPU non-determinism and designed same-seed paired comparisons to isolate weak signals. On resurrection: Opus 5 actively revisited previously dismissed β₂ tuning after the overall recipe changed and broke its own historical best. On combination: Fable 5 deliberately hunted for parameter pairs that looked worse in isolation but delivered gains together, netting 31 steps in one joint test.
The Princeton system failed precisely at archiving and resurrection. Its internal log explicitly flagged two major flaws in its approach, but with no mechanism to trigger a review when the recipe changed, it locked in its overall direction with 110 hours still on the clock. Even the extra 24 hours humans added didn't prompt a reopening of the main approach—work stayed on patching and rewriting the old draft.
The post proposes four automatable disciplines: noise baselining, fault attribution accounting, conditional negative-result logging, and weak-signal combination pools. But it also admits that in open-ended research without a live scoreboard, judging whether a flaw is a minor engineering issue or a fatal directional error still depends heavily on human research taste. I buy that. These four disciplines read more like a better lab notebook for AI researchers than a replacement for scientific judgment. The post doesn't disclose full model versions or complete config tables for the 153 runs, so the specific rank fluctuations on the leaderboard should be read with the 40-54 step variance range the article itself provides.