Breaking RLHF scaling bottlenecks: DeepMind raises data efficiency 10x with information-directed exploration
突破RLHF的规模化瓶颈 | DeepMind团队论文 | 数据利用效率极低 | 四种RLHF算法 | off-policy | 在线RLHF | 认知神经网络ENN | 信息导向探索 | 肯定性微调
A Google DeepMind team reports that online RLHF plus information-directed exploration on Gemma 9B reaches about 55% win rate with under 20k preference labels, versus about 200k for offline RLHF. The post describes four algorithms—offline, periodic, online, and information-directed exploration; online training uses batches of 64 prompts and 16 sampled responses per prompt, while the ENN head adds under 5% parameters. The key point is methodological, not that RLHF failed; the post also says results use Gemini 1.5 Pro simulated feedback, and the 1000x gain is an extrapolation toward 1M labels.
Why it matters: DeepMind compares four RLHF algorithms on Gemma 9B: online RLHF with information-directed exploration reaches the offline method's result with under 20,000 preference labels versus about 200,000, a tenfold gap in data efficiency.