NTU REI-Bench Tests Vague Human Instructions, With Success Rates Dropping Up to 36.9%

成功率最高暴跌36.9%！南洋理工首个“模糊指令”测试，直击具身智能落地软肋

NTU MARS Lab released REI-Bench, a benchmark with 9 ambiguity levels for vague human instructions. Tests used 4 robot planning frameworks and 6 small LLMs; LLaMA3.1-8B+SayCan fell from 57.7% to 46.9% in standard multi-turn context. The key issue is implicit reference resolution, where baseline success dropped 7.4% to 36.9%.

Why it matters: HKR-H/K/R all pass: the 36.9% drop is a strong hook, and the setup gives 9 ambiguity levels, 4 frameworks, and 6 models. This is a solid embodied-AI benchmark, not a major model release, so it fits the 78–84 band.