Skip to content
QbitAI · WeChat

NTU REI-Bench Tests Vague Human Instructions, With Success Rates Dropping Up to 36.9%

成功率最高暴跌36.9%!南洋理工首个“模糊指令”测试,直击具身智能落地软肋

NTU MARS Lab released REI-Bench, a benchmark with 9 ambiguity levels for vague human instructions. Tests used 4 robot planning frameworks and 6 small LLMs; LLaMA3.1-8B+SayCan fell from 57.7% to 46.9% in standard multi-turn context. The key issue is implicit reference resolution, where baseline success dropped 7.4% to 36.9%.

Why it matters: NTU's MARS lab released REI-Bench, grading ambiguous instructions into 9 levels and testing 4 planning frameworks with 6 small models, where one setup's multi-turn success rate fell from 57.7% to 46.9%.

Read the original ↗Export Markdown