Xinzhiyuan · WeChatMay 10, 2026, 12:49 UTC+8

Next-ToBE Targets Short-Sighted Next-Token Prediction in LLMs at ICLR 2026

大模型自信且短视！Next-ToBE破除Next Token预测诅咒 | ICLR'26

East China Normal University and Fudan University researchers proposed Next-ToBE, a training objective that keeps standard autoregressive inference while adding a soft target over future-token windows, and the article reports the method ranked best in 35 of 36 experiments across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Llama3.1-8B-Instruct.

Why it matters: HKR-H and HKR-K pass: the mechanism and 35/36 result are specific, and next-token training is a real debate. The item stays near the featured floor because no artifact, reproduction detail, or production claim is disclosed.

Read the original ↗Export Markdown

Take

Next-ToBE has a clean pitch: leave architecture and autoregressive inference alone, then replace the one-hot next-token target with a soft target over future-token windows. The summary gives one hard hook: across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Llama3.1-8B-Instruct, it ranks best in 35 of 36 experiments.

I cannot verify the full paper details here because the WeChat body is blocked by verification. That matters: task mix, window size, target construction, training budget, and statistical spread decide whether this is a durable objective or a narrow tuning win. I like this more than inference-time “think longer” hacks because the cost sits in training and serving stays unchanged. If the gains depend on math-heavy data or distilled labels, it stays a fine-tuning trick. If it transfers across general instruction tasks, it belongs in the pretraining-objective conversation.

Source: Xinzhiyuan · WeChat · mp.weixin.qq.com
Published: May 10, 2026, 12:49 UTC+8 · May 10, 2026