Skip to content
AI HOT (Curated Pool)

EMO: Expert Mixture Models for Emergent Modular Pretraining

EMO:为涌现模块化预训练的专家混合模型

AllenAI introduced EMO, a mixture-of-experts model with 14B total parameters and 1B active parameters, trained on 1 trillion tokens and able to use only 12.5% of its experts for specific tasks while retaining near-full-model performance.

Why it matters: EMO lets experts emerge from 1 trillion tokens instead of assigning them by subject, and activates only 12.5% of them at run time while nearing full-model quality.

Read the original ↗Export Markdown