I bolted an 8-arm reasoning MoE onto a frozen 1.4B Mamba backbone on a single RTX 3060
I bolted an 8-arm reasoning MoE onto a frozen 1.4B Mamba backbone on a single RTX 3060. Here’s the mechanistic autopsy of what broke and what worked.
The author trained Mamba-Titan-1.4B-Reasoning on a 12GB RTX 3060: a frozen 1.4B Mamba-1 backbone with 8 trainable MoE arms, 2.54B total parameters, Top-2 routing at layers 24/25, and about 50% math accuracy.
Why it matters: A single RTX 3060, a frozen 1.4B Mamba backbone and eight trainable expert arms at layers 24 and 25 reach 2.54B parameters and roughly 50% on math problems.