r/LocalLLaMAJun 1, 2026, 10:05 UTC+8

I bolted an 8-arm reasoning MoE onto a frozen 1.4B Mamba backbone on a single RTX 3060

I bolted an 8-arm reasoning MoE onto a frozen 1.4B Mamba backbone on a single RTX 3060. Here’s the mechanistic autopsy of what broke and what worked.

The author trained Mamba-Titan-1.4B-Reasoning on a 12GB RTX 3060: a frozen 1.4B Mamba-1 backbone with 8 trainable MoE arms, 2.54B total parameters, Top-2 routing at layers 24/25, and about 50% math accuracy.

Why it matters: HKR-H/K/R all pass via a numbered first-person experiment, but it is a single Reddit post with no independent replication and a fairly technical setup, so it stays in the low featured band.

Read the original ↗Export Markdown

Take

The useful part is not the claimed 50% math accuracy. It is the attempt to make “frozen backbone plus trainable experts” fit on a consumer GPU. The scraped summary gives real hooks: frozen Mamba-1 at 1.4B, eight trainable MoE arms, 2.54B total parameters, Top-2 routing at layers 24 and 25, and a 12GB RTX 3060. The post body is blocked, so the dataset, training steps, eval set, and ablations are missing.

I’d file this under the LoRA / adapter / small-MoE modification track, not under frontier reasoning. DeepSeek-style MoE is about scaling training and inference. This Reddit build is about structural grafting under brutal VRAM limits. If the promised autopsy has routing collapse, expert specialization, and Mamba state-transfer details, that is more useful than the headline accuracy.

Source: r/LocalLLaMA · reddit.com
Published: Jun 1, 2026, 10:05 UTC+8 · Jun 1, 2026