Skip to content
Hacker News front page

A small transformer trained on a 5090 hits 44% on ARC-AGI-1 for 67 cents

44% on ARC-AGI-1 in 67 cents

Mithil Vakde trained a small transformer from scratch on a 5090 in 1.5 hours for 67 cents, scoring 44% on ARC-AGI-1 public eval—matching TRM/HRM—and 7% on ARC-2. The method converts each puzzle into token sequences, uses 3D RoPE and per-task learnable embeddings for cross-task learning, and applies test-time augmentations with voting. Switching to a modern architecture (SwiGLU, RMSNorm) and using fewer augmentations drove the gains and cut costs. Training only on output tokens lifted the score from 40% to 44%, which the author doesn't fully understand yet. Code is open source; the union of solved tasks across runs reaches 55%, and the author sees room in better position embeddings and architecture tweaks.

Why it matters: Mithil Vakde trained a small transformer from scratch on one 5090 in 1.5 hours, hitting 44% on the ARC-AGI-1 public eval for 67 cents, with reproducible per-task embeddings and 3D RoPE.

Read the original ↗Export Markdown