Hacker News front pageJun 16, 2026, 22:50 UTC+8

SubQ 1.1 Small: Sparse attention cuts long-context compute by 64.5x at 1M tokens

Subquadratic – Introducing SubQ 1.1 Small

Subquadratic released the model card for SubQ 1.1 Small. It replaces quadratic dense attention with Subquadratic Sparse Attention (SSA) that routes based on content relevance, scaling linearly with context length. At 1M tokens, SSA uses 64.5x less compute than dense attention and runs 56x faster than FlashAttention-2. The model scores near-perfect on needle-in-a-haystack from 1M to 12M tokens and 99.12% on RULER at 128K. General reasoning holds: GPQA Diamond 85.4%, LiveCodeBench pass@4 89.7%, AutomationBench Finance 13%. Training started from an open-weight frontier model, replaced attention with SSA, then ran staged context extension up to 2M and ~1T tokens of continued pretraining on books, documents, and repo-scale code. The post does not name the base model. SubQ 1.1 Small is deploying with select design partners; a broader lineup from 2M to 12M tokens is planned later this year.

Why it matters: SubQ 1.1 Small ships a deployable sparse-attention model with a 64.5x compute reduction and near-perfect 12M-token retrieval. Held below 85 because it's still a model card + design-partner deployment — no open weights or public API yet, so the production story is incomplete.

Read the original ↗Export Markdown

Take

The reason this matters: it changes the cost structure of long-context models. Standard attention scales quadratically — double the text, quadruple the compute. SubQ's SSA routes attention based on content relevance instead, hitting 64.5x less compute than dense attention at 1M tokens.

The long-context retrieval numbers are genuinely strong: near-perfect needle-in-a-haystack from 1M to 12M tokens, 99.12% on RULER at 128K. General reasoning holds up too — 85.4% on GPQA Diamond and 89.7% pass@4 on LiveCodeBench put it in the mid-tier frontier range.

A few things I'd discount. The post doesn't name the base open-weight model they started from, so we can't separate what came from the original model vs. SSA and continued pretraining. The 12M-token NIAH result is impressive, but training was mostly at 1M — the 12x generalization relies on content-based routing rather than positional patterns, and I'd want to see more task diversity before calling it solved. AutomationBench Finance at 13% is still well below GPT-5.5's 18%, so the long-context advantage hasn't fully translated to complex agent tasks yet.

Right now it's only deploying with design partners. The broader 2M–12M token lineup is planned for later this year. If inference costs really stay linear, this could accelerate long-document agents and full-codebase reasoning significantly.

Source: Hacker News front page · subq.ai
Published: Jun 16, 2026, 22:50 UTC+8 · Jun 16, 2026