Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

441–460 of 609

May 9Saturday

AI HOT (Curated Pool)

EMO: Expert Mixture Models for Emergent Modular Pretraining

AllenAI introduced EMO, a mixture-of-experts model with 14B total parameters and 1B active parameters, trained on 1 trillion tokens and able to use only 12.5% of its experts for specific tasks while retaining near-full-model performance.

Why it matters: EMO lets experts emerge from 1 trillion tokens instead of assigning them by subject, and activates only 12.5% of them at run time while nearing full-model quality.

May 8Friday

r/LocalLLaMA

Gemma 4 26B Hits 600 Tok/s on One RTX 5090

chain-77 benchmarked Gemma 4 26B with vLLM 0.19.2rc1, and DFlash raised output throughput on one RTX 5090 from 228 tok/s to 578 tok/s under 256 input tokens, 1024 output tokens, concurrency 1, and num_speculative_tokens=13.

Why it matters: chain-77 ran Gemma 4 26B on a single RTX 5090 with vLLM 0.19.2rc1, and DFlash speculative decoding lifted output throughput from 228 to 578 tokens per second, with reproducible concurrency and draft-token settings.

Synced · WeChat

SGLang Team Launches RadixArk With $100M Seed Round

RadixArk announced a $100 million seed round on May 5 at a $400 million post-money valuation, while its SGLang inference project has 27K+ GitHub stars and deployments across 400K+ GPUs.

Why it matters: The SGLang team's new company raised a $100M seed at a $400M post-money valuation, with 27,000 GitHub stars and 400,000+ GPUs running its open inference framework.

Xinzhiyuan · WeChat

Token-Level Length Control: 3B Model Beats GPT 5.4 and Claude

UC Santa Barbara and Apple researchers introduced LenVM, which models remaining generation length as a token-level value function; Qwen2.5-3B with a 1.5B LenVM scored 62.6 on LIFEBench length control, above GPT-5.4 at 37.4 and Claude-Opus-4-6 at 35.5.

Why it matters: Modeling remaining generation length as a token-level value function lets a 3B model beat GPT-5.4 and Claude on length control.

QbitAI · WeChat

HIT and Huawei propose Dynamic-dLLM, a training-free acceleration framework with 4.48x speedup

HIT Shenzhen, Huawei, and Shenzhen Hetao College proposed Dynamic-dLLM, a training-free dLLM acceleration framework that raises LLaDA-8B-Instruct throughput on GSM8k from 8.32 TPS to 37.29 TPS with almost no accuracy loss.

Why it matters: Harbin Institute of Technology (Shenzhen) and Huawei propose Dynamic-dLLM, a training-free acceleration framework that lifts GSM8k throughput on LLaDA-8B-Instruct from 8.32 to 37.29 tokens per second with almost no accuracy loss.

Financial Times · Technology

Big Tech’s $725bn AI Spending Spree Sends Free Cash Flow to a Decade Low

Big Tech is spending $725 billion on AI infrastructure, and the title says free cash flow has fallen to a decade low; the RSS snippet says Silicon Valley giants shifted from asset-light cash generators to infrastructure investors, but it does not disclose the company list, time period, or accounting basis.

Why it matters: Big Tech has committed $725bn to AI infrastructure, pushing free cash flow to a decade low as these firms shift from asset-light cash cows to heavy infrastructure spending.

r/LocalLLaMA

11.67% ARC-AGI-2 Local Eval on a Single 4090: The TOPAS Recursive Architecture

Doug_Bitterbot says TOPAS scored 11.67% on ARC-AGI-2 using one RTX 4090 after about 14 days of training. The 100M-parameter checkpoint hit 36% locally, but recursive TTT caused null outputs on nearly half of Kaggle puzzles. The key detail is time management: the author expects 20% after threshold tuning and 3-5 more weeks of training.

Why it matters: About 14 days of training on one RTX 4090 got the 100M-parameter TOPAS to 11.67% on ARC-AGI-2, with local runs reaching 36%, while test-time training recursion left nearly half the Kaggle tasks with empty output.

Bloomberg Technology

Nvidia to Invest Up to $2.1 Billion in Data Center Firm IREN

Nvidia will invest up to $2.1 billion in IREN under an AI infrastructure partnership. The post discloses the cap and goal, but not equity terms, payment timing, or data center capacity.

Why it matters: Nvidia is putting up to $2.1 billion into IREN's AI data centers, though the filing leaves out equity terms and total compute.

The Verge · AI

SpaceX Has a $55 Billion Plan to Build AI Chips in Texas

SpaceX plans to invest at least $55 billion in its Terafab chip plant in Austin, Texas. A hearing notice says later phases could lift total investment to $119 billion. Musk said in March the target was chips for 200GW of compute per year; the post does not disclose process nodes.

Why it matters: SpaceX plans a chip plant in Austin, with hearing documents showing investment from at least $55 billion up to $119 billion and Musk targeting annual capacity for 200GW of compute.

AI HOT (Curated Pool)

Readable behavioral signals remain in frozen LLM hidden states, Cygnus boosts accuracy

Proprioceptive AI says Cygnus adds adapters to frozen LLMs and raises Qwen-32B on ARC-Challenge from 82.2% to 94.97%. It projects hidden states into a gl(4,R) Lie-algebra space to isolate “dark modes.” Watch replication; the post does not disclose full eval sets or controls.

Why it matters: Proprioceptive AI attaches a self-awareness adapter to a frozen model, reading behavior signals from hidden states without retraining, lifting Qwen-32B's ARC score from 82% to nearly 95%.

NVIDIA Blog

Powering the Next American Century: Chris Wright and NVIDIA’s Ian Buck on Genesis Mission

The U.S. DOE and NVIDIA are building two AI supercomputers at Argonne; Equinox uses 10,000 Grace Blackwell GPUs. Solstice will use 100,000 Vera Rubin GPUs, which Buck said reach 5,000 exaflops. The key bottleneck is grid work: Wright said AI can cut interconnection studies from years to weeks or hours.

Why it matters: The US Department of Energy and NVIDIA are deploying two AI supercomputers at Argonne National Laboratory, scaling from 10,000 Grace Blackwell chips to 100,000 Vera Rubin GPUs.

AI HOT (Curated Pool)

DeepSeek 4: Flash Local Inference Engine for Metal

DeepSeek 4 Flash is open-sourced on GitHub for offline inference on Apple Silicon Macs. The post says it uses Metal Performance Shaders to reduce latency and memory use, but discloses no benchmark numbers. The key item is the Metal local inference stack, not another model wrapper.

Why it matters: antirez, the creator of Redis, shipped ds4 to run DeepSeek 4 Flash locally on a Mac, calling Apple's Metal Performance Shaders directly to cut latency and memory use.

May 7Thursday

AI HOT (Curated Pool)

Notes From Inside China’s AI Labs

The author visited several leading Chinese AI labs and reported three patterns. The post says some Chinese tasks beat GPT-4, while firms build 100B-scale base models and 10B-scale vertical models. Watch compression and private deployment under compute constraints.

Why it matters: The author visited several leading Chinese AI labs and found students are core contributors, a fast-follow culture speeds up catching up, and original research gets suppressed in the process.

AI HOT (Curated Pool)

SenseNova-U1 Open-Sources 8-Step Distilled LoRA, Speeds Diffusion Inference by 11x

SenseNova-U1 open-sourced an 8-step distilled LoRA that cuts diffusion generation from 100 steps to 8. GPU inference time drops from 23 seconds to 2 seconds, with ComfyUI workflows for text-to-image, image editing, and interleaved generation. The key signal is distillation for latency, not parameter scale.

Why it matters: An open 8-step LoRA cuts diffusion sampling from 100 steps to 8, dropping GPU inference from 23 seconds to 2, with a ComfyUI workflow included.

QbitAI · WeChat

Zhejiang University and Alibaba MetaCompress reaches 90% token compression for multi-turn VQA

Zhejiang University and Alibaba proposed MetaCompress, a learned token-compression framework that generates a compression mapping from the input image alone for multi-turn VQA. The article says it can remove 90% of visual tokens while preserving accuracy, and reports only 1.71% overlap between optimally retained tokens and high-attention tokens.

Why it matters: Zhejiang University and Alibaba let a model decide which visual tokens to keep from the input image alone, cutting 90% of tokens in multi-turn VQA with no accuracy loss.

r/LocalLLaMA

Running Qwen3.5/Qwen3.6 with NextN MTP in llama.cpp on one RTX 3090 Ti

A Reddit user posted a llama.cpp guide for Qwen3.5/3.6 with NextN MTP on one RTX 3090 Ti. It requires two unmerged PRs, #22400 and #22673; Qwen3.6-35B-A3B-MTP reaches 157 tok/s at 350W, 1700MHz, with q8 KV. The key reproducible detail is nextn=q8_0 quant override; missing it yields “////” output.

Why it matters: A Reddit user posts a hands-on llama.cpp guide for running Qwen3.5 and 3.6 on a single RTX 3090 Ti, built on two unmerged PRs and covering NextN MTP speculative decoding setup.

Latent Space

Anthropic-SpaceXAI's 300MW/$5B/yr Deal for Colossus I, ARR Growth Is 8000% Annualized

Anthropic announced a SpaceX compute partnership, doubled Claude Code’s 5-hour limits for Pro, Max, Team, and seat-based Enterprise, raised Opus API limits, and said Claude inference would ramp on Colossus within days; the post treats the 300MW and $5B-per-year figures as widely circulated but not canonized in Anthropic’s own announcement.

Why it matters: Anthropic is renting xAI's 300MW Memphis Colossus I cluster for Claude inference, a deal estimated at $5B a year that puts xAI into the compute rental business.

Synced · WeChat

Musk Announces xAI Dissolution, Leasing 220,000 GPUs to Anthropic

Musk confirmed xAI will dissolve, with Grok and X-related operations folded into SpaceXAI. SpaceX and Anthropic signed a deal giving Claude access to Colossus 1’s 220,000+ Nvidia GPUs and 300 MW of compute. The key change is quota: Claude Code’s five-hour rate limit doubles, and Pro/Max peak-hour cuts are removed.

Why it matters: Musk confirmed xAI's dissolution, folding Grok and X into SpaceXAI while renting Colossus 1's 220,000 GPUs to Anthropic, a shift in two AI rivals' positions.

Computing Life · Share · Yage

Anthropic Locks Up Compute Channels as xAI Rents Its Castle to a Rival

Anthropic signed four compute contracts in six months covering AWS Trainium, Google TPU, SpaceXAI Colossus 1, and CoreWeave; during the same window, xAI rented the Colossus 1 supercomputing center to a competitor while GPU utilization stood at 11%.

Why it matters: Anthropic locked in four compute suppliers in six months, covering three mainstream chip architectures, while xAI rented its 220,000-GPU Colossus 1 to its rival.

Financial Times · Technology

Arm projects $2bn in sales of its new AI chip from next year

Arm projects $2bn in sales for its first in-house AI chip from next year. The RSS snippet says the SoftBank-backed UK group has strong demand; the post does not disclose customers, pricing, process node, or delivery cadence.

Why it matters: Arm is moving from licensing chip designs to building its own AI chip, with a $2bn sales projection for next year.