Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

361–380 of 609

May 19Tuesday

QbitAI · WeChat

World model supports multiplayer FPS gameplay before Fei-Fei Li

Odyssey released Agora-1, a world model that supports up to four human and AI players fighting in the same generated FPS world in real time. The system decouples simulation from rendering and trains on GoldenEye internal game states.

Why it matters: Odyssey released the Agora-1 world model, which puts four human or AI players in one FPS map, trains on internal game states from GoldenEye, and runs physics simulation separately from rendering.

Synced · WeChat

Recent LLM Architecture Changes: From Gemma 4 to DeepSeek V4

Jiqizhixin translated Sebastian Raschka’s blog on recent LLM architecture changes, covering long-context cost reductions in Gemma 4, Laguna XS.2, and ZAYA1-8B; the article states that Gemma 4 E2B saves about 2.7GB of KV cache at 128K context with bfloat16 precision.

Why it matters: Raschka walks through architecture changes in Gemma 4 through DeepSeek V4, all aimed at cutting long-context memory cost.

Financial Times · Technology

Google makes chip push with Blackstone-backed AI cloud group

A Blackstone-backed AI cloud group is set to receive a $5 billion investment to bring 500MW of data center capacity online next year; the post does not disclose the Google chip terms or deployment structure.

Why it matters: A Blackstone-backed AI cloud group raised $5bn and plans 500MW of data center capacity next year, with the Google chip tie-up still undisclosed.

AI HOT (Curated Pool)

Google and Blackstone form AI cloud company with $5B initial equity and 500 MW target by 2027

Google and Blackstone formed an AI cloud services company with Blackstone committing $5 billion in initial equity capital, an expected total investment of about $25 billion after leverage, and a plan to bring 500 megawatts of data center capacity online in 2027.

Why it matters: Google and Blackstone are forming an AI cloud company, with Blackstone putting in $5 billion of equity first against a $25 billion total, aiming for 500 MW of data centers by 2027.

Bloomberg Technology

Google, Blackstone to Create AI Cloud Firm With In-House Chips

Google agreed to create an AI cloud business with Blackstone using in-house chips to compete with CoreWeave; the post does not disclose ownership structure, investment size, launch timing, or chip specifications.

Why it matters: Google and Blackstone plan an AI cloud joint venture running on in-house TPUs against GPU cloud providers like CoreWeave, showing one route for big tech's own silicon into cloud compute.

Bloomberg Technology

Inside Meta’s $200 Billion Louisiana Data Center Bet

Meta is building an AI data center in Richland Parish, Louisiana, financed by a $200 billion private-capital deal, with power demand up to 7.5 gigawatts, including 5 gigawatts for computing, supplied by 10 new natural-gas plants.

Why it matters: Meta is raising $20 billion in private financing for a Louisiana AI data center and adding 10 gas plants for up to 7.5 GW, showing how closely compute growth is tied to fossil power.

Bloomberg Technology

Nvidia’s CEO Sees China Opening Market to AI Chips From US

Nvidia CEO Jensen Huang said Chinese authorities will eventually allow imports of US AI chips; the RSS snippet only says he made the comment days after joining President Donald Trump’s summit in China and does not disclose a timetable.

Why it matters: Nvidia's CEO said after a China trip with Trump that China will eventually open its market to US AI chips, without giving a timeline or evidence, while export controls stay tight.

r/LocalLLaMA

llama.cpp MTP support landed: Qwen3.6 27B reaches 2.44× on Strix Halo

llama.cpp merged MTP speculative decoding in PR #22673; Qwen3.6 27B Q8_0 rose from 7.4 to 18.1 tok/s on Strix Halo, while a dual RTX 3090 Q8_0 setup rose from 25.7 to 55.9 tok/s.

Why it matters: After llama.cpp merged MTP speculative decoding, Qwen3.6 27B measured 7.4 tok/s to 18.1 tok/s on Strix Halo, giving local inference a concrete speedup reference.

May 18Monday

Import AI (Jack Clark)

Import AI 457: AI Stuxnet, Cursed Muon Optimizer, and Positive Alignment

Import AI 457 covers fast16, Aurora, and positive alignment: SentinelOne found fewer than 10 matching files for fast16 signatures, while Tilde Research reports Aurora reached 2.26 loss on 1.1B-parameter transformers versus Muon’s 2.31 under a ~100B-token setup.

Why it matters: SentinelOne's teardown of fast16.sys shows it tampering with high-precision results in memory five years before Stuxnet, while Tilde Research finds Muon kills over a quarter of MLP neurons early in training.

r/LocalLLaMA

Qwen 3.6 27B on 24GB VRAM: backend comparisons, quant choice, and settings

The author tested Qwen 3.6 27B on an RTX 3090 24GB and kept ik_llama.cpp with Qwen3.6-27B-MTP-IQ4_KS.gguf; at 156k context with q8_0 KV and MTP, a ~5.9k-token prompt plus 1024-token output reached about 1261 tok/s prefill and 72.9 tok/s decode, while vLLM lacked a clean single-card long-context run.

Why it matters: The author benchmarked several backends and quantization schemes for Qwen 3.6 27B on a single 24GB RTX 3090, settling on ik_llama.cpp with IQ4_KS, as a single-card comparison for local deployment.

Synced · WeChat

ICML 2026: Huawei GTS proposes EDCO for dynamic curriculum fine-tuning

Huawei GTS proposed EDCO, a dynamic curriculum method that selects fine-tuning samples by inference entropy; prefix entropy estimation cuts per-sample scoring time from 2.24 seconds to 0.37 seconds.

Why it matters: Huawei GTS proposed EDCO, a dynamic curriculum fine-tuning method that uses reasoning entropy to judge sample difficulty and pick training data, with prefix-entropy estimation cutting per-sample scoring from 2.24 seconds to 0.37.

r/LocalLLaMA

Benchmarking vLLM vs SGLang vs llama.cpp on a mixed Blackwell/Ada cluster

The author benchmarked long-context prefill on a 7-GPU mixed Blackwell/Ada cluster; on Qwen3.5-397B-A17B with 75k tokens, vLLM reached 9.8s TTFT and 7,683 t/s, while llama.cpp took 57.2s and 1,319 t/s.

Why it matters: A hands-on test of long-context prefill across seven mixed Blackwell and Ada GPUs, with vLLM hitting 9.8s first-token latency at 75k tokens and 7,683 tok/s prefill, roughly 4 to 6 times llama.cpp.

r/LocalLLaMA

LLMs on Android: Snapdragon 8 Elite MoE Experience

A Reddit user tested MoE LLMs on an Honor Magic 7 Pro with Snapdragon 8 Elite and 24GB RAM; under Q4 quantization, LFM2-24b-a2b reached about 24 tokens/s while Gemma reached about 11 tokens/s, and CPU inference was still faster than NPU or GPU in the reported setup.

Why it matters: A Reddit user's real measurements of MoE models on a Snapdragon 8 Elite phone, where CPU inference beat the NPU and LFM2-24b-a2b ran near 24 tok/s at Q4.

May 17Sunday

QbitAI · WeChat

A Robot Dog Challenges Nvidia's Compute Lead

Weilan Technology unveiled BabyAlpha A3, a consumer quadruped robot using a six-chip heterogeneous cluster that runs a 7B-parameter model on-device at 280 TPS; the article says it has 66MP vision, 2.232 million point-cloud samples per second, and a planned Q3 launch.

Why it matters: Azurion runs a 7B model locally on a consumer robot dog using a six-chip heterogeneous cluster, claiming 280 TPS and giving on-device compute a concrete sample.

r/LocalLLaMA

Same Models Tested Across Strix Halo, RTX 3090, and RTX 5070

C_Coffie published 55 local inference benchmark runs across Strix Halo, RTX 3090, RTX 5070, five backends, and 0.35B to 35B-A3B models; RTX 5070 beats RTX 3090 on models fitting 12GiB, while RTX 3090 leads in the 14–31B band that exceeds 12GiB but fits 24GiB.

Why it matters: A user ran 55 local inference tests across three GPUs and five backends, and reports an RTX 5070 decoding faster than an RTX 3090 within 12GB of VRAM.

Dwarkesh Patel podcast

Notes on Pretraining Parallelisms and Failed Training Runs

Dwarkesh documents pretraining failure modes and parallelism tradeoffs: expert choice and token dropping can break causality in MoE routing, FP16 collectives can bias repeated additions after values exceed 1024, pretraining FLOPs are given as 6ND, B300 HBM is listed as 288GB, and FSDP communication can reach params × 3 with reduce-scatter.

Why it matters: The author breaks down two ways pretraining runs go off the rails, and why expert choice in MoE routing makes a token's destination depend on later routing decisions, so training sees information inference never gets.

May 16Saturday

Latent Space

Cerebras' $60B IPO: Slowly, then All at Once

Cerebras closed its IPO at a $60 billion market cap. CFO Bob Komin said it serves trillion-parameter models, including internal OpenAI 5.4 and 5.5 workloads, but the post does not disclose traffic share, latency tier, or cost per token.

Why it matters: Cerebras closed its IPO at a $60B market cap, and its CFO said publicly the company already serves trillion-parameter models, naming the 5.4 and 5.5 versions OpenAI uses internally.

r/LocalLLaMA

Orthrus-Qwen3-8B: Up to 7.8× tokens/forward on Qwen3-8B with frozen backbone

Orthrus-Qwen3-8B adds a trainable diffusion attention head to a frozen Qwen3-8B backbone, reaches up to 7.8× tokens per forward and about 6× wall-clock speedup on MATH-500, while training 16% of parameters with under 1B tokens over 24 hours on 8×H200 GPUs.

Why it matters: The team froze the Qwen3-8B backbone and trained only a diffusion attention head, reaching up to 7.8x token throughput per forward pass on MATH-500 and about 6x faster wall-clock time, with the output distribution unchanged.

Bloomberg Technology

Cerebras CEO Is Worth $3.2 Billion After Year’s Largest IPO

Cerebras Systems rose about 68% on Nasdaq after the year’s largest IPO, giving the company a market value of roughly $67 billion; the headline says its CEO is worth $3.2 billion after the listing.

Why it matters: Cerebras listed on Nasdaq and closed its first day up about 68%, giving a concrete public-market price for an AI compute company.

May 15Friday

r/LocalLLaMA

Fully Offline Suitcase Robot Built Around Jetson Orin NX SUPER 16GB

CreativelyBankrupt built Sparky as a fully offline suitcase robot on Jetson Orin NX SUPER 16GB, running Gemma 4 E4B Q4_K_M via llama.cpp with q8_0 KV cache, about 200 ms cached TTFT, 14-15 tok/s sustained output, 12K context, 30+ sensors, and no WiFi, Bluetooth, or cellular interface.

Why it matters: A solo developer runs Gemma 4 E4B fully offline on a Jetson Orin NX SUPER 16GB mounted on a suitcase, with about 200ms cached TTFT and 14 to 15 tokens per second.