Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

341–360 of 609

May 21Thursday

AI HOT (Curated Pool)

Context compression improves search efficiency and accuracy

Perplexity has deployed query-aware compression in production, reducing context tokens by up to 70% while improving search answer quality.

Why it matters: Perplexity uses query-aware compression in live search, cutting context tokens fed to the model by up to 70% while answer quality rises, a reference point for compressing without losing quality.

May 20Wednesday

Alibaba Technology · WeChat

Zhenwu M890 AI Chip Debuts as Agentic Compute Foundation

Alibaba released a 128-card supernode server based on T-Head’s Zhenwu M890 AI chip, with P2P latency below 150 ns and rack bandwidth at the Pb/s level; it is live on Alibaba Cloud Bailian and supports Qwen, DeepSeek, and Kimi.

Why it matters: Alibaba's T-Head unveiled the Zhenwu M890 AI chip, with a 128-card supernode server already live on the Bailian platform supporting Qwen, DeepSeek and Kimi, showing where domestic AI chips land in agent workloads.

Xinzhiyuan · WeChat

Behind Jensen Huang’s Douzhi Moment, Chinese GPUs Are Filling CUDA’s Moat

Moore Threads presented progress on its MUSA GPU ecosystem, with SDK 5.1.0 targeting CUDA 12.8 and supporting 761 driver and runtime APIs. The post says MUSA has entered SGLang’s mainline, is listed for 2026 Q2 hardware support, and supports automated library migration via MUSACODE.

Why it matters: Moore Threads SDK 5.1.0 targets CUDA 12.8 compatibility across 761 driver and runtime interfaces, SGLang mainline merged MUSA support, and hardware is slated for Q2 2026.

r/LocalLLaMA

Running DeepSeek-V4 locally on 4 legacy RTX 2080 Ti GPUs with W8A8 at 255 prefill tok/s

A Reddit user ran DeepSeek-V4-Flash locally on 4 RTX 2080 Ti GPUs, reporting 284B total parameters, 13B active parameters, a sub-$2,500 build, custom Turing CUDA kernels, W8A8 quantization, 1TB DDR4 ECC RAM, and about 255 prefill tokens/s.

Why it matters: A Reddit user ran the 284B-parameter DeepSeek-V4-Flash locally on four used RTX 2080 Ti cards for under $2,500, using W8A8 quantization and custom Turing CUDA kernels.

AI HOT (Curated Pool)

Unsustainable Subsidies

Google, OpenAI, and Anthropic diverged on model pricing: Gemini 3.1 Pro is priced at $2 input and $12 output, GPT-5.5 at $5 and $30 after a short subsidy, and Claude Opus 4.7 stayed at $5 and $25.

Why it matters: A side-by-side of three flagship model APIs: Gemini 3.1 Pro at $2 per million input tokens and $12 output, GPT-5.5 at $5 input and $30 output after subsidy.

AI HOT (Curated Pool)

Gemini 3.5 Flash price rises sharply as Google plans broad rollout

Google released Gemini 3.5 Flash at I/O with $1.50 per million input tokens and $9 per million output tokens, making it 3x and 6x the previous model’s pricing, while adding roughly 1 million input tokens and about 65,000 maximum output tokens.

Why it matters: Google skipped the preview and shipped Gemini 3.5 Flash at I/O, priced at $1.50 and $9 per million tokens, close to its own Pro model and far above the previous Flash and Flash-Lite.

AI HOT (Curated Pool)

OpenAI launches Guaranteed Capacity for long-term compute access

OpenAI launched Guaranteed Capacity, a service for customers to secure long-term access to OpenAI compute and plan critical workloads under capacity constraints; the post does not disclose pricing, contract duration, or quota levels.

Why it matters: OpenAI now sells reserved compute, a sign that capacity, not model quality, is what enterprise buyers are negotiating over.

AI HOT (Curated Pool)

Google Tensor ML SDK Beta Released

Google released the Tensor ML SDK beta, letting developers convert, compile, and run PyTorch or TFLite models on Pixel 10 TPUs through LiteRT, with a model library containing more than 100 classic and generative AI models, including Gemma 3.

Why it matters: Developers can now run their own PyTorch or TFLite models on the Pixel 10's TPU, which makes on-device inference testable without Google's cloud.

r/LocalLLaMA

Nemotron-Labs-Diffusion from NVIDIA

NVIDIA released the Nemotron-Labs-Diffusion 3B, 8B, and 14B dense model family with AR decoding, diffusion parallel decoding, and self-speculation; the 8B model reaches 850 tok/s on GB200 at concurrency 1, compared with 253 tok/s for AR and 360 tok/s for Eagle3.

Why it matters: NVIDIA's Nemotron-Labs-Diffusion ships three dense models from 3B to 14B that decode both autoregressively and by diffusion, plus a self-speculation speedup.

AI HOT (Curated Pool)

Gemini 3.5 Flash launches with stronger performance and speed

Google opened Gemini 3.5 Flash after Google I/O across its products and API; the post says it outperforms Gemini 3.1 Pro on most benchmarks and generates tokens 4x faster than other frontier models.

Why it matters: Gemini 3.5 Flash is now open to product teams and API users, with Google claiming it beats 3.1 Pro on most benchmarks.

AI HOT (Curated Pool)

Google releases Gemini 3.5 Flash with output speed about 4x GPT-5.5

Google introduced Gemini 3.5 Flash at I/O 2026, with output speed reaching 289 tokens per second, about 4x faster than Claude Opus 4.7 and GPT-5.5 xhigh under the cited comparison.

Why it matters: Google's Gemini 3.5 Flash hits 289 output tokens per second, about four times faster than GPT-5.5 xhigh and Claude Opus 4.7.

r/LocalLLaMA

KV cache quantization benchmarks: TurboQuant is overrated, q5 deserves attention, q8 may waste VRAM

Anbeeld benchmarked KV cache quantization for Qwen 3.6 27B on one RTX 3090 at 64k and 128k context, reporting q4_0 tail KLD 32% worse than q5_0 and turbo4 running 17% slower than q4_0 with little memory saving.

Why it matters: Anbeeld benchmarks Qwen 3.6 27B KV cache quantization on an RTX 3090, with numbers like q4_0's tail KLD 32% worse than q5_0.

AI HOT (Curated Pool)

Google launches Antigravity 2.0 platform, builds an OS in 12 hours

Google announced Antigravity 2.0 at I/O and demonstrated an agent building a runnable operating system from scratch in 12 hours, using 93 parallel sub-agents, more than 15,000 model calls, and 2.6 billion tokens, with API costs under $1,000.

Why it matters: Google's I/O demo used 93 parallel sub-agents, 15,000 model calls and 2.6 billion tokens to build a working OS in 12 hours for under $1,000 in API cost.

AI HOT (Curated Pool)

Gemini 3.5 Flash launches as an efficient option for task handling

Google released Gemini 3.5 Flash and calls it its best model so far for fast, efficient task completion. The post does not disclose pricing, context window size, benchmark scores, or API availability conditions.

Why it matters: Google's Gemini 3.5 Flash targets fast, efficient everyday tasks and multi-step creative work, with pricing and context window still undisclosed.

r/LocalLLaMA

Floor for local meeting summarization on a 6GB GPU: Qwen3.5 0.8B works in 57s, Granite 4 350M hallucinates

The author tested VoiceFlow 1.6.0 on an RTX 3060 Laptop 6GB, where Qwen3.5 0.8B summarized a 4-minute meeting in 57 seconds with 16K context, while Granite 4 350M returned summaries in 0.6-2.8 seconds but fabricated Binance and Star Trek content.

Why it matters: A user tested two small models on a 6GB RTX 3060 laptop: Qwen3.5 0.8B summarized a 4-minute meeting in 57 seconds, while the faster Granite 4 350M made things up.

AI HOT (Curated Pool)

NVIDIA open-sources first 4-bit infrastructure for ultra-long video generation

NVIDIA researchers open-sourced LongLive 2.0, an end-to-end long-video generation infrastructure covering training and inference with 4-bit quantization, FP4 quantization, parallel acceleration, KV-cache optimization, and 45.7 FPS generation on a 5B model.

Why it matters: NVIDIA open-sourced all training and inference code for LongLive 2.0, hitting 45.7 fps at 4-bit quantization on a 5B-parameter model, a reusable path to cheaper long-video generation.

May 19Tuesday

Hacker News front page

Show HN: Forge takes an 8B model from 53% to 99% on agentic tasks

Forge adds five guardrail layers to self-hosted LLM tool calling, raising Ministral 8B to 99.3% across 18 multi-step agentic scenarios, with the accepted ACM CAIS ’26 paper covering 97 model/backend configurations and 50 runs per scenario.

Why it matters: Forge used five layers of guardrails to lift Ministral 8B from 53% to 99.3% task success across 18 multi-step agent scenarios, covering 97 model and backend combinations.

AI HOT (Curated Pool)

Horizon Open-Sources 400M-Parameter Robot Control Model HoloMotion-1

Horizon Robotics Lab open-sourced HoloMotion-1, a 400M-parameter full-body humanoid control model that uses MoE sparse activation and KV-cache inference to reach about 300 FPS on-device, with code and a technical report released.

Why it matters: Horizon Robotics' lab open-sourced HoloMotion-1, a 400M-parameter whole-body control model for humanoid robots, reaching about 300FPS on-device via MoE sparse activation and KV-cache, well above the usual 50Hz control rate.

Latent Space

[AINews] How to Land a Job at a Frontier Lab (on Pretraining)

Latent Space says Vlad Feinberg’s pretraining job-prep notes reduce frontier-lab readiness to kernel-level performance work: derive Chinchilla laws, compare dense and MoE architectures, code the solution in JAX, then write a Pallas kernel that beats jax.lax.ragged_dot for F > D by fusing up/down projections.

Why it matters: A job-hunting note from Vlad Feinberg lays out what pretraining work at a frontier lab actually tests: low-level performance tuning, deriving Chinchilla scaling laws, and comparing dense models with MoE architectures.

QbitAI · WeChat

Chinese GPU vendor Moore Threads releases MT Lambda for embodied AI simulation

Moore Threads released MT Lambda, an embodied AI simulation platform that combines physics, rendering, and AI engines, and demonstrated the robot dog “Xiaofei” executing a Sim-to-Real policy trained 100% in simulation on domestic hardware.

Why it matters: Moore Threads released the MT Lambda simulation platform, which combines physics, rendering and AI training on its own GPUs; the robot dog Xiaofei learned to walk in simulation and moved straight to hardware.