Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

321–340 of 609

May 22Friday

AI HOT (Curated Pool)

Zhipu releases GLM-5.1-highspeed, claiming a large-model API speed record

Zhipu released the GLM-5.1-highspeed API to selected enterprise customers on May 22, with a claimed output speed of 400 tokens/s, built by the GLM team and TileRT team through system-level optimization.

Why it matters: Zhipu opened a high-speed GLM-5.1 endpoint to some enterprise customers at a claimed 400 tokens per second, a case for watching how flagship quality trades off against low latency in production.

New York Times Chinese

Trump Approved Nvidia Chip Sales to China. Why Is Beijing Reluctant?

Trump approved Nvidia H200 sales to China six months ago, but Beijing has not allowed any company to buy even one chip and is steering firms toward domestic alternatives from Huawei and Cambricon.

Why it matters: Six months after Trump approved H200 sales to China, Beijing has still not let any company place orders and is steering compute demand to Huawei, Cambricon and other domestic options.

Computing Life · Share · Yage

How to Run DeepSeek V4 Flash Locally on Mac: DS4 Engine Explained

DS4 provides a macOS local runtime path for DeepSeek V4 Flash; the post only discloses three mechanisms—multi-agent integration, KV cache disk persistence, and activation steering—and does not disclose performance numbers, hardware requirements, or pricing.

Why it matters: antirez wrote DS4, a pure C, Metal-first inference engine for DeepSeek V4 Flash at 284B total and 13B active parameters, filling a gap where mainstream engines cannot run it on a Mac.

Computing Life · Share · Yage

The Technology Behind GLM-5.1 Reaching 400 Tokens/s

Zhipu GLM-5.1 high-speed API claims 400 tokens/s, and the post says TileRT reconstructs GPU inference at the execution-model level; the RSS snippet does not disclose benchmark conditions, hardware, pricing, or latency distribution.

Why it matters: Zhipu released a high-speed GLM-5.1 API at a claimed 400 tokens per second, with its TileRT engine moving the GPU from batch processing to a continuous pipeline with no waits between steps.

r/LocalLLaMA

Interesting Paper Advocates Quantized Prefilling and Precise Decoding

arXiv 2605.20315 argues for W4A4 quantization during prefilling to target a theoretical 4x gain, while keeping decoding on the original high-precision path because activation errors can perturb sampled tokens and accumulate across autoregressive generation.

Why it matters: The paper proposes W4A4 quantization during prefill while keeping decoding at high precision, claiming up to 4x faster input processing without letting activation quantization error accumulate through autoregressive generation.

NVIDIA Blog

NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI

NVIDIA won four COMPUTEX 2026 Best Choice Awards for Vera Rubin NVL72, Jetson Thor, and Alpamayo; Vera Rubin NVL72 connects 36 Vera CPUs and 72 Rubin GPUs, and NVIDIA says it delivers up to 10x higher inference performance per watt and 10x lower cost per token.

Why it matters: NVIDIA took four Best Choice awards at COMPUTEX 2026 with Vera Rubin NVL72, Jetson Thor and Alpamayo, and NVL72's stated inference efficiency is 10x higher than the previous generation.

May 21Thursday

r/LocalLLaMA

Agent Execution Tax: New Procurement Metric for Browser Agent Benchmarks?

Fireworks ran 720 browser-agent tasks on WebVoyager and reported a 22.9% Agent Execution Tax, defined as wasted over productive inference; MiniMax M2.5 cost 2.3x less per successful task than Gemini, while GLM-5 reached 57.1% accuracy and Kimi K2.5 had 0% parse retries across 852 calls.

Why it matters: Fireworks ran 720 browser agent tasks on WebVoyager and found 22.9% of inference calls were wasted, giving buyers a way to price that hidden cost.

r/LocalLLaMA

LLM planner: pick a rig by use case, model, or budget, or pick models for your rig

totosse17 published the LLMRequirements hardware planner with 60+ build configs, 50+ models, 130 cited tokens-per-second sources, 150+ reviewer videos, multi-region prices, idle and active watts, and a public GitHub data repo.

Why it matters: A public dataset compares 60+ local model rigs and 50+ models on measured generation speed, idle and load power, and prices across regions.

The Verge · AI

Anthropic is paying $15 billion a year for access to Elon Musk’s data centers

SpaceX said in its S-1 filing that Anthropic agreed to pay $1.25 billion per month through May 2029 for access to Colossus I and II AI training centers in Memphis, totaling $15 billion annually.

Why it matters: Anthropic pays SpaceX $1.25B a month through May 2029 for two Memphis AI training sites, an annualized figure near twice SpaceX's 2025 revenue.

r/LocalLLaMA

Tencent Hy-MT2 30B/7B/1.8B

Tencent released Hy-MT2 translation models in 1.8B, 7B, and 30B-A3B sizes, supporting translation across 33 languages; AngelSlim 1.25-bit quantization reduces the 1.8B model’s storage requirement to 440 MB and raises inference speed by 1.5x.

Why it matters: Tencent's Hy-MT2 line covers 1.8B, 7B and 30B-A3B sizes with 33-language translation, and the quantized 1.8B build drops to 440MB of storage.

AI HOT (Curated Pool)

Tencent open-sources Hy-MT2 multilingual translation model

Tencent open-sourced the Hy-MT2 multilingual translation model with support for translation across 33 languages; its 1.8B version uses AngelSlim 1.25-bit quantization, occupies 440 MB of storage, and runs locally on mainstream mobile chipsets.

Why it matters: Tencent open-sourced Hy-MT2; the 1.8B build quantized to 1.25-bit is just 440MB and runs 33-language translation locally on mainstream phone chips.

Latent Space

OpenAI GPT-next Disproves 80-Year-Old Erdős Planar Unit Distance Problem for Under $1000

OpenAI said an internal general-purpose reasoning model disproved the 1946 Erdős planar unit distance problem by finding a new family of constructions; the reasoning summary reportedly spans about 125 pages, while outside observers speculate the run used under 32 hours or under $1,000.

Why it matters: A general OpenAI reasoning model disproved Erdős's 1946 planar unit distance conjecture, producing a roughly 125-page summary; outside observers estimate under 32 hours of runtime and under $1,000 in cost.

Synced · WeChat

VAST and Tsinghua propose density-controlled 3D Gaussian generation for SIGGRAPH 2026

VAST and Tsinghua propose DeG, a 3D Gaussian generation method that samples Gaussian centers from a learned density distribution and trains density control with a render loss contribution gradient; in some settings, it reaches TRELLIS-like visual quality with less than half the Gaussian count.

Why it matters: VAST and Tsinghua propose DeG, which allocates Gaussian points by object detail density and matches TRELLIS-level quality with under half the points.

Synced · WeChat

Zhipu deploys ZCube, raising inference throughput 15% on the same GPUs

Zhipu deployed ZCube in a thousand-GPU GLM-5.1 production inference cluster, replacing ROFT while keeping GPUs, software stack, and business code unchanged; throughput rose by over 15%, TTFT P99 fell 40.6%, and switch plus optical module costs dropped by one third.

Why it matters: Zhipu swapped in its own ZCube networking on a GLM-5.1 thousand-GPU inference cluster, raising throughput over 15% and cutting P99 first-token latency 40.6% without touching GPUs or business code.

Bloomberg Technology

Anthropic to Pay SpaceX Nearly $45 Billion for Computing Deal

Anthropic agreed to pay Elon Musk’s SpaceX nearly $45 billion over the next three years for computing resources to support its Claude AI software, according to a securities filing.

Why it matters: Anthropic signed a three-year compute deal with SpaceX worth about $45 billion, roughly $15 billion a year, giving a concrete reference for what top AI labs pay for compute.

TechCrunch · AI

Anthropic will pay xAI $1.25B per month for compute

Anthropic will pay xAI $1.25 billion per month for compute; the post discloses the deal value but does not disclose compute scale, contract length, or deployment conditions.

Why it matters: Anthropic is renting all the compute at xAI's Colossus 1 data center in Memphis for $1.25B a month through May 2029, per SpaceX filings, letting xAI monetize idle capacity.

r/LocalLLaMA

What happened to Cohere’s Command-A series of models?

Cohere launched Command A+, describing it as its first MoE model under the Apache 2.0 license, with quantization work that lets it run well on 1 or 2 GPUs; the post says top-line performance still needs work.

Why it matters: Cohere co-founder Nick Frosst posted Command A+ on Reddit, an Apache 2.0 open-source mixture-of-experts model that runs quantized on one or two consumer GPUs.

Bloomberg Technology

Nvidia Beats on Earnings, Revenue Projected at $91 Billion

Nvidia reported fiscal first-quarter earnings of $1.87 per share, above the $1.77 estimate; the company projected revenue of $91 billion for the quarter ending in July, above Wall Street expectations of about $87.4 billion.

Why it matters: Nvidia posted Q1 EPS of $1.87 against Wall Street's $1.77 estimate and guided next-quarter revenue to $91B, above the $87.4B consensus, a direct read on AI infrastructure demand.

AI HOT (Curated Pool)

Nvidia fiscal Q1 2027 net income reached $58.321 billion, up 211% YoY

Nvidia reported fiscal Q1 2027 revenue of $81.615 billion and net income of $58.321 billion, while data center revenue reached $75.2 billion and the company guided fiscal Q2 revenue to $91 billion.

Why it matters: Nvidia's fiscal 2027 Q1 net income hit $58.321B, up 211% year over year, with data center revenue of $75.2B, over 90% of total revenue, and gross margin near 75%.

AI HOT (Curated Pool)

Meta restructures 15,000 roles with layoffs and AI shift

Meta plans to cut about 8,000 jobs and move about 7,000 employees into AI-related roles, concentrating resources on AI infrastructure, foundation model development, and commercialization from model training to product work and profit generation.

Why it matters: Meta is cutting about 8,000 jobs and moving roughly 7,000 people into AI roles, concentrating resources on AI infrastructure, foundation models and commercialization.