Skip to content

#部署/工程

11 today

May 22Friday

r/LocalLLaMA

Interesting Paper Advocates Quantized Prefilling and Precise Decoding

arXiv 2605.20315 argues for W4A4 quantization during prefilling to target a theoretical 4x gain, while keeping decoding on the original high-precision path because activation errors can perturb sampled tokens and accumulate across autoregressive generation.

Why it matters: The paper proposes W4A4 quantization during prefill while keeping decoding at high precision, claiming up to 4x faster input processing without letting activation quantization error accumulate through autoregressive generation.

NVIDIA Blog

NVIDIA GTC Taipei at COMPUTEX: Live Updates on What’s Next in AI

NVIDIA won four COMPUTEX 2026 Best Choice Awards for Vera Rubin NVL72, Jetson Thor, and Alpamayo; Vera Rubin NVL72 connects 36 Vera CPUs and 72 Rubin GPUs, and NVIDIA says it delivers up to 10x higher inference performance per watt and 10x lower cost per token.

Why it matters: NVIDIA took four Best Choice awards at COMPUTEX 2026 with Vera Rubin NVL72, Jetson Thor and Alpamayo, and NVL72's stated inference efficiency is 10x higher than the previous generation.

May 21Thursday

r/LocalLLaMA

Agent Execution Tax: New Procurement Metric for Browser Agent Benchmarks?

Fireworks ran 720 browser-agent tasks on WebVoyager and reported a 22.9% Agent Execution Tax, defined as wasted over productive inference; MiniMax M2.5 cost 2.3x less per successful task than Gemini, while GLM-5 reached 57.1% accuracy and Kimi K2.5 had 0% parse retries across 852 calls.

Why it matters: Fireworks ran 720 browser agent tasks on WebVoyager and found 22.9% of inference calls were wasted, giving buyers a way to price that hidden cost.

r/LocalLLaMA

LLM planner: pick a rig by use case, model, or budget, or pick models for your rig

totosse17 published the LLMRequirements hardware planner with 60+ build configs, 50+ models, 130 cited tokens-per-second sources, 150+ reviewer videos, multi-region prices, idle and active watts, and a public GitHub data repo.

Why it matters: A public dataset compares 60+ local model rigs and 50+ models on measured generation speed, idle and load power, and prices across regions.

The Verge · AI

Anthropic is paying $15 billion a year for access to Elon Musk’s data centers

SpaceX said in its S-1 filing that Anthropic agreed to pay $1.25 billion per month through May 2029 for access to Colossus I and II AI training centers in Memphis, totaling $15 billion annually.

Why it matters: Anthropic pays SpaceX $1.25B a month through May 2029 for two Memphis AI training sites, an annualized figure near twice SpaceX's 2025 revenue.

r/LocalLLaMA

Tencent Hy-MT2 30B/7B/1.8B

Tencent released Hy-MT2 translation models in 1.8B, 7B, and 30B-A3B sizes, supporting translation across 33 languages; AngelSlim 1.25-bit quantization reduces the 1.8B model’s storage requirement to 440 MB and raises inference speed by 1.5x.

Why it matters: Tencent's Hy-MT2 line covers 1.8B, 7B and 30B-A3B sizes with 33-language translation, and the quantized 1.8B build drops to 440MB of storage.

AI HOT (Curated Pool)

Tencent open-sources Hy-MT2 multilingual translation model

Tencent open-sourced the Hy-MT2 multilingual translation model with support for translation across 33 languages; its 1.8B version uses AngelSlim 1.25-bit quantization, occupies 440 MB of storage, and runs locally on mainstream mobile chipsets.

Why it matters: Tencent open-sourced Hy-MT2; the 1.8B build quantized to 1.25-bit is just 440MB and runs 33-language translation locally on mainstream phone chips.

Latent Space

OpenAI GPT-next Disproves 80-Year-Old Erdős Planar Unit Distance Problem for Under $1000

OpenAI said an internal general-purpose reasoning model disproved the 1946 Erdős planar unit distance problem by finding a new family of constructions; the reasoning summary reportedly spans about 125 pages, while outside observers speculate the run used under 32 hours or under $1,000.

Why it matters: A general OpenAI reasoning model disproved Erdős's 1946 planar unit distance conjecture, producing a roughly 125-page summary; outside observers estimate under 32 hours of runtime and under $1,000 in cost.

Synced · WeChat

VAST and Tsinghua propose density-controlled 3D Gaussian generation for SIGGRAPH 2026

VAST and Tsinghua propose DeG, a 3D Gaussian generation method that samples Gaussian centers from a learned density distribution and trains density control with a render loss contribution gradient; in some settings, it reaches TRELLIS-like visual quality with less than half the Gaussian count.

Why it matters: VAST and Tsinghua propose DeG, which allocates Gaussian points by object detail density and matches TRELLIS-level quality with under half the points.

Synced · WeChat

Zhipu deploys ZCube, raising inference throughput 15% on the same GPUs

Zhipu deployed ZCube in a thousand-GPU GLM-5.1 production inference cluster, replacing ROFT while keeping GPUs, software stack, and business code unchanged; throughput rose by over 15%, TTFT P99 fell 40.6%, and switch plus optical module costs dropped by one third.

Why it matters: Zhipu swapped in its own ZCube networking on a GLM-5.1 thousand-GPU inference cluster, raising throughput over 15% and cutting P99 first-token latency 40.6% without touching GPUs or business code.

Bloomberg Technology

Anthropic to Pay SpaceX Nearly $45 Billion for Computing Deal

Anthropic agreed to pay Elon Musk’s SpaceX nearly $45 billion over the next three years for computing resources to support its Claude AI software, according to a securities filing.

Why it matters: Anthropic signed a three-year compute deal with SpaceX worth about $45 billion, roughly $15 billion a year, giving a concrete reference for what top AI labs pay for compute.

TechCrunch · AI

Anthropic will pay xAI $1.25B per month for compute

Anthropic will pay xAI $1.25 billion per month for compute; the post discloses the deal value but does not disclose compute scale, contract length, or deployment conditions.

Why it matters: Anthropic is renting all the compute at xAI's Colossus 1 data center in Memphis for $1.25B a month through May 2029, per SpaceX filings, letting xAI monetize idle capacity.

r/LocalLLaMA

What happened to Cohere’s Command-A series of models?

Cohere launched Command A+, describing it as its first MoE model under the Apache 2.0 license, with quantization work that lets it run well on 1 or 2 GPUs; the post says top-line performance still needs work.

Why it matters: Cohere co-founder Nick Frosst posted Command A+ on Reddit, an Apache 2.0 open-source mixture-of-experts model that runs quantized on one or two consumer GPUs.

Bloomberg Technology

Nvidia Beats on Earnings, Revenue Projected at $91 Billion

Nvidia reported fiscal first-quarter earnings of $1.87 per share, above the $1.77 estimate; the company projected revenue of $91 billion for the quarter ending in July, above Wall Street expectations of about $87.4 billion.

Why it matters: Nvidia posted Q1 EPS of $1.87 against Wall Street's $1.77 estimate and guided next-quarter revenue to $91B, above the $87.4B consensus, a direct read on AI infrastructure demand.

AI HOT (Curated Pool)

Nvidia fiscal Q1 2027 net income reached $58.321 billion, up 211% YoY

Nvidia reported fiscal Q1 2027 revenue of $81.615 billion and net income of $58.321 billion, while data center revenue reached $75.2 billion and the company guided fiscal Q2 revenue to $91 billion.

Why it matters: Nvidia's fiscal 2027 Q1 net income hit $58.321B, up 211% year over year, with data center revenue of $75.2B, over 90% of total revenue, and gross margin near 75%.

AI HOT (Curated Pool)

Meta restructures 15,000 roles with layoffs and AI shift

Meta plans to cut about 8,000 jobs and move about 7,000 employees into AI-related roles, concentrating resources on AI infrastructure, foundation model development, and commercialization from model training to product work and profit generation.

Why it matters: Meta is cutting about 8,000 jobs and moving roughly 7,000 people into AI roles, concentrating resources on AI infrastructure, foundation models and commercialization.

AI HOT (Curated Pool)

Context compression improves search efficiency and accuracy

Perplexity has deployed query-aware compression in production, reducing context tokens by up to 70% while improving search answer quality.

Why it matters: Perplexity uses query-aware compression in live search, cutting context tokens fed to the model by up to 70% while answer quality rises, a reference point for compressing without losing quality.

May 20Wednesday

Alibaba Technology · WeChat

Zhenwu M890 AI Chip Debuts as Agentic Compute Foundation

Alibaba released a 128-card supernode server based on T-Head’s Zhenwu M890 AI chip, with P2P latency below 150 ns and rack bandwidth at the Pb/s level; it is live on Alibaba Cloud Bailian and supports Qwen, DeepSeek, and Kimi.

Why it matters: Alibaba's T-Head unveiled the Zhenwu M890 AI chip, with a 128-card supernode server already live on the Bailian platform supporting Qwen, DeepSeek and Kimi, showing where domestic AI chips land in agent workloads.

Xinzhiyuan · WeChat

Behind Jensen Huang’s Douzhi Moment, Chinese GPUs Are Filling CUDA’s Moat

Moore Threads presented progress on its MUSA GPU ecosystem, with SDK 5.1.0 targeting CUDA 12.8 and supporting 761 driver and runtime APIs. The post says MUSA has entered SGLang’s mainline, is listed for 2026 Q2 hardware support, and supports automated library migration via MUSACODE.

Why it matters: Moore Threads SDK 5.1.0 targets CUDA 12.8 compatibility across 761 driver and runtime interfaces, SGLang mainline merged MUSA support, and hardware is slated for Q2 2026.

r/LocalLLaMA

Running DeepSeek-V4 locally on 4 legacy RTX 2080 Ti GPUs with W8A8 at 255 prefill tok/s

A Reddit user ran DeepSeek-V4-Flash locally on 4 RTX 2080 Ti GPUs, reporting 284B total parameters, 13B active parameters, a sub-$2,500 build, custom Turing CUDA kernels, W8A8 quantization, 1TB DDR4 ECC RAM, and about 255 prefill tokens/s.

Why it matters: A Reddit user ran the 284B-parameter DeepSeek-V4-Flash locally on four used RTX 2080 Ti cards for under $2,500, using W8A8 quantization and custom Turing CUDA kernels.

AI HOT (Curated Pool)

Unsustainable Subsidies

Google, OpenAI, and Anthropic diverged on model pricing: Gemini 3.1 Pro is priced at $2 input and $12 output, GPT-5.5 at $5 and $30 after a short subsidy, and Claude Opus 4.7 stayed at $5 and $25.

Why it matters: A side-by-side of three flagship model APIs: Gemini 3.1 Pro at $2 per million input tokens and $12 output, GPT-5.5 at $5 input and $30 output after subsidy.

AI HOT (Curated Pool)

Gemini 3.5 Flash price rises sharply as Google plans broad rollout

Google released Gemini 3.5 Flash at I/O with $1.50 per million input tokens and $9 per million output tokens, making it 3x and 6x the previous model’s pricing, while adding roughly 1 million input tokens and about 65,000 maximum output tokens.

Why it matters: Google skipped the preview and shipped Gemini 3.5 Flash at I/O, priced at $1.50 and $9 per million tokens, close to its own Pro model and far above the previous Flash and Flash-Lite.

AI HOT (Curated Pool)

OpenAI launches Guaranteed Capacity for long-term compute access

OpenAI launched Guaranteed Capacity, a service for customers to secure long-term access to OpenAI compute and plan critical workloads under capacity constraints; the post does not disclose pricing, contract duration, or quota levels.

Why it matters: OpenAI now sells reserved compute, a sign that capacity, not model quality, is what enterprise buyers are negotiating over.

AI HOT (Curated Pool)

Google Tensor ML SDK Beta Released

Google released the Tensor ML SDK beta, letting developers convert, compile, and run PyTorch or TFLite models on Pixel 10 TPUs through LiteRT, with a model library containing more than 100 classic and generative AI models, including Gemma 3.

Why it matters: Developers can now run their own PyTorch or TFLite models on the Pixel 10's TPU, which makes on-device inference testable without Google's cloud.

r/LocalLLaMA

Nemotron-Labs-Diffusion from NVIDIA

NVIDIA released the Nemotron-Labs-Diffusion 3B, 8B, and 14B dense model family with AR decoding, diffusion parallel decoding, and self-speculation; the 8B model reaches 850 tok/s on GB200 at concurrency 1, compared with 253 tok/s for AR and 360 tok/s for Eagle3.

Why it matters: NVIDIA's Nemotron-Labs-Diffusion ships three dense models from 3B to 14B that decode both autoregressively and by diffusion, plus a self-speculation speedup.

AI HOT (Curated Pool)

Gemini 3.5 Flash launches with stronger performance and speed

Google opened Gemini 3.5 Flash after Google I/O across its products and API; the post says it outperforms Gemini 3.1 Pro on most benchmarks and generates tokens 4x faster than other frontier models.

Why it matters: Gemini 3.5 Flash is now open to product teams and API users, with Google claiming it beats 3.1 Pro on most benchmarks.

AI HOT (Curated Pool)

Google releases Gemini 3.5 Flash with output speed about 4x GPT-5.5

Google introduced Gemini 3.5 Flash at I/O 2026, with output speed reaching 289 tokens per second, about 4x faster than Claude Opus 4.7 and GPT-5.5 xhigh under the cited comparison.

Why it matters: Google's Gemini 3.5 Flash hits 289 output tokens per second, about four times faster than GPT-5.5 xhigh and Claude Opus 4.7.

r/LocalLLaMA

KV cache quantization benchmarks: TurboQuant is overrated, q5 deserves attention, q8 may waste VRAM

Anbeeld benchmarked KV cache quantization for Qwen 3.6 27B on one RTX 3090 at 64k and 128k context, reporting q4_0 tail KLD 32% worse than q5_0 and turbo4 running 17% slower than q4_0 with little memory saving.

Why it matters: Anbeeld benchmarks Qwen 3.6 27B KV cache quantization on an RTX 3090, with numbers like q4_0's tail KLD 32% worse than q5_0.

AI HOT (Curated Pool)

Google launches Antigravity 2.0 platform, builds an OS in 12 hours

Google announced Antigravity 2.0 at I/O and demonstrated an agent building a runnable operating system from scratch in 12 hours, using 93 parallel sub-agents, more than 15,000 model calls, and 2.6 billion tokens, with API costs under $1,000.

Why it matters: Google's I/O demo used 93 parallel sub-agents, 15,000 model calls and 2.6 billion tokens to build a working OS in 12 hours for under $1,000 in API cost.

AI HOT (Curated Pool)

Gemini 3.5 Flash launches as an efficient option for task handling

Google released Gemini 3.5 Flash and calls it its best model so far for fast, efficient task completion. The post does not disclose pricing, context window size, benchmark scores, or API availability conditions.

Why it matters: Google's Gemini 3.5 Flash targets fast, efficient everyday tasks and multi-step creative work, with pricing and context window still undisclosed.

r/LocalLLaMA

Floor for local meeting summarization on a 6GB GPU: Qwen3.5 0.8B works in 57s, Granite 4 350M hallucinates

The author tested VoiceFlow 1.6.0 on an RTX 3060 Laptop 6GB, where Qwen3.5 0.8B summarized a 4-minute meeting in 57 seconds with 16K context, while Granite 4 350M returned summaries in 0.6-2.8 seconds but fabricated Binance and Star Trek content.

Why it matters: A user tested two small models on a 6GB RTX 3060 laptop: Qwen3.5 0.8B summarized a 4-minute meeting in 57 seconds, while the faster Granite 4 350M made things up.

AI HOT (Curated Pool)

NVIDIA open-sources first 4-bit infrastructure for ultra-long video generation

NVIDIA researchers open-sourced LongLive 2.0, an end-to-end long-video generation infrastructure covering training and inference with 4-bit quantization, FP4 quantization, parallel acceleration, KV-cache optimization, and 45.7 FPS generation on a 5B model.

Why it matters: NVIDIA open-sourced all training and inference code for LongLive 2.0, hitting 45.7 fps at 4-bit quantization on a 5B-parameter model, a reusable path to cheaper long-video generation.

May 19Tuesday

Hacker News front page

Show HN: Forge takes an 8B model from 53% to 99% on agentic tasks

Forge adds five guardrail layers to self-hosted LLM tool calling, raising Ministral 8B to 99.3% across 18 multi-step agentic scenarios, with the accepted ACM CAIS ’26 paper covering 97 model/backend configurations and 50 runs per scenario.

Why it matters: Forge used five layers of guardrails to lift Ministral 8B from 53% to 99.3% task success across 18 multi-step agent scenarios, covering 97 model and backend combinations.

AI HOT (Curated Pool)

Horizon Open-Sources 400M-Parameter Robot Control Model HoloMotion-1

Horizon Robotics Lab open-sourced HoloMotion-1, a 400M-parameter full-body humanoid control model that uses MoE sparse activation and KV-cache inference to reach about 300 FPS on-device, with code and a technical report released.

Why it matters: Horizon Robotics' lab open-sourced HoloMotion-1, a 400M-parameter whole-body control model for humanoid robots, reaching about 300FPS on-device via MoE sparse activation and KV-cache, well above the usual 50Hz control rate.

Latent Space

[AINews] How to Land a Job at a Frontier Lab (on Pretraining)

Latent Space says Vlad Feinberg’s pretraining job-prep notes reduce frontier-lab readiness to kernel-level performance work: derive Chinchilla laws, compare dense and MoE architectures, code the solution in JAX, then write a Pallas kernel that beats jax.lax.ragged_dot for F > D by fusing up/down projections.

Why it matters: A job-hunting note from Vlad Feinberg lays out what pretraining work at a frontier lab actually tests: low-level performance tuning, deriving Chinchilla scaling laws, and comparing dense models with MoE architectures.

QbitAI · WeChat

Chinese GPU vendor Moore Threads releases MT Lambda for embodied AI simulation

Moore Threads released MT Lambda, an embodied AI simulation platform that combines physics, rendering, and AI engines, and demonstrated the robot dog “Xiaofei” executing a Sim-to-Real policy trained 100% in simulation on domestic hardware.

Why it matters: Moore Threads released the MT Lambda simulation platform, which combines physics, rendering and AI training on its own GPUs; the robot dog Xiaofei learned to walk in simulation and moved straight to hardware.

QbitAI · WeChat

World model supports multiplayer FPS gameplay before Fei-Fei Li

Odyssey released Agora-1, a world model that supports up to four human and AI players fighting in the same generated FPS world in real time. The system decouples simulation from rendering and trains on GoldenEye internal game states.

Why it matters: Odyssey released the Agora-1 world model, which puts four human or AI players in one FPS map, trains on internal game states from GoldenEye, and runs physics simulation separately from rendering.

Synced · WeChat

Recent LLM Architecture Changes: From Gemma 4 to DeepSeek V4

Jiqizhixin translated Sebastian Raschka’s blog on recent LLM architecture changes, covering long-context cost reductions in Gemma 4, Laguna XS.2, and ZAYA1-8B; the article states that Gemma 4 E2B saves about 2.7GB of KV cache at 128K context with bfloat16 precision.

Why it matters: Raschka walks through architecture changes in Gemma 4 through DeepSeek V4, all aimed at cutting long-context memory cost.

Financial Times · Technology

Google makes chip push with Blackstone-backed AI cloud group

A Blackstone-backed AI cloud group is set to receive a $5 billion investment to bring 500MW of data center capacity online next year; the post does not disclose the Google chip terms or deployment structure.

Why it matters: A Blackstone-backed AI cloud group raised $5bn and plans 500MW of data center capacity next year, with the Google chip tie-up still undisclosed.

AI HOT (Curated Pool)

Google and Blackstone form AI cloud company with $5B initial equity and 500 MW target by 2027

Google and Blackstone formed an AI cloud services company with Blackstone committing $5 billion in initial equity capital, an expected total investment of about $25 billion after leverage, and a plan to bring 500 megawatts of data center capacity online in 2027.

Why it matters: Google and Blackstone are forming an AI cloud company, with Blackstone putting in $5 billion of equity first against a $25 billion total, aiming for 500 MW of data centers by 2027.