Skip to content

#部署/工程

11 today

May 27Wednesday

Synced · WeChat

AMD paper: FP4 training instability is not caused by insufficient randomness

AMD and Penn State pretrained Llama 3.1-8B with MXFP4 on MI355X native FP4 hardware, achieving 9-10% end-to-end speedup over an FP8 baseline, while the paper identifies Wgrad quantization as the bottleneck that raises token overhead to 26-27% without deterministic Hadamard stabilization.

Why it matters: AMD and Penn State pretrained Llama 3.1-8B on MI355X with native FP4, running 9-10% faster end to end than an FP8 baseline, while weight gradient quantization added 26-27% cost per token.

Latent Space

[AINews] New AI Infra Decacorns: Fireworks, Baseten, with OpenRouter on the Way

Latent Space says Fireworks is in talks for a $15 billion valuation round, Baseten is raising at an $11 billion valuation, and OpenRouter closed a $113 million Series C after volume grew 5x in six months.

Why it matters: Fireworks and Baseten are in funding talks at $15 billion and $11 billion valuations, with the jumps coming in 7 months and 3 months, showing how fast capital is concentrating on the inference layer.

AI HOT (Curated Pool)

Qualcomm and ByteDance reportedly partner on AI ASIC chips with millions of units planned

The title says Qualcomm and ByteDance reached an AI ASIC chip partnership with procurement in the millions of units; the post does not disclose chip specifications, unit pricing, delivery timing, or production conditions.

Why it matters: Qualcomm signed an AI ASIC custom chip deal with ByteDance for millions of units to run ByteDance's AI services, a concrete case of custom silicon moving from internal projects to volume production.

Bloomberg Technology

Fireworks AI in Talks for Funding at $15 Billion Valuation

Fireworks AI is in talks to raise a new funding round at a $15 billion valuation, according to people familiar with the matter; the post does not disclose the round size, investor names, or timeline.

Why it matters: Fireworks AI is negotiating a new round at a reported $15 billion post-money valuation, a number that can be compared against other AI infrastructure players.

AI HOT (Curated Pool)

Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL

Hugging Face merged TRL PR 5417 for delta weight sync, sending only changed weights as sparse safetensors via a Hugging Face Bucket; on Qwen3-0.6B, the per-step payload falls from 1.2GB to 20–35MB.

Why it matters: Hugging Face implemented incremental weight sync in TRL, sending only the weights that changed between optimizer steps, cutting the per-step payload for Qwen3-0.6B from 1.2GB to 20-35MB.

AI HOT (Curated Pool)

MiMo 2.5 Pro Gets Major Price Cut, Matching DeepSeek V4 Pro

Xiaomi permanently cut MiMo-V2.5 API prices by up to 99%, matched DeepSeek V4 Pro pricing, increased same-price token allowances by 5–8x, reset existing user quotas in full, and set the new pricing to take effect on May 26.

Why it matters: Xiaomi permanently cut MiMo-V2.5 series API prices, by up to 99%, matching DeepSeek V4 Pro, with 5 to 8 times more usable tokens at the same price.

r/LocalLLaMA

PrismML Released Binary and Ternary Bonsai Image 4B

PrismML released Binary and Ternary Bonsai Image 4B, 1-bit and ternary text-to-image diffusion transformers around 3GB, compared with FLUX.2 Klein 4B at about 16GB, with browser-local WebGPU demo links and an Apache-2.0 license disclosed in the Reddit snippet.

Why it matters: PrismML compressed Bonsai Image 4B's text-to-image diffusion transformer weights to 1-bit or ternary, about 3GB against FLUX.2 Klein 4B's roughly 16GB, and it runs locally in a browser on WebGPU.

May 26Tuesday

Bloomberg Technology

Qualcomm to Supply Chips to TikTok Owner ByteDance

Qualcomm will supply chips to ByteDance for artificial intelligence data centers, according to people familiar with the matter; the post does not disclose chip models, order volume, pricing, or delivery timing.

Why it matters: Bloomberg reports Qualcomm will supply AI data center chips to ByteDance, but the chip model, order size and delivery timeline are undisclosed, so only the direction of the deal is confirmed.

r/LocalLLaMA

[OSS] dlmserve: First Serving Engine for Diffusion Language Models

dlmserve released an MIT-licensed serving engine for diffusion language models, with LLaDA-8B-Instruct support and 2.5x HF throughput at batch=4. It exposes an OpenAI-compatible /v1/chat/completions API, batches at the denoising-step level, runs in 12GB VRAM, and adds about 1.8x throughput with optional LocalLeap acceleration.

Why it matters: A Reddit user open-sourced dlmserve, an OpenAI-compatible inference server for diffusion language models, reporting LLaDA-8B throughput at batch=4 that is 2.5x HuggingFace's.

AI HOT (Curated Pool)

OpenRouter Raises $113M Series B

OpenRouter raised a $113 million Series B led by CapitalG; its weekly volume rose from 5 trillion to 25 trillion tokens over the past 6 months.

Why it matters: OpenRouter raised a $113M Series B led by CapitalG, and its weekly token volume went from 5 trillion to 25 trillion in half a year, giving the model routing layer a concrete production-traffic figure.

Alibaba Technology · WeChat

Nearly 9x training speedup: residual streams in DiT are becoming a convergence bottleneck

Nanjing University LAMDA and Alibaba Intelligent Engine proposed DAR, a timestep-aware cross-layer routing method that replaces fixed residual accumulation in DiT; on ImageNet 256x256, it reduced SiT-XL/2 FID from 9.67 to 7.56 and reached baseline convergence quality with 8.75x fewer training iterations.

Why it matters: Nanjing University's LAMDA lab and Alibaba replace DiT's fixed residual accumulation with per-timestep cross-layer routing, reaching equal convergence on ImageNet in about one-ninth the training steps.

QbitAI · WeChat

Chinese AI-Written Pretraining Framework ForgeTrain Trains MiniCPM5-1B

ModelBest released ForgeTrain and MiniCPM5-1B, saying ForgeTrain was written by AI and trains 10% faster than NVIDIA Megatron under the same hardware conditions. MiniCPM5-1B is a 1B-parameter edge model with about 2GB FP16 weights and about 0.5GB INT4/Q4 weights.

Why it matters: ModelBest released ForgeTrain, a training framework whose code was written by AI, claiming 10% faster training than Nvidia's Megatron on the same hardware, plus the MiniCPM5-1B on-device model that compresses to about 0.5GB.

Synced · WeChat

AI-written training framework trains 1B edge model MiniCPM5-1B

ModelBest open-sourced MiniCPM5-1B and ForgeTrain; the 1B edge model scores 17.9 on AA-Index, while the AI-written ForgeTrain framework matches Megatron’s training results and runs 10% faster on Nvidia H100 under the article’s reported setup.

Why it matters: ModelBest open-sourced the MiniCPM5-1B on-device model and the ForgeTrain training framework, whose code was written by AI; the 1B model scores 17.9 on AA-Index.

AI HOT (Curated Pool)

ModelBest open-sources MiniCPM5-1B, topping sub-2B models on AA-Index

ModelBest open-sourced MiniCPM5-1B, a 1B-parameter edge language model that beats all sub-2B models on AA-Index, uses a 0.5GB weight file after INT4 quantization, and runs on phones and browsers.

Why it matters: ModelBest, Tsinghua and OpenBMB open-sourced the 1B MiniCPM5-1B, which tops the AA-Index among sub-2B models and runs on phones and browsers at 0.5GB after INT4 quantization.

Xinzhiyuan · WeChat

OpenAI Nearly Collapsed? President Says He Resigned the Day Altman Was Ousted

Greg Brockman recounted OpenAI’s 72-hour crisis: on November 17, 2023, the board removed Sam Altman as CEO and took Brockman off the board, after which Brockman resigned the same day and said he initially put the chance of taking the company back at 10%.

Why it matters: OpenAI president Greg Brockman recounts the 72 hours around Altman's November 2023 ouster, when he resigned that day and put the odds of retaking the company at one in ten.

r/LocalLLaMA

Shard - Getting to 10× KV Cache Compression

Shard reduces Llama-3.1-8B KV memory by about 10× at 8K context and 11× at 32K, with no measured drop on NIAH or LongBench, using PCA plus int4 quantization for K and Hadamard rotation plus vector quantization for V.

Why it matters: Shard compresses KV cache memory by about 90% on Llama-3.1-8B, roughly 10x at 8K context and 11x at 32K, using PCA plus int4 for keys and Hadamard rotation plus vector quantization for values.

AI HOT (Curated Pool)

OpenAI GPT-5.6 Reportedly Set for Next Month With 1.5M-Token Context

Developers found an unannounced OpenAI GPT-5.6 entry in Codex backend logs under the codename iris-alpha, with a 1.5 million-token context window, about 43% higher than GPT-5.5’s 1.05 million-token limit.

Why it matters: Developers dug an undisclosed model, iris-alpha, out of OpenAI Codex backend logs; it maps to GPT-5.6 and its 1.5M-token context window is about 43% above the 1.05M in the GPT-5.5 API.

AI HOT (Curated Pool)

Apple reportedly uses a custom 1.2T-parameter Google model for next-generation Siri

Apple is reportedly using a custom 1.2T-parameter Google model to run parts of the next-generation Siri, while simpler queries are expected to run on-device; the post says response speed for everyday questions is the key constraint.

Why it matters: Apple's next Siri is reported to use a custom Google model with 1.2 trillion parameters, far above the roughly 300 billion outsiders guessed, with complex tasks in the cloud and simple queries on device.

May 25Monday

r/LocalLLaMA

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

RTPurbo converts full-attention LLMs to sparse inference with a few hundred adaptation steps. It keeps the full KV cache only for retrieval heads, uses a 16-dimensional token indexer, and reports up to 9.36x prefill speedup at 1M context plus about 2.01x decode speedup on long-context and reasoning benchmarks.

Why it matters: RTPurbo converts a full-attention model to sparse inference by keeping full KV cache only for retrieval heads, cutting prefill time up to 9.36x at 1M tokens.

QbitAI · WeChat

Reasonix for DeepSeek V4 reaches 99.82% cache hit rate and cuts costs to 20%

Reasonix uses an append-only loop for DeepSeek V4 and reports a 99.82% cache hit rate in long coding sessions, cutting an example 400M-token bill from $61 to $12.

Why it matters: Reasonix caches DeepSeek V4 with an append-only loop that never rewrites, hitting a 99.82% cache rate in long code sessions and cutting cost to about a fifth.

Hacker News front page

Memory has grown to nearly two-thirds of AI chip component costs

Epoch AI says memory has grown to nearly two-thirds of AI chip component costs; the RSS body only lists the article URL, 68 points, and 71 comments, and the post does not disclose the methodology or sample scope.

Why it matters: Epoch AI estimates high-bandwidth memory rose from 52% to 63% of AI chip component costs between Q1 2024 and Q4 2025, while logic chips held near 13%.

May 24Sunday

r/LocalLLaMA

BitCPM-CANN: Native 1.58-Bit Large Language Model Training on Ascend NPU

OpenBMB released BitCPM-CANN, a 1.58-bit QAT training stack on Ascend NPU with 0.5B, 1B, 3B, and 8B models trained from scratch, where the 1B to 8B variants retain 95.7%–97.2% of full-precision MiniCPM4 performance across 11 benchmarks.

Why it matters: OpenBMB trained four 1.58-bit ternary models from 0.5B to 8B from scratch on Huawei Ascend NPUs, with the 8B retaining 97% of full-precision performance and compression done during training.

r/LocalLLaMA

It's OK to Quantize the KV Cache; Model Quant Matters More in Qwen3.6 27B KLD Tests

Reddit user hopbel tested Qwen3.6 27B with approximate KLD on wikitext-2 at 16k context, using Q5_K_M as the proxy baseline; Q5_K_S weights with q4_0 KV cache scored 0.016304, while Q4_K_XL with f16 KV cache scored 0.026067, so weight quant tier dominated KV-cache quant in this setup.

Why it matters: One Reddit user's KLD tests on Qwen3.6 27B give first-hand numbers on whether to quantize the KV cache or the weights for local deployment.

May 23Saturday

Synced · WeChat

FlashAR speeds up pretrained autoregressive image models by 22.9x using 0.05% data

Zhejiang University and the University of Adelaide introduced FlashAR, using 0.05% of the original training data to reduce Emu3.5-Image-34B 512×512 generation latency from 130.10 seconds to 5.68 seconds, while GenEval changed from 80.48 to 80.29.

Why it matters: Zhejiang University and the University of Adelaide propose FlashAR, which fine-tunes on just 0.05% of the original training data and cuts Emu3.5-Image-34B 512x512 generation from 130.10 seconds to 5.68 seconds.

Synced · WeChat

Bengio Paper Raises Recursive Reasoning Limits as Parallel Trajectories Beat Serial Reasoning

Yoshua Bengio’s team introduced GRAM, a generative recursive reasoning model that samples multiple latent trajectories; on Sudoku-Extreme, GRAM reached 97.0% accuracy with 16 recursive steps and 20 parallel samples, exceeding TRM’s 90.5% result at 320 serial recursive steps.

Why it matters: Bengio's team proposes GRAM, generating several reasoning traces in parallel and picking the best, hitting 97.0% on hard Sudoku in 16 recursive steps where serial models need 320 steps for 90.5%.

Mistral AI

Mistral to acquire physics AI company Emmi AI

Mistral AI said it has reached a definitive agreement to acquire Emmi AI, a physics AI pioneer, to strengthen its position as an AI transformation partner for industrial companies. Austria-based Emmi AI works on physics AI and large engineering models that speed up engineering workflows, replace multi-day computations with real-time simulation and build digital twins. Emmi's co-founders and more than 30 researchers and engineers will join Mistral's Science and Applied AI teams in May.

Why it matters: Mistral buys Austrian physical AI company Emmi AI, adding more than 30 researchers to its science and applied AI team in May to strengthen industrial simulation and digital twins.

QbitAI · WeChat

DeepSeek V4 cuts prices as CATL, JD.com and NetEase discuss investment; Liang Wenfeng targets AGI

DeepSeek-V4-Pro API will keep its promotional pricing from June 1, with cached input at RMB 0.025 per million tokens, while Bloomberg says DeepSeek is pursuing a RMB 70 billion round at a USD 45 billion pre-money valuation.

Why it matters: DeepSeek-V4-Pro's API keeps its promotional price from June 1, at 0.025 yuan per million cached input tokens, while Bloomberg says it is negotiating a 7 billion yuan round at a $45 billion pre-money valuation.

r/LocalLLaMA

club-rdna16: Practical 16GB AMD/Radeon local LLM testing repo

club-rdna16 publishes a practical 16GB Radeon local LLM testing repo, with an RX 6900 XT running llama.cpp on ROCm/HIP and Qwen3.6 35B-A3B reaching a stable 131k context using q8 KV cache.

Why it matters: A public test repo shows Qwen3.6 35B-A3B holding 131k context with q8 KV cache on an RX 6900 XT, giving 16GB AMD card owners a measured config to copy.

AI HOT (Curated Pool)

Jensen Huang Says Annual AI Infrastructure Spending Will Reach $4 Trillion

Jensen Huang predicted hyperscale cloud providers’ annual AI infrastructure spending will rise from $1 trillion to $3 trillion–$4 trillion, while Nvidia reported $81.6 billion in fiscal 2027 Q1 revenue and $75.2 billion from data centers.

Why it matters: Jensen Huang told Nvidia's earnings call that hyperscaler AI infrastructure spending will reach $3-4 trillion a year, roughly four times Wall Street's $1.03 trillion consensus for 2028.

AI HOT (Curated Pool)

Agent Workloads Quietly Reshape Inference Economics

SemiAnalysis analyzed 432,000 real coding-agent requests and found a median input length of 96,000 tokens, not 32,000 or 64,000. The post does not disclose the model mix, cost curve, sampling method, or time window.

Why it matters: SemiAnalysis counted 432,000 real coding-agent requests and found a median input of 96,000 tokens, well above the commonly cited 32,000 or 64,000.

r/LocalLLaMA

Experts first llama.cpp

comanderxv published a llama.cpp fork that caches MoE experts in 12GB VRAM; on an RTX 2060 with Qwen3.6-35B-A3B, throughput rose from 19/22 tk/s to 26 tk/s at about a 62% expert-cache hit rate.

Why it matters: comanderxv opened an experts first branch for llama.cpp that tunes MoE memory scheduling, lifting a 12GB RTX 2060 from 19/22 tk/s to 26 tk/s.

May 22Friday

Hacker News front page

DeepSeek Makes the V4 Pro Price Discount Permanent

DeepSeek will set deepseek-v4-pro API pricing to one quarter of the original price after the 75% promotion ends on 2026-05-31 at 15:59 UTC; the post does not disclose the exact per-token price.

Why it matters: DeepSeek made the V4 Pro 75% discount its standard price, cutting input and output tokens to $0.435 and $0.87 per million, so developers can recalculate their costs.

Dwarkesh Patel podcast

Reiner Pope – Chip Design from the Bottom Up

Dwarkesh Patel interviews MatX CEO Reiner Pope on chip design, starting with a 4-bit multiply and 8-bit accumulate example that uses 16 AND gates, then covering systolic arrays, pipeline registers, FPGAs versus ASICs, cache versus scratchpad, and why GPU cores are smaller than CPU cores.

Why it matters: MatX CEO Reiner Pope starts from AND, OR and NOT gates and walks through 4-bit by 4-bit multiply with 8-bit accumulate, the groundwork for how AI chips do matrix math.

Mistral AI

Mistral launches Connectors in Studio with built-in and custom MCP

Mistral launched Connectors in Studio. All built-in connectors and custom MCP are now callable through the API/SDK by every model and agent. New features include direct tool calling, human-in-the-loop approval flows, and programmatic access to create, modify, list and delete connectors.

Why it matters: Mistral registers built-in connectors and custom MCP servers in one place in Studio, with code samples for creating, calling and human approval, showing enterprise integration moving from repeated per-team work to one platform config.

AI HOT (Curated Pool)

BitCPM-CANN Released as First 1.58-bit Open Model Fully Trained on Huawei Ascend 910B NPU

ModelBest, Tsinghua University, and OpenBMB released BitCPM-CANN, a 0.5B-8B open model family trained natively on Huawei Ascend 910B NPUs with 1.58-bit ternary weights, cutting memory use by about 6x versus BF16 while retaining 95-97% of full-precision benchmark performance.

Why it matters: ModelBest, Tsinghua and OpenBMB released BitCPM-CANN, trained entirely on Huawei Ascend 910B, where 1.58-bit quantization saves about 6x memory over BF16 while benchmarks hold 95-97% of full-precision scores.

Latent Space

[AINews] New AI Infra Unicorns: Exa, Modal, TurboPuffer

Latent Space summarized AI News for May 20-21, 2026, confirming TurboPuffer reached $100 million ARR and profitability, Exa raised a $250 million Series C at a $2.2 billion valuation, and Modal raised a $355 million Series C at a $4.7 billion valuation.

Why it matters: TurboPuffer confirms $100M ARR and profitability, showing that vector databases can sustain paying customers and offering a reference point for how similar infrastructure companies commercialize.

AI HOT (Curated Pool)

Zhipu releases GLM-5.1-highspeed, claiming a large-model API speed record

Zhipu released the GLM-5.1-highspeed API to selected enterprise customers on May 22, with a claimed output speed of 400 tokens/s, built by the GLM team and TileRT team through system-level optimization.

Why it matters: Zhipu opened a high-speed GLM-5.1 endpoint to some enterprise customers at a claimed 400 tokens per second, a case for watching how flagship quality trades off against low latency in production.

New York Times Chinese

Trump Approved Nvidia Chip Sales to China. Why Is Beijing Reluctant?

Trump approved Nvidia H200 sales to China six months ago, but Beijing has not allowed any company to buy even one chip and is steering firms toward domestic alternatives from Huawei and Cambricon.

Why it matters: Six months after Trump approved H200 sales to China, Beijing has still not let any company place orders and is steering compute demand to Huawei, Cambricon and other domestic options.

Computing Life · Share · Yage

How to Run DeepSeek V4 Flash Locally on Mac: DS4 Engine Explained

DS4 provides a macOS local runtime path for DeepSeek V4 Flash; the post only discloses three mechanisms—multi-agent integration, KV cache disk persistence, and activation steering—and does not disclose performance numbers, hardware requirements, or pricing.

Why it matters: antirez wrote DS4, a pure C, Metal-first inference engine for DeepSeek V4 Flash at 284B total and 13B active parameters, filling a gap where mainstream engines cannot run it on a Mac.

Computing Life · Share · Yage

The Technology Behind GLM-5.1 Reaching 400 Tokens/s

Zhipu GLM-5.1 high-speed API claims 400 tokens/s, and the post says TileRT reconstructs GPU inference at the execution-model level; the RSS snippet does not disclose benchmark conditions, hardware, pricing, or latency distribution.

Why it matters: Zhipu released a high-speed GLM-5.1 API at a claimed 400 tokens per second, with its TileRT engine moving the GPU from batch processing to a continuous pipeline with no waits between steps.