Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

301–320 of 609

May 26Tuesday

AI HOT (Curated Pool)

OpenAI GPT-5.6 Reportedly Set for Next Month With 1.5M-Token Context

Developers found an unannounced OpenAI GPT-5.6 entry in Codex backend logs under the codename iris-alpha, with a 1.5 million-token context window, about 43% higher than GPT-5.5’s 1.05 million-token limit.

Why it matters: Developers dug an undisclosed model, iris-alpha, out of OpenAI Codex backend logs; it maps to GPT-5.6 and its 1.5M-token context window is about 43% above the 1.05M in the GPT-5.5 API.

AI HOT (Curated Pool)

Apple reportedly uses a custom 1.2T-parameter Google model for next-generation Siri

Apple is reportedly using a custom 1.2T-parameter Google model to run parts of the next-generation Siri, while simpler queries are expected to run on-device; the post says response speed for everyday questions is the key constraint.

Why it matters: Apple's next Siri is reported to use a custom Google model with 1.2 trillion parameters, far above the roughly 300 billion outsiders guessed, with complex tasks in the cloud and simple queries on device.

May 25Monday

r/LocalLLaMA

Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

RTPurbo converts full-attention LLMs to sparse inference with a few hundred adaptation steps. It keeps the full KV cache only for retrieval heads, uses a 16-dimensional token indexer, and reports up to 9.36x prefill speedup at 1M context plus about 2.01x decode speedup on long-context and reasoning benchmarks.

Why it matters: RTPurbo converts a full-attention model to sparse inference by keeping full KV cache only for retrieval heads, cutting prefill time up to 9.36x at 1M tokens.

QbitAI · WeChat

Reasonix for DeepSeek V4 reaches 99.82% cache hit rate and cuts costs to 20%

Reasonix uses an append-only loop for DeepSeek V4 and reports a 99.82% cache hit rate in long coding sessions, cutting an example 400M-token bill from $61 to $12.

Why it matters: Reasonix caches DeepSeek V4 with an append-only loop that never rewrites, hitting a 99.82% cache rate in long code sessions and cutting cost to about a fifth.

Hacker News front page

Memory has grown to nearly two-thirds of AI chip component costs

Epoch AI says memory has grown to nearly two-thirds of AI chip component costs; the RSS body only lists the article URL, 68 points, and 71 comments, and the post does not disclose the methodology or sample scope.

Why it matters: Epoch AI estimates high-bandwidth memory rose from 52% to 63% of AI chip component costs between Q1 2024 and Q4 2025, while logic chips held near 13%.

May 24Sunday

r/LocalLLaMA

BitCPM-CANN: Native 1.58-Bit Large Language Model Training on Ascend NPU

OpenBMB released BitCPM-CANN, a 1.58-bit QAT training stack on Ascend NPU with 0.5B, 1B, 3B, and 8B models trained from scratch, where the 1B to 8B variants retain 95.7%–97.2% of full-precision MiniCPM4 performance across 11 benchmarks.

Why it matters: OpenBMB trained four 1.58-bit ternary models from 0.5B to 8B from scratch on Huawei Ascend NPUs, with the 8B retaining 97% of full-precision performance and compression done during training.

r/LocalLLaMA

It's OK to Quantize the KV Cache; Model Quant Matters More in Qwen3.6 27B KLD Tests

Reddit user hopbel tested Qwen3.6 27B with approximate KLD on wikitext-2 at 16k context, using Q5_K_M as the proxy baseline; Q5_K_S weights with q4_0 KV cache scored 0.016304, while Q4_K_XL with f16 KV cache scored 0.026067, so weight quant tier dominated KV-cache quant in this setup.

Why it matters: One Reddit user's KLD tests on Qwen3.6 27B give first-hand numbers on whether to quantize the KV cache or the weights for local deployment.

May 23Saturday

Synced · WeChat

FlashAR speeds up pretrained autoregressive image models by 22.9x using 0.05% data

Zhejiang University and the University of Adelaide introduced FlashAR, using 0.05% of the original training data to reduce Emu3.5-Image-34B 512×512 generation latency from 130.10 seconds to 5.68 seconds, while GenEval changed from 80.48 to 80.29.

Why it matters: Zhejiang University and the University of Adelaide propose FlashAR, which fine-tunes on just 0.05% of the original training data and cuts Emu3.5-Image-34B 512x512 generation from 130.10 seconds to 5.68 seconds.

Synced · WeChat

Bengio Paper Raises Recursive Reasoning Limits as Parallel Trajectories Beat Serial Reasoning

Yoshua Bengio’s team introduced GRAM, a generative recursive reasoning model that samples multiple latent trajectories; on Sudoku-Extreme, GRAM reached 97.0% accuracy with 16 recursive steps and 20 parallel samples, exceeding TRM’s 90.5% result at 320 serial recursive steps.

Why it matters: Bengio's team proposes GRAM, generating several reasoning traces in parallel and picking the best, hitting 97.0% on hard Sudoku in 16 recursive steps where serial models need 320 steps for 90.5%.

Mistral AI

Mistral to acquire physics AI company Emmi AI

Mistral AI said it has reached a definitive agreement to acquire Emmi AI, a physics AI pioneer, to strengthen its position as an AI transformation partner for industrial companies. Austria-based Emmi AI works on physics AI and large engineering models that speed up engineering workflows, replace multi-day computations with real-time simulation and build digital twins. Emmi's co-founders and more than 30 researchers and engineers will join Mistral's Science and Applied AI teams in May.

Why it matters: Mistral buys Austrian physical AI company Emmi AI, adding more than 30 researchers to its science and applied AI team in May to strengthen industrial simulation and digital twins.

QbitAI · WeChat

DeepSeek V4 cuts prices as CATL, JD.com and NetEase discuss investment; Liang Wenfeng targets AGI

DeepSeek-V4-Pro API will keep its promotional pricing from June 1, with cached input at RMB 0.025 per million tokens, while Bloomberg says DeepSeek is pursuing a RMB 70 billion round at a USD 45 billion pre-money valuation.

Why it matters: DeepSeek-V4-Pro's API keeps its promotional price from June 1, at 0.025 yuan per million cached input tokens, while Bloomberg says it is negotiating a 7 billion yuan round at a $45 billion pre-money valuation.

r/LocalLLaMA

club-rdna16: Practical 16GB AMD/Radeon local LLM testing repo

club-rdna16 publishes a practical 16GB Radeon local LLM testing repo, with an RX 6900 XT running llama.cpp on ROCm/HIP and Qwen3.6 35B-A3B reaching a stable 131k context using q8 KV cache.

Why it matters: A public test repo shows Qwen3.6 35B-A3B holding 131k context with q8 KV cache on an RX 6900 XT, giving 16GB AMD card owners a measured config to copy.

AI HOT (Curated Pool)

Jensen Huang Says Annual AI Infrastructure Spending Will Reach $4 Trillion

Jensen Huang predicted hyperscale cloud providers’ annual AI infrastructure spending will rise from $1 trillion to $3 trillion–$4 trillion, while Nvidia reported $81.6 billion in fiscal 2027 Q1 revenue and $75.2 billion from data centers.

Why it matters: Jensen Huang told Nvidia's earnings call that hyperscaler AI infrastructure spending will reach $3-4 trillion a year, roughly four times Wall Street's $1.03 trillion consensus for 2028.

AI HOT (Curated Pool)

Agent Workloads Quietly Reshape Inference Economics

SemiAnalysis analyzed 432,000 real coding-agent requests and found a median input length of 96,000 tokens, not 32,000 or 64,000. The post does not disclose the model mix, cost curve, sampling method, or time window.

Why it matters: SemiAnalysis counted 432,000 real coding-agent requests and found a median input of 96,000 tokens, well above the commonly cited 32,000 or 64,000.

r/LocalLLaMA

Experts first llama.cpp

comanderxv published a llama.cpp fork that caches MoE experts in 12GB VRAM; on an RTX 2060 with Qwen3.6-35B-A3B, throughput rose from 19/22 tk/s to 26 tk/s at about a 62% expert-cache hit rate.

Why it matters: comanderxv opened an experts first branch for llama.cpp that tunes MoE memory scheduling, lifting a 12GB RTX 2060 from 19/22 tk/s to 26 tk/s.

May 22Friday

Hacker News front page

DeepSeek Makes the V4 Pro Price Discount Permanent

DeepSeek will set deepseek-v4-pro API pricing to one quarter of the original price after the 75% promotion ends on 2026-05-31 at 15:59 UTC; the post does not disclose the exact per-token price.

Why it matters: DeepSeek made the V4 Pro 75% discount its standard price, cutting input and output tokens to $0.435 and $0.87 per million, so developers can recalculate their costs.

Dwarkesh Patel podcast

Reiner Pope – Chip Design from the Bottom Up

Dwarkesh Patel interviews MatX CEO Reiner Pope on chip design, starting with a 4-bit multiply and 8-bit accumulate example that uses 16 AND gates, then covering systolic arrays, pipeline registers, FPGAs versus ASICs, cache versus scratchpad, and why GPU cores are smaller than CPU cores.

Why it matters: MatX CEO Reiner Pope starts from AND, OR and NOT gates and walks through 4-bit by 4-bit multiply with 8-bit accumulate, the groundwork for how AI chips do matrix math.

Mistral AI

Mistral launches Connectors in Studio with built-in and custom MCP

Mistral launched Connectors in Studio. All built-in connectors and custom MCP are now callable through the API/SDK by every model and agent. New features include direct tool calling, human-in-the-loop approval flows, and programmatic access to create, modify, list and delete connectors.

Why it matters: Mistral registers built-in connectors and custom MCP servers in one place in Studio, with code samples for creating, calling and human approval, showing enterprise integration moving from repeated per-team work to one platform config.

AI HOT (Curated Pool)

BitCPM-CANN Released as First 1.58-bit Open Model Fully Trained on Huawei Ascend 910B NPU

ModelBest, Tsinghua University, and OpenBMB released BitCPM-CANN, a 0.5B-8B open model family trained natively on Huawei Ascend 910B NPUs with 1.58-bit ternary weights, cutting memory use by about 6x versus BF16 while retaining 95-97% of full-precision benchmark performance.

Why it matters: ModelBest, Tsinghua and OpenBMB released BitCPM-CANN, trained entirely on Huawei Ascend 910B, where 1.58-bit quantization saves about 6x memory over BF16 while benchmarks hold 95-97% of full-precision scores.

Latent Space

[AINews] New AI Infra Unicorns: Exa, Modal, TurboPuffer

Latent Space summarized AI News for May 20-21, 2026, confirming TurboPuffer reached $100 million ARR and profitability, Exa raised a $250 million Series C at a $2.2 billion valuation, and Modal raised a $355 million Series C at a $4.7 billion valuation.

Why it matters: TurboPuffer confirms $100M ARR and profitability, showing that vector databases can sustain paying customers and offering a reference point for how similar infrastructure companies commercialize.