Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

261–280 of 609

May 30Saturday

r/LocalLLaMA

Testing MTP on vLLM and llama.cpp for Gemma 4 and Qwen 3.6

The author tested MTP on an RTX PRO 6000 Blackwell setup, where Gemma 4 31B on vLLM reached 132.52 tok/s versus a 39.69 tok/s baseline, a 3.34x speedup; the post reports 10 runs of 1,500 tokens each but does not provide a full quality or VRAM evaluation.

Why it matters: A user benchmarked Gemma 4 31B with vLLM on an RTX PRO 6000 with and without MTP, seeing throughput rise from 39.69 to 132.52 tokens per second on specific hardware and framework.

AI HOT (Curated Pool)

OpenAI launches real-time translation model with 70+ input languages

OpenAI launched gpt-realtime-translate, a speech translation model that accepts 70+ input languages and outputs speech in 13 target languages; the post says the feature is running on smart glasses.

Why it matters: OpenAI's gpt-realtime-translate takes speech input in over 70 languages and outputs 13, and it already runs on smart glasses, showing what a dedicated translation model trades off.

TechCrunch · AI

After Nvidia’s $20B Not-Acqui-Hire, AI Chip Startup Groq Reportedly Raising $650M

Axios says Groq is seeking $650 million in internal funding while shifting from hardware toward AI inference, after Nvidia’s reported $20 billion not-acqui-hire; the RSS snippet does not disclose Groq’s valuation, investor names, deal structure, or fundraising timeline.

Why it matters: Groq is seeking $650 million in internal funding and shifting its focus from building chips to AI inference, after Nvidia's $20 billion not-acqui-hire.

May 29Friday

Xinzhiyuan · WeChat

Three DeepSeek Models Enter OpenRouter Monthly Top 10 With Over 17 Trillion Tokens

DeepSeek placed three models in OpenRouter’s monthly top 10 with more than 17 trillion tokens combined, including V4 Flash at 9.13T tokens; the article says Ascend’s MegaMoE operator raised Prefill throughput by 20% to 30% on DeepSeek V3.1 and Qwen3-235B tests.

Why it matters: OpenRouter's monthly stats put three DeepSeek models in the top ten, showing how much traffic Chinese models draw on an aggregation platform.

r/LocalLLaMA

Liquid AI releases LFM2.5-8B-A1B

Liquid AI released LFM2.5-8B-A1B with a 128K context window, 38T pre-training tokens, large-scale reinforcement learning, doubled vocabulary for non-Latin tokenization, and availability on Hugging Face.

Why it matters: Liquid AI shipped LFM2.5-8B-A1B, 8 billion parameters with a 128K token context, a doubled vocabulary that tokenizes non-Latin scripts better, and the model is already on Hugging Face.

Synced · WeChat

A True 2-bit KV Quantization Algorithm for Long-context Reasoning Beyond TurboQuant

TogetherAI and collaborators released OSCAR, a 2.28 BPE INT2 KV Cache system integrated with SGLang, reporting up to 3× decode speedup at 100k context and up to 7× job-level throughput under a fixed memory budget.

Why it matters: TogetherAI released OSCAR, which compresses the KV cache to 2.28 bits per value on average and plugs into SGLang, decoding up to 3x faster at 100K tokens and up to 7x throughput at fixed memory.

Synced · WeChat

The Ma Jiaqi Failure Exposed an LLM Issue He Spotted in the Shower a Year Earlier

FaceMind links low-frequency token degradation to two papers: SLoW appeared at EMNLP 2025, Adam's Law was accepted as an ACL 2026 Oral, and high-frequency rewriting raised DeepSeek-V3 math accuracy from 63.55% to 71.54%.

Why it matters: The FaceMind team links large models failing on low-frequency words to the SLoW and Adam's Law papers, laying out a mechanistic account of low-frequency token degradation.

Bloomberg Technology

Samsung Takes Lead in Shipping Top-End AI Memory Chip Samples

Samsung Electronics has begun shipping samples of its most advanced memory to customers for AI accelerators from companies including Nvidia; the RSS snippet does not disclose the chip model, customer list, sample volume, pricing, or mass-production timeline.

Why it matters: Samsung has sent its most advanced memory chip samples to NVIDIA and other AI accelerator customers, reaching customer validation ahead of rivals.

Bloomberg Technology

Apollo Shops $36 Billion Debt Deal to Buy Google Chips for Anthropic

Apollo and Blackstone are seeking additional investors for about $36 billion in debt financing for Anthropic’s AI infrastructure; the title says the deal would buy Google chips, while the post does not disclose chip models, purchase volume, or timeline.

Why it matters: Apollo and Blackstone are raising about $36bn in debt for Anthropic to buy Google chips, a concrete case of an AI lab funding compute with debt.

AI HOT (Curated Pool)

Apple reportedly tries to fit Google's large Gemini model into iPhone for new Siri

Apple is trying to integrate a large Gemini model into the iPhone for new Siri features. The RSS snippet says full local processing is unlikely because of model size, and a cloud component is likely required; the post does not disclose parameters, latency targets, or a release timeline.

Why it matters: Apple is trying to distill Google's trillion-parameter Gemini into an on-device iPhone model for the new Siri, and the report judges full local execution unlikely, with cloud still needed.

May 28Thursday

QbitAI · WeChat

Behind DeepSeek V4's Chip-Model Co-Design, China's Compute Ecosystem Gains Speed

QbitAI says DeepSeek V4 validated Ascend chip-model co-design, with CANN open-sourcing 65 repositories and supporting day-zero adaptation for more than 70 mainstream models, while AIGCode reported 65% MFU in MoE pretraining on Ascend.

Why it matters: DeepSeek V4 was tuned against Huawei Ascend chips during training, and CANN open-sourced 65 repos with out-of-the-box support for over 70 mainstream models, giving domestic compute a concrete chip-to-framework-to-model path.

r/LocalLLaMA

Zai replaced the network architecture for GLM-5.1 inference, lifting throughput 15%

Zai replaced the ROFT network topology with ZCube on a thousand-GPU GLM-5.1 coding inference cluster, keeping the same GPUs, software stack, and model; the Reddit post cites 33% lower switch and optical module costs, 15% higher GPU inference throughput, and a 40.6% drop in first-token P99 tail latency under prefill-decode disaggregated inference.

Why it matters: Zai swapped ROFT for its own ZCube topology on a thousand-GPU GLM-5.1 inference cluster, with no other changes, and cut switch and optical module costs 33% while raising throughput 15%.

r/LocalLLaMA

Qwen3.6-35B-A3B-APEX Runs 128K Context on RTX 3060 12GB

old-mike ran mudler/Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf through spiritbuun’s llama.cpp fork on one RTX 3060 12GB, offloading a 17.3GB model and reaching 37.17 t/s generation at 72K filled context, 28.08 t/s at 129K, and PPL 3.2529 on an enwik8 64K-context perplexity test.

Why it matters: A user ran a 17.3GB Qwen model on one 12GB RTX 3060 at 37 tokens/s with 72K context filled, giving concrete settings for consumer-GPU inference.

AI HOT (Curated Pool)

AI Now Summit 2026

Mistral AI announced industrial AI work, a Vibe upgrade, and a 10 MW inference data center in Les Ulis at AI Now Summit 2026; it is working with Airbus, BMW Group, and ASML, and the data center is scheduled to start operating in Q3 2026.

Why it matters: Mistral is pairing industrial AI deals with Airbus, BMW and ASML with its own 10 MW inference data center near Paris, due in Q3 2026.

AI HOT (Curated Pool)

Mistral AI launches physics AI model for industrial engineering

Mistral AI integrated the Emmi AI team and launched a physics AI foundation model for industrial engineering, with the post saying it can learn from geometry, boundary conditions, or measurement data and predict full physical fields on a single GPU in seconds.

Why it matters: Mistral's physical AI foundation model for industrial engineering predicts full fields like temperature and stress in seconds on a single GPU, a route worth watching for industrial simulation.

Latent Space

Cognition Raises $1B in $26B Series D

Cognition raised a $1B Series D at a $26B valuation and projects more than $1B ARR by year-end; the post says its valuation rose 2.5× from the $10B Series C eight months earlier, while the rest of the issue summarizes agent, inference, benchmark, and multimodal AI updates from May 26–27, 2026.

Why it matters: Cognition raised $1 billion in a Series D at a $26 billion valuation, up from $10 billion in eight months, a concrete gauge of capital heat in coding agents.

Xinzhiyuan · WeChat

Tsinghua Team Open-Sources PilotDeck Agent System, Claims 70% Token Cost Reduction

Tsinghua THUNLP, ModelBest, OpenBMB, and AI9stars open-sourced PilotDeck; the article says its sub-agent routing reduced cost from $12.58 to $2.83 in a Xiaohongshu content-generation test, while preserving separate WorkSpaces, editable memory, and per-session routing logs.

Why it matters: Tsinghua NLP and other teams open-sourced PilotDeck, giving sub-agents separate workspaces and editable memory with scheduling logs, a reference implementation for multi-agent workflows.

Synced · WeChat

Mila and DeepMind Propose UNSL for Unified Multivariate Neural Scaling Laws

Mila and Google DeepMind proposed Unified Neural Scaling Law, a multivariate scaling-law form that models parameter count, token count, training steps, bottlenecks, overfitting, and adverse hyperparameter effects; UNSL achieved the best extrapolation on 60.87% of vision tasks and 88.89% of language tasks in the reported experiments.

Why it matters: Mila and DeepMind proposed UNSL, putting parameter count, token count and training steps into one scaling formula with bottlenecks and overfitting, a unified frame for training configuration.

TechCrunch · AI

Snowflake signs $6B deal with AWS for AI CPU chips

Snowflake signed a five-year, $6 billion AWS deal to secure chips for AI use; the post does not disclose chip models, delivery timing, or whether the capacity targets training, inference, or both.

Why it matters: Snowflake signed a $6 billion five-year AWS deal for AI chip capacity, a large cloud buyer looking beyond Nvidia for compute.

r/LocalLLaMA

Inferencing at 10.33 t/s on Qwen 3.5 35B on a $300 laptop

A Reddit user ran Qwen 3.5 35B Q4_K_S on a $300 Lenovo Ideapad Slim 3i and reported 10.33 t/s inference using ik_llama.cpp with two pinned CPU cores, MTP speculative decoding, 64 batch size, and Q8_0 KV cache.

Why it matters: A Reddit user ran the Q4_K_S quant of Qwen 3.5 35B on a $300 Lenovo Ideapad Slim 3i using CPU only, reaching 10.33 tokens per second.