Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

181–200 of 609

Jun 9Tuesday

AI HOT (Curated Pool)

Google Releases Gemini 3.5 Live Translate for Real-Time Speech Translation

Google released Gemini 3.5 Live Translate, a speech-to-speech translation model that supports more than 70 languages, starts translating before the speaker finishes, uses streaming updates, and runs through Gemini Live API, Google Meet preview, and Google Translate apps on iOS and Android.

Why it matters: Google released Gemini 3.5 Live Translate, covering more than 70 languages and translating as it listens instead of waiting for a full sentence, with latency down to a few seconds.

AI HOT (Curated Pool)

Google DeepMind Releases Gemma 4 12B, a Unified Encoder-Free Multimodal Model

Google DeepMind released Gemma 4 12B, a multimodal model with a unified encoder-free architecture, native audio input, Apache 2.0 licensing, and local laptop runtime with 16GB of VRAM or unified memory.

Why it matters: Google DeepMind open-sourced Gemma 4 12B, dropping separate vision and audio encoders so one Transformer handles images, text and speech, and it runs on 16GB of VRAM.

r/LocalLLaMA

Apple Announced New On-Device Inference Engine for Apple Silicon

Apple announced CoreAI at WWDC as a future CoreML replacement for Apple Silicon on-device inference; models require Python-script conversion, the supported list is mostly mid-2025 models, and the post does not disclose performance data.

Why it matters: Apple's CoreML successor runs models on Apple silicon, but models must first be converted with a Python script, and the support list is mostly mid-2025.

AI HOT (Curated Pool)

China Prepares $295 Billion Plan to Fund Nationwide AI Infrastructure Buildout

China plans to invest about 2 trillion yuan, or $295 billion, over five years to build nationwide data centers, with funding covering large-scale data center infrastructure for domestic AI development.

Why it matters: Bloomberg reports China is planning about 2 trillion yuan of AI infrastructure over five years, centered on large data centers to close the compute gap.

AI HOT (Curated Pool)

Xiaomi MiMo and TileRT Release UltraSpeed Mode, 1T Model Exceeds 1,000 Tokens/s

Xiaomi MiMo and TileRT released MiMo-V2.5-Pro-UltraSpeed, a 1T-parameter model mode exceeding 1,000 tokens/s, with API access open from June 9 to June 23, 2026, at 3× the MiMo-V2.5-Pro price and about 10× the speed.

Why it matters: Xiaomi MiMo and TileRT add an UltraSpeed mode to a 1T-parameter model, pushing output past 1000 tokens/s, with FP4 quantizing only MoE experts and block-level parallel speculative decoding.

AI HOT (Curated Pool)

Elon Musk Details SpaceX's AI1 Orbital AI Data Center Satellite Plan

Elon Musk detailed SpaceX’s AI1 orbital AI data center satellite plan, with 150 kW peak power per satellite, about 120 kW sustained compute power, and 6-8 ms round-trip latency at 600-800 km low Earth orbit.

Why it matters: Musk detailed SpaceX's AI1 orbital data center satellite, about 120 kW of sustained compute per unit, roughly one NVIDIA GB300 rack, with 6-8 ms laser link round-trip latency.

r/LocalLLaMA

2X tk/s on 1× MI50: Qwen3.6-27B inference rises from 19.4 to 38.1 tk/s

bigattichouse raised Qwen3.6-27B throughput on a single MI50 from 19.4 to 38.1 tk/s by running same-model parallel computations for Q8-or-lower quantization, exploiting unused compute lanes instead of adding a smaller speculative decoding model.

Why it matters: A user ran two concurrent Qwen3.6-27B computations on a single MI50, lifting generation from 19.4 to 38.1 tokens/s by exploiting compute small quants leave idle instead of loading a side model.

r/LocalLLaMA

New MLX LM Server From Apple

A Reddit post says Apple’s MLX LM Server uses continuous batching for concurrent sub-agent requests and supports distributed inference across multiple Macs via Thunderbolt RDMA.

Why it matters: Apple's MLX LM Server adds continuous batching and Thunderbolt-linked distributed inference across Macs, a concrete option for anyone running local models.

r/LocalLLaMA

Levi: Run AlphaEvolve on Your Local Qwen 30B

LEVI runs an AlphaEvolve-like search system with Qwen3-30B-A3B and reports tests on ADRS, IFBench, and HotpotQA, claiming up to 35x lower cost overall and up to 12x fewer evals under the same single-model, same-budget comparison.

Why it matters: The open-source Levi project runs AlphaEvolve-style search on a local Qwen3-30B-A3B, and its author measured up to 35x lower cost and 12x fewer evaluations at equal budget on three benchmarks.

Jun 8Monday

r/LocalLLaMA

Xiaomi claims 1,000+ tps on a 1T model using a standard 8-GPU server

Xiaomi MiMo claims MiMo-V2.5-Pro UltraSpeed runs a 1T-parameter MoE model above 1,000 output tokens per second on one standard 8-GPU node; the post does not disclose the GPU model, batch settings, or reproducible configuration.

Why it matters: Xiaomi's MiMo team says it hit over 1,000 tokens per second on a 1T-parameter MoE model on one standard 8-GPU server, without Cerebras or Groq custom hardware, a useful comparison against dedicated-chip routes.

Hacker News front page

MiMo-v2.5-Pro-UltraSpeed: 1T model with 1000 tokens per second

The title says Xiaomi MiMo-v2.5-Pro-UltraSpeed is a 1T model running at 1,000 tokens per second; the RSS body only provides the URL, Hacker News comments link, 66 points, and 14 comments, and the post does not disclose hardware, precision, context window, benchmark setup, or availability.

Why it matters: Xiaomi worked with TileRT on FP4-quantized MoE expert modules plus DFlash speculative decoding to push a 1T-parameter model past 1,000 tokens per second on 8 GPUs.

r/LocalLLaMA

Luce Spark: a 35B MoE on a 16 GB GPU, without the offload tax

Luce Spark runs Qwen3.6 35B-A3B at 13.3 GiB peak VRAM on an RTX 3090 by keeping hot experts on GPU, swapping cold experts through a bounded async cache, and using one fused graph for decode at about 100 tok/s.

Why it matters: A concrete recipe for running a 35B MoE on a 16 GB card, with the memory numbers and throughput to judge it by.

AI HOT (Curated Pool)

Xiaomi MiMo-V2.5-Pro-UltraSpeed Exceeds 1,000 Tokens/s

Xiaomi MiMo and TileRT_AI released MiMo-V2.5-Pro-UltraSpeed, running a 1T MoE model above 1,000 tokens/s on a single standard 8-GPGPU node, with UltraSpeed API priced at 3x and applications open from June 8 to 23 PDT.

Why it matters: Xiaomi and TileRT_AI ran a 1-trillion-parameter MoE model on a single 8-GPU node at over 1000 tokens per second without specialized hardware, offering a low-cost path for ordinary inference deployment.

r/LocalLLaMA

DFlash Speculative Decoding and KV Cache Compression on RTX 5090 Show 3.26x Speedup

The author tested Qwen3.6-27B on an RTX 5090 with DFlash plus KV cache compression, reaching up to 3.26x speedup; q4_0/turbo4 delivered 3.18x speedup with only +0.02% PPL on WikiText-2.

Why it matters: A single RTX 5090 runs DFlash speculative decoding with KV cache compression on Qwen3.6-27B, hitting 3.26x speedup while q4_0 quantization adds only 0.02% perplexity.

r/LocalLLaMA

Weird to get near-linear scaling by adding another GPU?

A Reddit user benchmarked qwen3.6-27b-autoround-int4 on 1x3090 versus 2x3090. Narrative decode rose from 53 TPS to 94 TPS, and code decode rose from 62 TPS to 120 TPS, under no NVLink, 8x/8x PCIe, P2P automatically enabled, tensor parallelism set to 2, and different KV-cache settings.

Why it matters: A Reddit user ran int4 qwen3.6-27b on two 3090s over plain PCIe without NVLink and nearly doubled decode speed over one card, a concrete data point for adding a second local GPU.

Synced · WeChat

openJiuwen proposes MANGO for multi-agent flow networks

openJiuwen proposed MANGO, a multi-agent flow-network framework that combines reinforcement learning, textual gradients, and a Skip-k mechanism; using GPT-4o-mini, it reports a 12.8% accuracy gain over MaAS on MATH500 and a 5.1% F1 gain over AFlow on DROP.

Why it matters: openJiuwen's MANGO multi-agent flow network combines reinforcement learning, textual gradients and a Skip-k mechanism so models learn their own teaming and division of labor.

Synced · WeChat

Alibaba RTPurboV2 uses hundreds of training steps for 10x sparse attention

Alibaba’s RTP team released RTPurboV2, replacing 85% of attention heads with SWA and compressing the remaining 15% retrieval heads using low-rank projection, clustering, and dynamic top-p; the adaptation uses about 600 training steps and roughly 1M label tokens, with reported Prefill speedup up to 9.36x.

Why it matters: Alibaba's RTP team swapped 85% of attention heads for sliding window attention, adapting with about 600 training steps and 1 million tokens, for up to 9.36x faster prefill inference.

AI HOT (Curated Pool)

Apple Releases Third-Generation Apple Foundation Models (AFM)

Apple released its third-generation AFM family with five models. The RSS snippet says they span on-device use and Private Cloud Compute servers, with Google involved in customization for Apple Intelligence, Siri, and system-level tools.

Why it matters: Apple released its third-generation Apple Foundation Models, customized with Google, spanning five on-device and cloud models, including the 3-billion-parameter dense on-device AFM 3 Core.

AI HOT (Curated Pool)

Nvidia and SK Hynix Sign Multi-Year Pact to Develop Next-Generation AI Memory Chips

Nvidia and SK Hynix signed a multi-year pact to co-design future generations of memory chips for AI applications; the RSS snippet does not disclose product specifications, production timelines, or financial terms.

Why it matters: Nvidia and SK Hynix signed a multi-year deal to co-develop next-generation AI memory chips from the design stage, a direction worth reading given their roles in AI memory supply.

Jun 7Sunday

r/LocalLLaMA

Qwen3.6 35B-A3B on a Laptop: My Zero-to-One Moment

A Reddit user ran Qwen3.6 35B-A3B on an ASUS Zenbook Pro 14 with RTX 4060 8GB VRAM and 64GB RAM, reaching about 27 TPS at 32k context and 18 TPS at 256k context. The setup uses llama.cpp, unsloth’s IQ3_XXS GGUF quantization, and a 262144-token context flag.

Why it matters: A Reddit user got Qwen3.6 35B-A3B running on an Asus Zenbook Pro 14 with an RTX 4060 8GB and 64GB of RAM, giving concrete hardware reference for anyone deploying large models on similar laptops.