Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

161–180 of 609

Aug 11Tuesday

Mistral AI

Mistral launches regional inference endpoints and a Priority Tier, adding third-party open models like GLM-5.2

Mistral announced general availability of Mistral Regional Endpoints, letting customers choose whether inference runs in Europe or the US. Mistral Priority Tier also entered public preview, offering custom rate limits and an availability commitment backed by an SLA.

Why it matters: Mistral puts regional inference endpoints, a Priority Tier with an SLA and third-party open models on one infrastructure, a concrete sample of what AI sovereignty looks like in practice.

Aug 8Saturday

Dwarkesh Patel podcast

The Era of Continual Learning: AI That Learns From Every Session

Dwarkesh Patel argues that once models can update weights continuously from deployment, the whole AI landscape shifts. Instead of train-then-deploy, models will learn from every interaction like a human practicing saxophone—notes alone can't transfer the skill. This breaks the current regulatory assumption of pre-deployment checks; monthly or quarterly risk inspections make more sense. Alignment research must pivot from controlling frozen weights to preventing jailbreaks or backdoors during constant updates. Commercially, the leading lab's advantage compounds: more usage yields more feedback, making the model smarter and pushing labs to ship their best models earlier. Switching costs become massive—ditching a model that has learned your org's context for months is like firing a veteran employee for a clueless intern, creating durable high margins. Enterprises will face a trade-off: accept lock-in for a model that improves with use, or lose access to top-tier AI. Labs may subsidize users who allow training on their sessions. Continual learning also increases AI mind diversity, breaking today's monoculture of a few similar base models. On the inference side, per-company full weight updates create huge batching economies; for a sparse model like DeepSeek v3, optimal batch size exceeds 2,400 concurrent sequences.

Why it matters: It asks what happens to AI safety rules when model weights keep changing after deployment, and proposes scheduled risk reviews instead of a one-time pre-launch check.

Aug 5Wednesday

AI HOT (Curated Pool)

SpecForge v0.3: LMSYS releases a disaggregated speculative decoding training stack and new open draft models

SpecForge v0.3 decouples target-model inference from draft-model training. Patched SGLang servers capture features, Mooncake transports tensors, and trainer workers consume them independently. On an 8×H20 testbed, 3 servers + 5 trainers deliver ~10% higher end-to-end training throughput than the previous colocated design. The runtime now supports six speculative decoding families—EAGLE3, DFlash, Domino, DSpark, and more—and ships community-contributed draft models trained entirely on open data.

Why it matters: LMSYS splits SpecForge draft-model training into three independent resource pools for feature capture, data transfer and training, reaching about 10% higher throughput than the bundled setup on 8 H20s.

Aug 4Tuesday

Latent Space

The Inference Engineering Masterclass with Baseten's Philip Kiely and Ali Taha

Baseten just raised a $13B Series F. Philip Kiely and Ali Taha explain why inference engineering is now its own discipline, covering quantization, speculative decoding, KV-cache movement, and disaggregated prefill/decode. In one GLM-5.2 experiment, quantizing more layers preserved benchmark quality while boosting throughput 20% because errors across layers canceled out. They also detail grafting Kimi's vision encoder onto GLM-5.2 without touching the language model, and note that inference optimizations can still deliver 20% to 200% gains. The conversation touches on NVIDIA Dynamo, Rubin, video generation, and local inference, but the post doesn't expand on those.

Why it matters: Baseten's Philip Kiely and Ali Taha break down quantization, speculative decoding and KV cache movement, with a GLM-5.2 test showing no benchmark quality loss after multi-layer quantization and 20% higher throughput.

Aug 3Monday

Hacker News front page

AirLLM runs 70B model inference on a single 4GB GPU

AirLLM is an open-source library that runs large models on consumer GPUs. It splits a model like Llama 3 70B into layers and loads them one at a time into VRAM, so a single 4GB GPU can handle inference without multi-GPU setups. It supports Llama, Mistral, ChatGLM, and other common architectures, and works with HuggingFace models. The trade-off is slower speed, but it lowers the hardware bar for local LLM inference to laptop level.

Why it matters: AirLLM loads a 70B model layer by layer so a 4GB GPU can run inference, dropping the hardware bar to laptop level at the cost of speed.

Jul 31Friday

Latent Space

GPT-5.6 price cut by 20%-80%: March's flagship intelligence now costs 1/13th the token price

OpenAI slashed GPT-5.6 Luna to $0.20/$1.20 per million tokens, an 80% drop. Terra fell 20%, and Sol got a 2.5x faster mode at 2x the price. Luna now matches GPT-5.4's March xhigh score of 51 on the AA benchmark, at roughly 1/13th the token cost. The cuts follow GPT-5.6 rewriting its own Triton and Gluon production kernels, saving 20% end-to-end, plus speculative decoding and KV cache improvements. The post notes an annualized ~2000x cost decline but warns public benchmarks like AA may be partially trained on, so discount the headline a bit.

Why it matters: OpenAI cut GPT-5.6 Luna input and output prices to $0.2 and $1.2 per million tokens, dropping the cost of equivalent intelligence 13-fold in four months.

Jul 24Friday

Hacker News front page

Echo routes prompts across open-weight models, claiming Fable-level results at 1/3 the cost

Echo is an experimental system from TracerML that pools open-weight models like GLM-5.2 and Kimi K2.7, then decides per request which models to invoke and how to combine their outputs. The author first computed a theoretical upper bound—if you could always pick the best model combination after seeing results, performance far exceeds any single model. Echo tries to approach that bound without knowing the answers in advance. On the author's own eval mix, Echo matched Fable's aggregate score at roughly one-third the inference cost. The post does not disclose specific benchmark names or absolute scores; methodology is at echo.tracerml.ai/eval. Known issues: routing and combination decisions sometimes fail, and the author is testing whether the approach holds for coding and agentic tasks. Caveat: the eval set is self-built, so saturation and representativeness are unknown—don't rush to benchmark against Fable yet.

Why it matters: TracerML pools open models like GLM-5.2 and Kimi K2.7 and picks the combination per request, aiming to approach the performance of picking the best model in hindsight without knowing the answer in advance.

Jul 15Wednesday

Hacker News front page

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

The author got Google's Gemma 4 26B MoE model running on a dual Xeon E5-2690 v2 server from 2013 with no GPU, costing under $300. The CPUs only support AVX1, but ik_llama.cpp's optimized kernels require AVX2, causing silent gibberish output. Claude diagnosed that the graph builder unconditionally emitted MOE_FUSED_UP_GATE ops while the dispatcher had no matching case, leaving ~240 tensors per forward pass reading uninitialized memory. After the fix, decode reaches ~5.2 tokens/sec and prompt eval ~16 tokens/sec. A PR is open but not yet merged. The post doesn't disclose quantized model memory usage or power draw.

Why it matters: The author runs the Gemma 4 26B mixture-of-experts model with no GPU on a 2013 dual-Xeon server costing under $300, generating about 5.2 tokens per second, close to human reading speed.

Jul 14Tuesday

Hacker News front page

MemStitch: Zero-copy KV cache stitching for vLLM cuts multi-agent TTFT by up to 25x

DaqulaLin open-sourced MemStitch, a gateway that sits in front of vLLM and stitches KV caches across requests at the memory level using PagedAttention. It skips redundant prefill in multi-agent workflows, cutting TTFT by up to 25x and saving over 40% VRAM. The post doesn't disclose the test model or GPU setup, so I'd discount the 25x claim until those details surface.

Why it matters: MemStitch stitches KV caches from different requests directly in vLLM's memory layer, skipping repeated prefill in multi-agent collaboration, cutting time to first token by up to 25x and memory use by over 40%.

Jul 9Thursday

Hacker News front page

Colibrì: A pure-C engine that runs GLM 5.2 on a 32 GB laptop with no GPU

Colibrì is a single-file C engine that runs the 744B-parameter GLM 5.2 model on a 32 GB laptop with no GPU. The dense part stays in RAM at int4 (~9.9 GB), while 21,504 routed experts stream from disk on demand with an LRU cache. Cold-start speed is 0.1 tok/s. The author built and tested it on a 12-core, 25 GB machine. The post does not include quality benchmarks against the full-precision model.

Why it matters: A 744B-parameter model running on a laptop with no GPU, by keeping the dense part in int4 memory and loading routed experts from disk.

Jul 7Tuesday

AI HOT (Curated Pool)

SGLang Integrates DSpark: Confidence-Driven, Variable-Length Speculative Decoding

LMSYS integrated DSpark speculative decoding into SGLang. The draft model's confidence head decides how many tokens to verify per request, instead of always verifying a full block. On H200 with DeepSeek-V4-Flash, it beats both MTP and non-spec baselines in throughput and per-user decode speed. The key engineering move is a ragged, variable-length verify under full CUDA graphs: when the scheduler trims the verify budget, the engine replays a genuinely smaller graph, not a padded one. A cost-table profiler lets the scheduler size each request's budget online, and a cap-accept mode exposes the acceptance ceiling hidden by trimming.

Why it matters: LMSYS plugs DSpark into SGLang, shortening verification for requests with low acceptance probability based on the draft model's confidence to save compute.

Jul 5Sunday

Computing Life · Share · Yage

Scaling Law's three corrections in five years: from bigger models to smaller models with more data

Scaling law is an empirically fitted curve, not a physical law. OpenAI's 2020 Kaplan paper concluded 'prioritize parameters' due to experimental biases, shaping GPT-3. DeepMind's 2022 Chinchilla corrected the ratio to 20:1, showing smaller models with more data outperform. Two 2024 replication studies confirmed that fixing Kaplan's setup reproduces Chinchilla's result—no fraud, just calibration. Since 2023, Meta and others deliberately deviate from Chinchilla: Llama 3 8B was trained on 15T tokens because the optimization target shifted from training cost to total cost of training plus inference. Tsinghua's Densing Law shows the parameter count needed for equal capability halves roughly every 3.5 months, but there is a floor: each parameter stores only ~2 bits of knowledge. The viral 'collapse' article cited a blog comment posted the same day as if it were peer-reviewed research; the post does not provide a paper source for that claim.

Why it matters: The piece traces OpenAI's Kaplan paper to a flawed experimental setup, explaining why it favored scaling parameters and how that shaped GPT-3.

Jun 30Tuesday

Hacker News front page

Moondream's Photon engine uses pipelined decoding to pop GPU bubbles, boosting decode throughput up to 35%

Moondream's Photon inference engine hits ~33ms VLM inference on NVIDIA B200. The bottleneck is GPU bubbles: the GPU idles while the CPU finishes bookkeeping between tokens. Photon pipelines the decode loop so the GPU starts the next forward before the CPU commits the current token, overlapping CPU housekeeping under GPU work. Three mechanisms make it safe: ping-pong slots prevent buffer collisions, forward-now-sample-later handles constrained decoding, and zombie cleanup deals with finished requests. The result is up to 35% higher decode throughput.

Why it matters: Moondream's Photon engine pushes VLM inference to about 33ms on a B200 and names the bottleneck: GPU idle bubbles while the CPU does bookkeeping, sampling and scheduling per token.

Hacker News front page

vLLM Semantic Router beats frontier models by making multiple models collaborate inside one API call

vLLM's Micro-Agent makes multiple models collaborate behind a single API endpoint. Users call one model name as usual; the router decides whether to try a cheaper model first, run several in parallel and aggregate, or let models review each other. The core idea is turning one API call into a bounded micro-collaboration with budget, topology, and failure policy. The post describes five looper patterns—Confidence, Ratings, ReMoM, Fusion, and Workflows—and claims they beat individual frontier models on benchmarks, though it does not name the benchmarks or provide scores.

Why it matters: vLLM turns semantic routing into a model collaboration engine, where one API call can pick a cheap model first, run parallel summaries or cross-review, all with failure handling.

Jun 19Friday

Hacker News front page

I restarted a 10-year-old Xeon 174 times to delete 12 flags and gain 4 tps

The author ablated all 25 flags from a previous Gemma 4 setup on a 2016 Xeon. Most did nothing. The real levers are flash attention, physical-core thread count, and workload-gated speculative decoding: on for code, off for long-doc summarization (54% faster without it). The auto-tune drafter setting is a crude per-request router that currently hurts long-context throughput.

Why it matters: A 174-run test of 25 startup flags finds only flash attention, physical core count and speculative decoding move speed, which helps 28% on code but hurts long-doc summaries by 54%.

Jun 14Sunday

r/LocalLLaMA

Xiaomi serves MiMo V2.5 at 1000–3000 tps with DFlash and Persistent Kernel

Xiaomi's MiMo V2.5 is live, claiming 1000–3000 tps via DFlash and Persistent Kernel. The DFlash model weights are out, and an open-source release is promised soon. The post body is blocked by Reddit security, so only the headline is available—no details on measured latency, concurrency, or hardware. I'd discount that tps figure: headline peaks usually assume optimal batching, and real single-user throughput is likely lower.

Why it matters: Xiaomi is serving MiMo V2.5 publicly at 1000 to 3000 tokens per second, has released the DFlash weights, and promises open-source code to follow.

Jun 13Saturday

Hacker News front page

Can I Buy Your KV Cache?

This paper proposes letting publishers precompute a document's KV cache so AI agents can buy and load it, skipping the most compute-heavy step: prefill. On Qwen3-4B, reuse is 9–50x cheaper than prefill with zero accuracy loss—token outputs match exactly. Shipping the KV cache fails because it's nearly incompressible and egress costs more than the prefill saved. The fix: host it provider-side, like production prompt caching. Serving one 3,774-token document to 80M agents costs ~$1.5M to re-prefill but only ~$30K via reuse, a 49.7x gap. The paper frames this as an agent-native prefill CDN and leaves lossless KV compression and cross-party payments as open problems.

Why it matters: The paper has publishers precompute document KV caches that other agents pay to load, skipping prefill; on Qwen3-4B reusing a cache costs 9 to 50 times less than re-prefilling, with identical output tokens.

Jun 12Friday

r/LocalLLaMA

MiniMax open-sources MSA, a sparse attention method that cuts attention compute by 28.4× at 1M tokens on a 109B model

MiniMax published a paper introducing MSA, a blockwise sparse attention built on GQA. A lightweight index branch scores KV blocks and picks a top-k subset per GQA group, then the main branch runs exact attention only on those blocks. With a co-designed GPU kernel, a 109B-parameter multimodal model achieves 14.2× prefill and 7.6× decoding wall-clock speedups on H800 at 1M context, matching full GQA quality. Code and inference kernel are open-sourced, along with a model called MiniMax-M3. The Reddit poster is curious whether the 109B model can run on consumer GPUs; the post doesn't say if weights will be released.

Why it matters: MiniMax's MSA scores KV cache blocks with an index branch and only runs exact attention on the top blocks, cutting attention compute 28.4x at a million tokens.

Jun 10Wednesday

AI HOT (Curated Pool)

OpenRouter Launches Advisor Tool for Low-Cost Models to Consult Stronger Models

OpenRouter released the Advisor server tool, letting GPT-4o Mini consult Claude Fable during generation, but the post does not disclose pricing, latency, or the routing policy.

Why it matters: OpenRouter planned an Advisor tool that would let cheap models like GPT-4o Mini call stronger ones such as Claude Fable mid-generation, but the blog page returns 404 and no pricing or routing details are public.

r/LocalLLaMA

ICML paper on predictable hallucination gate and ntkMirror open-weight implementation

An ICML 2026 paper presents an ISR=1 answer-abstain gate for evidence-grounded QA, and ntkMirror implements it for local open-weight models with multiple evidence orderings, reporting 0.0–0.7% hallucination at about 24% abstention in the held-out audit.

Why it matters: An ICML paper gates abstention on answer instability across repeated evidence orderings, and ships a training-free open implementation.