Skip to content

#部署/工程

11 today

Aug 3Monday

Hacker News front page

AirLLM runs 70B model inference on a single 4GB GPU

AirLLM is an open-source library that runs large models on consumer GPUs. It splits a model like Llama 3 70B into layers and loads them one at a time into VRAM, so a single 4GB GPU can handle inference without multi-GPU setups. It supports Llama, Mistral, ChatGLM, and other common architectures, and works with HuggingFace models. The trade-off is slower speed, but it lowers the hardware bar for local LLM inference to laptop level.

Why it matters: AirLLM loads a 70B model layer by layer so a 4GB GPU can run inference, dropping the hardware bar to laptop level at the cost of speed.

Jul 31Friday

Latent Space

GPT-5.6 price cut by 20%-80%: March's flagship intelligence now costs 1/13th the token price

OpenAI slashed GPT-5.6 Luna to $0.20/$1.20 per million tokens, an 80% drop. Terra fell 20%, and Sol got a 2.5x faster mode at 2x the price. Luna now matches GPT-5.4's March xhigh score of 51 on the AA benchmark, at roughly 1/13th the token cost. The cuts follow GPT-5.6 rewriting its own Triton and Gluon production kernels, saving 20% end-to-end, plus speculative decoding and KV cache improvements. The post notes an annualized ~2000x cost decline but warns public benchmarks like AA may be partially trained on, so discount the headline a bit.

Why it matters: OpenAI cut GPT-5.6 Luna input and output prices to $0.2 and $1.2 per million tokens, dropping the cost of equivalent intelligence 13-fold in four months.

Jul 24Friday

Hacker News front page

Echo routes prompts across open-weight models, claiming Fable-level results at 1/3 the cost

Echo is an experimental system from TracerML that pools open-weight models like GLM-5.2 and Kimi K2.7, then decides per request which models to invoke and how to combine their outputs. The author first computed a theoretical upper bound—if you could always pick the best model combination after seeing results, performance far exceeds any single model. Echo tries to approach that bound without knowing the answers in advance. On the author's own eval mix, Echo matched Fable's aggregate score at roughly one-third the inference cost. The post does not disclose specific benchmark names or absolute scores; methodology is at echo.tracerml.ai/eval. Known issues: routing and combination decisions sometimes fail, and the author is testing whether the approach holds for coding and agentic tasks. Caveat: the eval set is self-built, so saturation and representativeness are unknown—don't rush to benchmark against Fable yet.

Why it matters: TracerML pools open models like GLM-5.2 and Kimi K2.7 and picks the combination per request, aiming to approach the performance of picking the best model in hindsight without knowing the answer in advance.

Jul 15Wednesday

Hacker News front page

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

The author got Google's Gemma 4 26B MoE model running on a dual Xeon E5-2690 v2 server from 2013 with no GPU, costing under $300. The CPUs only support AVX1, but ik_llama.cpp's optimized kernels require AVX2, causing silent gibberish output. Claude diagnosed that the graph builder unconditionally emitted MOE_FUSED_UP_GATE ops while the dispatcher had no matching case, leaving ~240 tensors per forward pass reading uninitialized memory. After the fix, decode reaches ~5.2 tokens/sec and prompt eval ~16 tokens/sec. A PR is open but not yet merged. The post doesn't disclose quantized model memory usage or power draw.

Why it matters: The author runs the Gemma 4 26B mixture-of-experts model with no GPU on a 2013 dual-Xeon server costing under $300, generating about 5.2 tokens per second, close to human reading speed.

Jul 14Tuesday

Hacker News front page

MemStitch: Zero-copy KV cache stitching for vLLM cuts multi-agent TTFT by up to 25x

DaqulaLin open-sourced MemStitch, a gateway that sits in front of vLLM and stitches KV caches across requests at the memory level using PagedAttention. It skips redundant prefill in multi-agent workflows, cutting TTFT by up to 25x and saving over 40% VRAM. The post doesn't disclose the test model or GPU setup, so I'd discount the 25x claim until those details surface.

Why it matters: MemStitch stitches KV caches from different requests directly in vLLM's memory layer, skipping repeated prefill in multi-agent collaboration, cutting time to first token by up to 25x and memory use by over 40%.

Jul 9Thursday

Hacker News front page

Colibrì: A pure-C engine that runs GLM 5.2 on a 32 GB laptop with no GPU

Colibrì is a single-file C engine that runs the 744B-parameter GLM 5.2 model on a 32 GB laptop with no GPU. The dense part stays in RAM at int4 (~9.9 GB), while 21,504 routed experts stream from disk on demand with an LRU cache. Cold-start speed is 0.1 tok/s. The author built and tested it on a 12-core, 25 GB machine. The post does not include quality benchmarks against the full-precision model.

Why it matters: A 744B-parameter model running on a laptop with no GPU, by keeping the dense part in int4 memory and loading routed experts from disk.

Jul 7Tuesday

AI HOT (Curated Pool)

SGLang Integrates DSpark: Confidence-Driven, Variable-Length Speculative Decoding

LMSYS integrated DSpark speculative decoding into SGLang. The draft model's confidence head decides how many tokens to verify per request, instead of always verifying a full block. On H200 with DeepSeek-V4-Flash, it beats both MTP and non-spec baselines in throughput and per-user decode speed. The key engineering move is a ragged, variable-length verify under full CUDA graphs: when the scheduler trims the verify budget, the engine replays a genuinely smaller graph, not a padded one. A cost-table profiler lets the scheduler size each request's budget online, and a cap-accept mode exposes the acceptance ceiling hidden by trimming.

Why it matters: LMSYS plugs DSpark into SGLang, shortening verification for requests with low acceptance probability based on the draft model's confidence to save compute.

Jul 5Sunday

Computing Life · Share · Yage

Scaling Law's three corrections in five years: from bigger models to smaller models with more data

Scaling law is an empirically fitted curve, not a physical law. OpenAI's 2020 Kaplan paper concluded 'prioritize parameters' due to experimental biases, shaping GPT-3. DeepMind's 2022 Chinchilla corrected the ratio to 20:1, showing smaller models with more data outperform. Two 2024 replication studies confirmed that fixing Kaplan's setup reproduces Chinchilla's result—no fraud, just calibration. Since 2023, Meta and others deliberately deviate from Chinchilla: Llama 3 8B was trained on 15T tokens because the optimization target shifted from training cost to total cost of training plus inference. Tsinghua's Densing Law shows the parameter count needed for equal capability halves roughly every 3.5 months, but there is a floor: each parameter stores only ~2 bits of knowledge. The viral 'collapse' article cited a blog comment posted the same day as if it were peer-reviewed research; the post does not provide a paper source for that claim.

Why it matters: The piece traces OpenAI's Kaplan paper to a flawed experimental setup, explaining why it favored scaling parameters and how that shaped GPT-3.

Jun 30Tuesday

Hacker News front page

Moondream's Photon engine uses pipelined decoding to pop GPU bubbles, boosting decode throughput up to 35%

Moondream's Photon inference engine hits ~33ms VLM inference on NVIDIA B200. The bottleneck is GPU bubbles: the GPU idles while the CPU finishes bookkeeping between tokens. Photon pipelines the decode loop so the GPU starts the next forward before the CPU commits the current token, overlapping CPU housekeeping under GPU work. Three mechanisms make it safe: ping-pong slots prevent buffer collisions, forward-now-sample-later handles constrained decoding, and zombie cleanup deals with finished requests. The result is up to 35% higher decode throughput.

Why it matters: Moondream's Photon engine pushes VLM inference to about 33ms on a B200 and names the bottleneck: GPU idle bubbles while the CPU does bookkeeping, sampling and scheduling per token.

Hacker News front page

vLLM Semantic Router beats frontier models by making multiple models collaborate inside one API call

vLLM's Micro-Agent makes multiple models collaborate behind a single API endpoint. Users call one model name as usual; the router decides whether to try a cheaper model first, run several in parallel and aggregate, or let models review each other. The core idea is turning one API call into a bounded micro-collaboration with budget, topology, and failure policy. The post describes five looper patterns—Confidence, Ratings, ReMoM, Fusion, and Workflows—and claims they beat individual frontier models on benchmarks, though it does not name the benchmarks or provide scores.

Why it matters: vLLM turns semantic routing into a model collaboration engine, where one API call can pick a cheap model first, run parallel summaries or cross-review, all with failure handling.

Jun 19Friday

Hacker News front page

I restarted a 10-year-old Xeon 174 times to delete 12 flags and gain 4 tps

The author ablated all 25 flags from a previous Gemma 4 setup on a 2016 Xeon. Most did nothing. The real levers are flash attention, physical-core thread count, and workload-gated speculative decoding: on for code, off for long-doc summarization (54% faster without it). The auto-tune drafter setting is a crude per-request router that currently hurts long-context throughput.

Why it matters: A 174-run test of 25 startup flags finds only flash attention, physical core count and speculative decoding move speed, which helps 28% on code but hurts long-doc summaries by 54%.

Jun 14Sunday

r/LocalLLaMA

Xiaomi serves MiMo V2.5 at 1000–3000 tps with DFlash and Persistent Kernel

Xiaomi's MiMo V2.5 is live, claiming 1000–3000 tps via DFlash and Persistent Kernel. The DFlash model weights are out, and an open-source release is promised soon. The post body is blocked by Reddit security, so only the headline is available—no details on measured latency, concurrency, or hardware. I'd discount that tps figure: headline peaks usually assume optimal batching, and real single-user throughput is likely lower.

Why it matters: Xiaomi is serving MiMo V2.5 publicly at 1000 to 3000 tokens per second, has released the DFlash weights, and promises open-source code to follow.

Jun 13Saturday

Hacker News front page

Can I Buy Your KV Cache?

This paper proposes letting publishers precompute a document's KV cache so AI agents can buy and load it, skipping the most compute-heavy step: prefill. On Qwen3-4B, reuse is 9–50x cheaper than prefill with zero accuracy loss—token outputs match exactly. Shipping the KV cache fails because it's nearly incompressible and egress costs more than the prefill saved. The fix: host it provider-side, like production prompt caching. Serving one 3,774-token document to 80M agents costs ~$1.5M to re-prefill but only ~$30K via reuse, a 49.7x gap. The paper frames this as an agent-native prefill CDN and leaves lossless KV compression and cross-party payments as open problems.

Why it matters: The paper has publishers precompute document KV caches that other agents pay to load, skipping prefill; on Qwen3-4B reusing a cache costs 9 to 50 times less than re-prefilling, with identical output tokens.

Jun 12Friday

r/LocalLLaMA

MiniMax open-sources MSA, a sparse attention method that cuts attention compute by 28.4× at 1M tokens on a 109B model

MiniMax published a paper introducing MSA, a blockwise sparse attention built on GQA. A lightweight index branch scores KV blocks and picks a top-k subset per GQA group, then the main branch runs exact attention only on those blocks. With a co-designed GPU kernel, a 109B-parameter multimodal model achieves 14.2× prefill and 7.6× decoding wall-clock speedups on H800 at 1M context, matching full GQA quality. Code and inference kernel are open-sourced, along with a model called MiniMax-M3. The Reddit poster is curious whether the 109B model can run on consumer GPUs; the post doesn't say if weights will be released.

Why it matters: MiniMax's MSA scores KV cache blocks with an index branch and only runs exact attention on the top blocks, cutting attention compute 28.4x at a million tokens.

Jun 10Wednesday

AI HOT (Curated Pool)

OpenRouter Launches Advisor Tool for Low-Cost Models to Consult Stronger Models

OpenRouter released the Advisor server tool, letting GPT-4o Mini consult Claude Fable during generation, but the post does not disclose pricing, latency, or the routing policy.

Why it matters: OpenRouter planned an Advisor tool that would let cheap models like GPT-4o Mini call stronger ones such as Claude Fable mid-generation, but the blog page returns 404 and no pricing or routing details are public.

r/LocalLLaMA

ICML paper on predictable hallucination gate and ntkMirror open-weight implementation

An ICML 2026 paper presents an ISR=1 answer-abstain gate for evidence-grounded QA, and ntkMirror implements it for local open-weight models with multiple evidence orderings, reporting 0.0–0.7% hallucination at about 24% abstention in the held-out audit.

Why it matters: An ICML paper gates abstention on answer instability across repeated evidence orderings, and ships a training-free open implementation.

Jun 9Tuesday

AI HOT (Curated Pool)

Google Releases Gemini 3.5 Live Translate for Real-Time Speech Translation

Google released Gemini 3.5 Live Translate, a speech-to-speech translation model that supports more than 70 languages, starts translating before the speaker finishes, uses streaming updates, and runs through Gemini Live API, Google Meet preview, and Google Translate apps on iOS and Android.

Why it matters: Google released Gemini 3.5 Live Translate, covering more than 70 languages and translating as it listens instead of waiting for a full sentence, with latency down to a few seconds.

AI HOT (Curated Pool)

Google DeepMind Releases Gemma 4 12B, a Unified Encoder-Free Multimodal Model

Google DeepMind released Gemma 4 12B, a multimodal model with a unified encoder-free architecture, native audio input, Apache 2.0 licensing, and local laptop runtime with 16GB of VRAM or unified memory.

Why it matters: Google DeepMind open-sourced Gemma 4 12B, dropping separate vision and audio encoders so one Transformer handles images, text and speech, and it runs on 16GB of VRAM.

r/LocalLLaMA

Apple Announced New On-Device Inference Engine for Apple Silicon

Apple announced CoreAI at WWDC as a future CoreML replacement for Apple Silicon on-device inference; models require Python-script conversion, the supported list is mostly mid-2025 models, and the post does not disclose performance data.

Why it matters: Apple's CoreML successor runs models on Apple silicon, but models must first be converted with a Python script, and the support list is mostly mid-2025.

AI HOT (Curated Pool)

China Prepares $295 Billion Plan to Fund Nationwide AI Infrastructure Buildout

China plans to invest about 2 trillion yuan, or $295 billion, over five years to build nationwide data centers, with funding covering large-scale data center infrastructure for domestic AI development.

Why it matters: Bloomberg reports China is planning about 2 trillion yuan of AI infrastructure over five years, centered on large data centers to close the compute gap.

AI HOT (Curated Pool)

Xiaomi MiMo and TileRT Release UltraSpeed Mode, 1T Model Exceeds 1,000 Tokens/s

Xiaomi MiMo and TileRT released MiMo-V2.5-Pro-UltraSpeed, a 1T-parameter model mode exceeding 1,000 tokens/s, with API access open from June 9 to June 23, 2026, at 3× the MiMo-V2.5-Pro price and about 10× the speed.

Why it matters: Xiaomi MiMo and TileRT add an UltraSpeed mode to a 1T-parameter model, pushing output past 1000 tokens/s, with FP4 quantizing only MoE experts and block-level parallel speculative decoding.

AI HOT (Curated Pool)

Elon Musk Details SpaceX's AI1 Orbital AI Data Center Satellite Plan

Elon Musk detailed SpaceX’s AI1 orbital AI data center satellite plan, with 150 kW peak power per satellite, about 120 kW sustained compute power, and 6-8 ms round-trip latency at 600-800 km low Earth orbit.

Why it matters: Musk detailed SpaceX's AI1 orbital data center satellite, about 120 kW of sustained compute per unit, roughly one NVIDIA GB300 rack, with 6-8 ms laser link round-trip latency.

r/LocalLLaMA

2X tk/s on 1× MI50: Qwen3.6-27B inference rises from 19.4 to 38.1 tk/s

bigattichouse raised Qwen3.6-27B throughput on a single MI50 from 19.4 to 38.1 tk/s by running same-model parallel computations for Q8-or-lower quantization, exploiting unused compute lanes instead of adding a smaller speculative decoding model.

Why it matters: A user ran two concurrent Qwen3.6-27B computations on a single MI50, lifting generation from 19.4 to 38.1 tokens/s by exploiting compute small quants leave idle instead of loading a side model.

r/LocalLLaMA

New MLX LM Server From Apple

A Reddit post says Apple’s MLX LM Server uses continuous batching for concurrent sub-agent requests and supports distributed inference across multiple Macs via Thunderbolt RDMA.

Why it matters: Apple's MLX LM Server adds continuous batching and Thunderbolt-linked distributed inference across Macs, a concrete option for anyone running local models.

r/LocalLLaMA

Levi: Run AlphaEvolve on Your Local Qwen 30B

LEVI runs an AlphaEvolve-like search system with Qwen3-30B-A3B and reports tests on ADRS, IFBench, and HotpotQA, claiming up to 35x lower cost overall and up to 12x fewer evals under the same single-model, same-budget comparison.

Why it matters: The open-source Levi project runs AlphaEvolve-style search on a local Qwen3-30B-A3B, and its author measured up to 35x lower cost and 12x fewer evaluations at equal budget on three benchmarks.

Jun 8Monday

r/LocalLLaMA

Xiaomi claims 1,000+ tps on a 1T model using a standard 8-GPU server

Xiaomi MiMo claims MiMo-V2.5-Pro UltraSpeed runs a 1T-parameter MoE model above 1,000 output tokens per second on one standard 8-GPU node; the post does not disclose the GPU model, batch settings, or reproducible configuration.

Why it matters: Xiaomi's MiMo team says it hit over 1,000 tokens per second on a 1T-parameter MoE model on one standard 8-GPU server, without Cerebras or Groq custom hardware, a useful comparison against dedicated-chip routes.

Hacker News front page

MiMo-v2.5-Pro-UltraSpeed: 1T model with 1000 tokens per second

The title says Xiaomi MiMo-v2.5-Pro-UltraSpeed is a 1T model running at 1,000 tokens per second; the RSS body only provides the URL, Hacker News comments link, 66 points, and 14 comments, and the post does not disclose hardware, precision, context window, benchmark setup, or availability.

Why it matters: Xiaomi worked with TileRT on FP4-quantized MoE expert modules plus DFlash speculative decoding to push a 1T-parameter model past 1,000 tokens per second on 8 GPUs.

r/LocalLLaMA

Luce Spark: a 35B MoE on a 16 GB GPU, without the offload tax

Luce Spark runs Qwen3.6 35B-A3B at 13.3 GiB peak VRAM on an RTX 3090 by keeping hot experts on GPU, swapping cold experts through a bounded async cache, and using one fused graph for decode at about 100 tok/s.

Why it matters: A concrete recipe for running a 35B MoE on a 16 GB card, with the memory numbers and throughput to judge it by.

AI HOT (Curated Pool)

Xiaomi MiMo-V2.5-Pro-UltraSpeed Exceeds 1,000 Tokens/s

Xiaomi MiMo and TileRT_AI released MiMo-V2.5-Pro-UltraSpeed, running a 1T MoE model above 1,000 tokens/s on a single standard 8-GPGPU node, with UltraSpeed API priced at 3x and applications open from June 8 to 23 PDT.

Why it matters: Xiaomi and TileRT_AI ran a 1-trillion-parameter MoE model on a single 8-GPU node at over 1000 tokens per second without specialized hardware, offering a low-cost path for ordinary inference deployment.

r/LocalLLaMA

DFlash Speculative Decoding and KV Cache Compression on RTX 5090 Show 3.26x Speedup

The author tested Qwen3.6-27B on an RTX 5090 with DFlash plus KV cache compression, reaching up to 3.26x speedup; q4_0/turbo4 delivered 3.18x speedup with only +0.02% PPL on WikiText-2.

Why it matters: A single RTX 5090 runs DFlash speculative decoding with KV cache compression on Qwen3.6-27B, hitting 3.26x speedup while q4_0 quantization adds only 0.02% perplexity.

r/LocalLLaMA

Weird to get near-linear scaling by adding another GPU?

A Reddit user benchmarked qwen3.6-27b-autoround-int4 on 1x3090 versus 2x3090. Narrative decode rose from 53 TPS to 94 TPS, and code decode rose from 62 TPS to 120 TPS, under no NVLink, 8x/8x PCIe, P2P automatically enabled, tensor parallelism set to 2, and different KV-cache settings.

Why it matters: A Reddit user ran int4 qwen3.6-27b on two 3090s over plain PCIe without NVLink and nearly doubled decode speed over one card, a concrete data point for adding a second local GPU.

Synced · WeChat

openJiuwen proposes MANGO for multi-agent flow networks

openJiuwen proposed MANGO, a multi-agent flow-network framework that combines reinforcement learning, textual gradients, and a Skip-k mechanism; using GPT-4o-mini, it reports a 12.8% accuracy gain over MaAS on MATH500 and a 5.1% F1 gain over AFlow on DROP.

Why it matters: openJiuwen's MANGO multi-agent flow network combines reinforcement learning, textual gradients and a Skip-k mechanism so models learn their own teaming and division of labor.

Synced · WeChat

Alibaba RTPurboV2 uses hundreds of training steps for 10x sparse attention

Alibaba’s RTP team released RTPurboV2, replacing 85% of attention heads with SWA and compressing the remaining 15% retrieval heads using low-rank projection, clustering, and dynamic top-p; the adaptation uses about 600 training steps and roughly 1M label tokens, with reported Prefill speedup up to 9.36x.

Why it matters: Alibaba's RTP team swapped 85% of attention heads for sliding window attention, adapting with about 600 training steps and 1 million tokens, for up to 9.36x faster prefill inference.

AI HOT (Curated Pool)

Apple Releases Third-Generation Apple Foundation Models (AFM)

Apple released its third-generation AFM family with five models. The RSS snippet says they span on-device use and Private Cloud Compute servers, with Google involved in customization for Apple Intelligence, Siri, and system-level tools.

Why it matters: Apple released its third-generation Apple Foundation Models, customized with Google, spanning five on-device and cloud models, including the 3-billion-parameter dense on-device AFM 3 Core.

AI HOT (Curated Pool)

Nvidia and SK Hynix Sign Multi-Year Pact to Develop Next-Generation AI Memory Chips

Nvidia and SK Hynix signed a multi-year pact to co-design future generations of memory chips for AI applications; the RSS snippet does not disclose product specifications, production timelines, or financial terms.

Why it matters: Nvidia and SK Hynix signed a multi-year deal to co-develop next-generation AI memory chips from the design stage, a direction worth reading given their roles in AI memory supply.

Jun 7Sunday

r/LocalLLaMA

Qwen3.6 35B-A3B on a Laptop: My Zero-to-One Moment

A Reddit user ran Qwen3.6 35B-A3B on an ASUS Zenbook Pro 14 with RTX 4060 8GB VRAM and 64GB RAM, reaching about 27 TPS at 32k context and 18 TPS at 256k context. The setup uses llama.cpp, unsloth’s IQ3_XXS GGUF quantization, and a 262144-token context flag.

Why it matters: A Reddit user got Qwen3.6 35B-A3B running on an Asus Zenbook Pro 14 with an RTX 4060 8GB and 64GB of RAM, giving concrete hardware reference for anyone deploying large models on similar laptops.

QbitAI · WeChat

Chinese open-source framework targets stable 5-minute AI long-video generation

JD open-sourced JoyAI-Echo, a long audio-video generation framework for 5-minute consistent videos, using cross-modal memory, DMD post-training for about 7.5x faster inference, and real-time upscaling from 720P to 1K or 2K output.

Why it matters: JD's open-source JoyAI-Echo uses a cross-modal memory module to keep characters and scenes consistent across 5-minute videos, with roughly 7.5x faster inference.

AI HOT (Curated Pool)

AI Substitution Wave: Three Forces Reshape Cost Structures

Coinbase, Lindy, Harvey, and Cursor shifted workloads to cheaper models; Harvey reported Kimi 2.6 reached a 15% all-pass rate on Legal Agent Benchmark, versus Opus at 14%, with 100 tasks costing $84 versus $954.

Why it matters: Foundation labs moving up the stack, closed models raising prices and open models being good enough pushed Coinbase and Lindy to route or fully switch tasks to cheaper models.

Jun 6Saturday

AI HOT (Curated Pool)

OpenCV 5 Released with New DNN Engine and Native LLM Support

OpenCV 5 introduces a graph-based DNN engine, raising ONNX operator coverage from under 23% in 4.x to over 80%, with native support for Transformer, VLM, and LLM workloads.

Why it matters: OpenCV 5 moves the DNN engine to a compute graph with operator fusion, lifting ONNX operator coverage from under 23% in 4.x to over 80%, so models that previously would not import now run directly.

r/LocalLLaMA

Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding

Domino reports up to 5.8x throughput speedup on Qwen3 by decoupling causal modeling from autoregressive drafting in speculative decoding. The Reddit snippet links the arXiv paper, GitHub code, and Hugging Face models, but does not disclose hardware, baseline settings, dataset, or acceptance-rate details.

Why it matters: The paper proposes Domino, splitting causal modeling from autoregressive drafting in speculative decoding, measured at up to 5.8x throughput on Qwen3, a transferable split for inference optimization.