Skip to content

#部署/工程

11 today

May 4Monday

r/LocalLLaMA

Pushing a 5-Year-Old 6GB VRAM Laptop to Its Limits: Qwen3.6-35B-A3B

Reddit user abhinand05 ran Qwen3.6-35B-A3B on a 5-year-old Asus ROG Zephyrus G14, reaching about 23 t/s plugged in and 10+ t/s unplugged. The setup uses RTX 2060 Max-Q 6GB, 24GB DDR4, Ryzen 7, plus llama-server configs for 64k and 128k context. The key detail is the mix of CPU MoE, KV-cache quantization, and ngram speculative decoding.

Why it matters: A Reddit user got a 35B model running on a 5-year-old laptop with 6GB of VRAM, about 23 tokens per second plugged in, showing what CPU hybrid inference can do under tight memory.

r/LocalLLaMA

Could PC x64 Instruction Extensions Relieve Hardware Shortage?

Intel and AMD unveiled ACE, an x86 extension claiming 1,024 multiplications per clock. It uses 2D tile registers and outer-product algorithms, versus 64 multiplications for AVX. No ACE hardware is released; power, framework support, and shipping timelines are not disclosed.

Why it matters: Intel and AMD's ACE extensions use 2D tile registers and outer-product math to lift multiplies per clock from AVX's 64 to 1024, but no supporting hardware has shipped.

May 3Sunday

r/LocalLLaMA

Paper on Hummingbird+: low-cost FPGAs for LLM inference

A Hummingbird+ paper claims low-cost FPGAs run Qwen3-30B-A3B Q4 at 18 t/s generation. The title lists 24GB memory and an expected $150 mass-production cost; the post does not disclose FPGA model, power, or test conditions.

Why it matters: The Hummingbird+ paper runs a Q4 Qwen3-30B-A3B on a low-cost FPGA at 18 tokens per second with 24GB on board, and the authors expect roughly $150 at mass production.

Xinzhiyuan · WeChat

Claude Code helps Anthropic double revenue pace in two months

Semi Analysis says Anthropic’s ARR reached $44B, adding $35B over 12 months. Claude Code hit $2.5B annualized revenue by Feb 2026, while inference gross margin rose from 38% to over 70%. The key test is keeping enterprise usage, coding-agent revenue, and inference margin together.

Why it matters: Semi Analysis reports Anthropic at $44 billion annualized revenue, with Claude Code contributing $2.5 billion and inference gross margin up from 38% to over 70%.

QbitAI · WeChat

DeepSeek V4’s biggest omission

DeepSeek V4’s technical report omits Engram while listing mHC, CSA, HCA, Muon, and FP4. Engram was open-sourced by DeepSeek and Peking University in January, inserting lookup modules between Transformer layers 2 and 15; its 27B test raised MMLU by 3.4 and Multi-Query NIAH to 97.0%. The engineering signal is CXL pooling: 8 servers shared a 4TB memory pool with under 5% throughput loss.

Why it matters: The DeepSeek V4 technical report leaves out the Engram lookup module that Peking University open-sourced in January, which lifted MMLU by 3.4 points at 27B scale and hit 97.0% on multi-query needle-in-a-haystack.

r/LocalLLaMA

Implemented TurboQuant, but results do not fully match the paper

A Reddit user reimplemented TurboQuant and found the PROD variant reached about 95.8% correlation at 4-bit, below the paper’s 99%+ claim. They report degraded attention quality, with about 67% top-1 accuracy in a simple simulation. The key issue is correlation versus ranking preservation in KV cache quantization.

Why it matters: A Reddit user reproduced TurboQuant's KV cache quantization method and measured about 95.8% correlation with original precision for the 4-bit PROD variant, below the paper's claimed 99%-plus.

r/LocalLLaMA

Built a C++17 transformer from scratch with 0.83M params and CPU training

Reddit user Suspicious_Gap1121 released Quadtrix.cpp, a C++17 GPT-style model with 0.83M parameters. It uses 4 layers, 4 heads, 200d width, and a 128-character context; one CPU core trained on 31.4M characters for 76.2 minutes to 1.6371 nats val loss. The key detail is handwritten backprop for LayerNorm, attention, Q/K/V, dropout, and AdamW without PyTorch, BLAS, or autograd.

Why it matters: A from-scratch C++17 transformer with hand-written backprop shows what a small GPT actually costs to train without PyTorch or BLAS.

May 2Saturday

r/LocalLLaMA

Qwen3.6-27B hits 72 tok/s on RTX 3090 with native vLLM on Windows

Reddit user One_Slip1455 released a native Windows vLLM launcher for Qwen3.6-27B, reaching 72 tok/s on an RTX 3090. It reports 64.5 tok/s at ~25k tokens, 53.4 tok/s at 127k ctx on one GPU, and 160k ctx with PP=2 on 2×3090. The key detail is no WSL or Docker, an OpenAI-compatible endpoint, and an INT4 quant path.

Why it matters: A native Windows vLLM launcher runs Qwen3.6-27B on one RTX 3090 without WSL or Docker, hitting 72 tok/s at short context.

QbitAI · WeChat

Tencent Hunyuan open-sources 440MB offline translation model, claims Google Translate quality lead

Tencent Hunyuan open-sourced Hy-MT1.5-1.8B-1.25bit, compressing a 1.8B translation model to 440MB. It supports 33 languages and 1,056 directions, with an Android demo running offline on Snapdragon 888 and 8GB RAM. The key detail is Sherry 1.25-bit quantization: 3 of every 4 weights use 1 bit and 1 is zeroed.

Why it matters: Tencent open-sourced a 1.8B translation model that quantizes to 1.25 bits and 440MB, runs offline on a Snapdragon 888 phone, and covers 33 languages and 1,056 translation directions.

May 1Friday

r/LocalLLaMA

PFlash: 10x prefill speedup over llama.cpp at 128K on an RTX 3090

PFlash cuts Qwen3.6-27B Q4_K_M 128K TTFT to 24.8s on an RTX 3090, versus 248.4s cold for llama.cpp. It uses a Qwen3-0.6B drafter to score token importance, keeps 5% of spans, and runs C++/CUDA without Python, Triton, or PyTorch. The quality caveat is clear: only NIAH single-needle passes from 32K to 128K; RULER and multi-needle results are not disclosed.

Why it matters: PFlash uses Qwen3-0.6B to pick the 5% of chunks that matter, cutting first-token latency on a 128K-token RTX 3090 run from 248.4s to 24.8s.

r/LocalLLaMA

OpenAI's Privacy Filter vs GLiNER on 600 PII Samples

A Reddit user compared openai/privacy-filter and GLiNER large-v2.1 on 600 PII samples. On CPU, OpenAI's model ran 2.8 samples/s versus 1.1 for GLiNER; English boundary macro F1 was 0.498 versus 0.416. The key issue is tokenizer offset: strict matching drops openai/privacy-filter to 0.155.

Why it matters: A Reddit user ran OpenAI's Privacy Filter against GLiNER large-v2.1 on 600 PII samples on CPU only, at 2.8 versus 1.1 items per second.

r/LocalLLaMA

16x Spark Cluster Build Update

Reddit user Kurcide finished a 16-node DGX Spark cluster, with all nodes hitting line rate on the fabric. Each node uses one QSFP56 link to an FS N8510, showing 100–111 Gbps per rail and about 200 Gbps aggregate. The key angle is unified memory: 8 nodes served 434GB GLM-5.1-NVFP4, with DeepSeek and Kimi tests next.

Why it matters: A hands-on build showing how 16 DGX Sparks link over QSFP56 at line rate and run a 434GB model across 8 nodes, useful for anyone planning local multi-node inference.

Financial Times · Technology

Huawei’s AI chip sales surge as Nvidia stalls in China

Huawei received large AI processor orders from Chinese tech companies as Nvidia stalls in China. The post does not disclose order value, chip models, or delivery timing. The key issue is China’s domestic compute substitution path, not one sales headline.

Why it matters: Huawei landed large AI processor orders from Chinese tech companies while Nvidia is blocked in China by export controls, showing the shift in the domestic compute supply chain.

r/LocalLLaMA

32x AMD MI50 32GB runs Kimi K2.6 at 9.7 t/s TG and 264 t/s PP

Reddit user ai-infos ran Kimi K2.6 int4 on 32 AMD MI50 32GB GPUs, reaching 9.7 tok/s TG on 136 output tokens. PP hit 263 tok/s on 14,564 input tokens using vllm-gfx906-mobydick across two 16-GPU nodes over 10G Ethernet. Power was about 640W idle and 4,800W peak inference; PCIe bandwidth and the vLLM distributed stack are the real bottlenecks.

Why it matters: A concrete cost-per-token datapoint for anyone weighing used AMD MI50s against new accelerators for self-hosted inference.

r/LocalLLaMA

Follow-up: Qwen3.6-27B on 1× RTX 3090 reaches ~218K context and stable tool calls

A Reddit user ran Qwen3.6-27B on one RTX 3090, reporting ~218K context at 50/66 TPS. After fixing Genesis PN12 patch anchor drift, ~25K-token tool outputs stopped OOMing; 198K plus vision reached 51/68 TPS. Single-prompt single-GPU runs still hit a second memory cliff near 50–60K.

Why it matters: A single 24GB RTX 3090 running Qwen3.6-27B at roughly 218K context and 50 to 66 tokens per second, with tool-call memory overflow fixed.

r/LocalLLaMA

Long-context coding on RTX 5080 16GB: Qwen3.6-35B-A3B holds 30 t/s at 128K

A Reddit user tested a local coding-agent setup on RTX 5080 16GB; the title says Qwen3.6-35B-A3B reaches 30 t/s at 128K. The post lists Ryzen 9700X, 96GB DDR5, Windows 11, and CUDA 12.9.1 as required. Qwen3.6-27B dense hit only 3.2 t/s at 128K, so the key path is KV quantization plus MoE offload.

Why it matters: A 16GB RTX 5080 runs a Qwen3.6-35B-A3B coding agent at 128K context and still holds 30 t/s, with no reported quality drop.

NVIDIA Blog

Nemotron Labs: What OpenClaw Agents Mean for Every Organization

NVIDIA says OpenClaw reached 250,000 GitHub stars by March 2026, passing React within 60 days. OpenClaw is Peter Steinberger’s self-hosted persistent agent; NVIDIA introduced NemoClaw with OpenShell sandboxing and Nemotron models. The key issue is governance: the post claims reasoning AI raised token use 100x, and autonomous agents add another 1,000x.

Why it matters: OpenClaw's self-hosted agents hit 250,000 GitHub stars in 60 days, faster than React, prompting NVIDIA to weigh token costs and safety limits for always-on agents.

The Verge · AI

Elon Musk confirms xAI used OpenAI’s models to train Grok

Elon Musk testified Thursday in a California federal court that xAI used OpenAI models to improve Grok. The mechanism described is model distillation: a larger teacher model transfers knowledge to a smaller student model. The post does not disclose which OpenAI models, data volume, or training runs.

Why it matters: Musk testified in California federal court that xAI partly used OpenAI models to improve Grok through distillation, laying out how the two firms' models relate.

Apr 30Thursday

claude.dev Blog

Building Claude Code: designing agents around prompt caching

The Claude Code team shared engineering lessons on prompt caching, keeping the request prefix stable to cut latency and cost in long agent sessions. Put static content first, update state through messages, avoid switching models or adding and removing tools mid-session, and use defer_loading to load tools late. When compacting context, reuse the original session's system prompt, tool definitions and history, then append the compaction request so it hits the cache. The team also tracks cache hit rate and treats an abnormal drop as an incident.

Why it matters: The Claude Code team applies cache prefix stability to prompt ordering, tool loading and context compaction, a transferable caching method for long agent sessions.

Latent Space

[AINews] The Inference Inflection

Latent Space argues inference demand has hit an inflection point, citing its Apr 28-29, 2026 AINews roundup. Jensen Huang is quoted saying per-task compute rose about 10,000x in two years, with usage up about 100x. The key watchpoints are CPU sandboxes, agent harnesses, and split inference workloads.

Why it matters: Jensen Huang's GTC numbers show per-task compute up roughly 10,000x in two years and total usage up about 100x, and Latent Space ties that inference demand to real workloads like CPU sandboxes and agent scheduling.

Bloomberg Technology

Samsung’s Chip Profit Soars 48-Fold Due to AI Spending Spree

Samsung Electronics’ chip unit posted a 48-fold profit jump in the March quarter, driven by AI data-center orders. The RSS snippet says profit hit a record and beat expectations, but the post does not disclose profit value, memory type, or customers.

Why it matters: Samsung's chip division posted a 48-fold year-on-year profit jump, driven directly by AI data center memory buying that is tightening supply.

r/LocalLLaMA

Building a fully local PDF-to-audiobook workflow with Kokoro 82M, Qwen and llama.cpp

Reddit user purellmagents shared a local PDF-to-audiobook workflow using Kokoro 82M, Qwen 3.5 0.8B/2B, and llama.cpp. The Tauri 2.0 app runs on an M1 Mac, reads 15 initial sentences, then prepares the next 15. The hard parts are PDF-text alignment, code snippets, tables, and first-generation latency.

Why it matters: A Reddit user built a fully local PDF-to-audiobook pipeline on an M1 Mac with Kokoro 82M, Qwen and llama.cpp, reading 15 sentences while preparing the next 15 to cut waiting time.

Dwarkesh Patel podcast

Reiner Pope: The Math Behind How LLMs Are Trained and Served

Dwarkesh interviewed Reiner Pope in a 1-session blackboard lecture on LLM training and serving. The post lists 7 timestamps on batch size, MoE rack layout, pipeline parallelism, KV cache, and API pricing. The key mechanism is cost: without batching, serving economics can be 1,000x worse.

Why it matters: MatX CEO Reiner Pope breaks down the cost structure of large models from training to serving with formulas and chalk, showing batching affects serving economics by a factor of 1000.

Apr 29Wednesday

Xinzhiyuan · WeChat

Google Translate Turns 20 as Pichai Highlights Four AI Generations

Google Translate turned 20 on April 28, and Pichai said it now has 1B monthly users. The post traces four AI phases: SMT, GNMT, PaLM 2, and Gemini 2.5 Flash Native Audio, including 110 languages added in 2024. The key shift is native speech-to-speech translation that preserves intonation, pacing, and pitch.

Why it matters: Twenty years of Google Translate, traced by Sundar Pichai from statistical machine translation to Gemini 2.5 Flash native audio translation that handles sound directly to keep tone and rhythm.

Financial Times · Technology

How OpenAI’s $500bn Data Centre Venture Stargate Has Shifted Shape

OpenAI’s Stargate data centre venture is valued at $500bn. The RSS snippet says Sam Altman’s flexible infrastructure approach unsettles partners but boosts compute lead; the post does not disclose structure, partners, or timeline.

Why it matters: OpenAI's Stargate data center plan has changed shape several times, and Sam Altman's flexibility on infrastructure unsettles partners while helping OpenAI pull ahead on compute.

r/LocalLLaMA

Qwen3.6 27B on Dual RTX 5060 Ti 16GB with vLLM: ~60 tok/s, 204k Context Working

A user ran Qwen3.6 27B with vLLM on dual RTX 5060 Ti 16GB cards, reaching ~62–66 tok/s at 8K. The setup used 32GB VRAM, TP=2, fp8 KV cache, MTP 3 tokens, and a 204800 context window. The tight part is memory: after a 168k prefill, each GPU used ~15.65GiB with max_num_seqs=1.

Why it matters: A user ran Qwen3.6 27B on two RTX 5060 Ti 16GB cards with vLLM and measured 62–66 tok/s at 8K context, with the exact tensor-parallel, fp8 KV cache and MTP settings listed.

X · @dotey

Microsoft VibeVoice-ASR tested on Mac for a one-hour podcast

Simon Willison ran 4-bit VibeVoice-ASR on an M5 Max MacBook Pro and transcribed a one-hour podcast in 8m45s. The 9B MIT-licensed model supports 60-minute audio, 50+ languages, and structured speaker output. Memory is the constraint: prefill peaked at 61.5GB, making 32GB laptops impractical.

Why it matters: Simon Willison ran Microsoft's open-source VibeVoice-ASR on an M5 Max with 128GB of memory: 8 minutes 45 seconds to transcribe a one-hour podcast, with prefill memory peaking at 61.5GB.

Computing Life · Share · Yage

DeepSeek V4 Explained: Engineering Decisions Around Agentic Workloads

DeepSeek V4 targets long-horizon agent tasks with a 1M context. The snippet cites hybrid attention, OPD, Muon, and mHC; the post does not disclose size, data, pricing, or release timing.

Why it matters: The piece reads the DeepSeek V4 technical report as engineering trade-offs for long-horizon agent tasks, tracing how choices like hybrid attention play out over long chains.

Sinocism (Bill Bishop)

April Politburo Meeting, Manus Mess, and Possible New US Semiconductor Restrictions

China’s April Politburo meeting called for full implementation of the “AI+” initiative and listed computing power networks among six infrastructure networks. The readout signals no new stimulus, but stresses AI governance, supply-chain control, and rectifying involution-style competition.

Why it matters: The April Politburo meeting put compute networks alongside water and power grids as one of six infrastructure categories, moved "AI+" from slogan to full implementation, and added language on curbing cutthroat competition.

r/LocalLLaMA

XiaomiMiMo MiMo-V2.5: Sparse MoE with 310B total and 15B activated parameters

XiaomiMiMo shared MiMo-V2.5 with 310B total parameters and 15B activated parameters. The post only links Hugging Face and says it runs on more “human” configs than its larger sibling. It does not disclose VRAM needs, quantization, or benchmarks.

Why it matters: Xiaomi released the non-Pro MiMo-V2.5 on Hugging Face, a sparse MoE with 310B total parameters and 15B activated per inference, calling on large capacity at lower compute.

Apr 28Tuesday

Synced · WeChat

ACL 2026: Huawei Taylor Lab Proposes SHAPE, Adding a Reasoning Tax to LLM Inference

Huawei Taylor Lab, Peking University, and Shanghai University of Finance and Economics proposed SHAPE, accepted by ACL 2026, with about 3% average accuracy gain. It uses entropy segmentation, short rollouts for potential estimation, dynamic length discounts, and token-level credit assignment, cutting token use by about 30%. The key mechanism is a reasoning tax: long high-potential late-stage segments are penalized to reduce verbose confirmation loops.

Why it matters: Huawei's Taylor Lab and others propose SHAPE, which segments by entropy and probes short spans to estimate potential, then discounts long late sections so models answer math problems more accurately and more briefly.

Latent Space

Physical AI that Moves the World — Qasar Younis & Peter Ludwig, Applied Intuition

Applied Intuition’s founders reviewed a 10-year physical AI path, with the company valued at $15B. The post cites 30+ products, 18 of the top 20 non-Chinese automakers as customers, and L4 driverless trucks in Japan. The key constraint is onboard deployment: millisecond latency, low power, small models, and safety validation.

Why it matters: Applied Intuition's CEO and CTO trace the path from a YC self-driving toolchain to a $15B physical AI company, with 30-plus products and 18 of the top 20 non-Chinese automakers as customers.

Apr 27Monday

Hacker News front page

Show HN: Utilyze — an open-source GPU monitoring tool claiming higher accuracy than nvtop

Systalyze open-sourced Utilyze to measure real GPU compute efficiency in production, with negligible overhead claimed. The post says nvidia-smi and nvtop only check whether any kernel runs during the sampling window; an H100 has 132 SMs and 17,424 cores. The key issue is real throughput headroom, not binary utilization dashboards.

Why it matters: Utilyze measures GPU load by effective compute efficiency instead of the binary busy flag in nvidia-smi and nvtop, so a single busy core on an H100 no longer reads as 100%.

Hacker News front page

Running Local LLMs Offline on a Ten-Hour Flight

Dmitri Lerko ran Gemma 4 31B and Qwen 4.6 36B locally during a 10-hour flight with no Wi‑Fi. The MacBook Pro M5 Max had 128GB unified memory and a 40-core GPU; sustained load used about 1% battery per minute, and performance degraded past 100k tokens. The sharp finding is instrumentation: an iPhone cable delivered 60W, while a MacBook cable delivered 94W under the same load.

Why it matters: The author ran two local models offline on a maxed-out MacBook Pro M5 Max during a ten-hour flight, and logged how 60W versus 94W charging changed inference.

Mistral AI

Mistral AI opens public preview of Workflows

Mistral AI has put Workflows, its enterprise AI orchestration layer, into public preview. It offers durable execution, observability and human-in-the-loop approvals. ASML, ABANCA and CMA-CGM are already using it to automate critical processes.

Why it matters: Mistral AI opens Workflows, its enterprise AI orchestration layer, in public preview, with ASML and ABANCA already automating key processes and material showing durable execution, observability and human approval in freight release and KYC review.

Hacker News front page

AI can cost more than human workers now

Axios says some firms now spend more on AI than salaries; Nvidia's Bryan Catanzaro says compute costs exceed employee costs. Gartner forecasts 2026 IT spending at $6.31T, up 13.5%, driven by AI infrastructure, software, and cloud. Watch token costs: Uber's CTO has already exhausted the 2026 AI budget.

Why it matters: An Nvidia executive says the team's compute costs far exceed headcount, and Uber's CTO burned its entire 2026 AI budget on tokens, signs that AI bills now outrun payroll at specific companies.

QbitAI · WeChat

DeepSeek V4 Cuts Prices Permanently; Cached Inputs Get 90% Off, Coding Test Costs Drop 83%

DeepSeek V4 cut prices twice in two days: input/output pricing is 75% lower, with cached inputs getting another 90% off. QbitAI’s coding test fell from 31.73 yuan for 35M tokens to 5.34 yuan under new pricing, an 83% drop. The key case is high cache-hit workloads, with V4-Pro at about 95–96% cache hits.

Why it matters: DeepSeek V4 cut input and output prices 75% twice in two days, with cached input down another 90%, and QbitAI measured coding cost falling from 31.73 yuan to 5.34 yuan.

Hacker News front page

TurboQuant: A First-Principles Walkthrough

TurboQuant walkthrough explains compressing AI vectors to 2–4 bits per coordinate. It uses random rotation to map high-dimensional coordinates to a fixed distribution, then reuses one codebook with no scale overhead, training, or calibration.

Why it matters: TurboQuant compresses high-dimensional vectors to 2 to 4 bits per coordinate, fixing the coordinate distribution with a random rotation and using a universal codebook instead of per-group scaling, with no training or calibration.

Apr 26Sunday

Hacker News front page

DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles

SGLang and Miles added day-0 inference and RL support for DeepSeek-V4, covering 1.6T Pro and 284B Flash. The post cites a 1M-token context, FP4 MoE expert weights, 128-token SWA, and 4:1 or 128:1 KV compression. The key systems detail is ShadowRadix coherence across three KV pools and two compression-state pools.

Why it matters: SGLang and Miles shipped inference and RL training support on DeepSeek-V4's launch day, covering the 1.6T-parameter Pro and 284B-parameter Flash versions, and disclosed deployment details including hybrid sparse attention and FP4 expert weights.

Apr 25Saturday

Latent Space

DeepSeek V4 Pro and Flash released, runnable on Huawei Ascend chips

DeepSeek released V4 Pro and V4 Flash, with 1.6T/49B active and 284B/13B active parameters. Both support 1M-token context, Base/Instruct variants, and an MIT license; the report claims 27% FLOPs and 10% KV cache versus V3.2 at 1M tokens. The key point is Huawei CANN compatibility, not just benchmarks, because it reduces CUDA dependence.

Why it matters: DeepSeek's V4 Pro and V4 Flash both take a 1M-token context, cut long-text compute to 27% of V3.2, and run on Huawei Ascend chips.