Skip to content

#部署/工程

11 today

May 8Friday

QbitAI · WeChat

HIT and Huawei propose Dynamic-dLLM, a training-free acceleration framework with 4.48x speedup

HIT Shenzhen, Huawei, and Shenzhen Hetao College proposed Dynamic-dLLM, a training-free dLLM acceleration framework that raises LLaDA-8B-Instruct throughput on GSM8k from 8.32 TPS to 37.29 TPS with almost no accuracy loss.

Why it matters: Harbin Institute of Technology (Shenzhen) and Huawei propose Dynamic-dLLM, a training-free acceleration framework that lifts GSM8k throughput on LLaDA-8B-Instruct from 8.32 to 37.29 tokens per second with almost no accuracy loss.

Financial Times · Technology

Big Tech’s $725bn AI Spending Spree Sends Free Cash Flow to a Decade Low

Big Tech is spending $725 billion on AI infrastructure, and the title says free cash flow has fallen to a decade low; the RSS snippet says Silicon Valley giants shifted from asset-light cash generators to infrastructure investors, but it does not disclose the company list, time period, or accounting basis.

Why it matters: Big Tech has committed $725bn to AI infrastructure, pushing free cash flow to a decade low as these firms shift from asset-light cash cows to heavy infrastructure spending.

r/LocalLLaMA

11.67% ARC-AGI-2 Local Eval on a Single 4090: The TOPAS Recursive Architecture

Doug_Bitterbot says TOPAS scored 11.67% on ARC-AGI-2 using one RTX 4090 after about 14 days of training. The 100M-parameter checkpoint hit 36% locally, but recursive TTT caused null outputs on nearly half of Kaggle puzzles. The key detail is time management: the author expects 20% after threshold tuning and 3-5 more weeks of training.

Why it matters: About 14 days of training on one RTX 4090 got the 100M-parameter TOPAS to 11.67% on ARC-AGI-2, with local runs reaching 36%, while test-time training recursion left nearly half the Kaggle tasks with empty output.

Bloomberg Technology

Nvidia to Invest Up to $2.1 Billion in Data Center Firm IREN

Nvidia will invest up to $2.1 billion in IREN under an AI infrastructure partnership. The post discloses the cap and goal, but not equity terms, payment timing, or data center capacity.

Why it matters: Nvidia is putting up to $2.1 billion into IREN's AI data centers, though the filing leaves out equity terms and total compute.

The Verge · AI

SpaceX Has a $55 Billion Plan to Build AI Chips in Texas

SpaceX plans to invest at least $55 billion in its Terafab chip plant in Austin, Texas. A hearing notice says later phases could lift total investment to $119 billion. Musk said in March the target was chips for 200GW of compute per year; the post does not disclose process nodes.

Why it matters: SpaceX plans a chip plant in Austin, with hearing documents showing investment from at least $55 billion up to $119 billion and Musk targeting annual capacity for 200GW of compute.

AI HOT (Curated Pool)

Readable behavioral signals remain in frozen LLM hidden states, Cygnus boosts accuracy

Proprioceptive AI says Cygnus adds adapters to frozen LLMs and raises Qwen-32B on ARC-Challenge from 82.2% to 94.97%. It projects hidden states into a gl(4,R) Lie-algebra space to isolate “dark modes.” Watch replication; the post does not disclose full eval sets or controls.

Why it matters: Proprioceptive AI attaches a self-awareness adapter to a frozen model, reading behavior signals from hidden states without retraining, lifting Qwen-32B's ARC score from 82% to nearly 95%.

NVIDIA Blog

Powering the Next American Century: Chris Wright and NVIDIA’s Ian Buck on Genesis Mission

The U.S. DOE and NVIDIA are building two AI supercomputers at Argonne; Equinox uses 10,000 Grace Blackwell GPUs. Solstice will use 100,000 Vera Rubin GPUs, which Buck said reach 5,000 exaflops. The key bottleneck is grid work: Wright said AI can cut interconnection studies from years to weeks or hours.

Why it matters: The US Department of Energy and NVIDIA are deploying two AI supercomputers at Argonne National Laboratory, scaling from 10,000 Grace Blackwell chips to 100,000 Vera Rubin GPUs.

AI HOT (Curated Pool)

DeepSeek 4: Flash Local Inference Engine for Metal

DeepSeek 4 Flash is open-sourced on GitHub for offline inference on Apple Silicon Macs. The post says it uses Metal Performance Shaders to reduce latency and memory use, but discloses no benchmark numbers. The key item is the Metal local inference stack, not another model wrapper.

Why it matters: antirez, the creator of Redis, shipped ds4 to run DeepSeek 4 Flash locally on a Mac, calling Apple's Metal Performance Shaders directly to cut latency and memory use.

May 7Thursday

AI HOT (Curated Pool)

Notes From Inside China’s AI Labs

The author visited several leading Chinese AI labs and reported three patterns. The post says some Chinese tasks beat GPT-4, while firms build 100B-scale base models and 10B-scale vertical models. Watch compression and private deployment under compute constraints.

Why it matters: The author visited several leading Chinese AI labs and found students are core contributors, a fast-follow culture speeds up catching up, and original research gets suppressed in the process.

AI HOT (Curated Pool)

SenseNova-U1 Open-Sources 8-Step Distilled LoRA, Speeds Diffusion Inference by 11x

SenseNova-U1 open-sourced an 8-step distilled LoRA that cuts diffusion generation from 100 steps to 8. GPU inference time drops from 23 seconds to 2 seconds, with ComfyUI workflows for text-to-image, image editing, and interleaved generation. The key signal is distillation for latency, not parameter scale.

Why it matters: An open 8-step LoRA cuts diffusion sampling from 100 steps to 8, dropping GPU inference from 23 seconds to 2, with a ComfyUI workflow included.

QbitAI · WeChat

Zhejiang University and Alibaba MetaCompress reaches 90% token compression for multi-turn VQA

Zhejiang University and Alibaba proposed MetaCompress, a learned token-compression framework that generates a compression mapping from the input image alone for multi-turn VQA. The article says it can remove 90% of visual tokens while preserving accuracy, and reports only 1.71% overlap between optimally retained tokens and high-attention tokens.

Why it matters: Zhejiang University and Alibaba let a model decide which visual tokens to keep from the input image alone, cutting 90% of tokens in multi-turn VQA with no accuracy loss.

r/LocalLLaMA

Running Qwen3.5/Qwen3.6 with NextN MTP in llama.cpp on one RTX 3090 Ti

A Reddit user posted a llama.cpp guide for Qwen3.5/3.6 with NextN MTP on one RTX 3090 Ti. It requires two unmerged PRs, #22400 and #22673; Qwen3.6-35B-A3B-MTP reaches 157 tok/s at 350W, 1700MHz, with q8 KV. The key reproducible detail is nextn=q8_0 quant override; missing it yields “////” output.

Why it matters: A Reddit user posts a hands-on llama.cpp guide for running Qwen3.5 and 3.6 on a single RTX 3090 Ti, built on two unmerged PRs and covering NextN MTP speculative decoding setup.

Latent Space

Anthropic-SpaceXAI's 300MW/$5B/yr Deal for Colossus I, ARR Growth Is 8000% Annualized

Anthropic announced a SpaceX compute partnership, doubled Claude Code’s 5-hour limits for Pro, Max, Team, and seat-based Enterprise, raised Opus API limits, and said Claude inference would ramp on Colossus within days; the post treats the 300MW and $5B-per-year figures as widely circulated but not canonized in Anthropic’s own announcement.

Why it matters: Anthropic is renting xAI's 300MW Memphis Colossus I cluster for Claude inference, a deal estimated at $5B a year that puts xAI into the compute rental business.

Synced · WeChat

Musk Announces xAI Dissolution, Leasing 220,000 GPUs to Anthropic

Musk confirmed xAI will dissolve, with Grok and X-related operations folded into SpaceXAI. SpaceX and Anthropic signed a deal giving Claude access to Colossus 1’s 220,000+ Nvidia GPUs and 300 MW of compute. The key change is quota: Claude Code’s five-hour rate limit doubles, and Pro/Max peak-hour cuts are removed.

Why it matters: Musk confirmed xAI's dissolution, folding Grok and X into SpaceXAI while renting Colossus 1's 220,000 GPUs to Anthropic, a shift in two AI rivals' positions.

Computing Life · Share · Yage

Anthropic Locks Up Compute Channels as xAI Rents Its Castle to a Rival

Anthropic signed four compute contracts in six months covering AWS Trainium, Google TPU, SpaceXAI Colossus 1, and CoreWeave; during the same window, xAI rented the Colossus 1 supercomputing center to a competitor while GPU utilization stood at 11%.

Why it matters: Anthropic locked in four compute suppliers in six months, covering three mainstream chip architectures, while xAI rented its 220,000-GPU Colossus 1 to its rival.

Financial Times · Technology

Arm projects $2bn in sales of its new AI chip from next year

Arm projects $2bn in sales for its first in-house AI chip from next year. The RSS snippet says the SoftBank-backed UK group has strong demand; the post does not disclose customers, pricing, process node, or delivery cadence.

Why it matters: Arm is moving from licensing chip designs to building its own AI chip, with a $2bn sales projection for next year.

Financial Times · Technology

SpaceX to Rent Data Centre Capacity to Anthropic

SpaceX will rent data centre capacity to Anthropic, confirming a compute leasing arrangement. The RSS snippet says Anthropic is adding compute to match growth; the post does not disclose capacity, term, or pricing.

Why it matters: SpaceX will rent data center capacity to Anthropic, a new direction in their partnership as Anthropic adds compute to keep up with user growth.

r/LocalLLaMA

GB10 inference engine Atlas is open source, with Qwen3.6-35B-FP8 over 100 tok/s

Avarok open-sourced Atlas, an inference engine running Qwen3.5-35B at ~111 tok/s sustained on one DGX Spark. It uses Rust+CUDA, a ~2.5GB image, and sub-2-minute cold start; the author claims 3.0–3.3x vLLM in tests. The key details are Blackwell SM120/121 kernels, NVFP4/FP8, and MTP decoding.

Why it matters: Avarok open-sourced Atlas, a Rust and CUDA inference engine, hitting about 111 tok/s for Qwen3.5-35B on a DGX Spark, which the author puts at 3.0 to 3.3 times vLLM.

r/LocalLLaMA

Exaggerated PCI-E Bandwidth Concerns?

Reddit user ziphnor tested 2x RTX 5060 Ti 16GB with vLLM TP=2 and 32k-context prefill. PCIe peaked at 3–4 GB/s, about 40–50% of a PCIe 4.0 x4 link. Prefill reached ~840–850, 1500, and 1600–1700 t/s; the post does not disclose decode bandwidth.

Why it matters: User ziphnor ran vLLM tensor parallelism on two RTX 5060 Ti cards and measured PCIe peak bandwidth of only 3-4 GB/s during 32k-context prefill, 40-50% of the PCIe 4.0 x4 link.

r/LocalLLaMA

Analysis of 922 Agentic Task Traces Finds DeepSeek v4’s Cost Edge in Caching

A Reddit user analyzed 922 agentic task traces and reported $0.01 per task for DeepSeek v4 Flash versus $1.52 for Opus 4.7. Both used about 960K tokens per task, but DeepSeek showed a 97% cache hit rate versus 87%, with a 0.02 cache read/write price ratio versus 0.08. The key issue is caching, not headline pricing.

Why it matters: A Reddit user analyzed 922 agent task traces and compared per-task cost for DeepSeek v4 Flash and Opus 4.7, finding similar token use and the gap coming from cache hit rate and cache read-write pricing.

Bloomberg Technology

Anthropic Signs Computing Deal With SpaceX to Meet AI Demand

Anthropic signed a computing deal with Elon Musk’s SpaceX to support growing Claude demand. The post does not disclose capacity, contract value, deployment timing, or infrastructure details. The key issue is whether SpaceX enters Anthropic’s long-term training or inference supply chain.

Why it matters: Anthropic signed a compute deal with SpaceX to support growing Claude usage, but the size, value, timing and purpose are all undisclosed.

May 6Wednesday

r/LocalLLaMA

Qwen3.6 27B NVFP4 + MTP on a Single RTX 5090: 200k Context in vLLM

A Reddit user ran Qwen3.6 27B NVFP4 on one RTX 5090 32GB and validated 200k context in vLLM. The setup used fp8_e4m3 KV cache, FlashInfer, and MTP with 3 speculative tokens; a 10-run 200k pass completed with 73.6 tok/s mean generation and 70.2s TTFT. The key constraint is 32GB VRAM: logs showed 8.3GiB KV cache and about 30478MiB total GPU use.

Why it matters: A Reddit user ran Qwen3.6 27B NVFP4 on a single RTX 5090 in vLLM, verified 200k context, and posted the fp8_e4m3 KV cache and MTP speculative decoding config.

TechCrunch · AI

AI boom pushes Samsung to $1T

Samsung crossed a $1 trillion valuation after shares rose on AI chip demand. It is the second Asian company after TSMC to hit the mark; the post does not disclose share gains, revenue, or order mix.

Why it matters: Samsung crossed a $1T market cap, the second Asian company after TSMC, as memory chip demand from AI systems outran supply and pushed prices up.

NVIDIA Blog

NVIDIA Spectrum-X AI-Native Ethernet Fabric Adds MRC for Gigascale AI

NVIDIA added MRC support to Spectrum-X Ethernet, letting one RDMA connection spread traffic across multiple paths. MRC ran in Blackwell deployments, with microsecond failure bypass and hardware rerouting. The key detail is the OCP open specification and multiplane support for clusters up to hundreds of thousands of GPUs.

Why it matters: NVIDIA's Spectrum-X adds multipath RDMA so one RDMA connection spreads traffic across paths, already used in Blackwell clusters and pushed through OCP for larger GPU fabrics.

r/LocalLLaMA

2.5x Faster Inference with Qwen 3.6 27B Using MTP on 48GB

A llama.cpp PR adds MTP support for Qwen 3.6 27B, with a reported 2.5x inference speedup. The author measured 28 tok/s on a Mac M2 Max 96GB and shared GGUF builds, compile steps, and a 262144-context server command. The key detail is turbo4 4.25-bit KV cache: a 48GB Mac runs Q5_K_M at 262K context.

Why it matters: A llama.cpp PR adds MTP support for Qwen 3.6 27B, measured at 2.5x faster inference with 262k context on 48GB, a usable setup for local agentic coding.

Latent Space

AINews: Silicon Valley Gets Serious About Services

Anthropic and OpenAI announced enterprise services companies: Anthropic’s unnamed JV is funded with $1.5 billion, while OpenAI’s The Deployment Company has raised about $4 billion at a $10 billion pre-money valuation.

Why it matters: Anthropic and Blackstone and others formed a joint venture services firm with $1.5 billion, sending small teams in first to scope Claude use cases before engineers build custom systems.

Synced · WeChat

Two Chinese open-source projects turn Mac into a private AI workstation

Mininglamp open-sourced Cider and Mano-P 1.0 for Apple Silicon local inference and GUI agents. Cider speeds Qwen3-VL-2B prefill by 57%–61% on M5 Pro; Mano-P 1.0-72B scores 58.2% on OSWorld. The key constraint is W8A8 memory: on 16GB devices accuracy falls from 58.0% to 54.0%, so 32GB+ is recommended.

Why it matters: Minglue open-sources Cider and Mano-P 1.0, with Cider cutting Qwen3-VL-2B first-token latency about 60% on M5 Pro and the 72B Mano-P scoring 58.2 on OSWorld.

r/LocalLLaMA

DeepSeek V4 at 17x lower cost prompted a local-vs-cloud coding workflow test

Reddit user spencer_kw logged a 10-day coding workflow and retested 150 tasks on local Qwen 3.6 27B versus cloud models. Local was equivalent for 65% of tasks, acceptable for 20%, and cloud was needed for 15%; the API bill fell from $85/month to about $22. The useful signal is task-based routing, not headline model pricing alone.

Why it matters: A concrete split of which coding tasks run on a local Qwen model and which still need the cloud, with the API bill to match.

TechCrunch · AI

OpenAI releases GPT-5.5 Instant, a new default model for ChatGPT

OpenAI released GPT-5.5 Instant as ChatGPT’s new default model. The company says it reduces hallucinations in law, medicine, and finance while keeping prior low latency; the post does not disclose benchmarks, rollout scope, or pricing.

Why it matters: OpenAI swapped ChatGPT's default model from GPT-5.3 Instant to GPT-5.5 Instant, claiming fewer hallucinations in law, medicine and finance while keeping latency low.

r/LocalLLaMA

Gemma 4 MTP Released

Google released Gemma 4 MTP drafters with 4 Hugging Face checkpoints listed. MTP uses a smaller draft model to predict multiple tokens, then the target model verifies them in parallel, giving up to 2x decoding speedups with identical output quality.

Why it matters: Google posted four Gemma 4 MTP checkpoints on Hugging Face, where a small model guesses ahead and the main model verifies in parallel, doubling decode speed with unchanged output quality.

May 5Tuesday

r/LocalLLaMA

Heretic 1.3 Released: Reproducible Models, Integrated Benchmarks, Lower Peak VRAM

Heretic 1.3 adds reproducible runs, integrated benchmarks, lower peak VRAM, and broader model support. The project claims 20,000 GitHub stars and 13 million model downloads. Reproduce directories capture PyTorch, GPU, driver, and accelerator details; benchmarks use lm-evaluation-harness for MMLU, EQ-Bench, GSM8K, and HellaSwag. The post names Qwen3.5 and Gemma 4 support, but does not disclose VRAM reduction figures.

Why it matters: Heretic 1.3 records the PyTorch, GPU and driver environment so others can reproduce results, and adds built-in benchmarking with lower peak VRAM use.

OpenAI News

OpenAI Introduces MRC for Large-Scale AI Training Networks

OpenAI introduced MRC for large-scale AI training cluster networks. MRC stands for Multipath Reliable Connection and is released via OCP to improve resilience and performance; the post does not disclose throughput, latency, or cluster size.

Why it matters: OpenAI open-sourced MRC, a training network protocol that splits one transfer across hundreds of paths and reroutes around link failures in microseconds, useful for teams running their own large GPU clusters.

r/LocalLLaMA

vibevoice.cpp: Microsoft VibeVoice ported to ggml/C++ with no Python at inference

LocalAI released vibevoice.cpp, a ggml/C++ port of Microsoft VibeVoice for CPU, CUDA, Metal, and Vulkan inference. TTS uses a 30s reference clip for 24kHz cloned speech; ASR uses a 7B model with diarized JSON and was tested on 17min audio. The key constraint is memory: 17min CPU Q8_0 peaks near 26GB, with no streaming output yet.

Why it matters: LocalAI rewrote Microsoft VibeVoice's speech synthesis and long-audio transcription as a pure C++ version that runs on CPU, CUDA, Metal and Vulkan, so local deployment no longer needs a Python environment.

Synced · WeChat

Massive Idle Cluster: Musk’s 550,000 Nvidia GPUs Are Only 11% Utilized

The Information says xAI’s roughly 550,000 Nvidia GPUs have only 11% MFU, equal to about 60,000 effective GPUs. The post cites HBM I/O, inter-server communication, training idle time, and software-stack inconsistency; Meta and Google are listed at 43% and 46%.

Why it matters: xAI's roughly 550,000 Nvidia GPUs run at only 11% model FLOPs utilization, equal to about 60,000 cards doing real work, while Meta and Google sit at 43% and 46%.

r/LocalLLaMA

MTPLX: 2.24x Faster TPS Native MTP Inference Engine for Apple Silicon

MTPLX raises Qwen3.6-27B on a MacBook Pro M5 Max from 28 to 63 tok/s. The test used 4-bit MLX, temperature 0.6, top_p 0.95, top_k 20, with D3 as the best depth. The key detail is native MTP heads: no external drafter and no second-model memory.

Why it matters: MTPLX on a MacBook Pro M5 Max raised Qwen3.6-27B generation speed from 28 to 63 tokens per second, a 2.24x gain from built-in MTP heads with no separate drafter model to load.

TechCrunch · AI

OpenAI’s cozy partner Cerebras is on track for a blockbuster IPO

Cerebras is moving toward an IPO at a valuation of $26.6 billion or more. The snippet says its OpenAI relationship is deep, but does not disclose ownership, revenue, or timing. The key signal is OpenAI-linked supply-chain valuation, not just AI chips.

Why it matters: Cerebras is moving toward an IPO at a target valuation of at least $26.6 billion, a valuation driven mainly by its supply-chain tie to OpenAI rather than the chip's own technical numbers.

r/LocalLLaMA

FastDMS: 6.4X KV-cache compression running faster than vLLM BF16/FP8

FastDMS released an MIT implementation that cuts KV memory to 1/5–1/8 of vLLM BF16 at 8K context. A Llama-3.2-1B replication reports PPL 9.200 with 6.4x compression; Qwen3-8B c=1 drops KV from 1.406 GiB to 0.184 GiB. The key detail is physical reclamation of evicted slots, not just nominal KV-byte reduction.

Why it matters: FastDMS open-sourced an MIT-licensed KV cache compression implementation that cuts Llama-3.2-1B's cache to one-fifth to one-eighth at 8K context, with concrete perplexity and memory figures.

May 4Monday

r/LocalLLaMA

M3 Ultra + DGX Spark = M5 Ultra-lite?

A Reddit user benchmarked DGX Spark against M3 Ultra in llama.cpp at pp16384, with Spark 1.4× to 3.4× faster across 4 models. Qwen 27B hit 778 t/s vs 340 t/s, while Mistral 128B hit 241 t/s vs 72 t/s. The concrete tuning note is mmap=0: loading fell from minutes to about 20 seconds.

Why it matters: A Reddit user benchmarked DGX Spark against M3 Ultra at pp16384 context in llama.cpp, with Spark 1.4x to 3.4x faster on Qwen 27B and Mistral 128B.

r/LocalLLaMA

Mistral Medium 3.5 128B and Qwen 3.5 122B A10B on 4x RTX 3080 20GB

A Reddit user benchmarked Mistral Medium 3.5 128B and Qwen 3.5 122B A10B on 4x RTX 3080 20GB. llama.cpp tensor split raised Mistral tg128 from 10.37 to 21.59 t/s, but Qwen MoE fell from 60.08 to 53.49 t/s. vLLM served Qwen GPTQ-Int4 at 187.04 tok/s; the key signal is MoE sensitivity to parallel strategy.

Why it matters: A Reddit user ran two large models on 4 RTX 3080 20GB cards, and Mistral Medium 3.5 128B hit 21.59 tokens per second with tensor split, up from 10.37.

r/LocalLLaMA

450M On-Board VLM Wildfire Detection Pipeline with Sentinel-2 and LFM2.5-VL

PauLabartaBajo shared a wildfire detection PoC using 450M LFM2.5-VL on Sentinel-2 imagery. It pairs RGB and SWIR tiles, simulates orbit with SimSat, and covers 22 fire-prone sites. The key constraint is bandwidth: on-board inference downlinks only a JSON risk profile.

Why it matters: The author deployed a 450M vision-language model on the satellite itself for wildfire inference and downlinks only JSON risk assessments, sidestepping bandwidth limits on multispectral imagery.