Skip to content

#部署/工程

11 today

May 19Tuesday

Bloomberg Technology

Google, Blackstone to Create AI Cloud Firm With In-House Chips

Google agreed to create an AI cloud business with Blackstone using in-house chips to compete with CoreWeave; the post does not disclose ownership structure, investment size, launch timing, or chip specifications.

Why it matters: Google and Blackstone plan an AI cloud joint venture running on in-house TPUs against GPU cloud providers like CoreWeave, showing one route for big tech's own silicon into cloud compute.

Bloomberg Technology

Inside Meta’s $200 Billion Louisiana Data Center Bet

Meta is building an AI data center in Richland Parish, Louisiana, financed by a $200 billion private-capital deal, with power demand up to 7.5 gigawatts, including 5 gigawatts for computing, supplied by 10 new natural-gas plants.

Why it matters: Meta is raising $20 billion in private financing for a Louisiana AI data center and adding 10 gas plants for up to 7.5 GW, showing how closely compute growth is tied to fossil power.

Bloomberg Technology

Nvidia’s CEO Sees China Opening Market to AI Chips From US

Nvidia CEO Jensen Huang said Chinese authorities will eventually allow imports of US AI chips; the RSS snippet only says he made the comment days after joining President Donald Trump’s summit in China and does not disclose a timetable.

Why it matters: Nvidia's CEO said after a China trip with Trump that China will eventually open its market to US AI chips, without giving a timeline or evidence, while export controls stay tight.

r/LocalLLaMA

llama.cpp MTP support landed: Qwen3.6 27B reaches 2.44× on Strix Halo

llama.cpp merged MTP speculative decoding in PR #22673; Qwen3.6 27B Q8_0 rose from 7.4 to 18.1 tok/s on Strix Halo, while a dual RTX 3090 Q8_0 setup rose from 25.7 to 55.9 tok/s.

Why it matters: After llama.cpp merged MTP speculative decoding, Qwen3.6 27B measured 7.4 tok/s to 18.1 tok/s on Strix Halo, giving local inference a concrete speedup reference.

May 18Monday

Import AI (Jack Clark)

Import AI 457: AI Stuxnet, Cursed Muon Optimizer, and Positive Alignment

Import AI 457 covers fast16, Aurora, and positive alignment: SentinelOne found fewer than 10 matching files for fast16 signatures, while Tilde Research reports Aurora reached 2.26 loss on 1.1B-parameter transformers versus Muon’s 2.31 under a ~100B-token setup.

Why it matters: SentinelOne's teardown of fast16.sys shows it tampering with high-precision results in memory five years before Stuxnet, while Tilde Research finds Muon kills over a quarter of MLP neurons early in training.

r/LocalLLaMA

Qwen 3.6 27B on 24GB VRAM: backend comparisons, quant choice, and settings

The author tested Qwen 3.6 27B on an RTX 3090 24GB and kept ik_llama.cpp with Qwen3.6-27B-MTP-IQ4_KS.gguf; at 156k context with q8_0 KV and MTP, a ~5.9k-token prompt plus 1024-token output reached about 1261 tok/s prefill and 72.9 tok/s decode, while vLLM lacked a clean single-card long-context run.

Why it matters: The author benchmarked several backends and quantization schemes for Qwen 3.6 27B on a single 24GB RTX 3090, settling on ik_llama.cpp with IQ4_KS, as a single-card comparison for local deployment.

Synced · WeChat

ICML 2026: Huawei GTS proposes EDCO for dynamic curriculum fine-tuning

Huawei GTS proposed EDCO, a dynamic curriculum method that selects fine-tuning samples by inference entropy; prefix entropy estimation cuts per-sample scoring time from 2.24 seconds to 0.37 seconds.

Why it matters: Huawei GTS proposed EDCO, a dynamic curriculum fine-tuning method that uses reasoning entropy to judge sample difficulty and pick training data, with prefix-entropy estimation cutting per-sample scoring from 2.24 seconds to 0.37.

r/LocalLLaMA

Benchmarking vLLM vs SGLang vs llama.cpp on a mixed Blackwell/Ada cluster

The author benchmarked long-context prefill on a 7-GPU mixed Blackwell/Ada cluster; on Qwen3.5-397B-A17B with 75k tokens, vLLM reached 9.8s TTFT and 7,683 t/s, while llama.cpp took 57.2s and 1,319 t/s.

Why it matters: A hands-on test of long-context prefill across seven mixed Blackwell and Ada GPUs, with vLLM hitting 9.8s first-token latency at 75k tokens and 7,683 tok/s prefill, roughly 4 to 6 times llama.cpp.

r/LocalLLaMA

LLMs on Android: Snapdragon 8 Elite MoE Experience

A Reddit user tested MoE LLMs on an Honor Magic 7 Pro with Snapdragon 8 Elite and 24GB RAM; under Q4 quantization, LFM2-24b-a2b reached about 24 tokens/s while Gemma reached about 11 tokens/s, and CPU inference was still faster than NPU or GPU in the reported setup.

Why it matters: A Reddit user's real measurements of MoE models on a Snapdragon 8 Elite phone, where CPU inference beat the NPU and LFM2-24b-a2b ran near 24 tok/s at Q4.

May 17Sunday

QbitAI · WeChat

A Robot Dog Challenges Nvidia's Compute Lead

Weilan Technology unveiled BabyAlpha A3, a consumer quadruped robot using a six-chip heterogeneous cluster that runs a 7B-parameter model on-device at 280 TPS; the article says it has 66MP vision, 2.232 million point-cloud samples per second, and a planned Q3 launch.

Why it matters: Azurion runs a 7B model locally on a consumer robot dog using a six-chip heterogeneous cluster, claiming 280 TPS and giving on-device compute a concrete sample.

r/LocalLLaMA

Same Models Tested Across Strix Halo, RTX 3090, and RTX 5070

C_Coffie published 55 local inference benchmark runs across Strix Halo, RTX 3090, RTX 5070, five backends, and 0.35B to 35B-A3B models; RTX 5070 beats RTX 3090 on models fitting 12GiB, while RTX 3090 leads in the 14–31B band that exceeds 12GiB but fits 24GiB.

Why it matters: A user ran 55 local inference tests across three GPUs and five backends, and reports an RTX 5070 decoding faster than an RTX 3090 within 12GB of VRAM.

Dwarkesh Patel podcast

Notes on Pretraining Parallelisms and Failed Training Runs

Dwarkesh documents pretraining failure modes and parallelism tradeoffs: expert choice and token dropping can break causality in MoE routing, FP16 collectives can bias repeated additions after values exceed 1024, pretraining FLOPs are given as 6ND, B300 HBM is listed as 288GB, and FSDP communication can reach params × 3 with reduce-scatter.

Why it matters: The author breaks down two ways pretraining runs go off the rails, and why expert choice in MoE routing makes a token's destination depend on later routing decisions, so training sees information inference never gets.

May 16Saturday

Latent Space

Cerebras' $60B IPO: Slowly, then All at Once

Cerebras closed its IPO at a $60 billion market cap. CFO Bob Komin said it serves trillion-parameter models, including internal OpenAI 5.4 and 5.5 workloads, but the post does not disclose traffic share, latency tier, or cost per token.

Why it matters: Cerebras closed its IPO at a $60B market cap, and its CFO said publicly the company already serves trillion-parameter models, naming the 5.4 and 5.5 versions OpenAI uses internally.

r/LocalLLaMA

Orthrus-Qwen3-8B: Up to 7.8× tokens/forward on Qwen3-8B with frozen backbone

Orthrus-Qwen3-8B adds a trainable diffusion attention head to a frozen Qwen3-8B backbone, reaches up to 7.8× tokens per forward and about 6× wall-clock speedup on MATH-500, while training 16% of parameters with under 1B tokens over 24 hours on 8×H200 GPUs.

Why it matters: The team froze the Qwen3-8B backbone and trained only a diffusion attention head, reaching up to 7.8x token throughput per forward pass on MATH-500 and about 6x faster wall-clock time, with the output distribution unchanged.

Bloomberg Technology

Cerebras CEO Is Worth $3.2 Billion After Year’s Largest IPO

Cerebras Systems rose about 68% on Nasdaq after the year’s largest IPO, giving the company a market value of roughly $67 billion; the headline says its CEO is worth $3.2 billion after the listing.

Why it matters: Cerebras listed on Nasdaq and closed its first day up about 68%, giving a concrete public-market price for an AI compute company.

May 15Friday

r/LocalLLaMA

Fully Offline Suitcase Robot Built Around Jetson Orin NX SUPER 16GB

CreativelyBankrupt built Sparky as a fully offline suitcase robot on Jetson Orin NX SUPER 16GB, running Gemma 4 E4B Q4_K_M via llama.cpp with q8_0 KV cache, about 200 ms cached TTFT, 14-15 tok/s sustained output, 12K context, 30+ sensors, and no WiFi, Bluetooth, or cellular interface.

Why it matters: A solo developer runs Gemma 4 E4B fully offline on a Jetson Orin NX SUPER 16GB mounted on a suitcase, with about 200ms cached TTFT and 14 to 15 tokens per second.

AI HOT (Curated Pool)

X open-sources the “For You” feed recommendation algorithm

X open-sourced the For You recommendation pipeline on GitHub, using a Grok-based Phoenix Transformer to score candidate posts and predict engagement probabilities such as likes, replies, and reposts.

Why it matters: X published the recommendation pipeline behind its For You feed on GitHub, including code, pretrained models and content-understanding services, with a Grok-based Phoenix model scoring candidate posts.

r/LocalLLaMA

Evaluated a RAG Chatbot: The Most Expensive Model Was the Worst Performer

The author evaluated a customer-support RAG bot and raised the quality score from 6.62 to 7.88 while cutting per-session cost from $0.002420 to $0.000509, using retrieval logging, LLM-as-judge scoring, chunk deduplication, stricter grounding, and a five-model sweep.

Why it matters: A builder used a model as judge to tune a customer-service RAG bot, lifting quality from 6.62 to 7.88 and cutting cost per session by nearly 80%, while the priciest model performed worst.

r/LocalLLaMA

Used over a million tokens in three sessions to test Qwen 3.6 35B MTP

A Reddit user tested Qwen3.6-35B-A3B MTP across three million-token-scale sessions, using 300k context and KV Q8_0, and reported about 1.5x the tok/sec of earlier tests.

Why it matters: A Reddit user ran over a million tokens across three sessions to test the MTP build of Qwen3.6-35B-A3B at 300k context and KV Q8_0 quantization, measuring roughly 1.5x the generation speed of the earlier version.

Bloomberg Technology

OpenAI May Raise More Money as Compute Crunch Deepens, CFO Says

OpenAI CFO Sarah Friar said the company may raise more capital after completing what she described as the largest private fundraising round ever, as OpenAI seeks computing power to meet rising AI demand; the RSS snippet does not disclose the round size, target amount, or timeline.

Why it matters: OpenAI CFO Sarah Friar says the company may raise more money after its latest private round, because compute supply cannot keep up with AI demand growth.

AI HOT (Curated Pool)

Microsoft Has Invested Over $100 Billion in OpenAI, Nadella Says No One Wanted to Bet Then

Microsoft has invested more than $100 billion in OpenAI, including a $13 billion original investment and Azure infrastructure costs, while the partnership has generated about $30 billion in revenue; the renewed non-exclusive agreement caps OpenAI’s revenue share at $38 billion cumulatively through 2030.

Why it matters: Microsoft's corporate development lead confirmed in court that total OpenAI investment tops $100 billion, with $13 billion direct and most of the rest Azure infrastructure and hosting costs, largely spent before OpenAI began sharing revenue.

AI HOT (Curated Pool)

The First Derivative of Inference: Growth Logic in the AI Wave

Tom Tunguz says the AI inference market will reach $250 billion within seven years; Datadog’s LLM observability data volume nearly doubled in the latest quarter, and about 20% of its AI customers contribute roughly 80% of ARR.

Why it matters: Tom Tunguz floated the idea of a first derivative of inference: companies like Datadog that sell no compute but earn more as customer AI usage grows, with LLM monitoring data nearly doubling in a quarter.

AI HOT (Curated Pool)

API prompt precaching speeds up first-token generation

Claude API prewarms prompt cache with the system prompt, skips output, then hits cache on the real request.

Why it matters: The Claude API can write a system prompt into the cache on its own before the real request arrives and skip output, so later requests hit the cache and cut first-token latency on long prompts.

Bloomberg Technology

Cerebras Goes Public in Year's Biggest IPO | Bloomberg Tech 5/14/2026

Bloomberg Tech says Cerebras is going public in the year’s biggest IPO, while the RSS snippet does not disclose the fundraising amount, valuation, offer price, or exchange.

Why it matters: AI chip company Cerebras has gone public in the year's largest IPO, though the raise, valuation, offer price and exchange were not disclosed.

Bloomberg Technology

AI Chipmaker Cerebras Climbs 68% After Year’s Biggest IPO

Cerebras Systems shares closed up 68% in their trading debut after raising $5.5 billion in the year’s largest IPO, ending Thursday at $311.07 in New York versus the $185 IPO price.

Why it matters: Cerebras closed its first day of trading in New York up 68% after raising $5.5 billion, giving a concrete reference for public-market demand for AI chips outside Nvidia's path.

Bloomberg Technology

Cerebras CEO Is Worth $3.2 Billion After Year’s Largest IPO

Cerebras’ IPO valued CEO Andrew Feldman’s fortune at $3.2 billion, and the title identifies the listing as the year’s largest IPO; the RSS snippet only says Feldman previously sold three companies and took another public, and does not disclose the IPO price, proceeds, valuation, or share count.

Why it matters: Cerebras completed one of the year's largest IPOs, leaving CEO Andrew Feldman's stake worth $3.2 billion; he had already sold three companies and taken one public.

r/LocalLLaMA

I tracked EU GPU prices across 15 stores for 50+ days: RTX 5090 is the only card not dropping

Reddit user egudegi tracked EU GPU prices across 15 stores for more than 50 days with a 6-hour scrape cadence and about 126,000 readings; RTX 5090 average pricing rose from €3,392 to €3,487, a 3.0% increase.

Why it matters: A Reddit user scripted GPU price tracking across 15 EU stores for over 50 days, and among roughly 126,000 data points the RTX 5090 rose from 3,392 to 3,487 euros while every other model fell.

r/LocalLLaMA

The RTX 5000 PRO 48GB arrived and is better than expected

A Reddit user built a $5,600 RTX 5000 PRO 48GB PC and ran Qwen3.6-27B-FP8 with full-precision cache; they report up to 80 tok/s in TG, about 50–60 tok/s on very large prompts, 4,400 tok/s in prompt processing, and 200k tokens fitting in BF16 KV cache.

Why it matters: A builder with no prior experience spent $5,600 on a single RTX 5000 PRO 48GB machine, hitting 80 tok/s on a 27B model with 200k context at full precision.

TechCrunch · AI

Cerebras raises $5.5B, then stock pops 108%, in the first huge tech IPO of 2026

The title says Cerebras raised $5.5 billion and its stock rose 108% after the first major tech IPO of 2026; the post does not disclose the offer price, valuation, share count, or use of proceeds.

Why it matters: Cerebras raised $5.5B in the first big tech IPO of 2026 and its stock rose 108% on day one, a sample of how an AI chip company fares in public markets.

AI HOT (Curated Pool)

Accelerating On-Device AI: Arm and Google AI Edge Optimization Practices

Arm SME2 and Google AI Edge integrate with LiteRT, XNNPACK, and KleidiAI to optimize Stability AI’s stable-audio-open-small, delivering over 2x faster audio generation and 4x lower memory use on Arm-based mobile devices and laptops.

Why it matters: Arm and Google AI Edge use SME2 instructions to put matrix units in the CPU, more than doubling on-device audio model inference speed and cutting memory to a quarter.

May 14Thursday

Bloomberg Technology

AI Chipmaker Cerebras Climbs 68% After Year’s Biggest IPO

Cerebras Systems rose 68% in its trading debut after raising $5.5 billion in the year’s largest IPO; the post does not disclose the IPO price or valuation.

Why it matters: Cerebras listed on the public market with a $5.55 billion raise and closed its first day up 68%, taking a technical route on giant training chips that differs from Nvidia's.

QbitAI · WeChat

Chinese GPU Vendor Hosts Open Source Meetup With SGLang Core Developers

Moore Threads said at the SGLang × MUSA Meetup that the MUSA backend has been merged into SGLang mainline, with 47 PRs submitted and 41 merged as of May 12.

Why it matters: Moore Threads merged its MUSA backend for SGLang into the main branch, with 47 pull requests filed and 41 merged, a usable playbook for domestic GPUs joining mainstream inference frameworks.

AI HOT (Curated Pool)

OpenSquilla Open-Source Project Uses Smart Routing and Local Retrieval to Cut LLM Costs

OpenSquilla combines local model routing, vector retrieval, incremental sending, and cache hits to reduce transmitted tokens by more than 90%, while routing simple tasks to cheaper models and complex tasks to stronger models without spending tokens on the routing decision.

Why it matters: OpenSquilla runs model routing decisions locally, judging question difficulty without spending tokens, and cuts actual transfer volume below a tenth through incremental sending and cache hits.

AI HOT (Curated Pool)

UnslothAI Releases Qwen3.6 MTP GGUF Models With Over 1.4x Faster Inference

Daniel Han released experimental Qwen3.6 MTP GGUF models, with the 27B model reaching 140 tokens/s on one GPU and the 35B-A3B version reaching 220 tokens/s, using two draft tokens for speculative decoding.

Why it matters: Unsloth founder Daniel Han released an experimental Qwen3.6 MTP GGUF that drafts two tokens per step, trading accuracy for over 1.4x faster inference.

AI HOT (Curated Pool)

Moonshot AI founder Yang Zhilin releases a 40-minute video

Yang Zhilin explains Kimi K2 training in a 40-minute video, saying the model cost $4.6 million and beat GPT-5.5 and other competitors on coding tasks.

Why it matters: Yang Zhilin's 40-minute video puts Kimi K2's $4.6M training cost on the record and claims it beats GPT-5.5 on coding tasks.

AI HOT (Curated Pool)

Cost Analysis of AI Email

Top AI models process email at about $22 to $130 per month, with a $26 median; smaller models cut costs by 10 to 20 times, while local GPU execution can bring marginal cost close to zero.

Why it matters: A cost breakdown puts AI email handling at $22 to $130 per month in inference, and lays out smaller models and local GPUs as the ways down.

AI HOT (Curated Pool)

Unlocking Asynchrony in Continuous Batching

Hugging Face says an 8B model generating 8K tokens leaves the GPU idle for 24% of the time, and asynchronous batching uses CUDA streams to overlap CPU preparation for batch N+1 with GPU computation for batch N.

Why it matters: Hugging Face shows an 8B model idles its GPU 24% of the time generating 8,000 tokens, and explains how CUDA streams fill those gaps.

Financial Times · Technology

Cerebras boosts IPO price to raise $5.5bn

Cerebras raised its IPO price to seek $5.5bn in proceeds; the post discloses a $40bn valuation, but does not disclose the price range, share count, or listing date.

Why it matters: Cerebras raised its IPO price range to target $5.5bn, implying a roughly $40bn valuation and a read on demand for non-Nvidia AI chips.

r/LocalLLaMA

2x RTX 3090 setup for local Qwen 3.6 27B inference

A Reddit user ran Qwen 3.6 27B on a dual RTX 3090 Ubuntu setup, reporting 48GB VRAM, a 262k context window, no NVLink, about 4000 pp/s prompt processing, and 113 tk/s generation.

Why it matters: A Reddit user runs Qwen 3.6 27B on two RTX 3090s in Ubuntu, holding 260K context in 48GB of VRAM at 113 tokens per second.

r/LocalLLaMA

24+ tok/s from ~30B MoE models on an old GTX 1080

User mdda ran Qwen 3.6 35B-A3B on an i7-6700, GTX 1080, and 32GB RAM machine at about 24 tok/s with 128k context; the setup uses llama.cpp MoE offloading plus TurboQuant/RotorQuant KV cache quantization, with PCIe 3.0 x16 saturated and GPU utilization at about 40–50%.

Why it matters: A user runs roughly 30B-parameter MoE models on an i7-6700 with a GTX 1080 at about 24 tokens per second and 128k context, with the exact setup and numbers.