Skip to content

#部署/工程

11 today

Jun 2Tuesday

Bloomberg Technology

Nvidia’s AI Chips Sought by Chinese Labs With Ties to Military

Bloomberg says at least seven Chinese universities that support China’s armed forces and defense industry are seeking Nvidia H200 chips, based on a review of procurement records; the RSS snippet does not disclose order volumes, suppliers, or procurement status.

Why it matters: Bloomberg found purchase records showing at least seven military-linked Chinese universities seeking Nvidia H200 chips, a concrete look at where export-controlled compute demand goes.

r/LocalLLaMA

Computex 2026: Intel Launches Crescent Island GPU With Up to 480GB VRAM

Intel launched the Crescent Island GPU at Computex 2026 with up to 480GB of LPDDR5X VRAM, a 350W air-cooled TDP, Arc Xe 3P architecture, and datatype support from native FP4/MXFP4 to FP64.

Why it matters: Intel launched the Crescent Island GPU at Computex 2026, and up to 480GB of LPDDR5X memory lets local large-model runs hold bigger models and hit the disk less often.

Jun 1Monday

Latent Space

Why Video Agent Models Are Next — Ethan He on xAI Grok Imagine

Ethan He says a small xAI team built Grok Imagine from zero to one in 3 months, and the episode discusses video agents, audio-video alignment, inference speedups, and the storage, egress, and GPU-hour costs behind large video datasets.

Why it matters: xAI's Ethan He built Grok Imagine with a small team in three months and argues video-model intelligence comes mostly from language models.

AI HOT (Curated Pool)

Open and Closed Models Are on Different Exponentials

Nathan Lambert argues that closed frontier labs will capture high-margin demand in coding-agent workflows, citing a personal willingness to pay $2,000 per month and projecting OpenAI and Anthropic valuations of $2-10 trillion over 5-10 years.

Why it matters: Nathan Lambert argues users in coding-agent work will keep paying a premium for the smartest closed models, and closed labs will keep their best models back, which explains how open and closed could split.

AI HOT (Curated Pool)

OpenAI Starts Construction of Stargate 1GW Data Center in Michigan

OpenAI started the Stargate 1GW data center project in Michigan; the RSS snippet discloses the 1GW capacity but does not disclose investment size, construction timeline, or compute configuration.

Why it matters: OpenAI broke ground with Oracle and other partners on a 1GW data center campus in Saline, Michigan, saying power costs won't be passed to local residents and cooling uses a closed-loop design.

r/LocalLLaMA

Deepseek V4 Flash performance on DGX Spark

A Reddit user ran DeepSeek-V4-Flash with vLLM on two ASUS GX10 DGX Spark nodes and reported 1,680 prefill tokens/s plus 39.8 decode tokens/s at a 256K context with MTP=2; the setup uses TP=2 over RoCE, fp8 KV cache, and fits about 1M tokens safely in KV cache.

Why it matters: A Reddit user clustered two ASUS GX10 units and benchmarked DeepSeek-V4-Flash prefill and decode at 256K context, publishing the full tensor-parallel and RoCE network setup.

AI HOT (Curated Pool)

NVIDIA Releases RTX Spark and Local AI Agent Security and Performance Updates

NVIDIA released RTX Spark, a Windows PC for local AI agents with 1 petaflops of AI compute and 128GB of unified memory. OpenShell uses new Windows security primitives with Microsoft, while llama.cpp optimizations raise Qwen 27B throughput by up to 2x.

Why it matters: At Computex Nvidia showed RTX Spark, which runs large models locally with 1 petaflops and 128GB unified memory, plus an OpenShell runtime built with Microsoft to keep agents on-device.

AI HOT (Curated Pool)

Nvidia Enters Windows Laptop Market, Taking on Intel and AMD

Nvidia introduced one new PC-focused chip to enter the Windows laptop market and compete with Intel and AMD; the RSS snippet does not disclose specifications, pricing, launch timing, or AI compute metrics.

Why it matters: Nvidia is bringing a new PC chip into the Windows laptop processor market long held by Intel and AMD, a new variable for that competitive picture.

r/LocalLLaMA

I ported NVIDIA Parakeet speech-to-text to ggml: same output as NeMo, faster, GGUF-quantized, no Python

mudler_it ported NVIDIA Parakeet speech-to-text models to C++/ggml with no Python or PyTorch, reporting byte-for-byte NeMo parity on f32/f16, up to about 5x GPU speedups on larger TDT and hybrid models, and GGUF quantization across f16, q8_0, q6_k, q5_k, and q4_k.

Why it matters: A developer ported NVIDIA's Parakeet speech-to-text model to C++ and ggml, dropping Python and PyTorch, matching NeMo byte-for-byte at f32 and f16 and running about 5x faster on GPU.

May 31Sunday

AI HOT (Curated Pool)

Apple WWDC AI Upgrade: Gemini-Distilled Model Runs Locally, With Heavy External Dependencies

Apple will present Siri and on-device AI upgrades at next month’s WWDC, with iPhones running a smaller Gemini-distilled model locally while complex queries route to Google Cloud using Nvidia confidential computing.

Why it matters: Apple will show at WWDC that it distills Gemini into a small model running locally on iPhone, while complex requests still route to Google Cloud, leaving the stack clearly dependent on outside providers.

QbitAI · WeChat

NVIDIA’s MacBook Pro-like laptop reportedly uses an in-house CPU

NVIDIA, Microsoft, and Arm posted the same “new era of PC” teaser, and the article says the rumored N1X laptop may use a 20-core Arm CPU, a Blackwell GPU, 6,144 CUDA cores, and 128GB of LPDDR5X unified memory, while bandwidth and x86 translation remain the stated constraints.

Why it matters: Nvidia, Microsoft and Arm teased a PC shift the same day, pointing to Nvidia's N1X laptop with a 20-core Arm CPU and 128GB unified memory.

r/LocalLLaMA

Cost Analysis of My $6.4k Local LLM Server

The author runs Qwen3.6 27B on a $6,406.45 local server with 4 MI100 GPUs, processing 20.4M input tokens and 1.32M output tokens per day; using OpenRouter prices, the first-year local cost is $2,992.72 versus $3,701.10 for API use.

Why it matters: A full cost breakdown of a four-GPU local LLM server, showing first-year savings of just over $700 against API calls.

AI HOT (Curated Pool)

DynoSim: Simulation-Driven Inference Stack Optimization

NVIDIA released DynoSim for optimizing its Dynamo inference serving stack; the Rust-based tool models thousands of deployment configurations on a single virtual timeline and reached 1,500x real-time speed in tests.

Why it matters: NVIDIA's DynoSim simulates Dynamo deployment configs on a virtual timeline at 1500x real time, replacing trial-and-error on real hardware.

May 30Saturday

r/LocalLLaMA

Project Blackwell: Making an RTX Pro 6000 Run in a Dell R730 at 650K Context

The author installed an RTX Pro 6000 Blackwell in a 2016 Dell PowerEdge R730 and claims a 650K-context local AI box; the post describes fan-shroud modification, dual-riser power, PCIe BAR allocation failures, ACPI/DSDT inspection, MMIO aperture work, and Linux PCIe boot-flag testing as required conditions.

Why it matters: The author tried to fit an RTX Pro 6000 into a 2016 Dell R730 to run a 650K token context, hitting hardware obstacles around fan shrouds, power and PCIe BAR allocation.

AI HOT (Curated Pool)

xAI drops JAX GPU for an in-house training framework

SemiAnalysis says xAI dropped JAX GPU and moved to a C training framework written with Grok Build; the snippet claims xAI’s JAX stack had MFU below 10%, but the post does not disclose reproducible benchmark conditions.

Why it matters: xAI dropped JAX training on NVIDIA GPUs for its own C framework, Grok Build, after the old setup hit under 10% model FLOP utilization.

Synced · WeChat

Apple Uses AI to Rework Image Compression: Same Visual Quality at One-Third the File Size

Apple’s team published PICO, a perceptual image codec that uses 57%-70% fewer bits than AV1, VVC, and JPEG AI at the same subjective visual quality, while encoding a 12MP photo in 230 ms and decoding it in 150 ms on an iPhone 17 Pro Max.

Why it matters: Apple's paper claims 57% to 70% bitrate savings over AV1, VVC and JPEG AI at similar subjective quality, with on-device timings on iPhone 17 Pro Max.

r/LocalLLaMA

Testing MTP on vLLM and llama.cpp for Gemma 4 and Qwen 3.6

The author tested MTP on an RTX PRO 6000 Blackwell setup, where Gemma 4 31B on vLLM reached 132.52 tok/s versus a 39.69 tok/s baseline, a 3.34x speedup; the post reports 10 runs of 1,500 tokens each but does not provide a full quality or VRAM evaluation.

Why it matters: A user benchmarked Gemma 4 31B with vLLM on an RTX PRO 6000 with and without MTP, seeing throughput rise from 39.69 to 132.52 tokens per second on specific hardware and framework.

AI HOT (Curated Pool)

OpenAI launches real-time translation model with 70+ input languages

OpenAI launched gpt-realtime-translate, a speech translation model that accepts 70+ input languages and outputs speech in 13 target languages; the post says the feature is running on smart glasses.

Why it matters: OpenAI's gpt-realtime-translate takes speech input in over 70 languages and outputs 13, and it already runs on smart glasses, showing what a dedicated translation model trades off.

TechCrunch · AI

After Nvidia’s $20B Not-Acqui-Hire, AI Chip Startup Groq Reportedly Raising $650M

Axios says Groq is seeking $650 million in internal funding while shifting from hardware toward AI inference, after Nvidia’s reported $20 billion not-acqui-hire; the RSS snippet does not disclose Groq’s valuation, investor names, deal structure, or fundraising timeline.

Why it matters: Groq is seeking $650 million in internal funding and shifting its focus from building chips to AI inference, after Nvidia's $20 billion not-acqui-hire.

May 29Friday

Xinzhiyuan · WeChat

Three DeepSeek Models Enter OpenRouter Monthly Top 10 With Over 17 Trillion Tokens

DeepSeek placed three models in OpenRouter’s monthly top 10 with more than 17 trillion tokens combined, including V4 Flash at 9.13T tokens; the article says Ascend’s MegaMoE operator raised Prefill throughput by 20% to 30% on DeepSeek V3.1 and Qwen3-235B tests.

Why it matters: OpenRouter's monthly stats put three DeepSeek models in the top ten, showing how much traffic Chinese models draw on an aggregation platform.

r/LocalLLaMA

Liquid AI releases LFM2.5-8B-A1B

Liquid AI released LFM2.5-8B-A1B with a 128K context window, 38T pre-training tokens, large-scale reinforcement learning, doubled vocabulary for non-Latin tokenization, and availability on Hugging Face.

Why it matters: Liquid AI shipped LFM2.5-8B-A1B, 8 billion parameters with a 128K token context, a doubled vocabulary that tokenizes non-Latin scripts better, and the model is already on Hugging Face.

Synced · WeChat

A True 2-bit KV Quantization Algorithm for Long-context Reasoning Beyond TurboQuant

TogetherAI and collaborators released OSCAR, a 2.28 BPE INT2 KV Cache system integrated with SGLang, reporting up to 3× decode speedup at 100k context and up to 7× job-level throughput under a fixed memory budget.

Why it matters: TogetherAI released OSCAR, which compresses the KV cache to 2.28 bits per value on average and plugs into SGLang, decoding up to 3x faster at 100K tokens and up to 7x throughput at fixed memory.

Synced · WeChat

The Ma Jiaqi Failure Exposed an LLM Issue He Spotted in the Shower a Year Earlier

FaceMind links low-frequency token degradation to two papers: SLoW appeared at EMNLP 2025, Adam's Law was accepted as an ACL 2026 Oral, and high-frequency rewriting raised DeepSeek-V3 math accuracy from 63.55% to 71.54%.

Why it matters: The FaceMind team links large models failing on low-frequency words to the SLoW and Adam's Law papers, laying out a mechanistic account of low-frequency token degradation.

Bloomberg Technology

Samsung Takes Lead in Shipping Top-End AI Memory Chip Samples

Samsung Electronics has begun shipping samples of its most advanced memory to customers for AI accelerators from companies including Nvidia; the RSS snippet does not disclose the chip model, customer list, sample volume, pricing, or mass-production timeline.

Why it matters: Samsung has sent its most advanced memory chip samples to NVIDIA and other AI accelerator customers, reaching customer validation ahead of rivals.

Bloomberg Technology

Apollo Shops $36 Billion Debt Deal to Buy Google Chips for Anthropic

Apollo and Blackstone are seeking additional investors for about $36 billion in debt financing for Anthropic’s AI infrastructure; the title says the deal would buy Google chips, while the post does not disclose chip models, purchase volume, or timeline.

Why it matters: Apollo and Blackstone are raising about $36bn in debt for Anthropic to buy Google chips, a concrete case of an AI lab funding compute with debt.

AI HOT (Curated Pool)

Apple reportedly tries to fit Google's large Gemini model into iPhone for new Siri

Apple is trying to integrate a large Gemini model into the iPhone for new Siri features. The RSS snippet says full local processing is unlikely because of model size, and a cloud component is likely required; the post does not disclose parameters, latency targets, or a release timeline.

Why it matters: Apple is trying to distill Google's trillion-parameter Gemini into an on-device iPhone model for the new Siri, and the report judges full local execution unlikely, with cloud still needed.

May 28Thursday

QbitAI · WeChat

Behind DeepSeek V4's Chip-Model Co-Design, China's Compute Ecosystem Gains Speed

QbitAI says DeepSeek V4 validated Ascend chip-model co-design, with CANN open-sourcing 65 repositories and supporting day-zero adaptation for more than 70 mainstream models, while AIGCode reported 65% MFU in MoE pretraining on Ascend.

Why it matters: DeepSeek V4 was tuned against Huawei Ascend chips during training, and CANN open-sourced 65 repos with out-of-the-box support for over 70 mainstream models, giving domestic compute a concrete chip-to-framework-to-model path.

r/LocalLLaMA

Zai replaced the network architecture for GLM-5.1 inference, lifting throughput 15%

Zai replaced the ROFT network topology with ZCube on a thousand-GPU GLM-5.1 coding inference cluster, keeping the same GPUs, software stack, and model; the Reddit post cites 33% lower switch and optical module costs, 15% higher GPU inference throughput, and a 40.6% drop in first-token P99 tail latency under prefill-decode disaggregated inference.

Why it matters: Zai swapped ROFT for its own ZCube topology on a thousand-GPU GLM-5.1 inference cluster, with no other changes, and cut switch and optical module costs 33% while raising throughput 15%.

r/LocalLLaMA

Qwen3.6-35B-A3B-APEX Runs 128K Context on RTX 3060 12GB

old-mike ran mudler/Qwen3.6-35B-A3B-APEX-MTP-I-Compact.gguf through spiritbuun’s llama.cpp fork on one RTX 3060 12GB, offloading a 17.3GB model and reaching 37.17 t/s generation at 72K filled context, 28.08 t/s at 129K, and PPL 3.2529 on an enwik8 64K-context perplexity test.

Why it matters: A user ran a 17.3GB Qwen model on one 12GB RTX 3060 at 37 tokens/s with 72K context filled, giving concrete settings for consumer-GPU inference.

AI HOT (Curated Pool)

AI Now Summit 2026

Mistral AI announced industrial AI work, a Vibe upgrade, and a 10 MW inference data center in Les Ulis at AI Now Summit 2026; it is working with Airbus, BMW Group, and ASML, and the data center is scheduled to start operating in Q3 2026.

Why it matters: Mistral is pairing industrial AI deals with Airbus, BMW and ASML with its own 10 MW inference data center near Paris, due in Q3 2026.

AI HOT (Curated Pool)

Mistral AI launches physics AI model for industrial engineering

Mistral AI integrated the Emmi AI team and launched a physics AI foundation model for industrial engineering, with the post saying it can learn from geometry, boundary conditions, or measurement data and predict full physical fields on a single GPU in seconds.

Why it matters: Mistral's physical AI foundation model for industrial engineering predicts full fields like temperature and stress in seconds on a single GPU, a route worth watching for industrial simulation.

Latent Space

Cognition Raises $1B in $26B Series D

Cognition raised a $1B Series D at a $26B valuation and projects more than $1B ARR by year-end; the post says its valuation rose 2.5× from the $10B Series C eight months earlier, while the rest of the issue summarizes agent, inference, benchmark, and multimodal AI updates from May 26–27, 2026.

Why it matters: Cognition raised $1 billion in a Series D at a $26 billion valuation, up from $10 billion in eight months, a concrete gauge of capital heat in coding agents.

Xinzhiyuan · WeChat

Tsinghua Team Open-Sources PilotDeck Agent System, Claims 70% Token Cost Reduction

Tsinghua THUNLP, ModelBest, OpenBMB, and AI9stars open-sourced PilotDeck; the article says its sub-agent routing reduced cost from $12.58 to $2.83 in a Xiaohongshu content-generation test, while preserving separate WorkSpaces, editable memory, and per-session routing logs.

Why it matters: Tsinghua NLP and other teams open-sourced PilotDeck, giving sub-agents separate workspaces and editable memory with scheduling logs, a reference implementation for multi-agent workflows.

Synced · WeChat

Mila and DeepMind Propose UNSL for Unified Multivariate Neural Scaling Laws

Mila and Google DeepMind proposed Unified Neural Scaling Law, a multivariate scaling-law form that models parameter count, token count, training steps, bottlenecks, overfitting, and adverse hyperparameter effects; UNSL achieved the best extrapolation on 60.87% of vision tasks and 88.89% of language tasks in the reported experiments.

Why it matters: Mila and DeepMind proposed UNSL, putting parameter count, token count and training steps into one scaling formula with bottlenecks and overfitting, a unified frame for training configuration.

TechCrunch · AI

Snowflake signs $6B deal with AWS for AI CPU chips

Snowflake signed a five-year, $6 billion AWS deal to secure chips for AI use; the post does not disclose chip models, delivery timing, or whether the capacity targets training, inference, or both.

Why it matters: Snowflake signed a $6 billion five-year AWS deal for AI chip capacity, a large cloud buyer looking beyond Nvidia for compute.

r/LocalLLaMA

Inferencing at 10.33 t/s on Qwen 3.5 35B on a $300 laptop

A Reddit user ran Qwen 3.5 35B Q4_K_S on a $300 Lenovo Ideapad Slim 3i and reported 10.33 t/s inference using ik_llama.cpp with two pinned CPU cores, MTP speculative decoding, 64 batch size, and Q8_0 KV cache.

Why it matters: A Reddit user ran the Q4_K_S quant of Qwen 3.5 35B on a $300 Lenovo Ideapad Slim 3i using CPU only, reaching 10.33 tokens per second.

AI HOT (Curated Pool)

Open-source FastVideo Dreamverse real-time video generation tool

Hao AI Lab open-sourced FastVideo Dreamverse, a real-time video generation tool that generates a 30-second 1080p video in 7 seconds under the stated setup of one NVIDIA B200 GPU and LTX-2.

Why it matters: Hao AI Lab open-sourced FastVideo Dreamverse, which pairs LTX-2 with a single NVIDIA B200 to generate 30 seconds of 1080p video in 7 seconds, code and blog included.

May 27Wednesday

AI HOT (Curated Pool)

Perplexity open-sources Unigram tokenizer to reduce CPU usage

Perplexity open-sourced a rebuilt Unigram tokenizer that reduces CPU usage by 5-6x, targeting tokenization latency when small rerankers and embedding models run on GPUs in single-digit milliseconds.

Why it matters: Perplexity open-sourced a rewritten Unigram tokenizer that cuts CPU tokenization cost 5 to 6 times in reranking and embedding pipelines, so tokenization stops slowing down responses.

QbitAI · WeChat

Language Models Need Sleep: Let AI Nap Before Continuing Inference

Carnegie Mellon University and the University of Maryland propose a “sleep” mechanism for language models: when the context window is nearly full, the model stops accepting new tokens, runs multiple offline recursive forward passes to compress accumulated context into fast weights, clears the KV cache, and then resumes inference; tests cover cellular automata, multi-hop graph retrieval, and GSM-Infinite reasoning tasks.

Why it matters: Carnegie Mellon and the University of Maryland give LLMs a sleep-like mechanism that pauses and compresses memory as context fills, a transferable approach to long-context memory management.

Xinzhiyuan · WeChat

OpenRouter processes 100 trillion tokens monthly and raises $113M Series B

OpenRouter raised a $113 million Series B led by CapitalG, lifting its valuation to $1.3 billion; the platform processes 25 trillion tokens per week, about 100 trillion per month, and provides one API for more than 400 models.

Why it matters: OpenRouter routes 400-plus models through one API, handles 25 trillion tokens a week, and raised a $1.3B-valuation Series B led by CapitalG.