Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

401–420 of 609

May 14Thursday

AI HOT (Curated Pool)

Unlocking Asynchrony in Continuous Batching

Hugging Face says an 8B model generating 8K tokens leaves the GPU idle for 24% of the time, and asynchronous batching uses CUDA streams to overlap CPU preparation for batch N+1 with GPU computation for batch N.

Why it matters: Hugging Face shows an 8B model idles its GPU 24% of the time generating 8,000 tokens, and explains how CUDA streams fill those gaps.

Financial Times · Technology

Cerebras boosts IPO price to raise $5.5bn

Cerebras raised its IPO price to seek $5.5bn in proceeds; the post discloses a $40bn valuation, but does not disclose the price range, share count, or listing date.

Why it matters: Cerebras raised its IPO price range to target $5.5bn, implying a roughly $40bn valuation and a read on demand for non-Nvidia AI chips.

r/LocalLLaMA

2x RTX 3090 setup for local Qwen 3.6 27B inference

A Reddit user ran Qwen 3.6 27B on a dual RTX 3090 Ubuntu setup, reporting 48GB VRAM, a 262k context window, no NVLink, about 4000 pp/s prompt processing, and 113 tk/s generation.

Why it matters: A Reddit user runs Qwen 3.6 27B on two RTX 3090s in Ubuntu, holding 260K context in 48GB of VRAM at 113 tokens per second.

r/LocalLLaMA

24+ tok/s from ~30B MoE models on an old GTX 1080

User mdda ran Qwen 3.6 35B-A3B on an i7-6700, GTX 1080, and 32GB RAM machine at about 24 tok/s with 128k context; the setup uses llama.cpp MoE offloading plus TurboQuant/RotorQuant KV cache quantization, with PCIe 3.0 x16 saturated and GPU utilization at about 40–50%.

Why it matters: A user runs roughly 30B-parameter MoE models on an i7-6700 with a GTX 1080 at about 24 tokens per second and 128k context, with the exact setup and numbers.

Bloomberg Technology

AI Chipmaker Cerebras Raises $5.55 Billion in Year’s Biggest IPO

Cerebras Systems raised $5.55 billion in a US IPO, and the title calls it the year’s biggest IPO; the RSS snippet does not disclose the offering price or valuation.

Why it matters: Cerebras raised $5.55 billion in an IPO priced at $185 a share, money earmarked for expanding production of its giant AI training chips and customer growth.

AI HOT (Curated Pool)

Meta AI chief announces Incognito Chat for WhatsApp and Meta AI

Meta’s AI chief announced Incognito Chat for WhatsApp and Meta AI, with conversation inference running inside the phone’s hardware secure enclave, no server logs generated, and session data permanently deleted after the chat ends.

Why it matters: Meta launched Incognito Chat on WhatsApp, where inference runs inside the phone's secure enclave, servers keep no logs and sessions are deleted on exit, unlike temporary chats that only clear history.

May 13Wednesday

QbitAI · WeChat

ByteDance Proposes Generative Refinement Networks as a Third Route for Visual Generation

ByteDance’s commercial technology team proposed GRN, a visual generation architecture using HBQ, global refinement, and complexity-aware sampling to address quantization loss, error accumulation, and fixed-step inference; on a 130M model, adaptive sampling reduced inference from 50 steps to an average of 24, while gFID changed from 3.56 to 3.79.

Why it matters: ByteDance's GRN drafts a coarse image then refines it, with HBQ layered binary quantization cutting loss in discrete codes, a concrete third path beyond diffusion and autoregression.

r/LocalLLaMA

The Trillion-Parameter Dilemma: MiMo-V2.5-Pro Open-Sourced at 1.02T Parameters

Xiaomi open-sourced MiMo-V2.5-Pro with 1.02T parameters, 42B active parameters, a 1M context window, and an MIT license; the author ran 125 Claude Code sessions through the API, spending $70.12 for 387,380,436 tokens with a 96.3% cache hit rate.

Why it matters: A concrete cost benchmark for self-hosting versus API calls, built from 125 Claude Code tasks and 387M tokens.

New York Times Chinese

Jensen Huang Gets Last-Minute Invitation to Join Trump’s China Trip

Trump called Jensen Huang on Tuesday morning to invite him to join the China trip; the White House’s Monday list of 16 CEOs did not include him, while Nvidia is still seeking approval to sell AI chips to China.

Why it matters: Nvidia spent a year lobbying both Washington and Beijing on AI chip export licenses, and its CEO now joins the White House group.

New York Times Chinese

China Seeks AI Technology Self-Reliance, Weakening Washington’s Leverage Over Beijing

DeepSeek optimized its latest model for inference on Huawei chips for the first time, while two semiconductor sources said training still relies on Nvidia chips; Huawei says it plans to release a training chip this year, but matching current Nvidia performance will take another year.

Why it matters: DeepSeek's new model runs inference on Huawei chips while training still depends on Nvidia, and Huawei's training chips are a year behind, which shows where China's AI hardware decoupling stands and where it stalls.

r/LocalLLaMA

A real transformer language model running locally on a stock Game Boy Color

maddiedreese ran Andrej Karpathy’s TinyStories-260K on a stock Game Boy Color with INT8 weights, fixed-point math, an MBC5 ROM, bank-switched cartridge storage, and KV cache in cartridge SRAM; the demo uses no phone, PC, Wi‑Fi, link cable, or cloud inference, but output is extremely slow and gibberish.

Why it matters: HKR-H/K/R all pass: a named first-person experiment with concrete model and memory details. Impact stays low-featured because it is a Reddit hardware hack with slow, garbled output, not a usable product or model release.

Bloomberg Technology

CME Plans Computing Power Futures Market

CME Group and Silicon Data plan to create a futures market for computing power; the RSS snippet says Bloomberg Tech discusses the rationale and mechanics, but the post does not disclose contract specifications, launch timing, or pricing methodology.

Why it matters: CME and Silicon Data are working on a compute futures market that would let AI companies lock in GPU rental costs in advance, the way oil buyers hedge against price swings.

AI HOT (Curated Pool)

Claude Opus 4.7 Fast Mode Opens Research Preview

Claude Opus 4.7 Fast Mode is now available as a research preview in the API and Claude Code. The post does not disclose model parameters, pricing, rate limits, or a general availability date.

Why it matters: Anthropic opened a fast mode of Claude Opus 4.7 as a research preview for the API and Claude Code, which may cut waiting time on coding and tool calls.

Hacker News front page

Show HN: Needle Distills Gemini Tool Calling into a 26M Model

Cactus open-sourced Needle, a 26M-parameter tool-calling model that reaches 6,000 tok/s prefill and 1,200 tok/s decode on consumer devices, with MIT-licensed weights released on Hugging Face.

Why it matters: Cactus distilled Gemini's tool calling into a 26M-parameter model that runs locally on consumer devices at 6,000 tokens per second prefill and 1,200 per second generation.

r/LocalLLaMA

Needle: We Distilled Gemini Tool Calling Into a 26M Model

Cactus Compute open-sourced Needle, a 26M-parameter tool-calling model that reaches 6,000 tok/s prefill and 1,200 tok/s decode on consumer devices, using an attention-and-gating architecture with no MLPs.

Why it matters: Cactus Compute distilled Gemini's tool calling into a 26M-parameter model and reports 6,000 tokens per second prefill on consumer devices, a lightweight option for on-device tool use.

TechCrunch · AI

Report: Google and SpaceX in Talks to Put Data Centers Into Orbit

Google and SpaceX are discussing orbital data centers for AI compute, while the post says current space costs remain higher than ground infrastructure; the RSS snippet does not disclose cost gaps, deployment scale, or a timeline.

Why it matters: Google and SpaceX are weighing orbital data centers for AI compute, and TechCrunch's math shows launch and satellite costs still make space far pricier than the ground.

Financial Times · Technology

CME plans to launch futures market for AI computing power

CME plans to launch futures contracts tied to GPU rental prices, allowing traders and companies to bet on or hedge future costs; the RSS snippet does not disclose contract specifications, launch timing, or the reference index.

Why it matters: CME plans futures contracts tied to GPU rental prices, letting compute costs be hedged like oil, a step toward compute as a tradable financial asset.

May 12Tuesday

r/LocalLLaMA

Local LLM Autocomplete and Agentic Coding on a Single 16GB GPU + 64GB RAM

Reddit user grumd runs Qwen2.5-Coder-7B Q6 for autocomplete and Qwen3.6-35B-A3B Q8 for agentic coding on one RTX 5080 with RAM offloading; the post reports about 145k context, 56GB RAM used with other apps open, and Qwen3.6-35B-A3B speed of tg128 at 35.29 tokens/s.

Why it matters: One user runs both a code-completion model and an agentic coding model on a single 16GB GPU with 64GB RAM, and lists the models, quantization and context lengths.

Synced · WeChat

ByteDance Open-Sources DreamLite for Offline Mobile Image Generation and Editing

ByteDance open-sourced DreamLite, a 0.39B-parameter unified diffusion model that generates or edits a 1024×1024 image on an iPhone 17 Pro in about 3 seconds, using 4-step DMD2 distillation and on-device offline inference without cloud dependency.

Why it matters: ByteDance open-sourced DreamLite, a 390M-parameter unified diffusion model that generates or edits a 1024×1024 image locally on an iPhone 17 Pro in about 3 seconds, using DMD2 distillation to cut inference to 4 steps.

AI HOT (Curated Pool)

What Parameter Golf Taught Us About AI-Assisted Research

OpenAI’s Parameter Golf brought together over 1,000 participants and more than 2,000 submissions to test AI-assisted machine learning research, coding agents, model quantization, and model design under strict parameter constraints.

Why it matters: OpenAI ran a Parameter Golf challenge capping model plus code at 16MB and training at 10 minutes, drew over 1,000 entrants and 2,000-plus submissions, and recaps how participants used AI assistants to compress models.