Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

141–160 of 609

Oct 2Friday

Claude Blog

Customize Claude Code's behavior and interface with mods

Claude Code introduced mods, small TypeScript functions that customize tool behavior and the interface, now supported in the CLI and desktop app. Mods install and share with plugins, and can rewrite prompts, intercept tool calls, handle permission requests or replace built-in features. Claude Code can also write, install and hot-reload them.

Why it matters: Claude Code opens up event rewriting, interface customization and built-in feature replacement through mods, so developers can adjust tool behavior without waiting for official updates.

AI HOT · Tips & opinions

LangChain explains how to build a model router in an Agent Harness

LangChain described how it built a model router inside the Agent Harness of Open SWE. In A/B tests, median cost per thread dropped 64%, with no measurable change in quality.

Why it matters: LangChain combines task analysis, model tiers and result tracking in agent routing, a reusable way to control model cost by task complexity.

Oct 1Thursday

Latent Space

OpenAI interview: why it disputes Dwarkesh on computer use, and building a Jev rival in a week

In an interview, OpenAI's Ari Weinstein and Nikunj Handa explain progress in computer-use agents and look back at the Decisions API, started about a week earlier and inspired by Jev.

Why it matters: The OpenAI team combines screenshots, accessibility data and batched code execution to speed up computer use, cutting the waits from screen-by-screen scrolling and single-step calls.

AI HOT · Products

Factory opens Automations: Droid runs engineering workflows on a schedule or an event

Factory has opened Automations to all users. You describe a repetitive workflow, pick a schedule or an event trigger, and Droid runs it. Each automation has four parts: trigger, instructions, model and the machine it runs on. Model and reasoning level are set per task, with BYOK or Factory Router. Factory says that over the past 30 days, 52% of its enterprise users used automated code review, 48% security review and 35% AutoWiki, with automation sessions up 51% quarter over quarter.

Why it matters: Factory has opened Automations, letting Droid run code review and security audits on a schedule or event trigger, with each automation built from a trigger, instructions, a model and a runner.

Sep 30Wednesday

Hacker News front page

Text classification from bag-of-words to Jev: a language model that classifies without per-task fine-tuning

Jev turns text classification into a general decision service that needs no per-task fine-tuning. The article traces the history of classification techniques and tests Jev's limits. On the 25,000-review IMDb test set, the Choice API hit 96.47% accuracy at a total cost of $0.6492. The article notes it cannot confirm whether the test set was part of the training data.

Why it matters: It compares Jev with bag-of-words models, ModernBERT and general LLMs, and lays out the cost and fit of direct cross-task classification versus fine-tuning a dedicated model.

AI HOT · Products

Factory launches Automations, letting Droid run repeat engineering work on schedules or events

Factory has opened Automations to all users, letting Droid run repeat engineering work on a schedule or when an event fires. Users describe a flow in natural language or pick a template to handle ticket-to-PR work, code review, security checks, CI failure fixes and doc updates. Each automation sets its own model, reasoning level and run machine, runs as a user or service account, and leaves inspectable sessions, run history and failure notifications.

Why it matters: Factory turns repeat engineering tasks into scheduled or event-triggered automations, with per-task model and machine choices and an inspectable record of every run.

TechCrunch · AI

OpenAI adds reusable cloud dev environments to Codex across devices

OpenAI launched reusable cloud development environments for Codex that can be accessed across devices, making remote task environments more persistent and configurable, with shared settings and permissions for teams.

Why it matters: Reusable cloud dev environments for Codex let teams carry approved settings and permissions across devices, cutting the work of preparing an environment for every remote task.

Sep 29Tuesday

Hacker News front page

World Labs to join AMD; Fei-Fei Li becomes Chief Scientist

World Labs signed a definitive agreement to join AMD. Fei-Fei Li will become AMD's EVP and Chief Scientist, reporting to CEO Lisa Su. Justin Johnson and Ben Mildenhall will keep leading the team as it forms a new frontier research org inside AMD. The two companies started a technical partnership last year on model training and inference optimization on AMD GPUs. The goal is an end-to-end open AI ecosystem spanning hardware, software, and open models. The deal is expected to close by end of 2026, pending regulatory approvals.

Why it matters: World Labs is joining AMD outright, with Fei-Fei Li becoming EVP and chief scientist, after the two already worked on training and inference optimization on AMD GPUs.

Sep 25Friday

claude.dev Blog

How Opus 5.5 task cost in Claude Code is calculated

Task cost for Opus 5.5 in Claude Code comes from turns, cache reads, output tokens and model price, and cutting usage can raise retry cost. At API list prices, the same sample session falls from $3.50 on Opus 5 to $2.40 on Opus 5.5, about 31% lower, with cache writes excluded from the sample. The estimate of 40% lower cost on typical workloads also includes fewer task tokens under default settings.

Why it matters: The Claude Code cost guide bills turns, cache and thinking output separately, and uses the same task to separate a lower unit price from lower task consumption.

Sep 24Thursday

Google DeepMind

Google DeepMind adds secure server-side memory to Private AI Compute

Google DeepMind detailed a new capability for Private AI Compute: private, server-side persistent memory that lets an AI assistant keep context across devices. Data sits sealed in encrypted storage, and the unlock key stays only on the user's device. When the model needs access, an end-to-end encrypted channel carries it into a secure cloud enclave, where it is briefly decrypted in isolated memory and immediately re-encrypted.

Why it matters: Google DeepMind adds server-side persistent memory to Private AI Compute, keeping decryption keys on user devices so cloud assistants hold context across devices.

Sep 23Wednesday

AI HOT (Curated Pool)

Modal details how to serve trillion-parameter coding agents at trillion-token scale

Modal's engineering team published a deep-dive on serving Moonshot AI's Kimi K2.6 for coding agents. They boosted per-replica performance by 2.8x per user and 5.6x across users, turning a ruinously expensive service into a price-competitive one. One service processed hundreds of billions of tokens per day and trillions in aggregate. The post walks through workload analysis for hybrid-attention MoE models and the engineering optimizations applied. Exact GPU models and per-request latency numbers are not disclosed in the body.

Why it matters: Modal published its inference optimization work serving the Kimi K2.6 coding agent at trillion-token scale, reaching 2.8x per-user and 5.6x cross-user gains per replica and explaining how trillion-parameter serving spreads cost.

Sep 21Monday

AI HOT (Curated Pool)

AI Comes for the If Statement: Specialized Deciders Cut Classification Cost ~100x

Tomasz Tunguz tested Jev and SemIf on 98 production emails: classification accuracy jumped from 47% to over 80%, cost dropped to $0.0004 per call—76x to 209x cheaper than frontier models. These deciders skip text generation, run attention once, and output choice probabilities in hundreds of milliseconds. Tunguz sees this as a bifurcation: frontier models for discovery, specialized models for production, with if-then as the first optimized programming primitive.

Why it matters: Tunguz tests dedicated if-then decision makers like Jev and SemIf on a real email classification task from his own code, lifting accuracy from 47% to over 80% at roughly 99% lower cost than conventional AI.

Sep 19Saturday

Hacker News front page

C2C lets LLMs talk via KV-cache, 2.5× faster and 3–5% more accurate than text

This ICLR'26 paper proposes Cache-to-Cache (C2C), where multiple LLMs communicate by directly exchanging KV-cache instead of generating text. A neural network projects and fuses the source model's KV-cache into the target model, with a learnable gate selecting which layers benefit. C2C beats single models by 6.4–14.2% in average accuracy, outperforms text-based communication by ~3.1–5.4%, and delivers an average 2.5× latency speedup. Code is open at thu-nics/C2C.

Why it matters: Tsinghua and other institutions propose C2C, letting models pass semantics by projecting and merging KV-Cache directly instead of generating text, with 3.1-5.4% higher accuracy and 2.5x lower latency.

Sep 17Thursday

Hacker News front page

Ternary LLMs break the 1.58-bit floor: BITCOS hits 1.485 bits per weight by exploiting zero-weight density

Ternary models store weights as -1, 0, or +1, with a theoretical floor of ~1.585 bits and a practical 1.625 bits in five-trit packing. Intel authors measured 29 ternary LLMs and found up to 51.5% zeros. BITCOS replaces fixed packing with a presence bitmap plus a compacted sign vector, costing 2 minus zero-density bits per weight. It beats five-trit packing on 26 of 29 models and reaches 1.485 bits on the sparsest. Optimized unpacking on AVX-512, AVX2, and Xe2 GPUs yields up to 1.28× faster matrix-vector multiply; end-to-end decode throughput improves up to 1.18× on CPUs and 1.27× on GPUs. The paper does not name the models or disclose their parameter counts, nor whether they are publicly available.

Why it matters: Intel measured 29 ternary models and found zero weights take up as much as 51.5% of the total, which is what makes sub-1.58-bit storage possible.

Sep 15Tuesday

r/LocalLLaMA

DeepSeek V4.1 Flash Q4 hits 40 t/s on M3 Ultra with native DSpark multi-token prediction

A developer forked ds4 and tuned it for DeepSeek V4.1 Flash Q4 on a 512 GB M3 Ultra, lifting decode from 16.6 t/s to 31.3 t/s, and to 40.5 t/s with DSpark speculative decoding. The ~300 GB Q4 weights fit only the 512 GB M3 Ultra. The speedup comes from cutting Metal dispatch overhead: the 384-expert router went from 9 dispatches to 1, the shared expert gate+up+SwiGLU became a single kernel, and BF16 rounding moved inside producer kernels, removing ~770 re-round dispatches. At 300k context, compressed attention selection was the bottleneck; the fix scores only admitted blocks and uses a bounded radix select, keeping decode at 90% of the 8k rate. DSpark verifies 6 tokens per step with shared weight streams and overlapped Engram fetches, dropping verify latency from 177 ms to 112 ms. Output is byte-identical to upstream under greedy decode, with SHA-256 manifests provided. The branch is M3 Ultra only because the optimizations rely on measured behavior of this specific chip's dual-die memory, 80-core GPU scheduling, and Metal dispatch characteristics.

Why it matters: A hands-on Metal scheduler rewrite that doubles 8k-context decode speed on M3 Ultra, with a byte-identical check script against upstream.

Sep 10Thursday

DeepSeek · API updates

DeepSeek-V4.1-Flash ships with native vision and a lower API price

DeepSeek released DeepSeek-V4.1-Flash, the smallest model in its new architecture series, with native multimodal vision understanding. It is live on the DeepSeek API under deepseek-flash, and API prices dropped at the same time. The older V4 Flash and V4 Flash Vision Exp are retired, with their call names routed to V4.1 Flash for now.

Why it matters: DeepSeek routes the old model names to V4.1 Flash and cuts API prices, so existing calls keep their names but switch models.

Sep 2Wednesday

Hacker News front page

Slotstream runs the 104GB Qwen3.8-Flash-Next on a 48GB Mac at ~12 tok/s

carloslfu open-sourced Slotstream, an MLX + Swift tool that runs the 125B-parameter MoE model Qwen3.8-Flash-Next (104GB at 4-bit) on a Mac with only 48GB RAM. It streams expert modules from SSD on demand instead of loading everything into memory. Speed is ~12 tok/s, and it exposes an Ollama-compatible API. The post doesn't disclose time-to-first-token or the SSD model used, so real-world feel is still an open question.

Why it matters: Developer carloslfu open-sourced slotstream, which streams expert weights from SSD to run a 104GB Qwen model on a 48GB Mac at about 12 tok/s, with an Ollama-compatible API.

Aug 26Wednesday

Computing Life · Share · Yage

The term 'local LLM' conflates two separate markets

Yage breaks down 'local LLM' into two markets: a cost market buying 5–20× price gaps, and a control market buying 25–33-year certainty. Using a four-quadrant framework (open/closed weights × time/token billing), the piece explains why surging open-weight model usage on OpenRouter doesn't mean local deployment is winning. Self-hosting payback depends entirely on which cloud billing mode you replace—decades for subscriptions, months for high-cache-hit agent API calls. In July–August 2026, Anthropic and others made four moves at the inference layer: silently remapping parameters, repeatedly extending usage boosts, adding watermarks, and redefining self-hosting as 'your harness plus my inference.' But simultaneous deep price cuts mean the misalignment is real but direction is unresolved.

Why it matters: The piece splits local LLMs into cost and control markets, maps open versus closed weights against billing models, and notes OpenRouter's open-weight growth comes from remote APIs.

Aug 21Friday

Hacker News front page

Nari Labs pushes Qwen3-TTS to sub-50 ms time-to-first-audio at 10 RPS on a single H100

Nari Labs open-sourced a Qwen3-TTS 1.7B CustomVoice serving implementation that hits sub-50 ms p95 time-to-first-audio at 10 RPS on a single H100 SXM with zero underruns. They benchmarked against vLLM-Omni, SGLang-Omni, VoxServe, and M*—default p95 latencies ranged from 277 to 1,160 ms at 1 RPS. At full utilization the system costs roughly $2 per 1M characters, compared to $100 for ElevenLabs V3 and $49 for Cartesia Sonic 3.5. Key optimizations include dynamic leading-silence trimming (~80 ms saved) and tuned codec-frame accumulation. Code and benchmarks are public; the post does not disclose underrun details at higher concurrency or long-form performance.

Why it matters: Nari Labs open-sourced a Qwen3-TTS 1.7B deployment that holds p95 first-audio latency under 50 ms on a single H100 at 10 concurrent requests, and compared four engines at their defaults.

Aug 20Thursday

Hacker News front page

DFlash 2 pushes parallel drafting further: over 20% more output per verification pass for ~1% added latency

Inco AI released DFlash 2, adding a lightweight path selector on top of parallel speculative decoding. Instead of keeping only the top-1 candidate per position, it picks a coherent path from the top 16, raising accepted tokens per verification from 4.27 to 6.79. On Qwen3.8-27B with SGLang, throughput reaches 2.7–3.4× autoregressive decoding at batch size 1, with roughly 1% added cycle latency. SGLang, vLLM, llama.cpp, and oMLX already support it; DFlash models have been downloaded over 3.5 million times on Hugging Face.

Why it matters: Inco AI's DFlash 2 adds a path selector to parallel drafting, giving over 20% more useful output per verification round for roughly 1% more latency.