Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

81–100 of 609

Oct 21Wednesday

OpenAI News

OpenAI partners with Google Cloud to bring its models and Codex into Model Garden

OpenAI and Google Cloud announced a partnership to bring OpenAI's frontier models and Codex agent workflows into Google Cloud's Model Garden, with general availability planned in the coming weeks.

Why it matters: Enterprises can buy OpenAI models against existing cloud spending commitments and deploy them inside their current data governance setup.

Latent Space

DeepSeek open-sources DeepInfer, discloses 155T tokens/day inference on a single cluster with caching

DeepSeek released DeepInfer, the inference engine behind V4.1 Flash, along with deepseek-recipe, the component that handles API requests. The engine supports distributed inference, speculative decoding and structured output. DeepSeek also published production figures showing 155T tokens per day on a single cluster with caching.

Why it matters: The production numbers and cache hit rates let teams estimate inference margins, since cache billing and hardware cost assumptions drive the math.

Computing Life · Share · Yage

Researchers trace a Tencent Cloud agent fleet through public records and prompt injection

Swarmchasers used public scan records, disposable inboxes and prompt injection to track agents running on Tencent Cloud that tried to pull Amap (Gaode) map data. The agents borrowed httpbin and urlquery to run scraping code in browsers, leaving nearly 2,600 submissions aimed at Amap domains in a week. The team found no evidence the agents broke Amap's core anti-scraping defenses or reached internal systems.

Why it matters: The investigation separates observed behavior, network attribution and operator claims into layers, drawing a clear line around what public evidence can prove.

TechCrunch · AI

Anthropic's cloud deals get complicated as capacity purchases stall

A leaked S-1 reviewed by The Wall Street Journal says Anthropic cannot get enough cloud capacity from AWS, Google and Microsoft, which increasingly see it as a competitor. The company expects cloud contract spending to exceed $750 billion, including deals still under negotiation. It is also leasing its own data centers and working with third-party operators, but worries those operators have limited track records and may not be reliable.

Why it matters: Anthropic is turning to its own leased data centers while the big clouds treat it as a rival, and third-party reliability limits the backup.

AI HOT · Products

Meta's open-weight Muse Spark Open launches on Fireworks US serverless

Meta's open-weight model Muse Spark Open went live on Fireworks serverless on launch day, running on US capacity with no US premium. The MoE model has 816B parameters, activates 42B per token, and offers a 1M-token context window for long-horizon coding and agent tasks.

Why it matters: Fireworks runs Muse Spark Open on US capacity with zero data retention by default, giving regulated professional work clear data residency and retention boundaries.

AI HOT · Industry

SpaceX earnings show AI business beating expectations at 52% of revenue

SpaceX's earnings show AI now makes up 52% of revenue, with third-quarter revenue of $14.6 billion and net profit of $4.4 billion. Growth comes especially from the renamed Super Intelligence business, which covers the recently acquired Cursor, Grok and a new cloud business with 1.7 GW of compute.

Why it matters: SpaceX swung to a third-quarter profit, but it is still in the red for the first nine months and keeps spending on data centers.

Mistral AI

Mistral AI releases Voxtral 3 speech models and a voice agent SDK

Mistral AI released the Voxtral 3 family of speech recognition and speech generation models, plus a real-time voice agent SDK that combines listening, reasoning, tool calls and speaking. Transcription covers 50+ languages with under 500ms real-time latency. Speech generation covers 35+ languages and 50+ dialects, adjusting emotion, pace and accent to context.

Why it matters: Mistral folds recognition, reasoning, tool calls and speech generation into one streaming pipeline, with each stage swappable so a language or voice change doesn't mean rewriting the agent.

Oct 20Tuesday

AI HOT · Products

Cursor ships Agent SDK to define agents in code and deploy with one command

Cursor released its Agent SDK, which lets teams define agents in code and deploy them to Cursor infrastructure with agent-sdk deploy. The framework handles events like pull requests, task handoffs and parallel sub-agents, and includes built-in evals and event replay. Prompts and tools go through version control and review. Agents can also run as a single Node process on your own infrastructure.

Why it matters: Cursor's Agent SDK turns prompts, tools and evals into a code project, so custom agents follow the same version control and review as other software.

AI HOT · Products

Cursor launches Agent SDK for defining and deploying agents from a code folder

Cursor launched its Agent SDK, which lets teams define agents in a code folder, test them with evals, and deploy to Cursor or their own infrastructure.

Why it matters: Cursor's Agent SDK organizes prompts, tools and skills as a code project, testing agent behavior with local evals and replays of real pull requests.

Hacker News front page

Inception releases Mercury 3, scaling its text diffusion model past 40x Mercury 1

Inception released Mercury 3, a diffusion-based large language model. The company says it beats GPT-6 Luna and Claude Haiku 5.5 on several benchmarks and runs 4–8x faster.

Why it matters: Mercury 3 scales past 40x the parameters of Mercury 1 while keeping similar speed, giving a like-for-like read on text diffusion at larger size.

Oct 19Monday

AI HOT · Tips & opinions

Meta details gigawatt-scale Prometheus supercluster; first phase is training Muse models

Meta says the first phase of its gigawatt-scale Prometheus supercluster is complete and is being used to train the Muse model family. It has hundreds of thousands of GPUs now, with plans to reach about 1 million. Effective throughput tops 85% for Muse Wonder pretraining and 90% for RL training, and a year of engineering work cut cluster interruptions 99% from their peak.

Why it matters: Meta combines in-rack spare GPUs, parallelism-aware fault tolerance and asynchronous checkpointing into a practical method for cutting downtime and repeated compute in large training runs.

Hacker News front page

Stripe to acquire OpenRouter in reported $7.5B deal

Stripe has agreed to acquire OpenRouter, with The New York Times reporting a $7.5 billion price. A joint statement confirmed the deal but did not disclose the price. OpenRouter gives access to more than 400 AI models through a single API and, according to the two companies, handles over 10 trillion tokens a day for more than 10 million developers.

Why it matters: The deal puts model routing inside Stripe's payments and token billing, making routing neutrality and commercial incentives the central question after it closes.

Computing Life · Share · Yage

Anthropic's Scion source-control rebuild: which engineering calls transfer, which don't

Anthropic rebuilt its internal source control with Scion, and this analysis sorts the engineering calls other teams can reuse from the system design they shouldn't copy. The large monorepo's daily merges rose from 2,500 in May 2026 to more than 30,000 in September. Scion uses a thin client, server-side recomputation and three cache layers, and the report says source-control infrastructure CPU and memory use fell 200-fold.

Why it matters: The reusable parts for ordinary teams are pulling code on demand, reviewing the intent of a change and building automatic rollback guardrails.

Oct 17Saturday

Hacker News front page

Neuralwatt on how agent harnesses affect energy use and token efficiency

Neuralwatt analyzed cloud traffic and experimental requests and found that an agent harness editing an existing prompt breaks the cache, adding recomputation and energy use. With the same model and similar input and output lengths, requests that mostly miss the cache use about twice the energy of cache-hit requests, and about 2.5 times for GLM-5.3. When context passes 128k tokens, one broad re-read costs prefill compute roughly equal to 600 ordinary turns.

Why it matters: Neuralwatt traces cache misses request by request and frames stable prompt prefixes and appended state changes as ways to cut an agent's repeated computation.

The Decoder

China drafts export controls on open-weight models and 23 key AI technologies

China's commerce and science ministries released a draft revision of the technology export catalog that would put 23 key AI technologies under export control, covering large models, agents, compute clusters and data processing. A model's core technology is caught if it reaches 10^26 FLOPs of training compute, has at least 2 trillion parameters, or ranks in the global top 10. Models trained on China's "important data" and serving users from that training are also restricted.

Why it matters: China plans licensing thresholds for core large-model technology, with different procedures for overseas local servers and overseas cloud deployments, which directly shapes how models can be deployed abroad.

Oct 16Friday

AI HOT · Products

NVIDIA RTX Spark laptops go on sale today, up to 128GB unified memory

NVIDIA's RTX Spark laptops went on sale worldwide today, with up to 128GB of unified memory and 1 petaflop of FP4 AI performance, starting at $2,599. The machines pair a Blackwell RTX GPU with a Grace CPU and CUDA, run large language models and AI agents locally, and add agent permission controls plus inference serving over a home network.

Why it matters: RTX Spark puts local model running, coding and testing inside a Windows laptop, cutting how much the dev loop depends on cloud test environments.

AI HOT · Industry

China's Commerce and Science Ministries Propose Adding 23 AI Technologies to Export Control List

The Commerce Ministry and Science and Technology Ministry issued Announcement No. 48 of 2026, proposing to add 23 key AI technologies to China's catalog of technologies banned or restricted from export, and opened the draft to public comment. Interim measures require registration before export. The proposed scope covers large models, agents, compute clusters and information processing. Technologies already in the public domain through open source or academic papers would not be included.

Why it matters: The draft sets where restricted large-model overseas deployment and offshore API calls fall, and the line between technology transfer and service provision shapes compliance for companies going abroad.

AI HOT · Products

SGLang adds Day-0 support for Muse Spark Open, an 816B open-weight multimodal model

SGLang shipped Day-0 support for Muse Spark Open, an 816B open-weight multimodal model. The release adds sequence parallelism, NVFP4 MegaMoE, multimodal processing optimizations and DFlash V2 speculative decoding. Across seven concurrency levels, combining sequence parallelism with MegaMoE for long-prompt prefill cut p90 time to first token by 28.3% to 41.9%.

Why it matters: It pairs sequence parallelism with MegaMoE for long-prompt prefill and reports p90 time-to-first-token cuts of 28.3% to 41.9% across seven concurrency levels.

AI HOT · Tips & opinions

How Meta built its Forum app with AI-native engineering

Meta used AI agents in Forum's planning, implementation, testing and verification, and a small team reached parity with Facebook Groups' core features in 8 weeks. The team supported agent work with readable app state, automated verification and build isolation. Engineers still joined some stages to iterate, and signed off on every shipped result.

Why it matters: Meta wired automated tests, UI checks and verification records into the agent workflow, so engineers review code against checkable results and keep the release sign-off.