Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

461–480 of 609

May 7Thursday

Financial Times · Technology

SpaceX to Rent Data Centre Capacity to Anthropic

SpaceX will rent data centre capacity to Anthropic, confirming a compute leasing arrangement. The RSS snippet says Anthropic is adding compute to match growth; the post does not disclose capacity, term, or pricing.

Why it matters: SpaceX will rent data center capacity to Anthropic, a new direction in their partnership as Anthropic adds compute to keep up with user growth.

r/LocalLLaMA

GB10 inference engine Atlas is open source, with Qwen3.6-35B-FP8 over 100 tok/s

Avarok open-sourced Atlas, an inference engine running Qwen3.5-35B at ~111 tok/s sustained on one DGX Spark. It uses Rust+CUDA, a ~2.5GB image, and sub-2-minute cold start; the author claims 3.0–3.3x vLLM in tests. The key details are Blackwell SM120/121 kernels, NVFP4/FP8, and MTP decoding.

Why it matters: Avarok open-sourced Atlas, a Rust and CUDA inference engine, hitting about 111 tok/s for Qwen3.5-35B on a DGX Spark, which the author puts at 3.0 to 3.3 times vLLM.

r/LocalLLaMA

Exaggerated PCI-E Bandwidth Concerns?

Reddit user ziphnor tested 2x RTX 5060 Ti 16GB with vLLM TP=2 and 32k-context prefill. PCIe peaked at 3–4 GB/s, about 40–50% of a PCIe 4.0 x4 link. Prefill reached ~840–850, 1500, and 1600–1700 t/s; the post does not disclose decode bandwidth.

Why it matters: User ziphnor ran vLLM tensor parallelism on two RTX 5060 Ti cards and measured PCIe peak bandwidth of only 3-4 GB/s during 32k-context prefill, 40-50% of the PCIe 4.0 x4 link.

r/LocalLLaMA

Analysis of 922 Agentic Task Traces Finds DeepSeek v4’s Cost Edge in Caching

A Reddit user analyzed 922 agentic task traces and reported $0.01 per task for DeepSeek v4 Flash versus $1.52 for Opus 4.7. Both used about 960K tokens per task, but DeepSeek showed a 97% cache hit rate versus 87%, with a 0.02 cache read/write price ratio versus 0.08. The key issue is caching, not headline pricing.

Why it matters: A Reddit user analyzed 922 agent task traces and compared per-task cost for DeepSeek v4 Flash and Opus 4.7, finding similar token use and the gap coming from cache hit rate and cache read-write pricing.

Bloomberg Technology

Anthropic Signs Computing Deal With SpaceX to Meet AI Demand

Anthropic signed a computing deal with Elon Musk’s SpaceX to support growing Claude demand. The post does not disclose capacity, contract value, deployment timing, or infrastructure details. The key issue is whether SpaceX enters Anthropic’s long-term training or inference supply chain.

Why it matters: Anthropic signed a compute deal with SpaceX to support growing Claude usage, but the size, value, timing and purpose are all undisclosed.

May 6Wednesday

r/LocalLLaMA

Qwen3.6 27B NVFP4 + MTP on a Single RTX 5090: 200k Context in vLLM

A Reddit user ran Qwen3.6 27B NVFP4 on one RTX 5090 32GB and validated 200k context in vLLM. The setup used fp8_e4m3 KV cache, FlashInfer, and MTP with 3 speculative tokens; a 10-run 200k pass completed with 73.6 tok/s mean generation and 70.2s TTFT. The key constraint is 32GB VRAM: logs showed 8.3GiB KV cache and about 30478MiB total GPU use.

Why it matters: A Reddit user ran Qwen3.6 27B NVFP4 on a single RTX 5090 in vLLM, verified 200k context, and posted the fp8_e4m3 KV cache and MTP speculative decoding config.

TechCrunch · AI

AI boom pushes Samsung to $1T

Samsung crossed a $1 trillion valuation after shares rose on AI chip demand. It is the second Asian company after TSMC to hit the mark; the post does not disclose share gains, revenue, or order mix.

Why it matters: Samsung crossed a $1T market cap, the second Asian company after TSMC, as memory chip demand from AI systems outran supply and pushed prices up.

NVIDIA Blog

NVIDIA Spectrum-X AI-Native Ethernet Fabric Adds MRC for Gigascale AI

NVIDIA added MRC support to Spectrum-X Ethernet, letting one RDMA connection spread traffic across multiple paths. MRC ran in Blackwell deployments, with microsecond failure bypass and hardware rerouting. The key detail is the OCP open specification and multiplane support for clusters up to hundreds of thousands of GPUs.

Why it matters: NVIDIA's Spectrum-X adds multipath RDMA so one RDMA connection spreads traffic across paths, already used in Blackwell clusters and pushed through OCP for larger GPU fabrics.

r/LocalLLaMA

2.5x Faster Inference with Qwen 3.6 27B Using MTP on 48GB

A llama.cpp PR adds MTP support for Qwen 3.6 27B, with a reported 2.5x inference speedup. The author measured 28 tok/s on a Mac M2 Max 96GB and shared GGUF builds, compile steps, and a 262144-context server command. The key detail is turbo4 4.25-bit KV cache: a 48GB Mac runs Q5_K_M at 262K context.

Why it matters: A llama.cpp PR adds MTP support for Qwen 3.6 27B, measured at 2.5x faster inference with 262k context on 48GB, a usable setup for local agentic coding.

Latent Space

AINews: Silicon Valley Gets Serious About Services

Anthropic and OpenAI announced enterprise services companies: Anthropic’s unnamed JV is funded with $1.5 billion, while OpenAI’s The Deployment Company has raised about $4 billion at a $10 billion pre-money valuation.

Why it matters: Anthropic and Blackstone and others formed a joint venture services firm with $1.5 billion, sending small teams in first to scope Claude use cases before engineers build custom systems.

Synced · WeChat

Two Chinese open-source projects turn Mac into a private AI workstation

Mininglamp open-sourced Cider and Mano-P 1.0 for Apple Silicon local inference and GUI agents. Cider speeds Qwen3-VL-2B prefill by 57%–61% on M5 Pro; Mano-P 1.0-72B scores 58.2% on OSWorld. The key constraint is W8A8 memory: on 16GB devices accuracy falls from 58.0% to 54.0%, so 32GB+ is recommended.

Why it matters: Minglue open-sources Cider and Mano-P 1.0, with Cider cutting Qwen3-VL-2B first-token latency about 60% on M5 Pro and the 72B Mano-P scoring 58.2 on OSWorld.

r/LocalLLaMA

DeepSeek V4 at 17x lower cost prompted a local-vs-cloud coding workflow test

Reddit user spencer_kw logged a 10-day coding workflow and retested 150 tasks on local Qwen 3.6 27B versus cloud models. Local was equivalent for 65% of tasks, acceptable for 20%, and cloud was needed for 15%; the API bill fell from $85/month to about $22. The useful signal is task-based routing, not headline model pricing alone.

Why it matters: A concrete split of which coding tasks run on a local Qwen model and which still need the cloud, with the API bill to match.

TechCrunch · AI

OpenAI releases GPT-5.5 Instant, a new default model for ChatGPT

OpenAI released GPT-5.5 Instant as ChatGPT’s new default model. The company says it reduces hallucinations in law, medicine, and finance while keeping prior low latency; the post does not disclose benchmarks, rollout scope, or pricing.

Why it matters: OpenAI swapped ChatGPT's default model from GPT-5.3 Instant to GPT-5.5 Instant, claiming fewer hallucinations in law, medicine and finance while keeping latency low.

r/LocalLLaMA

Gemma 4 MTP Released

Google released Gemma 4 MTP drafters with 4 Hugging Face checkpoints listed. MTP uses a smaller draft model to predict multiple tokens, then the target model verifies them in parallel, giving up to 2x decoding speedups with identical output quality.

Why it matters: Google posted four Gemma 4 MTP checkpoints on Hugging Face, where a small model guesses ahead and the main model verifies in parallel, doubling decode speed with unchanged output quality.

May 5Tuesday

r/LocalLLaMA

Heretic 1.3 Released: Reproducible Models, Integrated Benchmarks, Lower Peak VRAM

Heretic 1.3 adds reproducible runs, integrated benchmarks, lower peak VRAM, and broader model support. The project claims 20,000 GitHub stars and 13 million model downloads. Reproduce directories capture PyTorch, GPU, driver, and accelerator details; benchmarks use lm-evaluation-harness for MMLU, EQ-Bench, GSM8K, and HellaSwag. The post names Qwen3.5 and Gemma 4 support, but does not disclose VRAM reduction figures.

Why it matters: Heretic 1.3 records the PyTorch, GPU and driver environment so others can reproduce results, and adds built-in benchmarking with lower peak VRAM use.

OpenAI News

OpenAI Introduces MRC for Large-Scale AI Training Networks

OpenAI introduced MRC for large-scale AI training cluster networks. MRC stands for Multipath Reliable Connection and is released via OCP to improve resilience and performance; the post does not disclose throughput, latency, or cluster size.

Why it matters: OpenAI open-sourced MRC, a training network protocol that splits one transfer across hundreds of paths and reroutes around link failures in microseconds, useful for teams running their own large GPU clusters.

r/LocalLLaMA

vibevoice.cpp: Microsoft VibeVoice ported to ggml/C++ with no Python at inference

LocalAI released vibevoice.cpp, a ggml/C++ port of Microsoft VibeVoice for CPU, CUDA, Metal, and Vulkan inference. TTS uses a 30s reference clip for 24kHz cloned speech; ASR uses a 7B model with diarized JSON and was tested on 17min audio. The key constraint is memory: 17min CPU Q8_0 peaks near 26GB, with no streaming output yet.

Why it matters: LocalAI rewrote Microsoft VibeVoice's speech synthesis and long-audio transcription as a pure C++ version that runs on CPU, CUDA, Metal and Vulkan, so local deployment no longer needs a Python environment.

Synced · WeChat

Massive Idle Cluster: Musk’s 550,000 Nvidia GPUs Are Only 11% Utilized

The Information says xAI’s roughly 550,000 Nvidia GPUs have only 11% MFU, equal to about 60,000 effective GPUs. The post cites HBM I/O, inter-server communication, training idle time, and software-stack inconsistency; Meta and Google are listed at 43% and 46%.

Why it matters: xAI's roughly 550,000 Nvidia GPUs run at only 11% model FLOPs utilization, equal to about 60,000 cards doing real work, while Meta and Google sit at 43% and 46%.

r/LocalLLaMA

MTPLX: 2.24x Faster TPS Native MTP Inference Engine for Apple Silicon

MTPLX raises Qwen3.6-27B on a MacBook Pro M5 Max from 28 to 63 tok/s. The test used 4-bit MLX, temperature 0.6, top_p 0.95, top_k 20, with D3 as the best depth. The key detail is native MTP heads: no external drafter and no second-model memory.

Why it matters: MTPLX on a MacBook Pro M5 Max raised Qwen3.6-27B generation speed from 28 to 63 tokens per second, a 2.24x gain from built-in MTP heads with no separate drafter model to load.

TechCrunch · AI

OpenAI’s cozy partner Cerebras is on track for a blockbuster IPO

Cerebras is moving toward an IPO at a valuation of $26.6 billion or more. The snippet says its OpenAI relationship is deep, but does not disclose ownership, revenue, or timing. The key signal is OpenAI-linked supply-chain valuation, not just AI chips.

Why it matters: Cerebras is moving toward an IPO at a target valuation of at least $26.6 billion, a valuation driven mainly by its supply-chain tie to OpenAI rather than the chip's own technical numbers.