Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

421–440 of 609

May 12Tuesday

r/LocalLLaMA

Prompt caching for RL training: 7.5x speedup on long-prompt, short-response workloads

The author proposes prompt caching for RL training. On Qwen3.5-4B, it reports a 7.5x speedup with 16k-token prompts and 64-token outputs, and the G=8 example with 1000-token prompts and 100-token responses reduces 8800 processed tokens to 1800 unique tokens.

Why it matters: The author moved prompt caching from inference into RL training, reusing repeated token computation for long-prompt short-response tasks, reaching a 7.5x training speedup on Qwen3.5-4B with 16k prompts and 64-token responses.

r/LocalLLaMA

Computer Build Using Intel Optane Persistent Memory Runs a 1T-Parameter Model at Over 4 Tokens/s

Reddit user APFrisco ran the 1T-parameter Kimi K2.5 Q2_K_XL quant locally at about 4 tokens/s using 768GB Intel Optane PMem, 192GB DDR4 ECC DRAM, and a 12GB RTX 3060 with llama.cpp hybrid GPU/CPU inference.

Why it matters: A Reddit build pairs 768GB of Intel Optane persistent memory with an RTX 3060 to run a 1-trillion-parameter model locally at over 4 tokens/sec, showing llama.cpp hybrid inference on cheap hardware.

May 11Monday

r/LocalLLaMA

ExLlamaV3 Major Updates

ExLlamaV3 added DFlash in v0.0.31, raising Coding throughput from 59.21 t/s to 177.67 t/s; v0.0.32 optimized five models, with Trinity-Nano gaining 72.4% on 6000 Pro², while v0.0.33 adds DFlash model quantization plus bug fixes and efficiency work.

Why it matters: ExLlamaV3 v0.0.31 adds DFlash, lifting coding throughput from 59.21 to 177.67 tokens per second, and v0.0.32 tunes five models, with Trinity-Nano up 72.4% on the 6000 Pro².

Synced · WeChat

ICML 2026: PRISM Brings Efficient Test-Time Scaling to dLLMs

PRISM raises LLaDA-8B-Instruct on GSM8K from 67.58% to 85.30% by combining hierarchical trajectory search, partial remasking, and self-verified feedback, reducing dLLM test-time scaling cost from O(NT) toward O(N+KT) under a final candidate width K.

Why it matters: An ICML 2026 paper proposes PRISM, a three-layer inference-time search using parallel generation, local rollback rewriting and self-scored selection, raising LLaDA-8B-Instruct accuracy on GSM8K from 67.58% to 85.30%.

QbitAI · WeChat

OpenAI backs Cerebras as the Nvidia challenger targets a $35B IPO valuation

Cerebras raised its IPO price range to $150-$160 per share, targeting about a $35 billion valuation at the top end, after OpenAI signed a 750-megawatt AI compute purchase agreement with deliveries through 2028.

Why it matters: After signing a 750 MW compute purchase deal with OpenAI, Cerebras raised its IPO price range to $150–160 per share, about $35 billion at the top end, with deliveries running into 2028.

QbitAI · WeChat

SpaceXAI Takes Shape as Elon Musk Files Trademark Applications

SpaceX filed two SpaceXAI trademark applications covering satellite-based data centers, orbital computing, AI SaaS, cloud storage, telecom hardware, and social networking; the post says xAI became a SpaceX subsidiary through an all-stock deal and cites a $250 billion xAI valuation.

Why it matters: SpaceX filed a SpaceXAI trademark covering satellite data centers, orbital computing and AI software services, and xAI has already become its subsidiary through an all-stock deal.

AI HOT (Curated Pool)

Cerebras IPO reportedly over 20 times oversubscribed, with pricing set to rise nearly 30%

Cerebras received more than 20 times oversubscription for its IPO and plans to raise the share count from 28 million to 30 million while increasing the price range to $150-$160.

Why it matters: Cerebras drew over 20x oversubscription for its IPO, plans to raise its price nearly 30% and sell more shares, targeting up to $4.8 billion, a concrete number for AI chip demand.

AI HOT (Curated Pool)

Local models handle half of daily tasks and respond faster than cloud models

A five-week experiment tested about 1,400 daily work tasks, where local 35B models such as Qwen 3.6 35B handled about 50% and averaged 2.8-second responses, 2.1 times faster than Claude Opus 4.5, while the cloud model still led complex reasoning by about 20%.

Why it matters: A five-week test of a local 35B model across about 1,400 daily requests completed half the tasks at 2.8 seconds average, 2.1x faster than cloud Claude Opus 4.5 at 5.8 seconds.

r/LocalLLaMA

MTP benchmark results: task type determines speculative inference speedups or slowdowns

A Reddit LocalLLaMA user ran 300+ tests on Qwen 3.6 27B MTP quants, finding coding draft acceptance at 79-89% and F16 coding speed up 171%, while Q4_K_M creative writing slowed down 9%.

Why it matters: More than 300 runs on a quantized Qwen 3.6 27B show speculative decoding's speedup depends on the task, not the setup.

May 10Sunday

r/LocalLLaMA

I have DeepSeek V4 Pro at home

Reddit user fairydreaming ran DeepSeek V4 Pro Q4_K_M with a modified llama.cpp CUDA repo on one RTX PRO 6000 Blackwell Max-Q workstation GPU, using an 859GB model file; the shared log reports a 1M context window and 8.6 tokens per second generation speed.

Why it matters: A Reddit user ran the 859GB DeepSeek V4 Pro on a single RTX PRO 6000 workstation card with a modified llama.cpp, a concrete reference for local inference hardware limits.

r/LocalLLaMA

NVIDIA AI Releases Star Elastic: One Checkpoint Contains 30B, 23B, and 12B Reasoning Models

NVIDIA AI released Star Elastic, a single checkpoint that can zero-shot slice 30B, 23B, and 12B reasoning models in BF16, FP8, and NVFP4; when the 23B submodel handles thinking and the 30B model handles final answers, reported accuracy rises 16% and latency drops 1.9× on AIME-2025, GPQA, LiveCodeBench v5, and MMLU-Pro.

Why it matters: NVIDIA's Star Elastic slices 30B, 23B and 12B reasoning models from one checkpoint with no extra training, the 23B thinking and the 30B answering.

r/LocalLLaMA

BeeLlama.cpp: DFlash and TurboQuant with reasoning and vision support

Anbeeld released BeeLlama.cpp, a llama.cpp fork that runs Qwen 3.6 27B Q5 with 200k context and vision on a single RTX 3090 or 4090; the title claims 2–3x faster than baseline and a 135 tps peak.

Why it matters: Anbeeld's llama.cpp fork BeeLlama.cpp adds DFlash memory-saving attention and TurboQuant quantization, letting Qwen 3.6 27B run 200k context on a single 3090 at 2 to 3 times baseline speed.

May 9Saturday

AI HOT (Curated Pool)

Redis founder uses a C inference engine to run a large model on a personal computer

Antirez open-sourced ds4, a native inference engine for DeepSeek V4 Flash that uses a few thousand lines of C to run a 1M-context model on a 128GB MacBook Pro at a reported 27 tok/s.

Why it matters: Antirez's ds4 runs DeepSeek V4 Flash's million-token context on a 128GB MacBook Pro in a few thousand lines of C, with 2-bit MoE quantization and KV cache spilled to SSD.

r/LocalLLaMA

80 tok/sec and 128K context on 12GB VRAM with Qwen3.6 35B A3B and llama.cpp MTP

Reddit user janvitos ran Qwen3.6-35B-A3B-MTP-GGUF with a llama.cpp MTP PR on an RTX 4070 Super. The posted benchmark shows 69.2-81.9 tok/s, 0.694-0.947 draft acceptance, 131072 context, and a -fitt 1536 setting that reserves 1536 MB for the draft model and KV cache.

Why it matters: Reddit user janvitos runs Qwen3.6 35B A3B on a 12GB RTX 4070 Super and posts measured settings for 69.2 to 81.9 tok/s at 128K context.

AI HOT (Curated Pool)

Baidu releases ERNIE 5.1 with compressed parameters and training cost

Baidu released ERNIE 5.1 with total parameters reduced to about one third of the original scale, active parameters to about one half, and pretraining cost to about 6% of same-scale models; the model is available on the ERNIE platform and Baidu AI Studio.

Why it matters: Baidu's ERNIE 5.1 cuts total parameters to about a third and pretraining cost to 6% of comparable models, with head-to-head numbers against DeepSeek-V4-Pro and Gemini 3.1 Pro.

Xinzhiyuan · WeChat

NVIDIA, AMD, and Intel Back RadixArk’s $100M Seed Round

RadixArk announced a $100 million seed round on May 5 at a $400 million post-money valuation, led by Accel and co-led by Spark Capital, with participation from NVentures, AMD, MediaTek, Databricks, and other investors tied to AI infrastructure.

Why it matters: RadixArk closed a $100 million seed at a $400 million post-money valuation, with Nvidia, AMD and Intel all on the cap table.

AI HOT (Curated Pool)

DeepSeek Raises $7 Billion at Record Scale, Founder Personally Invests $3 Billion

DeepSeek is raising up to $7 billion at a $50 billion valuation, with founder Liang Wenfeng personally contributing $3 billion, or 40% of the round, while the company says the funding will target large-scale compute, V4.1 model releases, enterprise products, and a path toward positive revenue.

Why it matters: DeepSeek is raising $7 billion at a $50 billion valuation, with founder Liang Wenfeng putting in $3 billion of his own money and keeping 90% of the shares.

r/LocalLLaMA

MTP + TurboQuant Running: Qwen3.6-27B Hits 80+ t/s on a Single RTX 4090

indrasmirror ran Qwen3.6-27B-Heretic-v2 on a single RTX 4090 with 262K context, TBQ4_0 KV cache, and MTP draft 3, improving throughput from about 43 t/s to 80-87 t/s with roughly 73% MTP draft acceptance.

Why it matters: indrasmirror pushed Qwen3.6-27B to 262K context on one RTX 4090, using MTP speculative decoding to lift generation from about 43 t/s to 80-87 t/s at a 73% draft acceptance rate.

The Verge · AI

All the Latest Updates on AI Data Centers

The Verge tracks AI data center disputes with specific updates: 43% of Americans blame data centers for rising power bills, a 40,000-acre Utah project won approval despite local opposition, and Anthropic says it will invest $50 billion in US AI data centers.

Why it matters: The Verge rounds up conflicts over AI data centers worldwide, including a poll where 43% of Americans blame data centers for higher power bills and a 40,000-acre Utah project approved despite community opposition.

Bloomberg Technology

Anthropic Inks $1.8 Billion Computing Deal With Akamai

Anthropic signed a $1.8 billion computing deal with Akamai to meet rising demand for its AI software; the post does not disclose capacity, contract duration, or deployment regions.

Why it matters: Anthropic signed an $1.8 billion computing deal with Akamai to meet growing demand for its AI software, putting Akamai into the role of AI compute supplier.