Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

455 picksRelated topicsOpen source Data & training On-device AI

Latest picks

281–300 of 455

May 10Sunday

r/LocalLLaMAMay 10, 08:48

NVIDIA AI Releases Star Elastic: One Checkpoint Contains 30B, 23B, and 12B Reasoning Models

NVIDIA AI released Star Elastic, a single checkpoint that can zero-shot slice 30B, 23B, and 12B reasoning models in BF16, FP8, and NVFP4; when the 23B submodel handles thinking and the 30B model handles final answers, reported accuracy rises 16% and latency drops 1.9× on AIME-2025, GPQA, LiveCodeBench v5, and MMLU-Pro.

Why it matters: HKR-H/K/R all pass: Star Elastic has a concrete mechanism and testable numbers for inference deployment. Its reach is still narrower than a frontier-model release, so it sits in the high-quality featured band.

r/LocalLLaMAMay 10, 00:05

BeeLlama.cpp: DFlash and TurboQuant with reasoning and vision support

Anbeeld released BeeLlama.cpp, a llama.cpp fork that runs Qwen 3.6 27B Q5 with 200k context and vision on a single RTX 3090 or 4090; the title claims 2–3x faster than baseline and a 135 tps peak.

Why it matters: HKR-H/K/R all pass, but the claims come from a Reddit title and summary without independent reproduction. Treat as a mid-weight open-source inference update, so it lands in the low featured band.

May 9Saturday

AI HOT (Curated Pool)May 9, 22:36

Redis founder uses a C inference engine to run a large model on a personal computer

Antirez open-sourced ds4, a native inference engine for DeepSeek V4 Flash that uses a few thousand lines of C to run a 1M-context model on a 128GB MacBook Pro at a reported 27 tok/s.

Why it matters: HKR-H/K/R all pass: Antirez open-sourced a native C inference engine with hardware, model, context, and speed numbers. Single-source X provenance keeps it below P1, but it is strong open-source inference signal.

r/LocalLLaMAMay 9, 19:57

80 tok/sec and 128K context on 12GB VRAM with Qwen3.6 35B A3B and llama.cpp MTP

Reddit user janvitos ran Qwen3.6-35B-A3B-MTP-GGUF with a llama.cpp MTP PR on an RTX 4070 Super. The posted benchmark shows 69.2-81.9 tok/s, 0.694-0.947 draft acceptance, 131072 context, and a -fitt 1536 setting that reserves 1536 MB for the draft model and KV cache.

Why it matters: HKR-H/K/R all pass with concrete single-user benchmark data and reproducible settings. Source is one Reddit post, so verification is thin; this lands above featured threshold, not in must-write range.

AI HOT (Curated Pool)May 9, 17:58

Baidu releases ERNIE 5.1 with compressed parameters and training cost

Baidu released ERNIE 5.1 with total parameters reduced to about one third of the original scale, active parameters to about one half, and pretraining cost to about 6% of same-scale models; the model is available on the ERNIE platform and Baidu AI Studio.

Why it matters: HKR-H/K/R all pass: Baidu ERNIE 5.1 is a domestic flagship-model release with concrete compression and 6% pretraining-cost claims. That puts it in the must-write band.

Take

Baidu is selling cost discipline, not a clean capability win. ERNIE 5.1 claims total parameters at about one third of the prior scale, active parameters at about one half, and pretraining cost at roughly 6% of same-scale models. If reproducible, that matters more than the vague “near Gemini 3.1 Pro” framing.

The weak spot is the evidence chain. The snippet says ERNIE 5.1 beats DeepSeek-V4-Pro on agent ability, trails only Gemini 3.1 Pro on reasoning, and ranks fourth globally in deep search. It gives no benchmark set, prompting protocol, judge, or eval owner. Baidu’s problem has rarely been a total absence of model quality; it has been developer trust in its scoreboards. Shipping on ERNIE platform and Baidu AI Studio helps, but outside agent-task runs will decide whether this is efficiency engineering or leaderboard packaging.

Xinzhiyuan · WeChatMay 9, 12:05

NVIDIA, AMD, and Intel Back RadixArk’s $100M Seed Round

RadixArk announced a $100 million seed round on May 5 at a $400 million post-money valuation, led by Accel and co-led by Spark Capital, with participation from NVentures, AMD, MediaTek, Databricks, and other investors tied to AI infrastructure.

Why it matters: HKR-H/K/R all pass: a $100M seed round, $400M post-money valuation, and chip/data investors create an AI-infra rivalry angle. It remains a single-company funding item with no product benchmarks or customer data, so it sits near the featured floor.

Take

RadixArk turned an open-source inference stack into hardware politics with a $100 million seed round. The $400 million post-money valuation matters less than the cap table: NVentures, AMD, MediaTek, Databricks, and Intel CEO Lip-Bu Tan as an angel. That reads like every vendor buying an option on SGLang before inference routing hardens around a rival stack.

The concrete hooks are strong: SGLang has 27K+ GitHub stars and the article claims deployment across 400,000 GPUs; Miles also supported DeepSeek-V4 RL training on launch day. I’m not taking the “trillions of tokens” claim at face value, because the source gives no third-party measurement, revenue, hosted product, or enterprise contract detail. vLLM already showed that open-source adoption is only the first inning. The paid game is SLA, managed serving, and ruthless hardware-specific performance work.

AI HOT (Curated Pool)May 9, 08:02

DeepSeek Raises $7 Billion at Record Scale, Founder Personally Invests $3 Billion

DeepSeek is raising up to $7 billion at a $50 billion valuation, with founder Liang Wenfeng personally contributing $3 billion, or 40% of the round, while the company says the funding will target large-scale compute, V4.1 model releases, enterprise products, and a path toward positive revenue.

Why it matters: HKR-H/K/R all pass: the $3B founder check is a strong hook, with concrete funding numbers and clear competitive resonance. Single X-source sourcing leaves lead investor, terms, and confirmation undisclosed, so it stays below P1.

r/LocalLLaMAMay 9, 05:15

MTP + TurboQuant Running: Qwen3.6-27B Hits 80+ t/s on a Single RTX 4090

indrasmirror ran Qwen3.6-27B-Heretic-v2 on a single RTX 4090 with 262K context, TBQ4_0 KV cache, and MTP draft 3, improving throughput from about 43 t/s to 80-87 t/s with roughly 73% MTP draft acceptance.

Why it matters: HKR-H/K/R all pass, backed by a numbered first-person experiment. The Reddit-only source and niche local-inference focus keep it below the 78–84 band for broader industry releases.

Take

I’d down-rank this before celebrating consumer-GPU long-context inference. The hard claim is specific: Qwen3.6-27B-Heretic-v2 on one RTX 4090, 262K context, TBQ4_0 KV cache, MTP draft 3, moving from 43 t/s to 80-87 t/s with about 73% draft acceptance. The body is only a Reddit 403, so there are no commands, quant settings, VRAM numbers, prompt mix, or prefill/decode split.

If the claim holds, the story is MTP plus compressed KV cache making a 27B long-context model usable on hobbyist hardware. It is not a fresh proof that Qwen suddenly jumped. DeepSeek already made the MTP engineering dividend obvious; the fragile part is whether that 73% acceptance survives code tasks, Chinese long-doc prompts, and multi-turn agent loops. Without reproduction, this is an attractive performance receipt, not a benchmark.

The Verge · AIMay 9, 02:45

All the Latest Updates on AI Data Centers

The Verge tracks AI data center disputes with specific updates: 43% of Americans blame data centers for rising power bills, a 40,000-acre Utah project won approval despite local opposition, and Anthropic says it will invest $50 billion in US AI data centers.

Why it matters: HKR-H/K/R all pass, but this is a Verge running roundup rather than a single breakout event. The concrete power-grid and capex numbers place it at the upper end of industry reporting.

Bloomberg TechnologyMay 9, 01:55

Anthropic Inks $1.8 Billion Computing Deal With Akamai

Anthropic signed a $1.8 billion computing deal with Akamai to meet rising demand for its AI software; the post does not disclose capacity, contract duration, or deployment regions.

Why it matters: HKR-H/K/R all pass: the $1.8B number is concrete, the Anthropic-Akamai pairing is fresh, and the story maps to Claude compute pressure. Missing scale, term, and regions keep it just above the featured threshold.

AI HOT (Curated Pool)May 9, 00:03

EMO: Expert Mixture Models for Emergent Modular Pretraining

AllenAI introduced EMO, a mixture-of-experts model with 14B total parameters and 1B active parameters, trained on 1 trillion tokens and able to use only 12.5% of its experts for specific tasks while retaining near-full-model performance.

Why it matters: HKR-H/K/R all pass, but this is an AllenAI/Hugging Face research release rather than a frontier model launch. The 14B/1B and 12.5% expert-activation claims justify the low featured band.

May 8Friday

r/LocalLLaMAMay 8, 22:13

Gemma 4 26B Hits 600 Tok/s on One RTX 5090

chain-77 benchmarked Gemma 4 26B with vLLM 0.19.2rc1, and DFlash raised output throughput on one RTX 5090 from 228 tok/s to 578 tok/s under 256 input tokens, 1024 output tokens, concurrency 1, and num_speculative_tokens=13.

Why it matters: HKR-H/K/R all pass: the single-GPU throughput hook is strong, and the post gives reproducible settings plus before/after speed. Reddit single-post evidence and one hardware setup keep it in the featured-threshold band.

Synced · WeChatMay 8, 17:06

SGLang Team Launches RadixArk With $100M Seed Round

RadixArk announced a $100 million seed round on May 5 at a $400 million post-money valuation, while its SGLang inference project has 27K+ GitHub stars and deployments across 400K+ GPUs.

Why it matters: HKR-H/K/R all pass: the round size, valuation, and deployment numbers are concrete, and SGLang is a known inference stack. It is still a startup funding and infra-roadmap story, not a major model release, so it stays in the 78–84 featured band.

Take

RadixArk’s strongest asset is not the $100M seed round; it is SGLang already sitting inside 400K+ GPUs. Infra startups usually burn time teaching the market. SGLang starts with 27K+ GitHub stars, claimed trillions of daily tokens, and day-zero DeepSeek-V4 inference plus RL support. That is stronger evidence than the cap table.

I don’t buy the “AI infrastructure as a public good” framing yet. Once an open-source inference engine gets tied to a hosted platform, neutrality becomes a product problem. Does SGLang stay vendor-neutral, or does RadixArk Cloud get the fast path? vLLM is fighting for the same open inference slot, and NVIDIA Triton still owns a lot of default hardware gravity. The article gives a platform signup and free credits, but no pricing, SLA, or enterprise deployment terms; that is the gap behind the $400M post-money valuation.

Xinzhiyuan · WeChatMay 8, 12:12

Token-Level Length Control: 3B Model Beats GPT 5.4 and Claude

UC Santa Barbara and Apple researchers introduced LenVM, which models remaining generation length as a token-level value function; Qwen2.5-3B with a 1.5B LenVM scored 62.6 on LIFEBench length control, above GPT-5.4 at 37.4 and Claude-Opus-4-6 at 35.5.

Why it matters: HKR-H/K/R all pass: the headline has a sharp small-model-vs-frontier hook, and the post gives LenVM's mechanism plus 62.6/37.4 benchmark numbers. The topic is narrow research, not a model or major product release, so it fits the 78-84 band.

Take

LenVM’s sharp edge is that it makes “write shorter” a decoding-time control problem, not another prompt-compliance hope. Qwen2.5-3B plus a 1.5B LenVM scores 62.6 on LIFEBench length control, ahead of GPT-5.4 at 37.4 and Claude-Opus-4-6 at 35.5. On GSM8K with a 200-token budget, Pass@1 rises from about 6% under hard truncation to about 63% with LenVM-guided decoding.

I would be careful with the “10x” headline, because the baseline is hard truncation, not a tuned compression decoder or distilled short-reasoning model. The mechanism still looks clean: assign each token a fixed negative reward, then predict discounted remaining length as a value function. For agent cost control, batching, and KV-cache planning, this token-level length signal is more usable than another long-CoT leaderboard jump.

QbitAI · WeChatMay 8, 12:05

HIT and Huawei propose Dynamic-dLLM, a training-free acceleration framework with 4.48x speedup

HIT Shenzhen, Huawei, and Shenzhen Hetao College proposed Dynamic-dLLM, a training-free dLLM acceleration framework that raises LLaDA-8B-Instruct throughput on GSM8k from 8.32 TPS to 37.29 TPS with almost no accuracy loss.

Why it matters: HKR-H/K/R all pass: the 4.48x speedup is clickable, and GSM8k TPS figures add concrete substance. It is inference-optimization research, not a mainstream model launch, so it fits the 78–84 band.

Financial Times · TechnologyMay 8, 12:00

Big Tech’s $725bn AI Spending Spree Sends Free Cash Flow to a Decade Low

Big Tech is spending $725 billion on AI infrastructure, and the title says free cash flow has fallen to a decade low; the RSS snippet says Silicon Valley giants shifted from asset-light cash generators to infrastructure investors, but it does not disclose the company list, time period, or accounting basis.

Why it matters: HKR-H/K/R all pass: the FT angle ties $725bn in AI infrastructure spending to decade-low free cash flow, hitting cost and ROI anxiety. Missing company list, period, and accounting scope keeps it in the 78–84 band.

r/LocalLLaMAMay 8, 04:56

11.67% ARC-AGI-2 Local Eval on a Single 4090: The TOPAS Recursive Architecture

Doug_Bitterbot says TOPAS scored 11.67% on ARC-AGI-2 using one RTX 4090 after about 14 days of training. The 100M-parameter checkpoint hit 36% locally, but recursive TTT caused null outputs on nearly half of Kaggle puzzles. The key detail is time management: the author expects 20% after threshold tuning and 3-5 more weeks of training.

Why it matters: HKR-H/K/R all pass, but this is a single Reddit post with unstable Kaggle submissions. It clears featured, not the higher research-release band.

Bloomberg TechnologyMay 8, 04:50

Nvidia to Invest Up to $2.1 Billion in Data Center Firm IREN

Nvidia will invest up to $2.1 billion in IREN under an AI infrastructure partnership. The post discloses the cap and goal, but not equity terms, payment timing, or data center capacity.

Why it matters: Bloomberg source plus Nvidia’s up-to-$2.1B investment clears HKR-H/K/R. Details stop at amount and partnership direction, with no stake, payment schedule, or capacity, so it stays at the featured threshold.

Take

Nvidia’s planned IREN investment, capped at $2.1 billion, is about locking physical bottlenecks before GPU demand hits the wall. The constraint for AI clusters is no longer just H100 or B-series allocation; it is power, land, cooling, interconnect, and deliverable racks. IREN’s crypto-mining roots matter because miners already know the ugly parts of power procurement.

The disclosure is too thin to price the impact. Bloomberg gives the cap and the AI infrastructure partnership, but no equity stake, payment schedule, megawatt capacity, GPU type, or delivery date. Compared with CoreWeave-style structures that tie GPUs, debt, cloud contracts, and customer demand together, this reads more like Nvidia pre-positioning its supply chain. Without terms, $2.1 billion is a ceiling-shaped headline, not usable capacity.

The Verge · AIMay 8, 03:26

SpaceX Has a $55 Billion Plan to Build AI Chips in Texas

SpaceX plans to invest at least $55 billion in its Terafab chip plant in Austin, Texas. A hearing notice says later phases could lift total investment to $119 billion. Musk said in March the target was chips for 200GW of compute per year; the post does not disclose process nodes.

Why it matters: HKR-H/K/R all pass on the SpaceX chip-plant hook, hard capex numbers, and compute-supply resonance. Not P1 because process node, timeline, and committed customers are not disclosed.

Take

SpaceX plans to put at least $55 billion into its Austin Terafab. My read is not “a new AI chipmaker has arrived.” It is Musk turning compute scarcity into capex theater. The disclosed numbers are huge: $55 billion upfront, $119 billion if later phases happen, and a March target of chips supporting 200GW of compute per year. The missing parts are more important: no process node, no packaging stack, no yield assumptions, no tool list, no HBM supplier, and no clarity on whether SpaceX means front-end wafer manufacturing or system-level assembly.

That distinction matters. A $55 billion number sounds like a national semiconductor program, not a normal corporate factory. TSMC’s Arizona expansion has been discussed around the $65 billion range. Intel’s Ohio plan started around $20 billion before delays and repricing. SpaceX is already talking above those levels, with a possible $119 billion ceiling. But the article only says “AI chips.” It does not say 3nm, 2nm, CoWoS, silicon interposers, or EUV. One High-NA EUV scanner costs above $300 million, and even regular EUV tools sit in the nine-figure range. Money buys a ticket into the room. It does not buy process integration, equipment lead times, materials discipline, or yield learning.

The 200GW claim is the part that makes me pause. That is a power and infrastructure number, not a clean chip manufacturing metric. A GB200 NVL72 rack sits roughly around the 100kW class. On that crude scale, 200GW maps to millions of high-end AI racks worth of power envelope. That is not only a chip question. It is substations, cooling, land, transmission, water, permits, and long-cycle utility planning. The article gives no reproducible definition. Is 200GW peak supported load? Annual added compute capacity? Internal demand across xAI, Tesla, and SpaceX? A future merchant supply business? Those answers produce very different business models.

The motive is easy to understand. xAI’s Colossus cluster has already been discussed at the hundred-thousand-GPU scale, with public ambition to expand further. Tesla tried Dojo, but Dojo has not displaced Nvidia in any visible way. Public procurement signals still keep Tesla and xAI tied to H100, H200, and now Blackwell allocation. Musk’s companies span rockets, cars, robots, satellite internet, social distribution, and frontier models. All of those lines increasingly consume inference. If Nvidia, TSMC, SK Hynix, and advanced packaging capacity control the calendar, Musk has a direct incentive to build bargaining leverage.

I do not buy the simple version where SpaceX suddenly becomes an advanced-node foundry. SpaceX is excellent at hard systems engineering, vertical integration, and fast iteration. Leading-edge semiconductor manufacturing is a different animal. Falcon 9 can learn through spectacular failures. A 2nm fab cannot run that playbook on yield. TSMC’s advantage is not one machine or one building. It is decades of process integration, customer trust, PDK maturity, packaging coordination, and operational discipline. Intel has spent years fighting its way back on EUV-era nodes. Samsung still fights yield and customer-confidence issues. SpaceX does not get to skip that queue because it is good at rockets.

The more plausible version is narrower and still significant. Terafab may focus on systems manufacturing, advanced packaging adjacency, boards, racks, power delivery, liquid cooling, or a back-end line for inference-optimized ASICs. In that model, SpaceX aggregates demand from xAI, Tesla, Starlink, and internal compute needs. It then uses Texas as the site for large-scale module and datacenter hardware production while locking wafers and memory upstream. That is far more believable than “SpaceX challenges TSMC.” The article does not disclose the node, so I would not read this as a clean foundry declaration.

There is also a tax-politics layer. The concrete source here is a Grimes County public hearing notice tied to tax breaks. Musk knows how to use giant investment figures to pressure local governments. $55 billion, $119 billion, and 200GW are numbers that land harder with Texas officials than with semiconductor engineers. The US has a CHIPS Act-era appetite for domestic chip projects, but SpaceX is not a proven fab operator. Winning incentives is one task. Delivering advanced AI silicon at scale is another.

So I would file Terafab under AI compute vertical integration, not under “new foundry competitor.” OpenAI has explored custom chips and giant datacenter projects. Google has TPU. Amazon has Trainium. Meta has MTIA. Tesla has Dojo. Musk’s version is louder and larger, with a stated ceiling that forces attention. The story still lacks the facts practitioners need: node, packaging, memory, customers, volume schedule, and first silicon timing. Believe the ambition. Do not believe the manufacturing story until those details show up.

AI HOT (Curated Pool)May 8, 03:22

Readable behavioral signals remain in frozen LLM hidden states, Cygnus boosts accuracy

Proprioceptive AI says Cygnus adds adapters to frozen LLMs and raises Qwen-32B on ARC-Challenge from 82.2% to 94.97%. It projects hidden states into a gl(4,R) Lie-algebra space to isolate “dark modes.” Watch replication; the post does not disclose full eval sets or controls.

Why it matters: HKR-H/K/R pass: the claim is novel, quantified, and practitioner-relevant. Kept at low featured because the source is an X post and full eval set, training details, and controls are not disclosed.

Take

Cygnus should not be converted into a capability story yet. A 12.77-point ARC-Challenge gain on frozen Qwen-32B is loud enough to demand replication first. The mechanism is specific: adapters project hidden states into a gl(4,R) Lie-algebra space and extract “dark modes.” If it holds, this is closer to test-time state correction than ordinary LoRA.

The eval boundary is the problem. The post gives one RTX 3090, 82.2% to 94.97%, coverage from 3B to 405B models, and 50,000 concurrent users. It does not give the split, prompt format, seed handling, or whether ARC validation was touched. ARC-style benchmarks have been over-optimized by reasoning wrappers for a year. Without an external rerun, this smells like a sharp interpretability-to-performance demo with a very fragile headline number.