Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

455 picksRelated topicsOpen source Data & training On-device AI

Latest picks

221–240 of 455

May 18Monday

Synced · WeChatMay 18, 12:47

ICML 2026: Huawei GTS proposes EDCO for dynamic curriculum fine-tuning

Huawei GTS proposed EDCO, a dynamic curriculum method that selects fine-tuning samples by inference entropy; prefix entropy estimation cuts per-sample scoring time from 2.24 seconds to 0.37 seconds.

Why it matters: HKR-H/K/R pass: the story has a lab-race hook, a concrete entropy-based mechanism, and a 2.24s→0.37s efficiency claim. It stays below 78 because it is still a training-method paper, not a major model or product release.

Take

EDCO’s useful move is not the “difficulty-adaptive training” label. It plugs sample selection into the fine-tuning loop and makes the scoring cost survivable. The concrete hook is strong: full-sequence entropy costs 2.24 seconds per sample, prefix entropy drops it to 0.37 seconds, and 8-GPU parallel scoring reaches 0.04 seconds. On Datacom RLFT, accuracy moves from 40.43% with random sampling to 46.96%.

I don’t buy the “new paradigm” framing yet. Active learning, curriculum learning, and hard-example mining have been circling this idea for years. The LLM-specific question is whether inference entropy stays a reliable proxy for useful gradient signal. The paper tests Qwen3-4B, Llama3.2-3B, and three vertical domains; it does not settle larger models, longer training, or production distribution drift.

r/LocalLLaMAMay 18, 06:57

Benchmarking vLLM vs SGLang vs llama.cpp on a mixed Blackwell/Ada cluster

The author benchmarked long-context prefill on a 7-GPU mixed Blackwell/Ada cluster; on Qwen3.5-397B-A17B with 75k tokens, vLLM reached 9.8s TTFT and 7,683 t/s, while llama.cpp took 57.2s and 1,319 t/s.

Why it matters: Single-source Reddit benchmark, so source authority keeps it near the threshold. HKR-H/K/R pass on the mixed 7-GPU setup, 397B at 75k tokens, and concrete TTFT/throughput numbers.

Take

vLLM exposes the ugly truth of local multi-GPU inference: heterogeneous rigs work only if the engine handles the pipeline sanely. On Qwen3.5-397B-A17B with 75k tokens across seven mixed Blackwell/Ada GPUs, vLLM hits 9.8s TTFT and 7,683 t/s. llama.cpp lands at 57.2s and 1,319 t/s, roughly a 6x gap.

The useful detail is not “vLLM is faster.” It is manual layer placement via VLLM_PP_LAYER_PARTITION, which balances fast Blackwell cards against slower 4090s doing FP4 emulation. SGLang looks fine on pure Blackwell, with 5.3s versus vLLM’s 5.0s on Qwen3.5-122B, then crashes when Ada enters because FP4 lacks a software fallback. Single Reddit benchmark, single topology, no independent replication; still, anyone stitching together used 4090s for 397B-class models should take the warning seriously.

r/LocalLLaMAMay 18, 06:22

LLMs on Android: Snapdragon 8 Elite MoE Experience

A Reddit user tested MoE LLMs on an Honor Magic 7 Pro with Snapdragon 8 Elite and 24GB RAM; under Q4 quantization, LFM2-24b-a2b reached about 24 tokens/s while Gemma reached about 11 tokens/s, and CPU inference was still faster than NPU or GPU in the reported setup.

Why it matters: HKR-H/K/R all pass: a named Reddit test gives hardware, quantization, and token/s figures. Single-device anecdote and weak source authority keep it at the low featured band.

May 17Sunday

QbitAI · WeChatMay 17, 18:22

A Robot Dog Challenges Nvidia's Compute Lead

Weilan Technology unveiled BabyAlpha A3, a consumer quadruped robot using a six-chip heterogeneous cluster that runs a 7B-parameter model on-device at 280 TPS; the article says it has 66MP vision, 2.232 million point-cloud samples per second, and a planned Q3 launch.

Why it matters: HKR-H/K/R pass: the robot-dog-versus-Nvidia angle is clickable, and 280 TPS on a local 7B model is concrete. Single-source summary lacks price, power draw, and benchmark setup, so it stays near the featured floor.

Take

Weilan is selling BabyAlpha A3 as an Nvidia bypass, and I don’t buy the crown-taking part yet. A six-chip heterogeneous cluster running a 7B model at 280 TPS is impressive, but the article gives no wattage, quantization, token quality, or end-to-end latency. A robot is not a chat box; perception, planning, and control fight for the same thermal and scheduling budget.

The sturdier evidence is the sensor stack: 66MP vision, HDR 140db, 2.232 million point-cloud samples per second, and a 12-mic array. That says more than the “compute throne” headline. The claimed cost gap also matters: roughly $300 for the domestic setup versus about $3,000 for Nvidia Jetson Thor T5000. But A3’s Q3 launch price is missing, and without system power plus real home-navigation clips, this is a strong product claim, not a dethroning.

r/LocalLLaMAMay 17, 07:57

Same Models Tested Across Strix Halo, RTX 3090, and RTX 5070

C_Coffie published 55 local inference benchmark runs across Strix Halo, RTX 3090, RTX 5070, five backends, and 0.35B to 35B-A3B models; RTX 5070 beats RTX 3090 on models fitting 12GiB, while RTX 3090 leads in the 14–31B band that exceeds 12GiB but fits 24GiB.

Why it matters: Hits HKR-H/K/R with a named first-person benchmark: 55 runs and concrete GPU crossover points. Source is a single Reddit post, so it stays in the low featured band.

Dwarkesh Patel podcastMay 17, 03:01

Notes on Pretraining Parallelisms and Failed Training Runs

Dwarkesh documents pretraining failure modes and parallelism tradeoffs: expert choice and token dropping can break causality in MoE routing, FP16 collectives can bias repeated additions after values exceed 1024, pretraining FLOPs are given as 6ND, B300 HBM is listed as 288GB, and FSDP communication can reach params × 3 with reduce-scatter.

Why it matters: HKR-H/K/R all pass: Dwarkesh’s notes expose concrete pretraining failure modes and numbers. The systems-training focus is specialized, so it sits in the high-quality band rather than same-day must-write.

May 16Saturday

Latent SpaceMay 16, 12:36

Cerebras' $60B IPO: Slowly, then All at Once

Cerebras closed its IPO at a $60 billion market cap. CFO Bob Komin said it serves trillion-parameter models, including internal OpenAI 5.4 and 5.5 workloads, but the post does not disclose traffic share, latency tier, or cost per token.

Why it matters: HKR-H/K/R all pass: a $60B Cerebras IPO is same-day material, and Bob Komin’s trillion-parameter/OpenAI 5.4–5.5 claim gives the story concrete compute-roadmap stakes.

Take

Cerebras at a $60B market cap is a call option on non-NVIDIA inference capacity, not a sudden mass conversion to wafer-scale purity. CFO Bob Komin said Cerebras is serving trillion-parameter models and named internal OpenAI 5.4 and 5.5 workloads; that is unusually specific for a hardware IPO narrative. The missing parts are the hard ones: traffic share, latency tier, cost per token, and whether those jobs are primary production paths or overflow capacity.

After NVIDIA bought Groq for $20B, the market already repriced low-latency inference supply. Cerebras is now selling itself as an independent shelf for frontier serving, and investors paid up because OpenAI’s name makes the story legible. I don’t buy the “no limit” phrasing. Engineers know every limit just moves into routing, sharding, memory bandwidth, and the bill.

r/LocalLLaMAMay 16, 03:07

Orthrus-Qwen3-8B: Up to 7.8× tokens/forward on Qwen3-8B with frozen backbone

Orthrus-Qwen3-8B adds a trainable diffusion attention head to a frozen Qwen3-8B backbone, reaches up to 7.8× tokens per forward and about 6× wall-clock speedup on MATH-500, while training 16% of parameters with under 1B tokens over 24 hours on 8×H200 GPUs.

Why it matters: HKR-H/K/R all pass: 7.8× speed is a strong hook, the post gives parameter, hardware, and benchmark conditions, and inference cost is a live practitioner pain. Single-source Reddit research keeps it in the lower good-quality band.

Take

Orthrus-Qwen3-8B attacks the ugly bottleneck directly: freeze Qwen3-8B, train 16% of parameters, then claim 7.8× tokens per forward and roughly 6× wall-clock speedup on MATH-500. That is a different bet from quantization, KV-cache tuning, or speculative decoding, because it changes how many tokens one forward pass can legally emit.

I do not buy “provably identical output distribution” on the headline alone. The summary gives 8×H200, 24 hours, under 1B training tokens, and a diffusion attention head; Reddit body is 403, so the proof assumptions, sampling settings, temperature, rejection behavior, and non-MATH regressions are hidden. Qwen3-8B is a strong base, and MATH-500 is structured enough to reward clever decoding. Code plus GSM8K, HumanEval, and long-context evals will decide whether this is an inference primitive or a benchmark dessert.

Bloomberg TechnologyMay 16, 01:07

Cerebras CEO Is Worth $3.2 Billion After Year’s Largest IPO

Cerebras Systems rose about 68% on Nasdaq after the year’s largest IPO, giving the company a market value of roughly $67 billion; the headline says its CEO is worth $3.2 billion after the listing.

Why it matters: HKR-H/K/R all pass: Cerebras' IPO gives AI compute a public-market price via a 68% jump and about $67B market cap. The Bloomberg item is wealth-video framed, so it lands in must-write range, not industry-shaking.

Take

Cerebras at a $67B market value is an aggressive public-market bid for non-Nvidia compute exposure. The snippet gives a 68% first-day jump, the year’s largest IPO, and a $3.2B CEO stake. It gives no revenue, gross margin, customer concentration, or inference pricing. That missing data matters more than the pop.

Cerebras has a real differentiation story with wafer-scale silicon and high-throughput inference. The hard part is that AI infrastructure is not a single-chip benchmark market. Nvidia’s moat sits in CUDA, networking, supply allocation, cloud procurement habits, and financing terms. If Cerebras cannot show $/token, uptime, and framework compatibility under production loads, $67B smells like a public-market premium for “anti-Nvidia beta,” not proof of platform escape velocity.

May 15Friday

r/LocalLLaMAMay 15, 23:09

Fully Offline Suitcase Robot Built Around Jetson Orin NX SUPER 16GB

CreativelyBankrupt built Sparky as a fully offline suitcase robot on Jetson Orin NX SUPER 16GB, running Gemma 4 E4B Q4_K_M via llama.cpp with q8_0 KV cache, about 200 ms cached TTFT, 14-15 tok/s sustained output, 12K context, 30+ sensors, and no WiFi, Bluetooth, or cellular interface.

Why it matters: HKR-H/K/R all pass, with a named hands-on build and concrete latency/sensor numbers. It stays in low featured because this is a Reddit project post, not a product launch or research release.

AI HOT (Curated Pool)May 15, 21:34

X open-sources the “For You” feed recommendation algorithm

X open-sourced the For You recommendation pipeline on GitHub, using a Grok-based Phoenix Transformer to score candidate posts and predict engagement probabilities such as likes, replies, and reposts.

Why it matters: HKR-H/K/R all pass, but the item only gives the open-source claim and Phoenix Transformer ranking mechanism; repo details, license, and reproducible tests are not disclosed, so it stays low-featured.

Take

X chose the boundary well: code, pretrained model, content understanding services, and ad mixing are public, while production power stays in live data and experiments. The For You stack covers candidate retrieval, context enrichment, Phoenix Transformer scoring, diversity adjustment, and spam filtering. It predicts likes, replies, and reposts, which is a thicker release than Twitter’s earlier partial ranking-code drop.

I don’t buy the “algorithmic transparency” framing. The sensitive parts of a recommender are not the Transformer skeleton. They are feature freshness, negative-feedback weights, safety and political policy knobs, ad auction hooks, and A/B bucketing. The snippet says the pipeline is runnable, but gives no online feature distributions, experiment configs, or policy thresholds. Developers can reproduce the shape; they cannot reproduce X’s feed.

r/LocalLLaMAMay 15, 20:24

Evaluated a RAG Chatbot: The Most Expensive Model Was the Worst Performer

The author evaluated a customer-support RAG bot and raised the quality score from 6.62 to 7.88 while cutting per-session cost from $0.002420 to $0.000509, using retrieval logging, LLM-as-judge scoring, chunk deduplication, stricter grounding, and a five-model sweep.

Why it matters: HKR-H/K/R all pass: counterintuitive model ranking, concrete quality and cost deltas, and direct RAG production relevance. Reddit source authority keeps it near the featured floor despite the first-person experiment signal.

Take

This reads like a slap at model-tier cargo culting: support RAG should spend first on retrieval observability and grounding, not the priciest model. The summary gives unusually concrete numbers: quality rose from 6.62 to 7.88, while per-session cost fell from $0.002420 to $0.000509, about a 79% cut. The levers were retrieval logging, LLM-as-judge scoring, chunk deduplication, stricter grounding, and a five-model sweep.

I’d treat this as an engineering-hygiene win, not a clean model leaderboard. The Reddit body is blocked by 403, so I can’t verify the judge rubric, sample size, model list, or how many support edge cases were covered. Without that, 7.88 is an internal score, not a portable benchmark. Still, the pattern matches most production RAG work: fix the evidence path before paying premium-token rent.

r/LocalLLaMAMay 15, 14:20

Used over a million tokens in three sessions to test Qwen 3.6 35B MTP

A Reddit user tested Qwen3.6-35B-A3B MTP across three million-token-scale sessions, using 300k context and KV Q8_0, and reported about 1.5x the tok/sec of earlier tests.

Why it matters: HKR-H/K/R all pass: the million-token test is clickable, 300k context and KV Q8_0 add testable detail, and local speed maps to cost. Source is one Reddit post, so it stays below the high-importance band.

Bloomberg TechnologyMay 15, 10:39

OpenAI May Raise More Money as Compute Crunch Deepens, CFO Says

OpenAI CFO Sarah Friar said the company may raise more capital after completing what she described as the largest private fundraising round ever, as OpenAI seeks computing power to meet rising AI demand; the RSS snippet does not disclose the round size, target amount, or timeline.

Why it matters: HKR-H/K/R pass: a named OpenAI CFO links more fundraising to the compute crunch. The score stays in the lower featured band because this is not a closed round and amount, investors, and timing are not disclosed.

AI HOT (Curated Pool)May 15, 10:27

Microsoft Has Invested Over $100 Billion in OpenAI, Nadella Says No One Wanted to Bet Then

Microsoft has invested more than $100 billion in OpenAI, including a $13 billion original investment and Azure infrastructure costs, while the partnership has generated about $30 billion in revenue; the renewed non-exclusive agreement caps OpenAI’s revenue share at $38 billion cumulatively through 2030.

Why it matters: HKR-H/K/R all pass: this is not a product launch, but the Microsoft-OpenAI economics include three concrete figures on spend, revenue, and revenue-share caps, placing it in the 78–84 quality band.

AI HOT (Curated Pool)May 15, 08:00

The First Derivative of Inference: Growth Logic in the AI Wave

Tom Tunguz says the AI inference market will reach $250 billion within seven years; Datadog’s LLM observability data volume nearly doubled in the latest quarter, and about 20% of its AI customers contribute roughly 80% of ARR.

Why it matters: HKR-H/K/R all pass: Tom Tunguz ties inference growth to Datadog volume and ARR concentration data. It stays in the 72–77 band because this is commentary, not a model, product, or protocol release.

Take

Tunguz’s strongest point is not the $250 billion inference TAM. It is Datadog’s revenue mix already tilting around AI workloads. In Q1 2026, Datadog said LLM Observability spans nearly tripled quarter over quarter, and 6,500-plus AI integration customers represent only 20% of customers but about 80% of ARR. That is not a nice attach-rate story; it is customer concentration arriving through AI usage.

I buy the “first derivative of inference” frame, but I don’t buy the claim that every pre-AI SaaS company has one escape route. Twilio’s AI voice and Datadog’s LLM tracing sit on runtime pain: latency, failures, cost, compliance, customer calls. Most SaaS vendors cannot slap token resale onto workflow software and get the same economics. Anthropic booking $9b and $10b in consecutive months sounds massive, but inference margins get competed down; observability and communications take the early toll.

AI HOT (Curated Pool)May 15, 07:35

API prompt precaching speeds up first-token generation

Claude API prewarms prompt cache with the system prompt, skips output, then hits cache on the real request.

Why it matters: HKR-H/K/R all pass: this is a concrete Claude API latency mechanism, not a vague product tease. It clears featured, but it is a mid-weight inference update rather than a major model or capability release.

Bloomberg TechnologyMay 15, 04:58

Cerebras Goes Public in Year's Biggest IPO | Bloomberg Tech 5/14/2026

Bloomberg Tech says Cerebras is going public in the year’s biggest IPO, while the RSS snippet does not disclose the fundraising amount, valuation, offer price, or exchange.

Why it matters: HKR-H/K/R all pass, but the item only gives a title-level Bloomberg claim with no proceeds, valuation, price, or exchange. Cerebras’ IPO matters for AI infrastructure markets, so it lands in the 78–84 band.

Take

Cerebras going public matters because public investors now have to price a non-Nvidia AI chip story. Bloomberg’s title says this is the year’s biggest IPO, but the article body is only a video shell. It gives no proceeds, valuation, offer price, exchange, revenue, customer concentration, or gross margin.

I’m skeptical of the victory framing. Cerebras has a differentiated wafer-scale architecture; WSE is not another GPU clone. But public buyers price revenue visibility, software attach, supply risk, and repeat workloads. Groq and SambaNova both sold fast-inference narratives, then ran into the same boring constraints: software, procurement, and utilization. Until the S-1 numbers are visible, this is less proof of Cerebras’ breakout and more a live market test for AI hardware multiples outside Nvidia’s gravity.

Bloomberg TechnologyMay 15, 04:12

AI Chipmaker Cerebras Climbs 68% After Year’s Biggest IPO

Cerebras Systems shares closed up 68% in their trading debut after raising $5.5 billion in the year’s largest IPO, ending Thursday at $311.07 in New York versus the $185 IPO price.

Why it matters: HKR-H/K/R all pass: Cerebras’ IPO gives AI infrastructure a public-market pricing signal, with a 68% debut gain and $5.5B raised. The chip-supply and NVIDIA-competition angle pushes it into p1.

Bloomberg TechnologyMay 15, 03:17

Cerebras CEO Is Worth $3.2 Billion After Year’s Largest IPO

Cerebras’ IPO valued CEO Andrew Feldman’s fortune at $3.2 billion, and the title identifies the listing as the year’s largest IPO; the RSS snippet only says Feldman previously sold three companies and took another public, and does not disclose the IPO price, proceeds, valuation, or share count.

Why it matters: HKR-H/K/R all pass: Cerebras’ “year’s largest IPO” is a real AI-infra capital-market signal. Score stays in 78–84 because the feed gives $3.2B net worth but not offer price, proceeds, or valuation.