Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

381–400 of 609

May 15Friday

AI HOT (Curated Pool)

X open-sources the “For You” feed recommendation algorithm

X open-sourced the For You recommendation pipeline on GitHub, using a Grok-based Phoenix Transformer to score candidate posts and predict engagement probabilities such as likes, replies, and reposts.

Why it matters: X published the recommendation pipeline behind its For You feed on GitHub, including code, pretrained models and content-understanding services, with a Grok-based Phoenix model scoring candidate posts.

r/LocalLLaMA

Evaluated a RAG Chatbot: The Most Expensive Model Was the Worst Performer

The author evaluated a customer-support RAG bot and raised the quality score from 6.62 to 7.88 while cutting per-session cost from $0.002420 to $0.000509, using retrieval logging, LLM-as-judge scoring, chunk deduplication, stricter grounding, and a five-model sweep.

Why it matters: A builder used a model as judge to tune a customer-service RAG bot, lifting quality from 6.62 to 7.88 and cutting cost per session by nearly 80%, while the priciest model performed worst.

r/LocalLLaMA

Used over a million tokens in three sessions to test Qwen 3.6 35B MTP

A Reddit user tested Qwen3.6-35B-A3B MTP across three million-token-scale sessions, using 300k context and KV Q8_0, and reported about 1.5x the tok/sec of earlier tests.

Why it matters: A Reddit user ran over a million tokens across three sessions to test the MTP build of Qwen3.6-35B-A3B at 300k context and KV Q8_0 quantization, measuring roughly 1.5x the generation speed of the earlier version.

Bloomberg Technology

OpenAI May Raise More Money as Compute Crunch Deepens, CFO Says

OpenAI CFO Sarah Friar said the company may raise more capital after completing what she described as the largest private fundraising round ever, as OpenAI seeks computing power to meet rising AI demand; the RSS snippet does not disclose the round size, target amount, or timeline.

Why it matters: OpenAI CFO Sarah Friar says the company may raise more money after its latest private round, because compute supply cannot keep up with AI demand growth.

AI HOT (Curated Pool)

Microsoft Has Invested Over $100 Billion in OpenAI, Nadella Says No One Wanted to Bet Then

Microsoft has invested more than $100 billion in OpenAI, including a $13 billion original investment and Azure infrastructure costs, while the partnership has generated about $30 billion in revenue; the renewed non-exclusive agreement caps OpenAI’s revenue share at $38 billion cumulatively through 2030.

Why it matters: Microsoft's corporate development lead confirmed in court that total OpenAI investment tops $100 billion, with $13 billion direct and most of the rest Azure infrastructure and hosting costs, largely spent before OpenAI began sharing revenue.

AI HOT (Curated Pool)

The First Derivative of Inference: Growth Logic in the AI Wave

Tom Tunguz says the AI inference market will reach $250 billion within seven years; Datadog’s LLM observability data volume nearly doubled in the latest quarter, and about 20% of its AI customers contribute roughly 80% of ARR.

Why it matters: Tom Tunguz floated the idea of a first derivative of inference: companies like Datadog that sell no compute but earn more as customer AI usage grows, with LLM monitoring data nearly doubling in a quarter.

AI HOT (Curated Pool)

API prompt precaching speeds up first-token generation

Claude API prewarms prompt cache with the system prompt, skips output, then hits cache on the real request.

Why it matters: The Claude API can write a system prompt into the cache on its own before the real request arrives and skip output, so later requests hit the cache and cut first-token latency on long prompts.

Bloomberg Technology

Cerebras Goes Public in Year's Biggest IPO | Bloomberg Tech 5/14/2026

Bloomberg Tech says Cerebras is going public in the year’s biggest IPO, while the RSS snippet does not disclose the fundraising amount, valuation, offer price, or exchange.

Why it matters: AI chip company Cerebras has gone public in the year's largest IPO, though the raise, valuation, offer price and exchange were not disclosed.

Bloomberg Technology

AI Chipmaker Cerebras Climbs 68% After Year’s Biggest IPO

Cerebras Systems shares closed up 68% in their trading debut after raising $5.5 billion in the year’s largest IPO, ending Thursday at $311.07 in New York versus the $185 IPO price.

Why it matters: Cerebras closed its first day of trading in New York up 68% after raising $5.5 billion, giving a concrete reference for public-market demand for AI chips outside Nvidia's path.

Bloomberg Technology

Cerebras CEO Is Worth $3.2 Billion After Year’s Largest IPO

Cerebras’ IPO valued CEO Andrew Feldman’s fortune at $3.2 billion, and the title identifies the listing as the year’s largest IPO; the RSS snippet only says Feldman previously sold three companies and took another public, and does not disclose the IPO price, proceeds, valuation, or share count.

Why it matters: Cerebras completed one of the year's largest IPOs, leaving CEO Andrew Feldman's stake worth $3.2 billion; he had already sold three companies and taken one public.

r/LocalLLaMA

I tracked EU GPU prices across 15 stores for 50+ days: RTX 5090 is the only card not dropping

Reddit user egudegi tracked EU GPU prices across 15 stores for more than 50 days with a 6-hour scrape cadence and about 126,000 readings; RTX 5090 average pricing rose from €3,392 to €3,487, a 3.0% increase.

Why it matters: A Reddit user scripted GPU price tracking across 15 EU stores for over 50 days, and among roughly 126,000 data points the RTX 5090 rose from 3,392 to 3,487 euros while every other model fell.

r/LocalLLaMA

The RTX 5000 PRO 48GB arrived and is better than expected

A Reddit user built a $5,600 RTX 5000 PRO 48GB PC and ran Qwen3.6-27B-FP8 with full-precision cache; they report up to 80 tok/s in TG, about 50–60 tok/s on very large prompts, 4,400 tok/s in prompt processing, and 200k tokens fitting in BF16 KV cache.

Why it matters: A builder with no prior experience spent $5,600 on a single RTX 5000 PRO 48GB machine, hitting 80 tok/s on a 27B model with 200k context at full precision.

TechCrunch · AI

Cerebras raises $5.5B, then stock pops 108%, in the first huge tech IPO of 2026

The title says Cerebras raised $5.5 billion and its stock rose 108% after the first major tech IPO of 2026; the post does not disclose the offer price, valuation, share count, or use of proceeds.

Why it matters: Cerebras raised $5.5B in the first big tech IPO of 2026 and its stock rose 108% on day one, a sample of how an AI chip company fares in public markets.

AI HOT (Curated Pool)

Accelerating On-Device AI: Arm and Google AI Edge Optimization Practices

Arm SME2 and Google AI Edge integrate with LiteRT, XNNPACK, and KleidiAI to optimize Stability AI’s stable-audio-open-small, delivering over 2x faster audio generation and 4x lower memory use on Arm-based mobile devices and laptops.

Why it matters: Arm and Google AI Edge use SME2 instructions to put matrix units in the CPU, more than doubling on-device audio model inference speed and cutting memory to a quarter.

May 14Thursday

Bloomberg Technology

AI Chipmaker Cerebras Climbs 68% After Year’s Biggest IPO

Cerebras Systems rose 68% in its trading debut after raising $5.5 billion in the year’s largest IPO; the post does not disclose the IPO price or valuation.

Why it matters: Cerebras listed on the public market with a $5.55 billion raise and closed its first day up 68%, taking a technical route on giant training chips that differs from Nvidia's.

QbitAI · WeChat

Chinese GPU Vendor Hosts Open Source Meetup With SGLang Core Developers

Moore Threads said at the SGLang × MUSA Meetup that the MUSA backend has been merged into SGLang mainline, with 47 PRs submitted and 41 merged as of May 12.

Why it matters: Moore Threads merged its MUSA backend for SGLang into the main branch, with 47 pull requests filed and 41 merged, a usable playbook for domestic GPUs joining mainstream inference frameworks.

AI HOT (Curated Pool)

OpenSquilla Open-Source Project Uses Smart Routing and Local Retrieval to Cut LLM Costs

OpenSquilla combines local model routing, vector retrieval, incremental sending, and cache hits to reduce transmitted tokens by more than 90%, while routing simple tasks to cheaper models and complex tasks to stronger models without spending tokens on the routing decision.

Why it matters: OpenSquilla runs model routing decisions locally, judging question difficulty without spending tokens, and cuts actual transfer volume below a tenth through incremental sending and cache hits.

AI HOT (Curated Pool)

UnslothAI Releases Qwen3.6 MTP GGUF Models With Over 1.4x Faster Inference

Daniel Han released experimental Qwen3.6 MTP GGUF models, with the 27B model reaching 140 tokens/s on one GPU and the 35B-A3B version reaching 220 tokens/s, using two draft tokens for speculative decoding.

Why it matters: Unsloth founder Daniel Han released an experimental Qwen3.6 MTP GGUF that drafts two tokens per step, trading accuracy for over 1.4x faster inference.

AI HOT (Curated Pool)

Moonshot AI founder Yang Zhilin releases a 40-minute video

Yang Zhilin explains Kimi K2 training in a 40-minute video, saying the model cost $4.6 million and beat GPT-5.5 and other competitors on coding tasks.

Why it matters: Yang Zhilin's 40-minute video puts Kimi K2's $4.6M training cost on the record and claims it beats GPT-5.5 on coding tasks.

AI HOT (Curated Pool)

Cost Analysis of AI Email

Top AI models process email at about $22 to $130 per month, with a $26 median; smaller models cut costs by 10 to 20 times, while local GPU execution can bring marginal cost close to zero.

Why it matters: A cost breakdown puts AI email handling at $22 to $130 per month in inference, and lays out smaller models and local GPUs as the ways down.