Skip to content

Deployment & engineering

Running models in practice: inference optimization, memory and cost, serving architecture and infrastructure choices.

Latest picks

281–300 of 609

May 28Thursday

AI HOT (Curated Pool)

Open-source FastVideo Dreamverse real-time video generation tool

Hao AI Lab open-sourced FastVideo Dreamverse, a real-time video generation tool that generates a 30-second 1080p video in 7 seconds under the stated setup of one NVIDIA B200 GPU and LTX-2.

Why it matters: Hao AI Lab open-sourced FastVideo Dreamverse, which pairs LTX-2 with a single NVIDIA B200 to generate 30 seconds of 1080p video in 7 seconds, code and blog included.

May 27Wednesday

AI HOT (Curated Pool)

Perplexity open-sources Unigram tokenizer to reduce CPU usage

Perplexity open-sourced a rebuilt Unigram tokenizer that reduces CPU usage by 5-6x, targeting tokenization latency when small rerankers and embedding models run on GPUs in single-digit milliseconds.

Why it matters: Perplexity open-sourced a rewritten Unigram tokenizer that cuts CPU tokenization cost 5 to 6 times in reranking and embedding pipelines, so tokenization stops slowing down responses.

QbitAI · WeChat

Language Models Need Sleep: Let AI Nap Before Continuing Inference

Carnegie Mellon University and the University of Maryland propose a “sleep” mechanism for language models: when the context window is nearly full, the model stops accepting new tokens, runs multiple offline recursive forward passes to compress accumulated context into fast weights, clears the KV cache, and then resumes inference; tests cover cellular automata, multi-hop graph retrieval, and GSM-Infinite reasoning tasks.

Why it matters: Carnegie Mellon and the University of Maryland give LLMs a sleep-like mechanism that pauses and compresses memory as context fills, a transferable approach to long-context memory management.

Xinzhiyuan · WeChat

OpenRouter processes 100 trillion tokens monthly and raises $113M Series B

OpenRouter raised a $113 million Series B led by CapitalG, lifting its valuation to $1.3 billion; the platform processes 25 trillion tokens per week, about 100 trillion per month, and provides one API for more than 400 models.

Why it matters: OpenRouter routes 400-plus models through one API, handles 25 trillion tokens a week, and raised a $1.3B-valuation Series B led by CapitalG.

Synced · WeChat

AMD paper: FP4 training instability is not caused by insufficient randomness

AMD and Penn State pretrained Llama 3.1-8B with MXFP4 on MI355X native FP4 hardware, achieving 9-10% end-to-end speedup over an FP8 baseline, while the paper identifies Wgrad quantization as the bottleneck that raises token overhead to 26-27% without deterministic Hadamard stabilization.

Why it matters: AMD and Penn State pretrained Llama 3.1-8B on MI355X with native FP4, running 9-10% faster end to end than an FP8 baseline, while weight gradient quantization added 26-27% cost per token.

Latent Space

[AINews] New AI Infra Decacorns: Fireworks, Baseten, with OpenRouter on the Way

Latent Space says Fireworks is in talks for a $15 billion valuation round, Baseten is raising at an $11 billion valuation, and OpenRouter closed a $113 million Series C after volume grew 5x in six months.

Why it matters: Fireworks and Baseten are in funding talks at $15 billion and $11 billion valuations, with the jumps coming in 7 months and 3 months, showing how fast capital is concentrating on the inference layer.

AI HOT (Curated Pool)

Qualcomm and ByteDance reportedly partner on AI ASIC chips with millions of units planned

The title says Qualcomm and ByteDance reached an AI ASIC chip partnership with procurement in the millions of units; the post does not disclose chip specifications, unit pricing, delivery timing, or production conditions.

Why it matters: Qualcomm signed an AI ASIC custom chip deal with ByteDance for millions of units to run ByteDance's AI services, a concrete case of custom silicon moving from internal projects to volume production.

Bloomberg Technology

Fireworks AI in Talks for Funding at $15 Billion Valuation

Fireworks AI is in talks to raise a new funding round at a $15 billion valuation, according to people familiar with the matter; the post does not disclose the round size, investor names, or timeline.

Why it matters: Fireworks AI is negotiating a new round at a reported $15 billion post-money valuation, a number that can be compared against other AI infrastructure players.

AI HOT (Curated Pool)

Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL

Hugging Face merged TRL PR 5417 for delta weight sync, sending only changed weights as sparse safetensors via a Hugging Face Bucket; on Qwen3-0.6B, the per-step payload falls from 1.2GB to 20–35MB.

Why it matters: Hugging Face implemented incremental weight sync in TRL, sending only the weights that changed between optimizer steps, cutting the per-step payload for Qwen3-0.6B from 1.2GB to 20-35MB.

AI HOT (Curated Pool)

MiMo 2.5 Pro Gets Major Price Cut, Matching DeepSeek V4 Pro

Xiaomi permanently cut MiMo-V2.5 API prices by up to 99%, matched DeepSeek V4 Pro pricing, increased same-price token allowances by 5–8x, reset existing user quotas in full, and set the new pricing to take effect on May 26.

Why it matters: Xiaomi permanently cut MiMo-V2.5 series API prices, by up to 99%, matching DeepSeek V4 Pro, with 5 to 8 times more usable tokens at the same price.

r/LocalLLaMA

PrismML Released Binary and Ternary Bonsai Image 4B

PrismML released Binary and Ternary Bonsai Image 4B, 1-bit and ternary text-to-image diffusion transformers around 3GB, compared with FLUX.2 Klein 4B at about 16GB, with browser-local WebGPU demo links and an Apache-2.0 license disclosed in the Reddit snippet.

Why it matters: PrismML compressed Bonsai Image 4B's text-to-image diffusion transformer weights to 1-bit or ternary, about 3GB against FLUX.2 Klein 4B's roughly 16GB, and it runs locally in a browser on WebGPU.

May 26Tuesday

Bloomberg Technology

Qualcomm to Supply Chips to TikTok Owner ByteDance

Qualcomm will supply chips to ByteDance for artificial intelligence data centers, according to people familiar with the matter; the post does not disclose chip models, order volume, pricing, or delivery timing.

Why it matters: Bloomberg reports Qualcomm will supply AI data center chips to ByteDance, but the chip model, order size and delivery timeline are undisclosed, so only the direction of the deal is confirmed.

r/LocalLLaMA

[OSS] dlmserve: First Serving Engine for Diffusion Language Models

dlmserve released an MIT-licensed serving engine for diffusion language models, with LLaDA-8B-Instruct support and 2.5x HF throughput at batch=4. It exposes an OpenAI-compatible /v1/chat/completions API, batches at the denoising-step level, runs in 12GB VRAM, and adds about 1.8x throughput with optional LocalLeap acceleration.

Why it matters: A Reddit user open-sourced dlmserve, an OpenAI-compatible inference server for diffusion language models, reporting LLaDA-8B throughput at batch=4 that is 2.5x HuggingFace's.

AI HOT (Curated Pool)

OpenRouter Raises $113M Series B

OpenRouter raised a $113 million Series B led by CapitalG; its weekly volume rose from 5 trillion to 25 trillion tokens over the past 6 months.

Why it matters: OpenRouter raised a $113M Series B led by CapitalG, and its weekly token volume went from 5 trillion to 25 trillion in half a year, giving the model routing layer a concrete production-traffic figure.

Alibaba Technology · WeChat

Nearly 9x training speedup: residual streams in DiT are becoming a convergence bottleneck

Nanjing University LAMDA and Alibaba Intelligent Engine proposed DAR, a timestep-aware cross-layer routing method that replaces fixed residual accumulation in DiT; on ImageNet 256x256, it reduced SiT-XL/2 FID from 9.67 to 7.56 and reached baseline convergence quality with 8.75x fewer training iterations.

Why it matters: Nanjing University's LAMDA lab and Alibaba replace DiT's fixed residual accumulation with per-timestep cross-layer routing, reaching equal convergence on ImageNet in about one-ninth the training steps.

QbitAI · WeChat

Chinese AI-Written Pretraining Framework ForgeTrain Trains MiniCPM5-1B

ModelBest released ForgeTrain and MiniCPM5-1B, saying ForgeTrain was written by AI and trains 10% faster than NVIDIA Megatron under the same hardware conditions. MiniCPM5-1B is a 1B-parameter edge model with about 2GB FP16 weights and about 0.5GB INT4/Q4 weights.

Why it matters: ModelBest released ForgeTrain, a training framework whose code was written by AI, claiming 10% faster training than Nvidia's Megatron on the same hardware, plus the MiniCPM5-1B on-device model that compresses to about 0.5GB.

Synced · WeChat

AI-written training framework trains 1B edge model MiniCPM5-1B

ModelBest open-sourced MiniCPM5-1B and ForgeTrain; the 1B edge model scores 17.9 on AA-Index, while the AI-written ForgeTrain framework matches Megatron’s training results and runs 10% faster on Nvidia H100 under the article’s reported setup.

Why it matters: ModelBest open-sourced the MiniCPM5-1B on-device model and the ForgeTrain training framework, whose code was written by AI; the 1B model scores 17.9 on AA-Index.

AI HOT (Curated Pool)

ModelBest open-sources MiniCPM5-1B, topping sub-2B models on AA-Index

ModelBest open-sourced MiniCPM5-1B, a 1B-parameter edge language model that beats all sub-2B models on AA-Index, uses a 0.5GB weight file after INT4 quantization, and runs on phones and browsers.

Why it matters: ModelBest, Tsinghua and OpenBMB open-sourced the 1B MiniCPM5-1B, which tops the AA-Index among sub-2B models and runs on phones and browsers at 0.5GB after INT4 quantization.

Xinzhiyuan · WeChat

OpenAI Nearly Collapsed? President Says He Resigned the Day Altman Was Ousted

Greg Brockman recounted OpenAI’s 72-hour crisis: on November 17, 2023, the board removed Sam Altman as CEO and took Brockman off the board, after which Brockman resigned the same day and said he initially put the chance of taking the company back at 10%.

Why it matters: OpenAI president Greg Brockman recounts the 72 hours around Altman's November 2023 ouster, when he resigned that day and put the odds of retaking the company at one in ten.

r/LocalLLaMA

Shard - Getting to 10× KV Cache Compression

Shard reduces Llama-3.1-8B KV memory by about 10× at 8K context and 11× at 32K, with no measured drop on NIAH or LongBench, using PCA plus int4 quantization for K and Hadamard rotation plus vector quantization for V.

Why it matters: Shard compresses KV cache memory by about 90% on Llama-3.1-8B, roughly 10x at 8K context and 11x at 32K, using PCA plus int4 for keys and Hadamard rotation plus vector quantization for values.