Skip to content

Data & training

The training side: datasets, synthetic data, pre- and post-training methods, compute and training cost.

Latest picks

101–120 of 173

May 25Monday

r/LocalLLaMA

The Financial Times published an article about Heretic

The Financial Times used Heretic to remove guardrails from Meta Llama 3.3 in under 10 minutes; creator Philipp Emanuel Weidmann said the tool has created over 3,500 decensored models and those modified systems have reached 13 million downloads.

Why it matters: The FT reports on Heretic, a tool its creator says strips Meta Llama 3.3's built-in limits in under 10 minutes.

May 24Sunday

r/LocalLLaMA

BitCPM-CANN: Native 1.58-Bit Large Language Model Training on Ascend NPU

OpenBMB released BitCPM-CANN, a 1.58-bit QAT training stack on Ascend NPU with 0.5B, 1B, 3B, and 8B models trained from scratch, where the 1B to 8B variants retain 95.7%–97.2% of full-precision MiniCPM4 performance across 11 benchmarks.

Why it matters: OpenBMB trained four 1.58-bit ternary models from 0.5B to 8B from scratch on Huawei Ascend NPUs, with the 8B retaining 97% of full-precision performance and compression done during training.

Synced · WeChat

Meta layoff survivors face a difficult choice

Meta is pushing some post-layoff employees into new roles: some engineering managers are returning to IC work, while some Infra and AI engineers are being reassigned to data labeling; the article cites a manager-to-report ratio shift from 1:8 to 1:50 and says Meta holds a 49% stake in Scale AI.

Why it matters: After layoffs, Meta moved some engineering managers back to writing code and reassigned infrastructure and AI engineers to data labeling, shifting the manager-to-report ratio from 1:8 to 1:50.

May 23Saturday

Mistral AI

Mistral to acquire physics AI company Emmi AI

Mistral AI said it has reached a definitive agreement to acquire Emmi AI, a physics AI pioneer, to strengthen its position as an AI transformation partner for industrial companies. Austria-based Emmi AI works on physics AI and large engineering models that speed up engineering workflows, replace multi-day computations with real-time simulation and build digital twins. Emmi's co-founders and more than 30 researchers and engineers will join Mistral's Science and Applied AI teams in May.

Why it matters: Mistral buys Austrian physical AI company Emmi AI, adding more than 30 researchers to its science and applied AI team in May to strengthen industrial simulation and digital twins.

May 21Thursday

Xinzhiyuan · WeChat

USTC Papers Study Lifelong Learning for LMMs via Multimodal Knowledge Injection

USTC researchers released MMEVOKE and KORE: MMEVOKE contains 9,422 samples across 159 subcategories, while KORE uses knowledge-tree augmentation and null-space constrained fine-tuning to reduce catastrophic forgetting during multimodal knowledge injection.

Why it matters: A USTC team's MMEVOKE benchmark and KORE fine-tuning method offer reusable evaluation and training ideas for forgetting in continual learning of multimodal models.

r/LocalLLaMA

HRM 1B

Sapientinc released HRM-Text 1B Base and its training code, and the paper claims competitive performance against 2–7B open models while using 100–900x fewer training tokens and 96–432x less estimated compute, with training on 16 H100 GPUs taking about 46 hours and costing about $1,472.

Why it matters: Sapientinc open-sourced HRM-Text 1B Base and its training code, claiming performance on par with 2B to 7B open models using 1/100th to 1/900th of their training data.

May 19Tuesday

QbitAI · WeChat

JD and CAS IIE Publish Three Papers Defining Self-Taught RLVR

JD and CAS IIE released three Self-Taught RLVR papers covering RLSD, NPO, and CoPD; RLSD reports that 200 training steps on Qwen3-VL-8B-Instruct exceed GRPO at 400 steps across 8 benchmarks.

Why it matters: JD and the Institute of Information Engineering at CAS published three papers on Self-Taught RLVR, a route where the model generates its own training signal during reinforcement learning, skipping human labels and external reward models.

Synced · WeChat

From Selling Tokens to Selling Outcomes: AI Companies Start Taking KPI Risk

Sierra raised $950 million in May at a valuation above $15 billion, while Lingxi says it reached scaled profitability and positive cash flow in 2025; the article uses both companies to frame RaaS as charging for measurable business outcomes rather than tokens or subscriptions.

Why it matters: Sierra charges per resolved support ticket and Lingxi says it hit profitable scale in 2025, a shift from per-call pricing to paying for outcomes.

AI HOT (Curated Pool)

Cursor releases Composer 2.5, calling it its strongest model yet

Cursor released Composer 2.5, claiming a 10x efficiency gain at comparable capability, with larger training scale, more complex reinforcement-learning environments, and a text-feedback mechanism.

Why it matters: Cursor moved Composer to 2.5, claiming better handling of long tasks and complex instructions and 10x efficiency at equal capability, on an architecture shared with Kimi K2.5.

AI HOT (Curated Pool)

Cursor releases Composer 2.5 coding model

Cursor released Composer 2.5, claiming up to 10x higher efficiency on long coding tasks; the model is further trained on Moonshot’s Kimi K2.5 and uses text feedback for 100k-token-scale trajectories.

Why it matters: Cursor shipped Composer 2.5, using text-feedback training to handle 100k-token long trajectories, with the company claiming up to 10x efficiency on complex coding over hundreds of steps.

AI HOT (Curated Pool)

NVIDIA fine-tunes Cosmos Predict 2.5 with LoRA/DoRA for robot video generation

NVIDIA published a Hugging Face post on fine-tuning Cosmos Predict 2.5 with LoRA and DoRA to generate robot first-person videos from text prompts; the post does not disclose dataset size, training cost, or evaluation results.

Why it matters: NVIDIA's technical blog shows lightweight LoRA and DoRA fine-tuning of Cosmos Predict 2.5 to generate first-person robot footage like arm grasps from text prompts, a cheap path to training video for embodied AI.

May 18Monday

Synced · WeChat

ICML 2026: Huawei GTS proposes EDCO for dynamic curriculum fine-tuning

Huawei GTS proposed EDCO, a dynamic curriculum method that selects fine-tuning samples by inference entropy; prefix entropy estimation cuts per-sample scoring time from 2.24 seconds to 0.37 seconds.

Why it matters: Huawei GTS proposed EDCO, a dynamic curriculum fine-tuning method that uses reasoning entropy to judge sample difficulty and pick training data, with prefix-entropy estimation cutting per-sample scoring from 2.24 seconds to 0.37.

r/LocalLLaMA

I trained TIME: short context-triggered thinking on Qwen instead of overthinking

An independent author trained TIME with QLoRA on Qwen3 4B/8B/14B/32B to trigger short mid-response reasoning when context changes; the post says datasets, notebooks, scripts, curriculum, and TIMEBench are public, with 24GB VRAM enough for training up to 14B.

Why it matters: The author trained TIME on the full Qwen3 range from 4B to 32B with QLoRA, triggering short mid-reasoning only when context changes, and released the dataset, scripts and TIMEBench.

AI HOT (Curated Pool)

Composer 2.5 release and technical analysis

Cursor released Composer 2.5, built on a Moonshot open-source checkpoint, trained with synthetic data from real codebases at 25 times the previous scale, and updated with text-feedback reinforcement learning and a sharded Muon optimizer.

Why it matters: Cursor moved Composer to version 2.5, using reinforcement learning from text feedback to insert hints where the model errs, and scaled synthetic data 25 times.

May 17Sunday

Google DeepMind

Google DeepMind launches Gemini for Science toolset

Google DeepMind released Gemini for Science, which includes three experimental tools on Google Labs: Hypothesis Generation, built on Co-Scientist.

Why it matters: DeepMind's research prototypes, including Co-Scientist and AlphaEvolve, become three tools open in Google Labs.

QbitAI · WeChat

TGO Aligns Visual Generative Models with Scalar Feedback Without Preference Pairs | ICML 2026

NUS proposed Threshold-Guided Optimization, which converts scalar feedback into positive or negative updates through a score-distribution threshold and was accepted by ICML 2026; experiments cover Stable Diffusion v1.5, FLUX, Wan 1.3B, and Meissonic across image and video generation settings.

Why it matters: NUS's TGO, accepted at ICML 2026, skips preference-pair construction and aligns visual generative models directly from scalar scores like aesthetics or image-text matching by thresholding.

Dwarkesh Patel podcast

Notes on Pretraining Parallelisms and Failed Training Runs

Dwarkesh documents pretraining failure modes and parallelism tradeoffs: expert choice and token dropping can break causality in MoE routing, FP16 collectives can bias repeated additions after values exceed 1024, pretraining FLOPs are given as 6ND, B300 HBM is listed as 288GB, and FSDP communication can reach params × 3 with reduce-scatter.

Why it matters: The author breaks down two ways pretraining runs go off the rails, and why expert choice in MoE routing makes a token's destination depend on later routing decisions, so training sees information inference never gets.

May 15Friday

r/LocalLLaMA

I trained Qwen3.5 to jailbreak itself with RL, then used the failures to improve its defenses

The author built an RL-based automated red-teaming loop for Qwen3.5, raising defense rate from 64% to 92% while benign accuracy fell from 92% to 88%, and the attacker found 7 tactic families.

Why it matters: The author trained an attack model with RL to jailbreak Qwen3.5 automatically, clustering and penalizing repeated tactics to force out 7 distinct attack types, with fictional-writing prompts at 34%.

r/LocalLLaMA

I Let a Small Model Train on Its Own Mistakes; It Reached 80% on HumanEval and Beat GPT-3.5 on Math

The author fine-tuned Qwen 2.5 7B base on self-mined mistake-correction pairs, raising HumanEval from 25/164 to 112/164; Qwen 2.5 14B used 100 pairs and a 95-minute H100 run costing $3.50.

Why it matters: The author had Qwen 2.5 7B write code and run tests on its own, turned the errors and fixes into training data, and fine-tuning lifted HumanEval pass rate from about 15% to about 68%.

r/LocalLLaMA

MOOSE-Star (ICML 2026): 7B Model and 108K-Paper Dataset for Scientific Hypothesis Discovery

MiroMind researchers released the MOOSE-Star collection with three 7B models and TOMATO-Star, a dataset of 108,717 NCBI papers. MS-IR-7B reaches 54.37% inspiration-retrieval accuracy, uses DeepSeek-R1-Distill-Qwen-7B as its base, runs at about 14GB fp16, and supports llama.cpp, vLLM, and SGLang.

Why it matters: MiroMind released three 7B MOOSE-Star models and the 108,717-paper TOMATO-Star dataset, with weights and data available for reuse in scientific hypothesis discovery.