Skip to content

Data & training

The training side: datasets, synthetic data, pre- and post-training methods, compute and training cost.

Latest picks

81–100 of 173

Jun 3Wednesday

The Verge · AI

Google Must Let Publishers Opt Out of AI Search Features, UK Rules

The UK CMA requires Google to let website owners exclude content from AI Search features, including AI Overviews, and prevent that content from being used for fine-tuning Google’s AI models.

Why it matters: The UK competition regulator told Google to let publishers pull their content out of AI search overviews and block it from fine-tuning its models, giving content owners a say over how their data is used.

Synced · WeChat

Understanding SFT Mechanisms in LLMs: Resolving Practice Disputes and Avoiding Wasted Compute

Junpeng Zhang and coauthors argue that SFT on highly homogeneous data has an effective window of only hundreds to about 1,000 training steps, and their interaction-based warning signal detects overfitting before loss gaps appear, saving roughly 30%–50% of training compute.

Why it matters: Shows SFT on homogeneous data has a narrow training window, with an early-warning method that flags overfitting before the loss gap widens.

Jun 2Tuesday

r/LocalLLaMA

I spent months inside verl, forked it, then stopped: internals, fork costs, and an NCCL bug

ReinforcedKnowledge analyzes ByteDance’s verl RLHF loop, covering DataProto plus rollout, reward, advantage, and update paths. The author stopped a private fork because near-daily upstream changes made sync cost exceed refactoring work, and describes an NCCL hang fixed on one node by setting NCCL_SOCKET_IFNAME=lo.

Why it matters: The author walks through verl's RLHF training flow, from the DataProto data protocol to rollout, reward, advantage computation and model updates, and logs the cost of a maintenance fork plus one NCCL bug.

AI HOT (Curated Pool)

NVIDIA Cosmos 3 Tops Open-Weight Image and Video Generation Rankings

NVIDIA Cosmos 3 ranked first in Artificial Analysis’s open-weight text-to-image and image-to-video categories, with 16B Nano and 64B Super variants, and the release includes weights, code, curated datasets, and fine-tuning recipes under the OpenMDW 1.1 license.

Why it matters: NVIDIA's Cosmos 3 took first on Artificial Analysis's open-weights board for both text-to-image and image-to-video, with a Mixture-of-Transformers design pairing an autoregressive reasoner with a diffusion generator.

Jun 1Monday

AI HOT (Curated Pool)

OpenBMB Releases Two UltraData Open Datasets, Tops HuggingFace Trending

OpenBMB, Tsinghua NLP, and Modelbest released two UltraData open datasets: Ultra-FineWeb-L3 contains 600B+ tokens, including 400B+ English and 200B+ Chinese tokens, while UltraData-SFT-2605 contains 15M+ SFT samples with thinking and non-thinking labels.

Why it matters: OpenBMB with Tsinghua NLP and Modelbest released Ultra-FineWeb-L3 and a second dataset, including over 200B tokens of synthetic Chinese pretraining data ready for pretraining and instruction tuning.

r/LocalLLaMA

I bolted an 8-arm reasoning MoE onto a frozen 1.4B Mamba backbone on a single RTX 3060

The author trained Mamba-Titan-1.4B-Reasoning on a 12GB RTX 3060: a frozen 1.4B Mamba-1 backbone with 8 trainable MoE arms, 2.54B total parameters, Top-2 routing at layers 24/25, and about 50% math accuracy.

Why it matters: A single RTX 3060, a frozen 1.4B Mamba backbone and eight trainable expert arms at layers 24 and 25 reach 2.54B parameters and roughly 50% on math problems.

May 31Sunday

QbitAI · WeChat

Fudan and Tongyi introduce ToolCUA for GUI-Tool path selection in agents

Fudan University and Tongyi Lab introduced ToolCUA-8B, which reaches 46.85% accuracy on OSWorld-MCP after training with about 4k synthetic tools and 180k interleaved GUI-Tool trajectory steps.

Why it matters: Fudan and the Qwen team train an agent to pick the right on-screen tool with 4,000 synthetic tools and 180,000 cross-trajectory steps, hitting 46.85% on OSWorld-MCP with an 8B model.

May 30Saturday

Synced · WeChat

CUHK Pion optimizer updates LLMs on iso-spectral manifolds to address AdamW and Muon instability

CUHK and collaborators introduced Pion, an optimizer that preserves weight singular values through orthogonal equivalence transformations, and reported that it kept a 60M normalization-free LLaMA-like model stable for 9.6B training tokens while AdamW and Muon collapsed with NaNs.

Why it matters: A CUHK team's Pion optimizer keeps singular values fixed and updates only via orthogonal transforms, training a 60M-parameter model for 9.6B tokens where AdamW and Muon hit NaN.

May 29Friday

AI HOT (Curated Pool)

Adam's Law: Prompts Written with High-Frequency Words Work Better

FaceMind tested 100 languages and four core tasks, finding that, with semantics unchanged, prompts or fine-tuning text using higher-frequency expressions from pretraining data improves large language model performance.

Why it matters: The FaceMind team tested 100 languages across four task types and found that phrasing prompts the way high-frequency pretraining text does improves model performance, adding a frequency angle to prompt writing.

Synced · WeChat

The Ma Jiaqi Failure Exposed an LLM Issue He Spotted in the Shower a Year Earlier

FaceMind links low-frequency token degradation to two papers: SLoW appeared at EMNLP 2025, Adam's Law was accepted as an ACL 2026 Oral, and high-frequency rewriting raised DeepSeek-V3 math accuracy from 63.55% to 71.54%.

Why it matters: The FaceMind team links large models failing on low-frequency words to the SLoW and Adam's Law papers, laying out a mechanistic account of low-frequency token degradation.

May 28Thursday

QbitAI · WeChat

Behind DeepSeek V4's Chip-Model Co-Design, China's Compute Ecosystem Gains Speed

QbitAI says DeepSeek V4 validated Ascend chip-model co-design, with CANN open-sourcing 65 repositories and supporting day-zero adaptation for more than 70 mainstream models, while AIGCode reported 65% MFU in MoE pretraining on Ascend.

Why it matters: DeepSeek V4 was tuned against Huawei Ascend chips during training, and CANN open-sourced 65 repos with out-of-the-box support for over 70 mainstream models, giving domestic compute a concrete chip-to-framework-to-model path.

Synced · WeChat

Chinese pretrained embodied model Wall-OSS-0.5 is open sourced

X Square Robot open sourced Wall-OSS-0.5, a VLA model whose 400k pretraining checkpoint scored above 80 on 4 of 17 real-robot zero-shot tasks, with weights, code, training recipe, ablations, and a DMuon optimizer implementation released.

Why it matters: X Square Robot open-sources Wall-OSS-0.5, a vision-language-action model with a 400k-step pretrained checkpoint, weights, code and training recipe; 4 of 17 real-robot zero-shot tasks top 80% success.

AI HOT (Curated Pool)

NVIDIA Releases AI Framework Polar, Raising Codex Benchmark Score by 594.74%

NVIDIA’s research team open-sourced Polar, an agent reinforcement learning framework that connects GRPO training at the model API boundary without rewriting Codex CLI, Claude Code, Qwen Code, or Pi; on Qwen3.5-4B, Polar raised Codex pass@1 on SWE-Bench Verified from 3.8% to 26.4%, while prefix_merging cut training steps from 1,185 to 218.

Why it matters: NVIDIA open-sources the Polar framework, letting Codex train with reinforcement learning without rewriting code, a transferable path to upgrading existing coding tools.

r/LocalLLaMA

I built a 103B-token Usenet corpus from 1980–2013

OwnerByDane released a 103.1B-token Usenet corpus covering 1980–2013, 408M posts, and 18,347 newsgroups, with free 5K-post-per-hierarchy samples and full-corpus licensing available.

Why it matters: The corpus holds 103.1 billion tokens across 408 million posts from 1980 to 2013, all real Usenet speech, giving researchers text free of AI-generated content.

May 27Wednesday

Synced · WeChat

AMD paper: FP4 training instability is not caused by insufficient randomness

AMD and Penn State pretrained Llama 3.1-8B with MXFP4 on MI355X native FP4 hardware, achieving 9-10% end-to-end speedup over an FP8 baseline, while the paper identifies Wgrad quantization as the bottleneck that raises token overhead to 26-27% without deterministic Hadamard stabilization.

Why it matters: AMD and Penn State pretrained Llama 3.1-8B on MI355X with native FP4, running 9-10% faster end to end than an FP8 baseline, while weight gradient quantization added 26-27% cost per token.

AI HOT (Curated Pool)

AI Builds AI: ModelBest Open-Sources ForgeTrain, a Training Framework Written by AI

ModelBest, Tsinghua University, and OpenBMB open-sourced ForgeTrain, described as the first production-grade LLM training framework written entirely by AI with zero human code, and ModelBest used it to pretrain MiniCPM5-1B on Huawei Ascend chips.

Why it matters: ModelBest, Tsinghua and OpenBMB open-sourced ForgeTrain, a training framework written entirely by AI, and used it to pretrain MiniCPM5-1B on Huawei Ascend chips, a concrete path to automating training pipelines.

AI HOT (Curated Pool)

Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL

Hugging Face merged TRL PR 5417 for delta weight sync, sending only changed weights as sparse safetensors via a Hugging Face Bucket; on Qwen3-0.6B, the per-step payload falls from 1.2GB to 20–35MB.

Why it matters: Hugging Face implemented incremental weight sync in TRL, sending only the weights that changed between optimizer steps, cutting the per-step payload for Qwen3-0.6B from 1.2GB to 20-35MB.

May 26Tuesday

AI HOT (Curated Pool)

SenseNova-U1 full training code open-sourced for multimodal multitask training

OpenSenseNova released the full SenseNova-U1 training code on GitHub under Apache-2.0, supporting an 8B dense model, an A3B MoE architecture, and multimodal tasks such as text-to-image generation, image editing, interleaved generation, and text-visual understanding.

Why it matters: SenseTime open-sourced SenseNova-U1's full training code on GitHub under Apache-2.0, covering 8B dense and A3B MoE variants, with one framework handling text-to-image, image editing, interleaved generation and understanding.

Synced · WeChat

Grok keeps updating after xAI disbandment as Musk announces a new model

Elon Musk said the 1.5T-parameter Grok V9-Medium has finished training, will enter reinforcement learning in a few days, and is planned for release in two to three weeks. Grok Build supports up to 8 parallel sub-agents, a 256K-token context window, Plan Mode, Arena Mode, MCP, and ACP.

Why it matters: Musk says the 1.5-trillion-parameter Grok V9-Medium has finished training and ships in two to three weeks, with Grok Build supporting 8 sub-agents and a 256K-token context.

r/LocalLLaMA

Update on a 12×32GB SXM V100 Cluster for Local Legal Drafting

A lawyer runs a local legal-drafting pipeline across 16 GPUs, with Qwen3.5-122B-A10B reaching about 50 tok/s on four V100s, while a verifier blocks ungrounded citations, dates, and Bates numbers before any final document is used.

Why it matters: A lawyer built a local cluster from 12 V100s with 32GB of VRAM each, running Qwen3.5-122B on 4 cards at roughly 50 tokens/sec to draft legal documents, showing a workable local setup for professional use.