Skip to content

#数据/训练

6 today

Jun 3Wednesday

AI HOT (Curated Pool)

Build 2026: Microsoft tops Google in image generation while catching up on reasoning

Microsoft announced seven in-house AI models at Build 2026, including its first reasoning model, one new tuning method, and one autonomous background AI agent; the RSS snippet does not disclose model names, benchmarks, or release dates.

Why it matters: Microsoft shipped seven in-house models at Build 2026, and its trillion-parameter MAI-Thinking-1 reasoning model beat Anthropic Sonnet 4.6 in internal blind tests while landing near DeepSeek V3.2 on public benchmarks.

The Verge · AI

Google Must Let Publishers Opt Out of AI Search Features, UK Rules

The UK CMA requires Google to let website owners exclude content from AI Search features, including AI Overviews, and prevent that content from being used for fine-tuning Google’s AI models.

Why it matters: The UK competition regulator told Google to let publishers pull their content out of AI search overviews and block it from fine-tuning its models, giving content owners a say over how their data is used.

Synced · WeChat

Understanding SFT Mechanisms in LLMs: Resolving Practice Disputes and Avoiding Wasted Compute

Junpeng Zhang and coauthors argue that SFT on highly homogeneous data has an effective window of only hundreds to about 1,000 training steps, and their interaction-based warning signal detects overfitting before loss gaps appear, saving roughly 30%–50% of training compute.

Why it matters: Shows SFT on homogeneous data has a narrow training window, with an early-warning method that flags overfitting before the loss gap widens.

Jun 2Tuesday

r/LocalLLaMA

I spent months inside verl, forked it, then stopped: internals, fork costs, and an NCCL bug

ReinforcedKnowledge analyzes ByteDance’s verl RLHF loop, covering DataProto plus rollout, reward, advantage, and update paths. The author stopped a private fork because near-daily upstream changes made sync cost exceed refactoring work, and describes an NCCL hang fixed on one node by setting NCCL_SOCKET_IFNAME=lo.

Why it matters: The author walks through verl's RLHF training flow, from the DataProto data protocol to rollout, reward, advantage computation and model updates, and logs the cost of a maintenance fork plus one NCCL bug.

AI HOT (Curated Pool)

NVIDIA Cosmos 3 Tops Open-Weight Image and Video Generation Rankings

NVIDIA Cosmos 3 ranked first in Artificial Analysis’s open-weight text-to-image and image-to-video categories, with 16B Nano and 64B Super variants, and the release includes weights, code, curated datasets, and fine-tuning recipes under the OpenMDW 1.1 license.

Why it matters: NVIDIA's Cosmos 3 took first on Artificial Analysis's open-weights board for both text-to-image and image-to-video, with a Mixture-of-Transformers design pairing an autoregressive reasoner with a diffusion generator.

Jun 1Monday

AI HOT (Curated Pool)

OpenBMB Releases Two UltraData Open Datasets, Tops HuggingFace Trending

OpenBMB, Tsinghua NLP, and Modelbest released two UltraData open datasets: Ultra-FineWeb-L3 contains 600B+ tokens, including 400B+ English and 200B+ Chinese tokens, while UltraData-SFT-2605 contains 15M+ SFT samples with thinking and non-thinking labels.

Why it matters: OpenBMB with Tsinghua NLP and Modelbest released Ultra-FineWeb-L3 and a second dataset, including over 200B tokens of synthetic Chinese pretraining data ready for pretraining and instruction tuning.

r/LocalLLaMA

I bolted an 8-arm reasoning MoE onto a frozen 1.4B Mamba backbone on a single RTX 3060

The author trained Mamba-Titan-1.4B-Reasoning on a 12GB RTX 3060: a frozen 1.4B Mamba-1 backbone with 8 trainable MoE arms, 2.54B total parameters, Top-2 routing at layers 24/25, and about 50% math accuracy.

Why it matters: A single RTX 3060, a frozen 1.4B Mamba backbone and eight trainable expert arms at layers 24 and 25 reach 2.54B parameters and roughly 50% on math problems.

May 31Sunday

QbitAI · WeChat

Fudan and Tongyi introduce ToolCUA for GUI-Tool path selection in agents

Fudan University and Tongyi Lab introduced ToolCUA-8B, which reaches 46.85% accuracy on OSWorld-MCP after training with about 4k synthetic tools and 180k interleaved GUI-Tool trajectory steps.

Why it matters: Fudan and the Qwen team train an agent to pick the right on-screen tool with 4,000 synthetic tools and 180,000 cross-trajectory steps, hitting 46.85% on OSWorld-MCP with an 8B model.

May 30Saturday

Synced · WeChat

CUHK Pion optimizer updates LLMs on iso-spectral manifolds to address AdamW and Muon instability

CUHK and collaborators introduced Pion, an optimizer that preserves weight singular values through orthogonal equivalence transformations, and reported that it kept a 60M normalization-free LLaMA-like model stable for 9.6B training tokens while AdamW and Muon collapsed with NaNs.

Why it matters: A CUHK team's Pion optimizer keeps singular values fixed and updates only via orthogonal transforms, training a 60M-parameter model for 9.6B tokens where AdamW and Muon hit NaN.

May 29Friday

AI HOT (Curated Pool)

Adam's Law: Prompts Written with High-Frequency Words Work Better

FaceMind tested 100 languages and four core tasks, finding that, with semantics unchanged, prompts or fine-tuning text using higher-frequency expressions from pretraining data improves large language model performance.

Why it matters: The FaceMind team tested 100 languages across four task types and found that phrasing prompts the way high-frequency pretraining text does improves model performance, adding a frequency angle to prompt writing.

Synced · WeChat

The Ma Jiaqi Failure Exposed an LLM Issue He Spotted in the Shower a Year Earlier

FaceMind links low-frequency token degradation to two papers: SLoW appeared at EMNLP 2025, Adam's Law was accepted as an ACL 2026 Oral, and high-frequency rewriting raised DeepSeek-V3 math accuracy from 63.55% to 71.54%.

Why it matters: The FaceMind team links large models failing on low-frequency words to the SLoW and Adam's Law papers, laying out a mechanistic account of low-frequency token degradation.

May 28Thursday

QbitAI · WeChat

Behind DeepSeek V4's Chip-Model Co-Design, China's Compute Ecosystem Gains Speed

QbitAI says DeepSeek V4 validated Ascend chip-model co-design, with CANN open-sourcing 65 repositories and supporting day-zero adaptation for more than 70 mainstream models, while AIGCode reported 65% MFU in MoE pretraining on Ascend.

Why it matters: DeepSeek V4 was tuned against Huawei Ascend chips during training, and CANN open-sourced 65 repos with out-of-the-box support for over 70 mainstream models, giving domestic compute a concrete chip-to-framework-to-model path.

Synced · WeChat

Chinese pretrained embodied model Wall-OSS-0.5 is open sourced

X Square Robot open sourced Wall-OSS-0.5, a VLA model whose 400k pretraining checkpoint scored above 80 on 4 of 17 real-robot zero-shot tasks, with weights, code, training recipe, ablations, and a DMuon optimizer implementation released.

Why it matters: X Square Robot open-sources Wall-OSS-0.5, a vision-language-action model with a 400k-step pretrained checkpoint, weights, code and training recipe; 4 of 17 real-robot zero-shot tasks top 80% success.

AI HOT (Curated Pool)

NVIDIA Releases AI Framework Polar, Raising Codex Benchmark Score by 594.74%

NVIDIA’s research team open-sourced Polar, an agent reinforcement learning framework that connects GRPO training at the model API boundary without rewriting Codex CLI, Claude Code, Qwen Code, or Pi; on Qwen3.5-4B, Polar raised Codex pass@1 on SWE-Bench Verified from 3.8% to 26.4%, while prefix_merging cut training steps from 1,185 to 218.

Why it matters: NVIDIA open-sources the Polar framework, letting Codex train with reinforcement learning without rewriting code, a transferable path to upgrading existing coding tools.

r/LocalLLaMA

I built a 103B-token Usenet corpus from 1980–2013

OwnerByDane released a 103.1B-token Usenet corpus covering 1980–2013, 408M posts, and 18,347 newsgroups, with free 5K-post-per-hierarchy samples and full-corpus licensing available.

Why it matters: The corpus holds 103.1 billion tokens across 408 million posts from 1980 to 2013, all real Usenet speech, giving researchers text free of AI-generated content.

May 27Wednesday

Synced · WeChat

AMD paper: FP4 training instability is not caused by insufficient randomness

AMD and Penn State pretrained Llama 3.1-8B with MXFP4 on MI355X native FP4 hardware, achieving 9-10% end-to-end speedup over an FP8 baseline, while the paper identifies Wgrad quantization as the bottleneck that raises token overhead to 26-27% without deterministic Hadamard stabilization.

Why it matters: AMD and Penn State pretrained Llama 3.1-8B on MI355X with native FP4, running 9-10% faster end to end than an FP8 baseline, while weight gradient quantization added 26-27% cost per token.

AI HOT (Curated Pool)

AI Builds AI: ModelBest Open-Sources ForgeTrain, a Training Framework Written by AI

ModelBest, Tsinghua University, and OpenBMB open-sourced ForgeTrain, described as the first production-grade LLM training framework written entirely by AI with zero human code, and ModelBest used it to pretrain MiniCPM5-1B on Huawei Ascend chips.

Why it matters: ModelBest, Tsinghua and OpenBMB open-sourced ForgeTrain, a training framework written entirely by AI, and used it to pretrain MiniCPM5-1B on Huawei Ascend chips, a concrete path to automating training pipelines.

AI HOT (Curated Pool)

Shipping a Trillion Parameters With a Hub Bucket: Delta Weight Sync in TRL

Hugging Face merged TRL PR 5417 for delta weight sync, sending only changed weights as sparse safetensors via a Hugging Face Bucket; on Qwen3-0.6B, the per-step payload falls from 1.2GB to 20–35MB.

Why it matters: Hugging Face implemented incremental weight sync in TRL, sending only the weights that changed between optimizer steps, cutting the per-step payload for Qwen3-0.6B from 1.2GB to 20-35MB.

May 26Tuesday

AI HOT (Curated Pool)

SenseNova-U1 full training code open-sourced for multimodal multitask training

OpenSenseNova released the full SenseNova-U1 training code on GitHub under Apache-2.0, supporting an 8B dense model, an A3B MoE architecture, and multimodal tasks such as text-to-image generation, image editing, interleaved generation, and text-visual understanding.

Why it matters: SenseTime open-sourced SenseNova-U1's full training code on GitHub under Apache-2.0, covering 8B dense and A3B MoE variants, with one framework handling text-to-image, image editing, interleaved generation and understanding.

Synced · WeChat

Grok keeps updating after xAI disbandment as Musk announces a new model

Elon Musk said the 1.5T-parameter Grok V9-Medium has finished training, will enter reinforcement learning in a few days, and is planned for release in two to three weeks. Grok Build supports up to 8 parallel sub-agents, a 256K-token context window, Plan Mode, Arena Mode, MCP, and ACP.

Why it matters: Musk says the 1.5-trillion-parameter Grok V9-Medium has finished training and ships in two to three weeks, with Grok Build supporting 8 sub-agents and a 256K-token context.

r/LocalLLaMA

Update on a 12×32GB SXM V100 Cluster for Local Legal Drafting

A lawyer runs a local legal-drafting pipeline across 16 GPUs, with Qwen3.5-122B-A10B reaching about 50 tok/s on four V100s, while a verifier blocks ungrounded citations, dates, and Bates numbers before any final document is used.

Why it matters: A lawyer built a local cluster from 12 V100s with 32GB of VRAM each, running Qwen3.5-122B on 4 cards at roughly 50 tokens/sec to draft legal documents, showing a workable local setup for professional use.

May 25Monday

r/LocalLLaMA

The Financial Times published an article about Heretic

The Financial Times used Heretic to remove guardrails from Meta Llama 3.3 in under 10 minutes; creator Philipp Emanuel Weidmann said the tool has created over 3,500 decensored models and those modified systems have reached 13 million downloads.

Why it matters: The FT reports on Heretic, a tool its creator says strips Meta Llama 3.3's built-in limits in under 10 minutes.

May 24Sunday

r/LocalLLaMA

BitCPM-CANN: Native 1.58-Bit Large Language Model Training on Ascend NPU

OpenBMB released BitCPM-CANN, a 1.58-bit QAT training stack on Ascend NPU with 0.5B, 1B, 3B, and 8B models trained from scratch, where the 1B to 8B variants retain 95.7%–97.2% of full-precision MiniCPM4 performance across 11 benchmarks.

Why it matters: OpenBMB trained four 1.58-bit ternary models from 0.5B to 8B from scratch on Huawei Ascend NPUs, with the 8B retaining 97% of full-precision performance and compression done during training.

Synced · WeChat

Meta layoff survivors face a difficult choice

Meta is pushing some post-layoff employees into new roles: some engineering managers are returning to IC work, while some Infra and AI engineers are being reassigned to data labeling; the article cites a manager-to-report ratio shift from 1:8 to 1:50 and says Meta holds a 49% stake in Scale AI.

Why it matters: After layoffs, Meta moved some engineering managers back to writing code and reassigned infrastructure and AI engineers to data labeling, shifting the manager-to-report ratio from 1:8 to 1:50.

May 23Saturday

Mistral AI

Mistral to acquire physics AI company Emmi AI

Mistral AI said it has reached a definitive agreement to acquire Emmi AI, a physics AI pioneer, to strengthen its position as an AI transformation partner for industrial companies. Austria-based Emmi AI works on physics AI and large engineering models that speed up engineering workflows, replace multi-day computations with real-time simulation and build digital twins. Emmi's co-founders and more than 30 researchers and engineers will join Mistral's Science and Applied AI teams in May.

Why it matters: Mistral buys Austrian physical AI company Emmi AI, adding more than 30 researchers to its science and applied AI team in May to strengthen industrial simulation and digital twins.

May 21Thursday

Xinzhiyuan · WeChat

USTC Papers Study Lifelong Learning for LMMs via Multimodal Knowledge Injection

USTC researchers released MMEVOKE and KORE: MMEVOKE contains 9,422 samples across 159 subcategories, while KORE uses knowledge-tree augmentation and null-space constrained fine-tuning to reduce catastrophic forgetting during multimodal knowledge injection.

Why it matters: A USTC team's MMEVOKE benchmark and KORE fine-tuning method offer reusable evaluation and training ideas for forgetting in continual learning of multimodal models.

r/LocalLLaMA

HRM 1B

Sapientinc released HRM-Text 1B Base and its training code, and the paper claims competitive performance against 2–7B open models while using 100–900x fewer training tokens and 96–432x less estimated compute, with training on 16 H100 GPUs taking about 46 hours and costing about $1,472.

Why it matters: Sapientinc open-sourced HRM-Text 1B Base and its training code, claiming performance on par with 2B to 7B open models using 1/100th to 1/900th of their training data.

May 19Tuesday

QbitAI · WeChat

JD and CAS IIE Publish Three Papers Defining Self-Taught RLVR

JD and CAS IIE released three Self-Taught RLVR papers covering RLSD, NPO, and CoPD; RLSD reports that 200 training steps on Qwen3-VL-8B-Instruct exceed GRPO at 400 steps across 8 benchmarks.

Why it matters: JD and the Institute of Information Engineering at CAS published three papers on Self-Taught RLVR, a route where the model generates its own training signal during reinforcement learning, skipping human labels and external reward models.

Synced · WeChat

From Selling Tokens to Selling Outcomes: AI Companies Start Taking KPI Risk

Sierra raised $950 million in May at a valuation above $15 billion, while Lingxi says it reached scaled profitability and positive cash flow in 2025; the article uses both companies to frame RaaS as charging for measurable business outcomes rather than tokens or subscriptions.

Why it matters: Sierra charges per resolved support ticket and Lingxi says it hit profitable scale in 2025, a shift from per-call pricing to paying for outcomes.

AI HOT (Curated Pool)

Cursor releases Composer 2.5, calling it its strongest model yet

Cursor released Composer 2.5, claiming a 10x efficiency gain at comparable capability, with larger training scale, more complex reinforcement-learning environments, and a text-feedback mechanism.

Why it matters: Cursor moved Composer to 2.5, claiming better handling of long tasks and complex instructions and 10x efficiency at equal capability, on an architecture shared with Kimi K2.5.

AI HOT (Curated Pool)

Cursor releases Composer 2.5 coding model

Cursor released Composer 2.5, claiming up to 10x higher efficiency on long coding tasks; the model is further trained on Moonshot’s Kimi K2.5 and uses text feedback for 100k-token-scale trajectories.

Why it matters: Cursor shipped Composer 2.5, using text-feedback training to handle 100k-token long trajectories, with the company claiming up to 10x efficiency on complex coding over hundreds of steps.

AI HOT (Curated Pool)

NVIDIA fine-tunes Cosmos Predict 2.5 with LoRA/DoRA for robot video generation

NVIDIA published a Hugging Face post on fine-tuning Cosmos Predict 2.5 with LoRA and DoRA to generate robot first-person videos from text prompts; the post does not disclose dataset size, training cost, or evaluation results.

Why it matters: NVIDIA's technical blog shows lightweight LoRA and DoRA fine-tuning of Cosmos Predict 2.5 to generate first-person robot footage like arm grasps from text prompts, a cheap path to training video for embodied AI.

May 18Monday

Synced · WeChat

ICML 2026: Huawei GTS proposes EDCO for dynamic curriculum fine-tuning

Huawei GTS proposed EDCO, a dynamic curriculum method that selects fine-tuning samples by inference entropy; prefix entropy estimation cuts per-sample scoring time from 2.24 seconds to 0.37 seconds.

Why it matters: Huawei GTS proposed EDCO, a dynamic curriculum fine-tuning method that uses reasoning entropy to judge sample difficulty and pick training data, with prefix-entropy estimation cutting per-sample scoring from 2.24 seconds to 0.37.

r/LocalLLaMA

I trained TIME: short context-triggered thinking on Qwen instead of overthinking

An independent author trained TIME with QLoRA on Qwen3 4B/8B/14B/32B to trigger short mid-response reasoning when context changes; the post says datasets, notebooks, scripts, curriculum, and TIMEBench are public, with 24GB VRAM enough for training up to 14B.

Why it matters: The author trained TIME on the full Qwen3 range from 4B to 32B with QLoRA, triggering short mid-reasoning only when context changes, and released the dataset, scripts and TIMEBench.

AI HOT (Curated Pool)

Composer 2.5 release and technical analysis

Cursor released Composer 2.5, built on a Moonshot open-source checkpoint, trained with synthetic data from real codebases at 25 times the previous scale, and updated with text-feedback reinforcement learning and a sharded Muon optimizer.

Why it matters: Cursor moved Composer to version 2.5, using reinforcement learning from text feedback to insert hints where the model errs, and scaled synthetic data 25 times.

May 17Sunday

Google DeepMind

Google DeepMind launches Gemini for Science toolset

Google DeepMind released Gemini for Science, which includes three experimental tools on Google Labs: Hypothesis Generation, built on Co-Scientist.

Why it matters: DeepMind's research prototypes, including Co-Scientist and AlphaEvolve, become three tools open in Google Labs.

QbitAI · WeChat

TGO Aligns Visual Generative Models with Scalar Feedback Without Preference Pairs | ICML 2026

NUS proposed Threshold-Guided Optimization, which converts scalar feedback into positive or negative updates through a score-distribution threshold and was accepted by ICML 2026; experiments cover Stable Diffusion v1.5, FLUX, Wan 1.3B, and Meissonic across image and video generation settings.

Why it matters: NUS's TGO, accepted at ICML 2026, skips preference-pair construction and aligns visual generative models directly from scalar scores like aesthetics or image-text matching by thresholding.

Dwarkesh Patel podcast

Notes on Pretraining Parallelisms and Failed Training Runs

Dwarkesh documents pretraining failure modes and parallelism tradeoffs: expert choice and token dropping can break causality in MoE routing, FP16 collectives can bias repeated additions after values exceed 1024, pretraining FLOPs are given as 6ND, B300 HBM is listed as 288GB, and FSDP communication can reach params × 3 with reduce-scatter.

Why it matters: The author breaks down two ways pretraining runs go off the rails, and why expert choice in MoE routing makes a token's destination depend on later routing decisions, so training sees information inference never gets.

May 15Friday

r/LocalLLaMA

I trained Qwen3.5 to jailbreak itself with RL, then used the failures to improve its defenses

The author built an RL-based automated red-teaming loop for Qwen3.5, raising defense rate from 64% to 92% while benign accuracy fell from 92% to 88%, and the attacker found 7 tactic families.

Why it matters: The author trained an attack model with RL to jailbreak Qwen3.5 automatically, clustering and penalizing repeated tactics to force out 7 distinct attack types, with fictional-writing prompts at 34%.

r/LocalLLaMA

I Let a Small Model Train on Its Own Mistakes; It Reached 80% on HumanEval and Beat GPT-3.5 on Math

The author fine-tuned Qwen 2.5 7B base on self-mined mistake-correction pairs, raising HumanEval from 25/164 to 112/164; Qwen 2.5 14B used 100 pairs and a 95-minute H100 run costing $3.50.

Why it matters: The author had Qwen 2.5 7B write code and run tests on its own, turned the errors and fixes into training data, and fine-tuning lifted HumanEval pass rate from about 15% to about 68%.