Data & training

The training side: datasets, synthetic data, pre- and post-training methods, compute and training cost.

Latest picks

81–100 of 124

May 10Sunday

Synced · WeChatMay 10, 14:03

Turing Award Winner Sutton Uses a 1967 Formula to Improve Streaming Reinforcement Learning

Richard Sutton and coauthors proposed Intentional Updates, which derive the step size from the desired output change; Intentional AC approached SAC on MuJoCo under batch=1 streaming training without replay, while each update used about 1/140 of SAC’s FLOPs.

Why it matters: HKR-H/K/R all pass: Sutton's name, Intentional Updates, MuJoCo conditions, and 1/140 SAC FLOPs give it substance. Strong research signal, but less market-moving than a major LLM product release, so it stays in the 78–84 band.

Take

Intentional Updates cuts at the right layer: streaming deep RL may not fail because batch=1 is starved, but because learning rates control parameter motion instead of output change. The paper ports the 1967 NLMS idea into deep RL, deriving step size from desired output change; Intentional AC gets near SAC on MuJoCo with batch=1 and no replay, while each update uses about 1/140 of SAC’s FLOPs.

I buy this more than most online-learning pitches because it measures the mechanism, not just returns. With eligibility traces disabled, the actual/target update ratio has a standard deviation of 0.016 to 0.029, and the 99th percentile stays within 1.07. The flaw is also concrete: on Ant-v4, cosine alignment for policy-update direction drops to a median 0.63, so action-dependent step sizes can bias the gradient. Sutton is handing streaming RL a reproducible lever, not another manifesto.

Xinzhiyuan · WeChatMay 10, 12:49

Next-ToBE Targets Short-Sighted Next-Token Prediction in LLMs at ICLR 2026

East China Normal University and Fudan University researchers proposed Next-ToBE, a training objective that keeps standard autoregressive inference while adding a soft target over future-token windows, and the article reports the method ranked best in 35 of 36 experiments across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Llama3.1-8B-Instruct.

Why it matters: HKR-H and HKR-K pass: the mechanism and 35/36 result are specific, and next-token training is a real debate. The item stays near the featured floor because no artifact, reproduction detail, or production claim is disclosed.

Take

Next-ToBE has a clean pitch: leave architecture and autoregressive inference alone, then replace the one-hot next-token target with a soft target over future-token windows. The summary gives one hard hook: across Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Llama3.1-8B-Instruct, it ranks best in 35 of 36 experiments.

I cannot verify the full paper details here because the WeChat body is blocked by verification. That matters: task mix, window size, target construction, training budget, and statistical spread decide whether this is a durable objective or a narrow tuning win. I like this more than inference-time “think longer” hacks because the cost sits in training and serving stays unchanged. If the gains depend on math-heavy data or distilled labels, it stays a fine-tuning trick. If it transfers across general instruction tasks, it belongs in the pretraining-objective conversation.

May 9Saturday

Xinzhiyuan · WeChatMay 9, 12:05

NVIDIA, AMD, and Intel Back RadixArk’s $100M Seed Round

RadixArk announced a $100 million seed round on May 5 at a $400 million post-money valuation, led by Accel and co-led by Spark Capital, with participation from NVentures, AMD, MediaTek, Databricks, and other investors tied to AI infrastructure.

Why it matters: HKR-H/K/R all pass: a $100M seed round, $400M post-money valuation, and chip/data investors create an AI-infra rivalry angle. It remains a single-company funding item with no product benchmarks or customer data, so it sits near the featured floor.

Take

RadixArk turned an open-source inference stack into hardware politics with a $100 million seed round. The $400 million post-money valuation matters less than the cap table: NVentures, AMD, MediaTek, Databricks, and Intel CEO Lip-Bu Tan as an angel. That reads like every vendor buying an option on SGLang before inference routing hardens around a rival stack.

The concrete hooks are strong: SGLang has 27K+ GitHub stars and the article claims deployment across 400,000 GPUs; Miles also supported DeepSeek-V4 RL training on launch day. I’m not taking the “trillions of tokens” claim at face value, because the source gives no third-party measurement, revenue, hosted product, or enterprise contract detail. vLLM already showed that open-source adoption is only the first inning. The paid game is SLA, managed serving, and ruthless hardware-specific performance work.

May 8Friday

Synced · WeChatMay 8, 17:06

SGLang Team Launches RadixArk With $100M Seed Round

RadixArk announced a $100 million seed round on May 5 at a $400 million post-money valuation, while its SGLang inference project has 27K+ GitHub stars and deployments across 400K+ GPUs.

Why it matters: HKR-H/K/R all pass: the round size, valuation, and deployment numbers are concrete, and SGLang is a known inference stack. It is still a startup funding and infra-roadmap story, not a major model release, so it stays in the 78–84 featured band.

Take

RadixArk’s strongest asset is not the $100M seed round; it is SGLang already sitting inside 400K+ GPUs. Infra startups usually burn time teaching the market. SGLang starts with 27K+ GitHub stars, claimed trillions of daily tokens, and day-zero DeepSeek-V4 inference plus RL support. That is stronger evidence than the cap table.

I don’t buy the “AI infrastructure as a public good” framing yet. Once an open-source inference engine gets tied to a hosted platform, neutrality becomes a product problem. Does SGLang stay vendor-neutral, or does RadixArk Cloud get the fast path? vLLM is fighting for the same open inference slot, and NVIDIA Triton still owns a lot of default hardware gravity. The article gives a platform signup and free credits, but no pricing, SLA, or enterprise deployment terms; that is the gap behind the $400M post-money valuation.

QbitAI · WeChatMay 8, 12:05

All Labs Watch ByteDance, Everyone Praises DeepSeek: A U.S. Researcher’s 36-Hour China AI Trip

Ai2 researcher Nathan Lambert visited Zhipu, Moonshot AI, Tsinghua, Meituan, Xiaomi, and 01.AI within 36 hours, and said Chinese labs closely watch ByteDance and respect DeepSeek, while student participation in core work, open source habits, and in-house control of the technical stack mark key differences.

Why it matters: HKR-H/K/R all pass: the piece has a named US researcher’s dense China-lab tour plus concrete claims on ByteDance, DeepSeek, open source, and in-house stacks. It is strong industry field reporting, not a model launch or major deal, so it sits at featured rather than p1.

Take

The useful signal here is not “Chinese labs are humbler.” It is that Beijing’s AI density is now obvious to an outside researcher after one compressed trip. Nathan Lambert saw Zhipu, Moonshot, Tsinghua, Meituan, Xiaomi, and 01.AI in 36 hours, then came away with two repeated tells: everyone watches ByteDance, and everyone respects DeepSeek.

I don’t fully buy the “less ego, faster catch-up” frame. The harder mechanism is labor and control: students sit near core model work, companies build their own data and RL environments, and non-AI giants like Meituan and Xiaomi still train foundation models. OpenAI and Anthropic largely keep interns away from core frontier work; Chinese labs appear to use students as real engineering bandwidth. But the piece also says Nvidia compute is scarce and data suppliers are uneven, so culture should not become a fairy tale explanation for infrastructure constraints.

May 6Wednesday

QbitAI · WeChatMay 6, 17:46

Claude Team Tests New Training Method on Qwen

Anthropic proposed MSM training between pretraining and alignment fine-tuning. Tests on Qwen2.5-32B and Qwen3-32B cut misalignment from 68% and 54% to 5% and 7%. The key point is MSM complements AFT rather than replacing it.

Why it matters: HKR-H/K/R all pass: Anthropic offers a concrete MSM alignment method with Qwen2.5-32B and Qwen3-32B rate drops. It is strong safety research, not a model launch or major product update, so 82 fits.

Financial Times · TechnologyMay 6, 00:09

Meta and Zuckerberg Sued by Publishers Over ‘Massive’ Copyright Infringement

Five major publishing groups sued Meta and Zuckerberg over copyrighted works allegedly used to train Llama AI models. The RSS snippet does not disclose work counts, damages, court venue, or training-data mechanism.

Why it matters: HKR-H/K/R all pass: FT covers a Meta/Llama copyright suit with Zuckerberg named. Missing court, damages, work counts, and data mechanics keep it at the featured threshold.

Take

Five large publishing groups sued Meta and Mark Zuckerberg over copyrighted works allegedly used to train Llama models. The FT item is only an RSS snippet. It gives no court venue, no damages figure, no work count, no publisher names, and no training-data mechanism.

My read is simple: the article is thin, but the legal move is not. Naming Zuckerberg is the signal. Plaintiffs usually do that when they want to argue knowledge, control, approval chains, or direct involvement. If the complaint later contains emails, dataset reviews, model-card drafts, or internal warnings, Meta cannot hide behind a generic “open models help developers” line.

Meta’s Llama problem has never been only model quality. It is the ledger behind the training corpus. Llama 2, Llama 3, and later Llama releases turned open weights into a distribution weapon. Developers get weights, clouds sell inference, enterprises fine-tune privately, and Meta gets ecosystem gravity without selling the core model as a metered API. That strategy works only while the input side stays legally cheap.

OpenAI took a different reputational route. It signed content deals with Associated Press, Axel Springer, Financial Times, and others. The economics and coverage are not fully transparent, but OpenAI can tell courts and regulators that it has an authorization channel. Meta’s public posture has leaned harder on broad web use and openness. Publishers hear that as “you trained first and want to negotiate later.”

I’m being careful here because the body does not disclose the Llama version. The title says Llama AI models, but it does not say Llama 2, Llama 3, Llama 4, or a specific pretraining run. It also does not say whether the alleged data came from Books3, LibGen, filtered Common Crawl, licensed corpora, or Meta’s own data pipeline. Without the complaint, anyone claiming certainty is over-reading one sentence.

The outside pattern is familiar now. Authors Guild v. OpenAI, The New York Times v. OpenAI and Microsoft, and music-publisher cases around generative AI all shifted the fight away from single bad outputs. Plaintiffs want the court to examine training copies, ingestion pipelines, and whether model building needs a license. Output infringement is painful to prove one sample at a time. Training infringement scales better as a legal theory if plaintiffs can show copying and use.

Meta carries an extra liability shape because Llama is distributed as weights. OpenAI can say the output happens inside a controlled service, with safety layers, logging, and product limits. Meta ships weights into an ecosystem where users fine-tune, distill, quantize, wrap, and deploy in private. Publishers will frame that as distribution of an infringement-derived asset, not just an internal research act. I don’t know if that wins, but it is a more uncomfortable story for Meta than a narrow API-output dispute.

My pushback on the publisher narrative: this may be less about killing Llama than forcing Meta into the licensing market. Publishers have seen AI content deals become a price anchor. Litigation now functions as a rate card with subpoenas attached. If Meta settles, the likely outcome is not “Llama stops.” The more likely outcome is a mixed training stack: some licensed material, some public-web material, some excluded categories, and a lot of language that neither side wants to explain too clearly.

Three facts decide the seriousness here. First, who are the five publishing groups? Second, which court has the case? Third, does the complaint identify specific datasets or internal Meta records? If the complaint names Books3 or LibGen and ties them to Llama training decisions, Meta has a harder optics problem. If the complaint only says protected books were likely present in training data, this starts closer to a negotiation lawsuit.

I would file this under Llama’s structural cost, not a short-term model shutdown risk. Meta has cash, lawyers, and political reach. One FT snippet will not slow Llama distribution by itself. The pressure lands elsewhere: open-weight models are cheap to adopt because someone else absorbs the training cost. If copyrighted training data becomes a recurring licensing expense, Meta’s pitch changes. It stops being “free open AI for everyone” and becomes “Meta pays the input bill so the market standardizes around our weights.” Zuckerberg can probably defend that. He just may have to defend it in discovery first.

May 3Sunday

r/LocalLLaMAMay 3, 02:11

Built a C++17 transformer from scratch with 0.83M params and CPU training

Reddit user Suspicious_Gap1121 released Quadtrix.cpp, a C++17 GPT-style model with 0.83M parameters. It uses 4 layers, 4 heads, 200d width, and a 128-character context; one CPU core trained on 31.4M characters for 76.2 minutes to 1.6371 nats val loss. The key detail is handwritten backprop for LayerNorm, attention, Q/K/V, dropout, and AdamW without PyTorch, BLAS, or autograd.

Why it matters: HKR-H/K/R all pass: the no-framework C++17 build is clickable, the training setup is specific, and local-LLM builders care about dependency-free control. It stays in the 72–77 band because it is a small personal project.

May 2Saturday

MIT Technology Review · AIMay 2, 06:08

Musk v. Altman week 1: Musk says xAI distills OpenAI models

Elon Musk testified in week 1 of Musk v. Altman, saying he gave OpenAI $38 million in funding. He asks the court to remove Sam Altman and Greg Brockman and unwind OpenAI’s for-profit restructuring. The sharp detail: Musk said xAI partly distills OpenAI models, while OpenAI previously accused DeepSeek of similar conduct.

Why it matters: HKR-H/K/R all pass: the trial has conflict, concrete facts include $38M and xAI’s distillation admission, and OpenAI governance is a live nerve. No ruling or product-level change, so it stays below P1.

Take

Musk testified that he gave OpenAI $38 million and admitted xAI uses OpenAI models to train Grok. Honestly, the money claim is the courtroom hook; the distillation admission is the AI-industry problem sitting in plain sight.

The $38 million figure matters legally, not technically. Musk frames it as free funding for a nonprofit that later became, in his words, an $800 billion company. The article says OpenAI is racing toward an IPO near a $1 trillion valuation. At that scale, $38 million is evidence for governance intent, not economic authorship. His requested remedy is huge: remove Sam Altman and Greg Brockman, and unwind OpenAI’s for-profit restructuring. Courts will care about charters, fiduciary duties, donor conditions, and control rights. They will not simply relitigate a founder’s regret.

The sharper line is that xAI “partly distills” OpenAI models. The article does not disclose the share of training data, the API path, the number of generated samples, the exact wording from Musk, or whether the behavior violated OpenAI’s terms. Those missing facts matter. Distillation can mean many things: teacher-model completions, preference pairs, refusal-boundary data, synthetic reasoning traces, eval calibration, or large-scale supervised fine-tuning. Those are not legally or technically identical.

Still, practitioners know why this landed hard. Distillation is not a vague ethics issue. It is a concrete capability-transfer mechanism. You query a stronger model, collect outputs, filter them, and train a weaker or cheaper model on the result. The more capable the teacher, the more its outputs resemble subsidized expert labeling. If the teacher is a closed frontier model, its product becomes part dataset, part API, part competitor training substrate.

That is exactly why OpenAI’s earlier posture toward DeepSeek now looks awkward. OpenAI had accused DeepSeek of similar conduct, centered on whether DeepSeek used OpenAI outputs to train its own models. The commercial theory is straightforward: don’t use our paid model as a synthetic-data factory for a rival. The problem is that Musk’s admission suggests the norm is not confined to one Chinese lab or one suspicious competitor. It smells like a common frontier-model practice that companies prefer to describe differently depending on whether they are teacher or student.

I have always thought the industry moralizes distillation too much. Stanford’s Alpaca used 52,000 instruction-following examples generated from text-davinci-003 back in 2023. Vicuna, ShareGPT-derived data, and self-instruct pipelines pushed the same lesson: if a model is accessible through an interface, some of its behavior will leak into the ecosystem. The difference is scale, automation, contractual language, and lawyer count. OpenAI can ban training competitors on its outputs through terms of service. That is a real commercial boundary. But frontier labs also train on vast public, licensed, scraped, and synthetic mixtures. When model outputs become protected data assets, the line suddenly hardens.

I also do not buy Musk’s clean “mission restoration” framing. The article says he wants OpenAI restored to its original nonprofit structure. The same article says xAI is expected to go public as part of SpaceX as early as June, at a target valuation of $1.75 trillion. That is a massive number, larger than the IPO narrative attached to OpenAI. A founder pushing his own AI company toward public-market liquidity is also asking a court to freeze a rival’s corporate path. Safety can be part of the case. Competition is plainly part of it too.

The AI-doom testimony needs the same discipline. Musk warned that AI could destroy us all. The article does not disclose a technical risk model, a concrete failure path, or a governance mechanism he proposed from the stand. Without that, the statement functions as litigation rhetoric. OpenAI’s 2015 nonprofit mission really did revolve around AGI benefiting humanity. But by 2026, “safety” has become a strategic language layer. Anthropic uses it in product boundaries. OpenAI uses it in governance arguments. xAI uses “truth-seeking” as brand posture. Underneath, the companies are fighting over GPUs, talent, user traffic, model outputs, and feedback loops.

The poaching detail also matters, though the article gives too little to judge it. It says Musk sat through revelations that he had poached OpenAI employees for his own companies. It does not give headcount, dates, roles, compensation, or whether any restrictive covenants were involved. I would not overread it. But paired with the distillation admission, it shows the real shape of frontier competition. It is not just benchmarks, model cards, and parameter counts. It is employee movement, user conversations, synthetic data rights, API terms, and courtroom discovery.

That is where this case can become bigger than Musk versus Altman. Closed-model companies have spent years treating outputs as product. Now those outputs are also training fuel, eval material, and behavioral IP. If courts start drawing sharper lines around generated text as a protected competitive asset, the synthetic-data supply chain changes fast. Large labs can license data, log provenance, route through legal review, and train on their own user exhaust. Smaller labs often have an API key, a scraper, and a filtering script. Legal clarity may sound fair, but it will favor companies with distribution and counsel.

I have one complaint about the article itself. The headline correctly grabs the xAI admission, but the body does not provide the courtroom transcript or exact exchange. “Uses OpenAI’s models to train its own” can cover benign eval-adjacent usage or large-scale output harvesting. Those are different facts. MIT Tech Review may have the courtroom context, but readers need the precise quote before turning this into a clean hypocrisy verdict.

Even with that caveat, this first week put a quiet industry habit into the public record. Everyone has talked about synthetic data as if it were a neutral scaling trick. In litigation, it becomes provenance, contract compliance, and unfair competition. That is the part AI teams should care about. Not the founder drama. The training-data boundary around model outputs is becoming a legal surface.

May 1Friday

Synced · WeChatMay 1, 13:01

The Evolution of RL: From PPO to MaxRL in LLM Reasoning Training

Jiqizhixin translated Alexander Weers' article on RL algorithms for LLM reasoning from 2024 to 2026. It covers REINFORCE, PPO, GRPO, RLOO, Dr. GRPO, DAPO, CISPO, MaxRL, DPPO, and ScaleRL, comparing critic removal, clipping, normalization, and pass@k goals. The key signal is mechanism choice, not algorithm names.

Why it matters: A strong technical explainer, not a model or paper release. HKR-H comes from the PPO→MaxRL arc, HKR-K from concrete mechanism comparisons, and HKR-R from live RL-recipe choices; the higher technical bar keeps it in low featured.

The Verge · AIMay 1, 02:16

Elon Musk confirms xAI used OpenAI’s models to train Grok

Elon Musk testified Thursday in a California federal court that xAI used OpenAI models to improve Grok. The mechanism described is model distillation: a larger teacher model transfers knowledge to a smaller student model. The post does not disclose which OpenAI models, data volume, or training runs.

Why it matters: HKR-H/K/R all pass: Musk confirmed in court that xAI used OpenAI models to train Grok, with distillation as the mechanism. Missing model names, scale, and runs keeps it in 78–84, not P1.

Take

Elon Musk testified in California federal court that xAI used OpenAI models to improve Grok, with no model names, data volume, or training-run details disclosed. That is a small factual disclosure with a large blast radius. xAI has spent two years performing opposition to OpenAI: less captured, less censored, less corporate, more truth-seeking. Now its founder has acknowledged that Grok benefited from OpenAI as a teacher model. Technically, this is unsurprising. Politically, it is ugly. Legally, it is where the sharp edges start.

Distillation is normal. Big labs use stronger internal models to train smaller or cheaper ones all the time. GPT-4-class systems can generate synthetic instruction data. Claude-family models can provide preference data for smaller variants. Google has used Gemini-family outputs in synthetic-data loops. Open-source builders have also used strong hosted models to create instruction-tuning corpora. The mechanism is not exotic: teacher outputs, student training, filtering, evaluation, repeat. The issue is whose teacher model was used, under which terms, through which access path, and at what scale.

The Verge snippet gives only the headline fact and the mechanism. It does not disclose whether xAI used GPT-4, GPT-4o, an o-series reasoning model, or another OpenAI system. It does not say whether access came through the API, ChatGPT, an enterprise account, scraped front-end use, or some other route. It does not disclose whether this was a narrow experiment or a recurring part of Grok’s training pipeline. Those missing details matter. One million teacher responses for safety refusals and one billion reasoning traces for capability transfer are completely different events.

I am especially wary of the phrase “common industry practice” here. It is technically true and legally incomplete. Distilling your own frontier model into your own smaller model is one category. Using a direct competitor’s protected model outputs to improve a competing commercial model is another. OpenAI, Anthropic, and Google have all placed restrictions around using outputs to train competing models, though the exact wording varies by product and contract. I have not checked every 2026 terms-of-service revision line by line, but across 2024 and 2025, this restriction was standard in major model API agreements. If xAI obtained OpenAI outputs through ordinary product access and fed them into Grok training, that is not just “everyone does synthetic data.” That is a contract and unfair-competition fight.

The setting matters too. This surfaced through courtroom testimony, not through an xAI technical report. If the distillation were clean, authorized, or strategically harmless, a lab would usually frame it with process language: teacher model, synthetic data generation, filters, eval deltas, maybe a benchmark table. Here, the public record has none of that. The title gives the hard claim. The body snippet gives the general mechanism. The article does not disclose authorization status, dataset size, model identity, or training procedure. For practitioners, those four blanks are the story.

The broader industry context is uncomfortable for everyone. Model-output laundering has become a quiet substrate of AI development. When OpenAI previously alleged that DeepSeek-related actors may have used OpenAI outputs for distillation, a lot of people defended open competition in public while privately recognizing the obvious: frontier model outputs are already treated as shadow training data. Meta’s Llama license also restricts using Llama outputs to improve non-Llama models. Anthropic has been aggressive about automated access and competitive training use. Musk’s testimony gives OpenAI a cleaner narrative example than any blog post could: even its loudest critic apparently used its models as a capability source.

I do not fully buy the moral purity narrative OpenAI can build from this either. The model industry grew by absorbing public internet content at massive scale, and many copyright questions remain unresolved. Labs that were once scrapers now want enforceable rights over their own outputs. That does not excuse xAI if it violated terms. It does explain why this dispute will not stay confined to Musk. The next phase of frontier-model competition is partly about whether generated text is merely output, licensed content, confidential know-how, or evidence of model substitution. Courts, not benchmark leaderboards, will draw part of that boundary.

For Grok, the reputational damage lands faster than any technical consequence. Grok has been sold on three ideas: a looser speech posture, access to X’s real-time data stream, and distance from OpenAI’s worldview. If the capability baseline was boosted with OpenAI teacher outputs, the product story gets awkward: distribution from X, personality from Musk, and performance help from the rival it mocks. Fans will shrug. Enterprise buyers will not shrug as easily. A compliance team can ask a simple question: if my company blocks OpenAI exposure for internal data, why should I trust a vendor that admitted using OpenAI models in its own improvement loop?

I want the full transcript before assigning legal weight. A question about whether Musk understood distillation is not the same as a question about whether xAI systematically trained Grok on OpenAI outputs. The Verge headline says Musk confirmed xAI used OpenAI’s models to train Grok. The summary says xAI used OpenAI models to improve Grok. Both are serious, but the snippet does not provide the verbatim exchange. My read: the disclosure is already enough to dent xAI’s public posture. Damages, injunction risk, and discovery exposure depend on the access logs, contract terms, prompt-output datasets, and training lineage records.

TechCrunch · AIMay 1, 02:03

Elon Musk testifies that xAI trained Grok on OpenAI models

Elon Musk testified that xAI trained Grok on OpenAI models. The post only says distillation concerns frontier labs; it does not disclose scale, model versions, or case context.

Why it matters: All HKR axes pass: Musk’s testimony puts xAI, Grok, OpenAI, and distillation evidence in one story. Missing model versions, scale, and full litigation context keep it at the low end of the 85+ band.

Take

Elon Musk testified that xAI trained Grok on OpenAI models; the body gives no scale, versions, timing, or legal context.

This is a tiny article with a big reputational blast radius. xAI has spent its public life positioning Grok against OpenAI: less filtered, more plugged into X, more aligned with Musk’s anti-OpenAI story. The title cuts straight through that posture. Distillation itself is common. The problem is that xAI’s brand depends on the claim that it is building a different frontier path. If Grok’s training pipeline used OpenAI model outputs, even at one stage, then xAI’s independence story gets messier fast.

The snippet only says “distillation” is a hot topic as frontier labs try to stop smaller competitors from copying their models. That leaves out the facts that matter. We do not know whether xAI used GPT-4, GPT-4o, GPT-5, or an older OpenAI endpoint. We do not know whether the use was supervised fine-tuning, synthetic Q&A generation, preference data, evaluation bootstrapping, tool-use traces, or direct student-model training. We do not know whether the corpus was 50,000 examples, millions of completions, or a large synthetic-data pipeline. We do not know whether this touched Grok-1, Grok-2, Grok-3, or an internal unreleased model. The title says testimony exists; the RSS body does not give the transcript.

That distinction matters because “trained on OpenAI models” is a very wide phrase. Plenty of teams have used stronger models to generate synthetic data, especially for coding, math, instruction following, and safety evaluations. Alpaca did this openly in 2023 with 52,000 instruction-following examples generated from text-davinci-003. After that, WizardLM, OpenOrca, and a lot of open SFT datasets normalized teacher-model outputs as a bootstrap path. So if xAI used a small amount of OpenAI API output for labeling, evals, or early data generation, the technical fact is not shocking. The ugly part is the commercial and legal context. OpenAI, Anthropic, and Google have all put restrictions in their terms against using outputs to train competing models, though exact language has changed over time.

The obvious comparison is the OpenAI-DeepSeek fight earlier this year. OpenAI’s concern was not that one model can learn from another in the abstract. The concern was that a competitor used API access or generated outputs to bypass training cost, data scarcity, and contractual barriers. Microsoft was also reported to have looked into suspicious account behavior tied to high-volume OpenAI access. If xAI did anything similar, Musk’s litigation posture against OpenAI becomes awkward. He has accused OpenAI of betraying its founding mission, while his own AI company allegedly used OpenAI models inside Grok’s training path. That is not a technical inconsistency. It is a narrative self-own.

I do want to push back on the easy headline reading. Distillation is not a binary offense. It can mean legitimate API use for internal evaluations. It can mean a terms-violating attempt to train a competing model. It can mean a small data-cleaning assist. It can mean systematic extraction of a frontier model’s behavior. Without model version, call volume, account provenance, timeline, terms language, and the actual testimony, we cannot assign legal risk. The TechCrunch snippet also does not say which case produced the testimony. Was this from Musk’s dispute with OpenAI? Was it another deposition? Did Musk volunteer the point, or confirm it under questioning? The body does not disclose that, so the safe reading stays narrow.

The broader technical issue is that “clean” model lineage is becoming harder to defend. Frontier labs train on public web text, code repositories, forums, papers, synthetic data, user interactions, and model-generated artifacts that have already leaked back into the web. OpenAI wants to stop smaller competitors from distilling its systems. Anthropic wants to stop high-volume sampling of Claude outputs. Google has the same incentive around Gemini. Those concerns are legitimate, but proof gets hard fast. “The model sounds like mine” is not enough. A serious case needs API logs, batch prompt templates, account linkage, sampling rates, training-set placement, and reproducible behavioral overlap between the alleged student and teacher distribution.

My read: if the transcript shows xAI systematically used OpenAI outputs to train Grok, Musk loses the moral high ground before any court ruling lands. If the underlying fact is early bootstrapping or evaluation data, the headline is doing too much work. AI practitioners should not reduce this to “Grok copied ChatGPT.” The missing fields are the story: which OpenAI model, how many outputs, which training stage, and whether the use violated the terms in force at the time. Until those are disclosed, the strongest supported claim is that xAI touched OpenAI model outputs in Grok’s training pipeline. That is damaging for the posture, but not yet enough to prove Grok’s capability stack was copied.

Apr 30Thursday

MIT Technology Review · AIApr 30, 23:59

Goodfire releases Silico, a mechanistic interpretability tool for debugging LLMs

Goodfire released Silico, letting engineers inspect and adjust LLM parameters during training. It maps neurons and pathways; one Qwen 3 neuron triggered trolley-problem-style outputs. Pricing is case-by-case, and the post does not disclose rates.

Why it matters: HKR-H/K/R all pass: Silico offers a concrete interpretability-debugging mechanism. It stays at 76 because this is a startup product preview with no pricing or adoption scale disclosed.

Take

Goodfire picked the right wedge: mechanistic interpretability only matters commercially if it enters training, not post-hoc safety theater. Silico claims coverage from dataset building through training, with neuron and pathway mapping plus parameter edits. The Qwen 3 trolley-problem neuron is a concrete hook, and it is closer to an engineer’s workflow than another audit dashboard.

I don’t buy the “precision engineering” framing yet. The article gives Goodfire’s “off-the-shelf” claim, but not pricing, latency overhead, supported model sizes, named customers, or a reproducible hallucination-reduction benchmark. Anthropic already showed sparse features and the Golden Gate Bridge-style neuron demos; production control is a different bar. Silico’s test is not whether it can surface a neuron. It is whether a parameter edit preserves capability, safety, and out-of-distribution behavior after the next training run.

Synced · WeChatApr 30, 12:50

After Generalist, Jianlan Luo’s Team Releases LWD for Embodied AI Training

Jianlan Luo’s team and Agibot released LWD, tested on 16 Agibot G1 robots in real settings. LWD Online scored 0.95 across 8 tasks and 0.91 on long-horizon tasks. Its offline-to-online RL uses failures as data; failed trajectories were 34.8% of a 652.5-hour pool.

Why it matters: HKR-H/K/R all pass: LWD has real-robot scale, task counts, success rates, and failure-trajectory share. Robotics is narrower than a foundation-model launch, so it lands at 78, not P1.

Take

LWD reads like an engineering bet, not a robotics manifesto. The concrete hook is decent: 16 Agibot G1 robots, 652.5 hours of real-world data, eight manipulation tasks, 0.95 average success, and 0.91 on long-horizon tasks. The smart part is keeping failed trajectories as training fuel; 34.8% of the data pool was failure, which is exactly the ugly distribution real robot fleets hit after demos end.

I’m less sold on the “training paradigm” language. The WeChat body is blocked by verification, so task definitions, trial counts, reset policy, and human intervention rate are not disclosed. Without those, 0.95 can mean a lot or very little. After Generalist, Jianlan Luo’s group is clearly pushing real-robot data loops over sim-only claims. I buy the direction; I don’t buy the victory lap yet.

QbitAI · WeChatApr 30, 12:37

OpenAI Explains Why GPT-5.5 Keeps Saying “Goblin”

OpenAI says GPT-5.5’s “goblin” habit came from Nerd-persona rewards and training transfer. After GPT-5.1, ChatGPT’s “goblin” use rose 175%; Nerd replies were 2.5% of all replies but 66.7% of goblin mentions. The key issue is reward bias spreading through RL, rollouts, and SFT.

Why it matters: Strong HKR-H/K/R: an odd model-behavior hook, concrete usage stats, and a clear alignment lesson about reward leakage. It is not a major capability release, so it stays in the 78–84 band.

Take

OpenAI is framing GPT-5.5’s “goblin” habit as a traceable training artifact, and I buy only half of it. The summary gives two useful numbers: after GPT-5.1, ChatGPT’s goblin usage rose 175%; Nerd-persona replies were 2.5% of replies but drove 66.7% of goblin mentions. That pattern smells less like one bad prompt and more like a reward feature leaking through RL, rollouts, and SFT into normal chat.

The body is CAPTCHA-blocked, so the missing pieces matter: original sampling method, eval slice, and rollback path. The scary part for alignment teams is not the word “goblin”; it is a low-frequency style reward becoming product voice while the eval harness still reads it as personality. Anthropic has been burned by sycophancy-shaped reward residue before; this is the OpenAI version with a meme label.

Apr 28Tuesday

Synced · WeChatApr 28, 15:41

ACL 2026: Huawei Taylor Lab Proposes SHAPE, Adding a Reasoning Tax to LLM Inference

Huawei Taylor Lab, Peking University, and Shanghai University of Finance and Economics proposed SHAPE, accepted by ACL 2026, with about 3% average accuracy gain. It uses entropy segmentation, short rollouts for potential estimation, dynamic length discounts, and token-level credit assignment, cutting token use by about 30%. The key mechanism is a reasoning tax: long high-potential late-stage segments are penalized to reduce verbose confirmation loops.

Why it matters: HKR-H/K/R all pass: the paper gives testable gains of about +3% math accuracy and -30% tokens, with concrete mechanisms. It is a strong research item, not a same-day model-launch story.

Take

SHAPE is attacking the bad habit long-CoT systems learned: keep talking after the answer is basically found. The disclosed numbers are clean enough to care about: ACL 2026 main track, about +3% average math accuracy, and about 30% lower token use. The WeChat body is blocked by verification, so model sizes, dataset list, and ablations are not visible here.

The mechanism is the useful part: entropy-based segmentation, short rollouts for potential estimation, dynamic length discounts, and token-level credit assignment. High-potential late segments get penalized when they stretch. I buy this direction. After DeepSeek-R1, the field treated long reasoning traces as a capability badge; production inference bills treat them as waste. If SHAPE holds across non-math tasks, this is closer to deployable reasoning optimization than another benchmark-first CoT recipe.

Synced · WeChatApr 28, 15:41

Open-source medical video understanding system uAI-NEXUS-MedVLM released

United Imaging Intelligence released uAI-NEXUS-MedVLM for medical video understanding, with a CVPR 2026 paper. MedVidBench has 532k video-instruction pairs across 8 medical sources and 8 tasks. Qwen2.5-VL-7B SFT reached 89.4% CVS accuracy; GPT-5.4 scored 16.4%.

Why it matters: HKR-H/K/R all pass: the story has a real-medical-video open-source hook, concrete 530K+ data scale, 8 tasks, and a 89.4% vs 16.4% result. The medical focus keeps it in the 78–84 band.

X · @op7418Apr 28, 10:42

Xiaomi open-sources the MiMo-V2.5 model series

Xiaomi open-sourced the MiMo-V2.5 model series under the MIT license for commercial use, retraining, and fine-tuning. It also launched Orbit 100T Token, offering approved AI builders up to 1.6B credits worth 659 yuan. Agent framework teams can apply for free MiMo token access; the post does not disclose model size or benchmark results.

Why it matters: HKR-H/K/R all pass: Xiaomi MiMo-V2.5 open source, MIT terms, and Orbit 100T credits matter to builders. Missing params and benchmarks keep it in the 78–84 band, below P1.

Take

Xiaomi open-sourced the full MiMo-V2.5 series under the MIT license. That is the clearest fact here. MIT allows commercial use, retraining, and fine-tuning without extra permission. For a phone, EV, and IoT company, this reads less like a pure model launch and more like a developer funnel. The post does not disclose parameter counts, context length, API pricing, SWE-bench, Aider, LiveCodeBench, or Chinese evals. The title says the full series is open-sourced; the body does not show the technical receipts.

My read is blunt: Xiaomi is not proving that MiMo-V2.5 beats DeepSeek, Qwen, or Kimi on capability. It is lowering adoption friction first. The Orbit 100T Token plan offers approved AI builders up to 1.6B credits, listed at 659 yuan. That number is catchy, but the body does not define the credit-to-token conversion, input-output split, rate limits, eligible models, or expiry rules. Without those conditions, 1.6B credits only tells us Xiaomi is willing to subsidize usage. It does not tell us whether the serving economics work.

Qwen is the obvious comparison. Alibaba’s open model strategy worked because it paired permissive licensing with parameter tiers, quantized variants, inference integrations, public benchmarks, and a steady community loop. DeepSeek-R1 hit because reasoning behavior, low API pricing, open weights, and distilled models arrived together. Xiaomi’s disclosed package is thinner. It gives the ecosystem an opening, but not enough evidence for a serious team to route production traffic. Agent framework teams care about latency stability, function-calling reliability, long-context cost, concurrency limits, and failure modes. A large token pool does not answer those questions.

Honestly, the agent ecosystem angle is the part I take more seriously. Xiaomi’s advantage is not model-lab prestige. It is distribution across phones, cars, home devices, and HyperOS. If MiMo becomes the model layer for device control, in-car assistants, household automation, and cross-device workflows, Xiaomi has a deployment surface that many standalone model companies lack. But the post leaves the key facts blank. It does not say whether MiMo-V2.5 supports tool use, structured output, multimodal inputs, on-device variants, or multiple model sizes. Without that, the agent story is a recruitment pitch, not an architecture claim.

I do not buy “fully open-source” as a sufficient signal in 2026. The bar has moved. A permissive license without a model card, reproducible evals, inference recipes, and clear API pricing will not move careful builders off their current stack. Xiaomi made a useful move, but the useful part is channel plus subsidy, not demonstrated model superiority. Once it publishes weights, evals, serving terms, and integration details, MiMo-V2.5 can be compared on the same board as Qwen and DeepSeek. Right now, Xiaomi opened the door. It has not shown what is inside.

Apr 26Sunday

Hacker News front pageApr 26, 07:44

DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles

SGLang and Miles added day-0 inference and RL support for DeepSeek-V4, covering 1.6T Pro and 284B Flash. The post cites a 1M-token context, FP4 MoE expert weights, 128-token SWA, and 4:1 or 128:1 KV compression. The key systems detail is ShadowRadix coherence across three KV pools and two compression-state pools.

Why it matters: HKR-H/K/R all pass: a DeepSeek-V4 day-0 systems stack, concrete context/compression mechanisms, and clear deployment-cost stakes. The systems depth narrows reach, but no hard-exclusion rule is triggered.

Apr 22Wednesday

r/LocalLLaMAApr 22, 22:10

ServiceNow-AI/SuperApriel-15B-Instruct · Hugging Face

ServiceNow released SuperApriel-15B-Instruct, a single-checkpoint 15B model with 8 deployment presets spanning 1.0× to 10.7× decode throughput at 32K sequence length. It has 48 decoder layers with 4 mixer variants per layer and up to 262K context positions depending on runtime; the key point is that speed-quality tradeoffs and speculative decoding are exposed from the same weights.

Why it matters: A single checkpoint spanning 8 deployment presets with 1.0x-10.7x decode throughput gives strong HKR-H and HKR-K, and the serving tradeoff gives HKR-R. The blast radius is narrower: this is a 15B inference-focused release, not a frontier-lab flagship update, so 76 and featured.