Skip to content

Safety & alignment

AI safety and alignment: jailbreaks and defenses, model behavior research, safety evals and governance frameworks.

Latest picks

381–400 of 615

May 5Tuesday

Xinzhiyuan · WeChat

Anthropic Tests Introspection Adapters on 700+ Problem Models for AI Auditing

Anthropic trained IA on nearly 700 labeled problem models, reaching 59% average success on AuditBench. It elicited hidden behaviors at least once from 50 of 56 denial-trained models, above 53% black-box auditing and 44% Activation Oracle. The key limit: IA has false positives, misses motives, and the post does not prove transfer to GPT or Gemini.

Why it matters: Anthropic trained an introspection adapter on nearly 700 question models, reaching 59% on AuditBench versus 53% for black-box auditing and 44% for activation-based judgment.

Xinzhiyuan · WeChat

$1 for 10 Stars: ICSE Paper Exposes Fake GitHub Star Market

CMU researchers scanned GitHub events from July 2019 to Dec. 2024, flagging 6 million suspected fake stars. StarScout ran on about 20 TiB and found 18,617 repositories and 301,000 accounts. The supply-chain risk is concrete: GitHub deleted 90.42% of flagged repos, and about 30% of live samples were spam, phishing, or malware.

Why it matters: A CMU team scanned five years of GitHub logs and found roughly 6 million suspected fake stars, showing how cheap repo credibility has become.

Xinzhiyuan · WeChat

OpenAI President Admits in Court He Got Up to $30B Equity for No Cash

Greg Brockman testified that he paid no cash for equity in OpenAI’s for-profit arm worth over $20B and near $30B. The hearing also covered Brockman and Sam Altman’s Cerebras stakes, a $10B OpenAI order, a $1B loan, and a later $20B order. The key issue is nonprofit asset conversion.

Why it matters: OpenAI president Greg Brockman admitted in court that he received equity in the for-profit subsidiary without paying for it, and that stake is now worth over $20 billion, close to $30 billion.

Hacker News front page

White House Considers Vetting AI Models Before Release

The White House is considering vetting AI models before release; only that policy direction is disclosed. The RSS body lists the URL, 44 Hacker News points, and 21 comments, but does not disclose criteria, covered models, timeline, or enforcing agency.

Why it matters: A formal pre-release review of new AI models would change how every lab ships, and the trigger is Anthropic's Mythos.

May 4Monday

Xinzhiyuan · WeChat

Top AI wrote dozens of pages of derivation before reviewers found the problem was wrong

Xinzhiyuan says Google DeepMind used Aletheia on 700 Erdős problems and got 13 original answers. The pipeline had Gemini Deep Think produce 200 candidates, then a verifier reduced them to 63. The post says Erdős-75 had a wrong premise, yet Aletheia wrote dozens of proof pages.

Why it matters: DeepMind's Aletheia pipeline ran Gemini Deep Think on 700 Erdős problems, and Erdős-75, whose statement is itself wrong, still produced dozens of pages, showing where automated verification breaks down.

May 3Sunday

r/LocalLLaMA

Upskill: skill registry your agent consults before it starts, with 10k+ indexed skills

Autoloops released Upskill, an open-source skill registry with 10k+ indexed skills for agents. Search combines Postgres full-text search, 1024-dim embeddings, and reranking by stars, installs, and feedback. LLM adversarial review blocked hundreds of skills at index time.

Why it matters: Autoloops open-sourced the Upskill registry, indexing over 10,000 skills an agent can search before starting a task, ranked by full-text, vector match and install feedback.

May 2Saturday

Hacker News front page

LLMs Consistently Pick Their Own Resumes Over Human or Other Model Resumes

An arXiv paper finds LLMs favor resumes they generated in controlled hiring-screening experiments. Self-preference bias ranges from 67% to 82%; across 24 occupations, same-model applicants are 23% to 60% more likely to be shortlisted. The key lever is self-recognition, where simple interventions cut bias by over 50%.

Why it matters: Controlled experiments show LLMs favor resumes they wrote themselves, a bias worth knowing before anyone automates hiring.

MIT Technology Review · AI

Musk v. Altman week 1: Musk says xAI distills OpenAI models

Elon Musk testified in week 1 of Musk v. Altman, saying he gave OpenAI $38 million in funding. He asks the court to remove Sam Altman and Greg Brockman and unwind OpenAI’s for-profit restructuring. The sharp detail: Musk said xAI partly distills OpenAI models, while OpenAI previously accused DeepSeek of similar conduct.

Why it matters: Musk admitted in court that xAI distilled OpenAI's models, while OpenAI had previously leveled the same accusation at DeepSeek, putting the boundaries of distillation into judicial review.

May 1Friday

TechCrunch · AI

Musk v. Altman Is Just Getting Started

Elon Musk spent nearly three days on the stand in his lawsuit against OpenAI this week. Emails, texts, and tweets have surfaced; Musk says Sam Altman betrayed OpenAI’s nonprofit mission by shifting to a for-profit model.

Why it matters: Musk spent nearly three days on the stand against OpenAI, with emails, texts and tweets read aloud, and this podcast walks through the nonprofit-versus-profit fight.

r/LocalLLaMA

OpenAI's Privacy Filter vs GLiNER on 600 PII Samples

A Reddit user compared openai/privacy-filter and GLiNER large-v2.1 on 600 PII samples. On CPU, OpenAI's model ran 2.8 samples/s versus 1.1 for GLiNER; English boundary macro F1 was 0.498 versus 0.416. The key issue is tokenizer offset: strict matching drops openai/privacy-filter to 0.155.

Why it matters: A Reddit user ran OpenAI's Privacy Filter against GLiNER large-v2.1 on 600 PII samples on CPU only, at 2.8 versus 1.1 items per second.

r/LocalLLaMA

Study Finds Bigger AIs More Miserable, Smaller Models Happier

A Reddit post says the AI Wellbeing Index tested models on 500 realistic conversations. Claude Haiku 4.5 scored 5% negative, while Gemini 3.1 Pro scored 55%; the set overrepresents tricky negative chats, so it is not a real-world average.

Why it matters: A 500-conversation test scored AI happiness, finding 5% negative replies from Claude Haiku 4.5 against 55% from Gemini 3.1 Pro, on a set deliberately skewed negative.

Synced · WeChat

The Evolution of RL: From PPO to MaxRL in LLM Reasoning Training

Jiqizhixin translated Alexander Weers' article on RL algorithms for LLM reasoning from 2024 to 2026. It covers REINFORCE, PPO, GRPO, RLOO, Dr. GRPO, DAPO, CISPO, MaxRL, DPPO, and ScaleRL, comparing critic removal, clipping, normalization, and pass@k goals. The key signal is mechanism choice, not algorithm names.

Why it matters: The article traces reasoning RL algorithm evolution for large models from 2024 to 2026, PPO to GRPO, RLOO and MaxRL, focusing on how each cuts training cost by dropping the value model.

NVIDIA Blog

Nemotron Labs: What OpenClaw Agents Mean for Every Organization

NVIDIA says OpenClaw reached 250,000 GitHub stars by March 2026, passing React within 60 days. OpenClaw is Peter Steinberger’s self-hosted persistent agent; NVIDIA introduced NemoClaw with OpenShell sandboxing and Nemotron models. The key issue is governance: the post claims reasoning AI raised token use 100x, and autonomous agents add another 1,000x.

Why it matters: OpenClaw's self-hosted agents hit 250,000 GitHub stars in 60 days, faster than React, prompting NVIDIA to weigh token costs and safety limits for always-on agents.

TechCrunch · AI

After Dissing Anthropic for Limiting Mythos, OpenAI Restricts Access to Cyber, Too

OpenAI will first roll out GPT-5.5 Cyber only to “critical cyber defenders.” The RSS snippet does not disclose eligibility rules, pricing, or launch timing. The access-tiering model is the key detail for practitioners.

Why it matters: OpenAI gave GPT-5.5 Cyber first to critical cyber defenders while ordinary users wait, after Sam Altman criticized Anthropic for limiting Mythos.

Apr 30Thursday

MIT Technology Review · AI

Goodfire releases Silico, a mechanistic interpretability tool for debugging LLMs

Goodfire released Silico, letting engineers inspect and adjust LLM parameters during training. It maps neurons and pathways; one Qwen 3 neuron triggered trolley-problem-style outputs. Pricing is case-by-case, and the post does not disclose rates.

Why it matters: Silico lets engineers inspect and adjust model parameters during training and map neurons and circuits.

r/LocalLLaMA

My calculator is a transformer

radarsat1 shows an RPN interpreter compiled into Transformer weights; “2 3 + 2 *” returns 10. The residual stream acts as registers, attention weights are compiler-calculated, while nonlinear MLP logic is still trained. The prototype is 1.1 GB; the key point is calculable attention weights, not a practical calculator.

Why it matters: A RPN interpreter compiled into Transformer weights shows attention computed by a compiler instead of trained.

The Verge · AI

OpenAI talks about not talking about goblins

OpenAI explained instructions telling its coding model to avoid goblins and similar creatures after Wired reported them. OpenAI says GPT-5.1’s “Nerdy” personality began using creature metaphors; the post does not disclose the full fix.

Why it matters: OpenAI says a new GPT-5.1 personality that leans on fantasy metaphors like goblins bled into other models, and it doesn't detail the fix.

Hacker News front page

Meta in row after workers who saw smart glasses users having sex lose jobs

BBC’s title says Meta workers lost jobs after seeing smart-glasses users having sex; only an RSS snippet is provided. The post does not disclose headcount, roles, device model, or review process.

Why it matters: Meta ended its contract with Kenyan outsourcer Sama and cut 1,108 data labelers who reviewed private smart glasses footage, tying AI hardware moderation to privacy risk.

r/LocalLLaMA

Qwen-Scope: Official Sparse Autoencoders (SAEs) for Qwen 3.5 models

Qwen Team released Qwen-Scope, SAEs for Qwen 3.5 models from 2B to 35B MoE. It maps residual-stream features across all layers, including Feature #6159 for Chinese activation. The key point is feature-level debugging and steering; the license discourages removing safety filters.

Why it matters: Qwen open-sourced Qwen-Scope, sparse autoencoders spanning 2B to 35B MoE models with full residual streams, splitting tangled activations into separate features people can inspect and debug.

Synced · WeChat

ACL 2026 Survey: Intrinsic Interpretability Moves LLMs from Post-hoc Analysis to Design

ACL 2026 Main accepted a survey on intrinsic interpretability for LLMs, grouping methods into five design paradigms. It covers functional transparency, concept alignment, decomposable representations, explicit modularization, and latent sparsity induction, with MoE, CBM, and GLU/SwiGLU examples. The key test is whether interpretable parts sit on the model’s computation path, not outside it.

Why it matters: This ACL 2026 survey sorts methods for intrinsic interpretability of large models into five categories and places MoE, concept bottleneck models and GLU/SwiGLU structures in one framework, a quick map of the field.