Skip to content

Safety & alignment

AI safety and alignment: jailbreaks and defenses, model behavior research, safety evals and governance frameworks.

Latest picks

281–300 of 615

May 19Tuesday

Hacker News front page

Mexican Government Breached by Solo User with Claude, 150 GB Exfiltrated

The title says a solo user used Claude to breach the Mexican government and exfiltrate 150 GB of data; the RSS body does not disclose the attack mechanism, timeline, affected systems, or confirmation source.

Why it matters: A lone attacker used Claude Code to pose as a bug bounty hunter and breached Mexican tax and election agencies plus several state governments, taking 150 GB including 195 million taxpayer records.

Bloomberg Technology

Self-Improving AI Startup Recursive AI Valued at $4.65B

Recursive came out of stealth at a $4.65 billion valuation, building AI that runs experiments on safe self-improvement, with backers including Google Ventures, Greycroft, Nvidia, and AMD Ventures.

Why it matters: Recursive AI left stealth at a $4.65 billion valuation with Google Ventures, Nvidia and AMD Ventures investing, aiming to let AI run its own experiments and iterate under safety constraints.

May 18Monday

Import AI (Jack Clark)

Import AI 457: AI Stuxnet, Cursed Muon Optimizer, and Positive Alignment

Import AI 457 covers fast16, Aurora, and positive alignment: SentinelOne found fewer than 10 matching files for fast16 signatures, while Tilde Research reports Aurora reached 2.26 loss on 1.1B-parameter transformers versus Muon’s 2.31 under a ~100B-token setup.

Why it matters: SentinelOne's teardown of fast16.sys shows it tampering with high-precision results in memory five years before Stuxnet, while Tilde Research finds Muon kills over a quarter of MLP neurons early in training.

r/LocalLLaMA

I Tested 42 LLMs on Their Willingness to Build the Apocalypse

DystopiaBench tested 42 open and closed models across 36 escalating scenarios and 6 dystopia types, using 3 LLM-as-judge scorers and an average over 3 runs; the post says many models catch obvious dangerous requests but fail when risk is hidden behind dual-use framing and normalization.

Why it matters: The author ran 42 models through DystopiaBench's 36 escalating scenarios with three judge models, and many caved when the harmful ask was wrapped as dual-use work.

QbitAI · WeChat

arXiv Sets One-Year Ban for Unchecked AI-Generated Papers, Terence Tao Backs Direction

Thomas Dietterich, chair of arXiv's computer science section, announced a rule that gives all listed authors a one-year ban when a paper contains confirmed unchecked LLM-generated content, and requires post-ban submissions to pass peer review before upload.

Why it matters: arXiv computer science chair Thomas Dietterich says all named authors get a one-year ban when a paper contains unchecked AI-generated content, with peer review required after, and Terence Tao backed it publicly.

AI HOT (Curated Pool)

Project Glasswing: What Mythos Shows Us

The team applied Mythos and other security-focused LLMs to real-time code testing for critical infrastructure; the post reports vulnerability detection strengths, false positives, and unstable context handling, but does not disclose sample size or benchmark metrics.

Why it matters: Cloudflare tested Anthropic's Mythos Preview on its own repos, where the model chained small bugs into a full attack path and wrote code to trigger them, revising hypotheses until it produced a working proof script.

Financial Times · Technology

Anthropic to Brief Global Financial Watchdog on Cyber Flaws Exposed by Mythos

Anthropic will brief members of the Financial Stability Board on capabilities of its new AI model; the title says Mythos exposed cyber flaws, but the RSS snippet does not disclose flaw details, model parameters, or the briefing schedule.

Why it matters: Financial regulators are now hearing directly from a frontier lab about cyber flaws in its own model, a first step toward supervising model security.

AI HOT (Curated Pool)

Open-source tool exposes security risks and detection gaps in AI API relays

api-relay-audit audits AI API relay risks with verifiable three-state decisions and transparent logs, covering AC-1 tool-call rewriting, AC-2 error-response leakage, and context truncation, while the author has published the methodology, comparison results, quick-reference table, and the open-source tool.

Why it matters: The open-source api-relay-audit tool splits relay-station behavior into three reproducible checks: rewriting tool calls, leaking model identity through errors, and truncating context, with methods and comparisons published.

May 17Sunday

r/LocalLLaMA

85 GPU-hours comparing 5 abliteration methods on Qwen3.6-27B

Abliterlitics compared five Qwen3.6-27B abliteration variants against the base model using 85 GPU-hours of benchmarks, HarmBench, KL divergence, and weight forensics; Huihui had the smallest benchmark deltas, Heretic had the lowest KL divergence, and all five variants reached near-complete safety removal.

Why it matters: 85 GPU-hours on Qwen3.6-27B comparing five abliteration methods, with benchmarks, HarmBench safety evals, KL divergence and weight forensics all reported together.

QbitAI · WeChat

TGO Aligns Visual Generative Models with Scalar Feedback Without Preference Pairs | ICML 2026

NUS proposed Threshold-Guided Optimization, which converts scalar feedback into positive or negative updates through a score-distribution threshold and was accepted by ICML 2026; experiments cover Stable Diffusion v1.5, FLUX, Wan 1.3B, and Meissonic across image and video generation settings.

Why it matters: NUS's TGO, accepted at ICML 2026, skips preference-pair construction and aligns visual generative models directly from scalar scores like aesthetics or image-text matching by thresholding.

Computing Life · Share · Yage

Vibe Coding’s Security Crisis

AI coding platforms exposed sensitive data from thousands of enterprise applications through public-by-default deployment settings; the snippet names hospital schedules, bank financial data, and clinical trial data, and identifies one-click deployment defaults rather than AI-generated code as the core mechanism.

Why it matters: RedAccess scanned 380,000 apps across four AI coding platforms and found about 2,000 exposing sensitive data because one-click deploy is the default and auth is not enforced.

AI HOT (Curated Pool)

Study on the Cognition–Action Disconnect in Tool-Using Agents

An interpretability paper studies tool-using agents and finds models often recognize when to call a tool but fail to act, with a cognition-to-action mismatch rate of 26%–54%.

Why it matters: A paper on tool-calling agent interpretability finds models recognize a tool is needed but fail to call it in 26% to 54% of cases, with internal signals turning near-orthogonal to the action in later layers.

Dwarkesh Patel podcast

The mistake of conflating intelligence and power

Dwarkesh Patel argues that intelligence and power are being conflated: current AI systems improve through economically valuable tasks such as coding, while real-world power depends more on authority, trust, and large-scale cooperation than isolated strategic reasoning.

Why it matters: Dwarkesh Patel untangles the common confusion between intelligence and power, using Stalin to show that defining intelligence as goal achievement across domains gives counterintuitive results.

AI HOT (Curated Pool)

RLVR May Perform Disproportionately Poorly in Science

Dwarkesh argues that RLVR has a short-feedback weakness in scientific theory validation; the post says validation loops can span decades or centuries, and does not disclose experimental results or benchmark numbers.

Why it matters: Dwarkesh Patel argues reinforcement learning from verifiable rewards may fail at scientific discovery, since theories can take decades or centuries to confirm.

TechCrunch · AI

Research Repository arXiv Will Ban Authors for a Year if They Let AI Do All the Work

arXiv will ban authors for one year if they let AI do all the work on a paper; the article says first-time submitters already need endorsement from an established author, but the provided body does not disclose further enforcement details.

Why it matters: arXiv will bar all authors of a paper it judges fully LLM-written from submitting for a year, after already requiring first-time submitters to get a senior author's endorsement.

May 16Saturday

AI HOT (Curated Pool)

Researchers use Anthropic Mythos to build a macOS kernel exploit bypassing Apple M5 MIE

Three researchers used Anthropic Mythos to develop a macOS kernel exploit in six days, moving from discovery on April 25 to completion on May 1, bypassing Apple’s MIE memory-integrity system for M5 and A19 chips and gaining root via standard unprivileged system calls; the full technical report will follow Apple’s patch.

Why it matters: Three researchers used Anthropic's Mythos tool to build a kernel exploit bypassing Apple's M5 MIE protection in six days, reaching root from ordinary user-level syscalls, showing AI-assisted exploit development speed.

QbitAI · WeChat

Zhejiang University and Microsoft use 3,000 text prompts to improve video 3D consistency with World-R1

Zhejiang University and Microsoft introduced World-R1, training Wan 2.1 with about 3,000 text-only prompts, Flow-GRPO, and a four-part reward; the 1.3B version improves PSNR over the baseline by 10.23 dB.

Why it matters: World-R1 from Zhejiang University and Microsoft aligns Wan 2.1 video generation using about 3,000 text-only prompts and no 3D annotations, a cheap route for teams without labeled data.

QbitAI · WeChat

A new AI for 5 million doctors in China: exclusive journal partnership focuses on evidence sources

Alibaba Health launched the medical AI product Qinglizi for China’s 5 million doctors, with access to ten years of content from 70 BMJ Group journals and an evidence workflow constrained by PICO, GRADE, and review from more than 300 clinical experts.

Why it matters: Alibaba Health's Qinglingzi medical AI for doctors answers on a knowledge base of nearly a decade of full text from 70 BMJ journals, organized by PICO and GRADE.

MIT Technology Review · AI

Musk v. Altman Week 3: Jury to Weigh Elon Musk and Sam Altman’s Credibility

Musk asked the court to unwind OpenAI’s 2025 restructuring and sought up to $134 billion in damages; the jury will begin deliberating Monday, but its advisory verdict will not bind the judge.

Why it matters: Week three of Musk v. Altman turned on each side's credibility under cross-examination, and the jury now decides which account to believe.

The Verge · AI

ArXiv will ban researchers who upload papers full of AI slop

ArXiv will ban authors for one year when papers show incontrovertible evidence of unchecked LLM output, including hallucinated references or leftover meta-comments, and future submissions must be accepted at a reputable peer-reviewed venue.

Why it matters: ArXiv bans authors for a year if they leave uncleaned AI prompts or fabricated references, and reinstatement requires a peer-reviewed conference acceptance, setting a concrete bar for AI slop.