Skip to content
Hacker News front page

Why prompt injection keeps working: ICML paper traces it to role confusion in LLMs

A Theory of Why Prompt Injection Works

This ICML 2026 paper reframes prompt injection as role confusion. LLMs receive a single text stream with role tags (system, user, think, tool) to separate instructions from external data. Attackers hide commands inside tool-tagged webpage content; the model fails when it misreads tool text as a user instruction. The authors argue current models rely on attack memorization—scoring well on static benchmarks but easily bypassed by human red-teamers who rephrase attacks. The robust fix is role perception: respecting role tags instead of pattern-matching known attacks. The post does not disclose a concrete defense or deployment timeline.

Why it matters: An ICML 2026 paper traces prompt injection to the model's confusion over role tags, arguing that when an attacker hides instructions inside a tool-tagged web page, the model treats them as a real user command.

Read the original ↗Export Markdown