Cloudflare splits AI crawlers into Search, Agent, and Training so site owners can allow or charge by use case

Cloudflare 推出全新AI流量管理选项：区分搜索、智能体与训练爬虫，保护广告页面

Building on last year's one-click AI bot block, Cloudflare now classifies automated traffic into three buckets: Search (indexing, should bring referral traffic), Agent (real-time tasks on a user's behalf, like ChatGPT-User or browser-driving agents), and Training (crawling to train models). The taxonomy aims to fix the small-site dilemma—block crawlers and lose discoverability, or allow them and risk free training. Cloudflare urges bot operators to split crawlers by purpose so site owners can manage access per use case. The post does not disclose pricing or a launch date for the new controls.

Why it matters: Cloudflare upgrades AI crawler management from a binary block to a three-category split, directly addressing content sites' core anxiety. Score stays at featured threshold because this is a traffic management tool update, not a model or protocol breakthrough, but the framing a...