Your robots.txt is a 2023 war memorial — most sites ignore answer-time bots

Sites that block AI training crawlers mostly ignore the answer time bots

Sitedex scanned the top 10,000 sites' robots.txt files. 38% of dated GPTBot block rules were written in Q4 2023, right after GPTBot launched and the NYT sued. 87% of those sites later added new rules, but almost all target training crawlers. Anthropic, OpenAI, and Perplexity each run two bots: one for training, one for fetching pages live when a user asks a question. Among sites that block the training crawler, 71% have no rule for Anthropic's answer-time bot, 53% for OpenAI's, and 50% for Perplexity's. Fewer than 4% deliberately allow the answer bot while blocking training. The post does not disclose Cloudflare's new billing scheme pricing or launch date.

Why it matters: Data-backed, opinionated, and revealing a real gap: site owners rushed to block training crawlers but missed answer-time bots entirely. Score stays below 80 because Sitedex isn't a top-tier authority and the full body wasn't provided, so we can't verify the data depth.