AI crawlers are hammering git.kernel.org with billions of requests
Konstantin Ryabitsev shares hard numbers: git.kernel.org gets 6M daily requests, 98% from AI scrapers. Instead of cloning repos, scrapers render every commit as HTML, generating billions of valid URLs from 922 forks of linux.git. IP bans and ASN blocks failed once bots moved to residential proxy SDKs in TVs and phones. Anubis proof-of-work challenges worked briefly, but bots now solve difficulty 5. Across 5 geo-distributed nodes with 90 cores, 14–16 cores are constantly busy rendering commits for scrapers—more CPU than all legitimate access combined.
Why it matters: Kernel.org maintainer publishes first hard numbers on AI crawler impact: 14 CPU cores wasted 24/7 rendering commits for scrapers. High signal density and strong industry resonance. Slight discount because the topic is infra/ops rather than a model or product update, but the op...