AI crawlers are hammering git.kernel.org with billions of requests
Creepy Crawlies
Konstantin Ryabitsev shares hard numbers: git.kernel.org gets 6M daily requests, 98% from AI scrapers. Instead of cloning repos, scrapers render every commit as HTML, generating billions of valid URLs from 922 forks of linux.git. IP bans and ASN blocks failed once bots moved to residential proxy SDKs in TVs and phones. Anubis proof-of-work challenges worked briefly, but bots now solve difficulty 5. Across 5 geo-distributed nodes with 90 cores, 14–16 cores are constantly busy rendering commits for scrapers—more CPU than all legitimate access combined.
Why it matters: Konstantin Ryabitsev disclosed that git.kernel.org gets 6 million requests a day, 98% of them from AI crawlers that render every commit as HTML, and linux.git's 922 branches can generate billions of valid URLs.