OpenAI, Google, Microsoft and Anthropic Back 1 Opt-Out Standard for AI Scraping
Updated
Updated · The Verge · Oct 30
OpenAI, Google, Microsoft and Anthropic Back 1 Opt-Out Standard for AI Scraping
3 articles · Updated · The Verge · Oct 30
Summary
Four major AI firms said they will honor a shared “policy token” in robots.txt, letting publishers set one opt-out signal across multiple AI crawlers instead of separate rules for each company.
The IETF-backed standard targets a growing publisher complaint: AI bots scraping sites for model training, prompting defenses such as Cloudflare’s default blocking and licensing efforts like RSL.
Anthropic already supports the signal, OpenAI said it will implement it in coming months, and Google and Microsoft plan support next year.
Microsoft said publishers will get finer controls to block AI training or AI grounding, while their sites can still appear in Bing search results.