OpenAI, Google, Microsoft and Anthropic have announced that they will adopt a common mechanism allowing website owners to block their content from being harvested by artificial-intelligence training systems. The initiative, first reported by Search Engine Roundtable, aims to replace the current practice of negotiating separate opt-out arrangements with each provider.
The technical proposal introduces a “policy token” that can be embedded in a site’s robots.txt file, the long-standing directive used to guide web crawlers. Kevin Kelley, a standards lead at Anthropic, is steering the effort through the Internet Engineering Task Force, where the specification will be formalized for broad adoption.
Publishers today must convey their AI-training preferences to each company individually, a process that has grown cumbersome as the number of autonomous crawlers rises. Earlier initiatives such as the Really Simple Licensing standard have offered a way to charge visiting bots, while services like Cloudflare have begun blocking AI crawlers by default, yet no universal signal existed.
Microsoft has said it will provide granular settings that let site operators refuse the use of their pages for AI model training or for grounding, the practice of linking model outputs to verified sources. The company clarified that opting out will not remove a site from Bing’s search index, preserving visibility while restricting training use.
Google has already rolled out comparable controls that allow publishers to decide whether their material appears in AI-enhanced search features or is harvested for training, without affecting the site’s ranking in standard search results. These mechanisms operate through the same robots.txt extensions, ensuring that a single declaration can satisfy both search and AI use cases.
By consolidating opt-out preferences into a single policy token, the industry hopes to reduce administrative overhead for content creators and provide clearer signals for AI developers. If widely adopted, the standard could curb unauthorized data harvesting, influence training data composition, and set a precedent for coordinated governance of web-scale AI resources.