Google to Support Shared AI Training Control in robots.txt by 2027
Google has announced plans to support a shared industry-wide AI training control in robots.txt, allowing website owners to express their preferences across multiple AI providers in a single step. This initiative is grounded in the work of the AIPREF working group at the Internet Engineering Task Force (IETF) and builds on Google-Extended, the company's existing AI training control launched in 2023. The new mechanism is expected to be fully supported by Google and other AI providers in 2027.
Key points
- Google plans to support a shared 'policy token' in robots.txt that lets website owners opt out of AI training across multiple providers simultaneously, rather than managing separate directives for each company.
- When a website uses the shared policy token to opt out and Google-Extended is not explicitly specified, Google will interpret this as a disallow instruction for Google-Extended, its own AI training crawler introduced in 2023.
- Existing company-specific controls such as Google-Extended will continue to function alongside the new shared signal, giving publishers the ability to set broad preferences while fine-tuning instructions per provider.
- Websites that specifically allow Google-Extended are eligible to appear in AI-generated responses and receive clicks through grounding in Gemini Apps, creating a direct link between robots.txt configuration and AI-driven traffic.
- The proposal is consistent with the EU Code of Practice for General Purpose AI Model Providers under the EU AI Act, which formally recognizes robots.txt as a primary method for machine-readable publisher preferences.
- Google aims to implement this new control alongside other AI providers in 2027, with updates to be provided as development progresses through the IETF AIPREF working group.
Analysis
The announcement marks a significant shift in how the web publishing ecosystem manages relationships with AI systems. Until now, website owners who wanted to restrict AI training had to research and configure separate crawl directives for every AI provider, a fragmented and time-consuming process. The introduction of a shared policy token in robots.txt dramatically simplifies this by offering a single, standardized opt-out mechanism that multiple providers agree to honor.
The interplay between the shared token and Google-Extended reveals an important hierarchy of instructions. Google's interpretation is explicit: if a site uses the shared opt-out token without specifying Google-Extended, the company treats it as a Google-Extended disallow. This means publishers must understand that broad opt-outs will affect their eligibility for AI-generated responses in Gemini Apps, which now represent a meaningful source of referral traffic. The decision to opt out carries concrete visibility trade-offs that were not present in the pre-AI search era.
The IETF AIPREF working group context is critical for understanding the long-term trajectory of this change. By anchoring the proposal in a multistakeholder standards body, Google and other participating companies are signaling that this is intended to become a durable, cross-industry norm rather than a proprietary solution. The draft has incorporated feedback from publishers, civil society organizations, and regulators, which strengthens its legitimacy and increases the likelihood of broad adoption across the AI industry.
The alignment with the EU AI Act's Code of Practice for General Purpose AI Model Providers adds a regulatory dimension that marketing and SEO teams cannot ignore. European regulators have formally recognized robots.txt as a machine-readable preference mechanism, which means that how a website configures this file may have compliance implications beyond mere crawl management. Companies operating in the EU should treat robots.txt configuration as part of their broader AI governance and legal review processes.
For agencies managing large publisher portfolios, the 2027 timeline provides a window of preparation but also introduces risk if clients delay action. The new mechanism is being rolled out progressively as more AI providers adopt it, meaning the scope of the shared opt-out will expand over time. Clients who set broad opt-outs today may find that those preferences apply to an increasing number of AI systems, while those who actively allow Google-Extended may benefit from growing AI-driven discovery surfaces.
What to do
- Audit your current robots.txt files immediately to identify any existing Google-Extended directives and assess whether they reflect the organization's intended stance on AI training, before the shared token standard arrives in 2027.
- Decide explicitly whether to opt in or opt out of AI training at the organizational level, since websites that allow Google-Extended gain eligibility for AI-generated responses in Gemini Apps, which represents a measurable traffic and visibility opportunity.
- Avoid setting a broad shared opt-out token without first reviewing the Google-Extended implications, because Google will interpret an unspecified shared opt-out as a Google-Extended disallow, potentially removing the site from AI-driven response surfaces.
- Monitor the IETF AIPREF working group publications and Google Search Central updates regularly to stay ahead of vocabulary changes and additional participating providers, so that robots.txt configurations can be updated proactively rather than reactively.
- For clients operating under EU jurisdiction, involve legal and compliance teams in the robots.txt review process, given that the EU AI Act's Code of Practice formally recognizes this file as a compliance-relevant preference signal for AI model providers.
- Develop a documented AI crawl policy for each client or property that defines allowed and disallowed AI training use cases, so that when the shared token is officially supported in 2027, implementation can be executed quickly and consistently across all managed sites.
Websites that explicitly allow Google-Extended remain eligible to appear in AI-generated responses and receive traffic via grounding in Gemini Apps, making the opt-in decision a direct factor in AI-driven visibility. Publishers who fail to review and update their robots.txt configuration risk being inadvertently excluded from AI training pipelines or, conversely, included without their consent once the new standard goes live.