Back to All Insights
Web Architecture & Performance October 31, 2026 12 min read 41 reads

AI Training Signals in robots.txt: A Kenyan SME Checklist

Google, OpenAI and Microsoft are adding AI-training controls. Learn what robots.txt signals do, what they do not do and what Kenyan SMEs should check.

U

UniqueTechCamp Editorial

Lead Systems Architect • UniqueTechCamp Engineering Unit

AI Training Signals in robots.txt: A Kenyan SME Checklist

Should a Kenyan business block artificial-intelligence bots from its website, or will that make the site harder to find? A new shared preference announced by major AI and search companies on 30 October 2026 is meant to make one part of that decision simpler. It is not yet a universal, live switch, and it is not the same thing as hiding a page from search.

For an SME owner, the useful question is more specific: which public pages may be used to train generative AI, which should remain discoverable in ordinary search, and which might be used as sources in AI-generated answers? Those are separate choices. Treating them as one on/off setting can cost you visibility without protecting anything private.

What was actually announced?

On 30 October, OpenAI announced its plan to honour ↗ the shared AI-training preference, and Google announced plans to support it ↗ in robots.txt. The proposal is intended to let a site state a content-use preference across participating providers, instead of writing a separate training rule for each company. The mechanism is based on work at the Internet Engineering Task Force (IETF).

The rollout is not immediate or uniform. OpenAI says it aims to implement the signal “in the coming months” and will publish implementation details later. Google says it aims to support it in 2027; its crawler documentation describes the target as early 2027. Microsoft has also announced that it will honour the shared training preference from 1 February 2027 as it crawls, alongside separate page-level controls for how content can be used to ground Microsoft AI answers. These are provider-specific schedules, not a single launch date.

The underlying IETF document is an Internet-Draft, not a finished standard: the IETF Datatracker labels it an individual draft, says it is not endorsed by the IETF and has no formal standing in the standards process. The draft may change. So do not copy a snippet into a live site and assume every AI company will honour it today. OpenAI says its existing crawler controls continue to apply, while Google says its existing controls will work alongside the new signal.

In outline, the draft uses a policy token in an ordinary robots.txt group. The current example looks like this:

User-Agent: __usage_ai-training
Disallow: /

This is a reference to the draft’s proposed training preference, not a universal instruction to deploy now. The token expresses a preference about content use; it is not, by itself, a command that prevents a crawler from fetching a page. Providers need to implement and explain how they interpret it. Read the IETF draft status and current text ↗, then check each provider’s implementation guidance before making a production change.

Training, search and grounding are different

Before editing anything, separate the jobs people often call “AI crawling”. The word bot can describe very different uses, and a rule for one purpose does not automatically control the others.

1. Ordinary search crawling and indexing

A search engine fetches pages to understand, store and rank them for search results. A robots.txt rule can tell a compliant crawler which URLs it may fetch. Google cautions that the file is not a reliable way to remove a URL from Google Search, and it is not a security feature. A blocked URL may still be known from links elsewhere, and robots rules are not a substitute for a password or server-side access control. For pages that must remain private, do not publish them publicly and hope a crawler will stay away.

2. AI model training

The IETF working definition describes training as using an asset to modify the learned parameters of a generative AI model. In plain language, the page may contribute to how a future model is trained. That is different from a model retrieving a page to answer one person’s question. The new shared signal is specifically about this training use.

3. AI grounding and citations

Grounding means a service consults web content while preparing a response, often so the answer can cite a source or link back to it. It is not the same as training. Microsoft’s new controls explicitly separate training from grounding; its guidance says choosing a grounding restriction can reduce a brand’s visibility in Microsoft AI experiences, even though traditional Bing search eligibility is not changed. Google’s existing Google-Extended token currently covers both training and some Gemini grounding, but its new shared proposal is for training.

4. A page opened because a user asked

Some assistants fetch a page after an individual asks them to look at it. OpenAI documents a separate ChatGPT-User agent for certain user-initiated actions. It is not an automatic crawler for search, and OpenAI notes that robots.txt rules may not apply to those user-triggered requests. That is another reason not to treat one file as a complete privacy or access-control system.

What the current provider controls mean

OpenAI: separate controls for training and ChatGPT Search

OpenAI’s current crawler documentation distinguishes GPTBot, which may fetch content for foundation-model training, from OAI-SearchBot, which helps surface websites in ChatGPT Search. OpenAI says these controls are independent: a site can allow OAI-SearchBot while disallowing GPTBot. That is a useful example of how an owner can separate a training choice from a discovery choice. The shared token is additional future support, not a replacement already operating across the web.

For current settings, review OpenAI’s crawler guide for GPTBot, OAI-SearchBot and ChatGPT-User ↗. Do not assume that blocking a training bot also removes your pages from ChatGPT Search, or that allowing a search bot means you agreed to every other kind of use.

Google: Google-Extended is not Googlebot

Google-Extended is a robots.txt product token, not a separate crawler with its own network identity. Google says it can manage whether content collected by Google may be used for future Gemini model training and for certain grounding uses. Crucially, Google says Google-Extended does not affect a site’s inclusion in Google Search or serve as a Search ranking signal. Google says it plans to interpret the shared training preference alongside this existing control, not instead of it.

Check the current scope in Google’s official crawler documentation ↗. The exact combination matters: the shared training signal is not a promise that Google Search, Gemini grounding and every Google crawler will all behave identically.

Microsoft Bing: note the 2027 transition if you use legacy tags

Microsoft’s announcement describes a shared training preference in robots.txt and separate page-level preferences for Microsoft AI grounding. Microsoft says its new preferences will begin taking effect as it crawls from 1 February 2027. If your site already uses Microsoft’s older NoArchive or NoCache settings, Microsoft asks you to follow its migration steps before 1 March 2027; for some existing grounding restrictions, a temporary transition control in Bing Webmaster Tools also matters.

That deadline does not mean every Kenyan business must make a change. Microsoft says the transition applies to sites using those legacy settings. If you are unsure whether your CMS or developer added them, ask for a copy of the current robots and page-level rules before changing anything. Microsoft’s full training, grounding and migration guidance ↗ sets out the dates and conditions.

A practical decision for a Kenyan SME

There is no single correct choice for every business. A public service page or FAQ exists to help prospective customers understand what you do. You may value being found in search and being cited in an answer, while still wanting a say in whether that content is used to train a future model. A private customer file, invoice or staff record should not be made public just because you want to influence AI use; keep it behind authentication and limit access at the server or application level.

Think in page groups rather than fear-based headlines. For example, list your public service pages, public educational articles, product documentation, customer case studies and any restricted or licensed material separately. Ask who owns the rights to each item, whether it contains personal information, and what business outcome you want from search and AI answer visibility. A site-level rule may be too broad if only one section needs special treatment.

For a practical website and analytics review, a business owner can work with UniqueTechCamp’s AI Solutions Desk ↗ to map technical controls against the customer journey. The important handover is not merely “we added a line”; it should say which paths are affected, which providers’ current documentation was checked, and how search access was tested.

Six checks before you change robots.txt

  1. Find the real owner and the correct file. Confirm which person or vendor controls your domain, hosting, content-management system and deployment process. The file normally applies at a site’s root, so a staging domain, a subdomain and the production website may each have different rules. Do not edit a copy that is not the one crawlers receive.
  2. Save the current configuration. Download the existing robots.txt and any relevant page-level meta tags or HTTP headers before touching them. Keep a timestamped backup and note who approved the change. This lets your developer restore the previous state if a search crawler is blocked accidentally.
  3. Make a content-and-purpose map. For each group of URLs, record whether it is public marketing material, original editorial content, licensed material, customer-generated content or private information. Then decide separately about training and answer grounding. If an article contains a customer’s name or testimonial, review permission and your privacy process; a robots instruction does not itself establish consent or a licence.
  4. Preserve search and access rules deliberately. Compare any proposed policy token with the existing groups for Googlebot, Google-Extended, GPTBot, OAI-SearchBot and Bingbot. Keep ordinary search crawling open where you want it, unless you have a separate, tested reason to restrict it. A broad wildcard rule can have effects beyond the AI-training preference you intended.
  5. Wait for provider-specific implementation details. As of 31 October 2026, the shared signal is not a universal live control. OpenAI says “in the coming months”; Google says 2027; Microsoft gives 1 February 2027 as its date for honouring its new preferences while crawling. Mark those dates for review, but follow updated official docs before relying on a cross-provider rule. If you use Microsoft’s legacy tags, put the March migration date on the checklist.
  6. Test after publishing, then measure the outcome you care about. Ask a developer to verify the live file, check that the right crawler groups are parsed, and test a representative public URL and a restricted URL. Review Search Console, server logs and analytics for crawl errors or unexpected drops. If the business goal is leads, track qualified enquiries and booked work by source; a policy preference does not guarantee an AI citation, a click or a sale.

For a separate guide to interpreting AI-answer visibility, see the Bing AI Performance content-growth guide ↗. It explains why citations and customer enquiries are different; this article is about who may use content and for what purpose.

What this signal will not do

A robots.txt preference is not a password, copyright licence, payment contract or technical barrier. It communicates a requested use to providers that choose to honour it. It cannot make private information safe if the page is publicly accessible, and it cannot guarantee what every crawler or third party will do. Google’s general robots.txt guide ↗ explains the limits of crawler rules. For private customer or staff data, use authentication, permissions and secure storage; for rights or contractual questions, seek advice from a qualified professional.

Nor does opting out of training automatically mean your pages disappear from search. OpenAI separates GPTBot from OAI-SearchBot, and Google says Google-Extended does not change Google Search inclusion or ranking. Microsoft says its new preferences do not alter eligibility for traditional Bing results. On the other hand, restrictions on grounding may reduce visibility in some AI-generated experiences. Decide which trade-off you want, provider by provider, and do not assume that all companies use the same controls.

Frequently asked questions

Will the shared AI-training signal hide my website from Google?

Google says its planned shared training control will be separate from Google Search inclusion and ranking. Its current Google-Extended control also does not affect Google Search inclusion or ranking. Keep ordinary search crawler rules intact if you want search visibility, and verify Google’s live implementation instructions when the shared signal launches.

Can robots.txt protect invoices, customer details or internal documents?

No. Robots.txt communicates crawler preferences; it is not access control. Do not expose private files on a public URL. Use authentication and server-side permissions, and remove public access if a document should not be available to anyone with its link.

Can I use the shared token today and assume all providers will follow it?

No. It is an IETF Internet-Draft and provider implementations have different schedules. OpenAI says it aims to implement in the coming months, Google aims for 2027, and Microsoft has specified 1 February 2027 for its new preferences to take effect as pages are crawled. Check the provider’s updated documentation before relying on the token.

Should every small business opt out of AI training?

There is no universal answer. Consider the value of your public content, your licensing and privacy responsibilities, and whether you want to be discoverable or cited in AI experiences. Make the choice by content type, keep private data private, and measure actual customer enquiries rather than guessing at traffic effects.

Conclusion

The 30 October announcements make a shared AI-training preference more plausible, but they do not turn robots.txt into a universal switch. For now, the sensible move is to inventory your current rules, separate training from search and grounding, keep private material behind proper access controls, and monitor each provider’s rollout. Make a change only when you can explain which pages it covers and what outcome you expect.

Need this built for your business?

If you run a business website and want help checking crawler settings, preserving search access and measuring real enquiries, start with UniqueTechCamp’s AI Solutions Desk ↗. For an ongoing view of technical health, search traffic and conversions, see SEO Reporting ↗. You can book a consultation ↗ to scope the work with a technical team. Ask for a clear handover covering the pages changed, the provider documents used, search-access tests and a rollback plan.

Found this analysis valuable?

Share with other business owners and technology leaders.

Ready To Implement This In Your Business?

Deploy An Autonomous AI Lead Gen System Today

We engineer high-converting web applications with integrated 24/7 WhatsApp qualification bots and multi-channel follow-up drips.

24/7 AI Solutions Architect
UTC AI
Brian K. Verified
7s ago
Nairobi, Kenya

Started consultation for custom web system

Click to consult with AI Architect Open Chat →