← Back to blog
Marketing · Colin Kleine ·

AI crawlers and robots.txt: decide what to allow

AI crawler rules in robots.txt should match what your company wants to allow: search discovery, model training or a page fetch a user asked for. Identify the provider and the purpose before you change robots.txt, because one allow-or-block line can cover very different uses.

Write the policy before the technical change

You should be able to say which public pages you want buyers to find and which uses need a separate decision. Give your website owner that policy in writing, with the name of the person who can approve changes.

Keep private information behind access controls. Robots.txt states crawl preferences. It does not replace a login, and it is the wrong place to list confidential paths. Ask your website owner to test what can actually be reached, beyond what the file says.

Separate the access decisions

Provider or controlDocumented purposeDecision to record
OpenAI OAI-SearchBot and GPTBotSearch and training, handled separatelyA preference for each purpose
PerplexityBot and Perplexity-UserSearch crawling and fetches a user requestedCheck each request type on its own
GooglebotGoogle Search crawlingKeep the search access you intend
Google-ExtendedA product control for Gemini training and groundingDecide this apart from Google Search inclusion

Use the provider's own definitions

OpenAI distinguishes its search crawler from its training crawler. Perplexity documents search crawling and user-requested fetching separately. Both sources checked on 16 November 2026.

Google documents Google-Extended as a control for Gemini training and grounding that does not determine Google Search inclusion or ranking; checked on 16 November 2026. Blocking it does not switch off all Google AI search visibility, so do not describe it that way.

Put the source link next to each policy entry. When a provider changes its documentation, the owner can revisit that one decision without rewriting the whole policy.

Check the response the crawler actually receives

A correct robots.txt does not prove a request can reach the page. A firewall challenge, a login screen or a failed response can get in the way. Ask for a test of a typical public URL and keep the response as evidence.

A user-agent label alone does not tell you who sent a request. Follow the provider's verification guidance before you read anything into server logs. Label uncertain requests as uncertain instead of counting them as confirmed visits from a platform.

Keep access evidence separate from visibility

A successful fetch shows a request reached your content. It does not show that an answer cited you, that a buyer saw it or that a visit turned into an enquiry. Report each of those in its own section.

Use the AI visibility audit to record answers and the AEO checklist to judge the content a reader gets. Access problems go to the website owner; unclear claims go to the content owner.

Make changes easy to review and reverse

Ask whoever makes the change to show the current rule, the proposed rule and the public URLs they will test. Note the reason and who follows up. After the change, test the search behaviour you wanted; a syntax check alone is not enough.

Keep this work inside your wider B2B SEO plan. A bot-access task needs a specific problem and a test for done, or it drifts into an open-ended promise of AI visibility.

People also ask

Should we allow every AI crawler? No. Decide provider by provider and purpose by purpose. Someone who owns the business decision should approve the policy before the website team puts it in place.

Does allowing a crawler guarantee that our company appears in answers? No. Access is one prerequisite for being found. Review the answers themselves separately from crawler requests.

Is robots.txt enough to protect private content? No. Private material needs authentication and proper access controls. Have your website owner check what a logged-out request can actually reach.

Hiring a marketing leader to own this work? Brief Scalerr on the role or use the enquiry form below.

Hiring someone to own search?

Scalerr recruits marketing leaders for tech companies, from a first Head of Marketing to a CMO. Brief us on a marketing leadership hire, or use the form below.

Request a call

Brief us on a marketing leadership hire

By sending, you agree to be contacted by the Scalerr team about your enquiry.