Should you block AI crawlers? Deciding what AI systems may read on your website.

The question arrives in different forms. A board member has read that AI companies train on website content without paying for it. A marketing manager wants to know why a competitor is cited in ChatGPT and the business is not. A developer has noticed unfamiliar user agents in the server logs and a rise in requests. Each conversation ends in the same place: should we block AI crawlers? The honest answer is that AI crawlers are not one thing, and a single yes or no usually gets at least part of the decision wrong.

This article separates the jobs those crawlers do, explains what the main controls actually change, and sets out a way to decide that an executive can sign off and a developer can implement in an afternoon.

The short answer.

Separate three jobs before you touch robots.txt: training, search indexing and user-initiated fetching. Most established businesses that win work or sell through their website should allow the search indexing crawlers and the user-initiated fetchers, and make a deliberate, documented choice about training. Publishers, marketplaces and any business whose content is the product face a different trade-off and may reasonably block more.

Whatever you decide, apply it by named crawler and by section of the website rather than as a blanket rule, and confirm in the server logs that the crawlers are behaving as the documentation says they should.

Three kinds of AI visitor, three different jobs.

The major AI companies now publish separate user agents for separate purposes, and each has its own robots.txt behaviour. The names matter because a rule written for one does nothing to another.

• Training crawlers collect content that may be used to train future models. OpenAI uses GPTBot, Anthropic uses ClaudeBot, Google uses Google-Extended and Common Crawl uses CCBot. Disallowing these tells the vendor that your content should not be used for training. It does not remove you from any search product.
• Search indexing crawlers build the index behind AI search answers. OpenAI uses OAI-SearchBot for ChatGPT search, Anthropic uses Claude-SearchBot and Perplexity uses PerplexityBot. OpenAI states that websites opted out of OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links. Anthropic says that disabling Claude-SearchBot may reduce visibility and accuracy in results.
• User-initiated fetchers visit a page because a person asked a question that needs it. OpenAI uses ChatGPT-User, Anthropic uses Claude-User and Perplexity uses Perplexity-User. OpenAI and Perplexity both note that robots.txt rules may not apply to these fetchers because a user requested the page. Anthropic says Claude-User does respect robots.txt.

Blocking a user-initiated fetcher rarely protects anything. It mostly means that a person who asked about your business, your product or your category receives an answer assembled from other sources, and you have no influence over which ones.

What Google-Extended does and does not do.

Google-Extended is the most misunderstood control of the group. Google documents it as the way to manage whether crawled content is used for training future Gemini models and for grounding in Gemini apps and Vertex AI. Google also states that it does not affect a website's inclusion in Google Search and is not used as a ranking signal.

AI Overviews and AI Mode sit inside Google Search and draw on the ordinary Googlebot crawl. Blocking Google-Extended does not keep you out of them. The only way to reduce how your pages appear in those features is to restrict snippets with the nosnippet or max-snippet controls, which also strips your listings in ordinary results. For nearly every business that is a poor trade. Treat Google-Extended as a training and grounding decision and nothing more.

The case for allowing.

Buyers increasingly research suppliers, platforms and agencies in AI tools before they ever reach a search results page. Being cited in those answers is becoming the equivalent of making the shortlist. Being absent does not stop the answer being written. It means the answer is built from competitors, directories, review platforms and old press coverage, and the model has no access to your own explanation of what you do.

Accuracy follows the same logic. When a model can read your service pages, case studies and articles, it has current first-party material to work from. When it cannot, it relies on third-party descriptions that are more likely to be dated or simply wrong. Comparison and procurement questions, the ones that shape a buying decision, are exactly where that gap hurts most.

We have written separately about how visibility in AI search is changing and how to measure it. The crawler decision is the precondition for both: nothing in either article helps if the relevant crawler cannot read the page.

The case for blocking, and who it applies to.

There are legitimate reasons to block, and they cluster around businesses where the content itself is what customers pay for. Subscription research, proprietary datasets, course material and editorial businesses that monetise attention lose something real when a model reproduces the substance of a page without a visit. Some organisations are also in licensing discussions with AI companies and want a clear position on record while those talks continue.

Load is a separate concern. Aggressive or poorly behaved crawlers can add meaningful server load, particularly on large catalogues with faceted navigation where every filter combination is another URL. That is a rate and cost problem rather than a visibility problem, and it is usually better solved with rate limiting and proper handling of parameter URLs than with a blanket disallow.

Even where blocking is justified, block by job. Refusing training crawlers while allowing search indexing and user-initiated fetching is a coherent position for many content businesses. Refusing everything is a decision to be invisible in a channel your customers are already using.

Decide by section, not by website.

A good robots.txt already distinguishes between parts of a website. Search result pages, cart and account areas, staging environments and internal tools should be disallowed for every crawler, AI or otherwise, and customer portals should sit behind authentication where no crawler reaches them. The AI decision is then only about the public pages.

For most businesses the public marketing pages, service pages, case studies and articles are the pages you most want AI systems to read, because they carry your own account of your work. Gated resources, downloadable reports and pages you would not want summarised out of context deserve a specific decision. Robots.txt supports path-level rules per user agent, so the same file can allow OAI-SearchBot across the website while disallowing GPTBot from a resources directory.

Robots.txt is a request, not a lock.

The documented crawlers of the major AI companies say they honour robots.txt, and in our experience the named agents generally do. It remains a voluntary convention. Unlisted bots, scrapers pretending to be browsers and tools that have never read the standard will ignore it entirely.

If a rule must be enforced rather than requested, enforcement belongs at the server or content delivery network layer: web application firewall rules, bot management and verification of published IP ranges where a vendor provides them. Cloudflare moved this into the mainstream in July 2025 when it began blocking AI crawlers by default for newly signed up domains and introduced a pay per crawl model. Enforcement has side effects, though. Over-broad rules block the legitimate fetchers you decided to allow, and a misconfigured challenge page can silently remove you from an index you wanted to be in.

Check the logs before and after.

Before changing anything, take a baseline from the server logs: which AI user agents are visiting, how often, which paths they request and whether any of them are behaving badly. After the change, confirm that the disallowed agents have stopped and the allowed ones are still fetching the pages you care about.

In analytics, watch referral traffic from AI products alongside the Search Console generative AI reporting. A drop after a robots.txt change is evidence that a rule reached further than intended. Keep the baseline and the change date together so the comparison is honest.

A practical decision record.

The most useful output of this exercise is a short document that anyone in the business can read. For each named crawler it should record:

• The decision: allow or disallow, and for which paths.
• The job that crawler does, in one line, so the reasoning survives staff changes.
• The reason for the decision and who approved it.
• How the decision is implemented: robots.txt, firewall rule or both.
• The date it was made and the date it will be reviewed.

Revisit the record at least every six months. Vendors add and retire user agents regularly. Anthropic replaced its earlier agents with the current three, and OpenAI introduced OAI-SearchBot after GPTBot was already widely blocked. A rule written for last year's names may not describe this year's behaviour.

How we can help.

We can review your current robots.txt, server logs and content delivery network rules, document what each AI user agent is actually doing on your website and prepare a decision record for approval. Where enforcement is needed we can specify the firewall or bot management rules and verify them against the vendors' published ranges.

The smallest credible pathway is a log review and a written recommendation per crawler. From there the implementation is usually a small change, and the value is in having made the decision deliberately rather than by default.

If you want to know what the AI systems can currently read on your website and whether that matches what you would choose, book a meeting and bring your robots.txt.