Should You Block AI Crawlers Like GPTBot and ClaudeBot From Your Site?

GPTBot trains AI models, OAI-SearchBot powers ChatGPT search, and blocking the wrong one by mistake can erase your whole site from every AI-generated answer.

By Yannis Spanenburg · 2026-09-13 · 7 min read

Every AI crawler touching your site is either training a model, building a search index, or fetching a page for someone's live question right now. Treat all three as one bot and you either block the traffic you want or hand over training rights you never meant to give.

What are AI crawlers like GPTBot and ClaudeBot?

AI crawlers are automated bots that fetch pages from your site on behalf of an AI company. OpenAI runs GPTBot, OAI-SearchBot, and ChatGPT-User. Anthropic runs ClaudeBot, Claude-SearchBot, and Claude-User. Each bot does one job only (training, search indexing, or a live user fetch), and each has its own user agent line in your robots.txt file, though OpenAI says robots.txt rules may not apply to ChatGPT-User because a person starts each of its fetches.

Why would you want to block any of them?

The concern is consent: whether your product copy, pricing pages, and blog posts get folded into a model's training set without payment or credit. Server load rarely comes into it. A growing number of publishers also want a bargaining position: if an AI company wants to use their content, they want a deal in place before any crawl.

Which bots actually control training versus search visibility?

OpenAI splits the job three ways. GPTBot collects content for training future models, OAI-SearchBot powers what shows up when someone searches inside ChatGPT, and ChatGPT-User fetches a page live the moment a person asks about it in a chat, as laid out in OpenAI's own crawler documentation. Disallowing GPTBot does not touch the other two.

Anthropic mirrors that structure. Search Engine Journal reported that ClaudeBot handles training collection, Claude-SearchBot builds Claude's search index, and Claude-User retrieves a page the instant someone asks Claude about it. Blocking ClaudeBot alone does nothing to the other two, since each user agent needs its own line in robots.txt.

What changed with Cloudflare's default AI crawler block?

As Search Engine Land reported, Cloudflare started defaulting to blocking mixed-use AI crawlers (bots that combine search indexing with training data collection) on ad-monetized pages from September 15, 2026 onward. That default applies to new customers, new sites, and every free-tier account. Cloudflare also replaced its original Pay Per Crawl model with Pay Per Use, which pays a publisher when an AI answer cites their content instead of every time a bot fetches a page, with Ceramic.ai and You.com as launch partners.

Does blocking AI crawlers hurt your AI search visibility?

Blocking the wrong bot can cut you out of exactly the traffic you want. Block Claude-SearchBot, OAI-SearchBot, or PerplexityBot along with the training bots and you disappear from AI-generated answers entirely, as well as from future training runs. Keeping content out of training data and staying visible in AI answers are two different goals, and each needs its own line in robots.txt.

How to set up robots.txt for AI crawlers in 6 steps

1. Pull your server or CDN logs and filter user agents for GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, and PerplexityBot.
2. Check your current robots.txt for a blanket Disallow: / that blocks more bots than you meant to.
3. Disallow GPTBot, ClaudeBot, and CCBot if you want to keep your copy out of model training.
4. Allow OAI-SearchBot so your pages can appear in ChatGPT search, and ChatGPT-User so ChatGPT can fetch them when someone asks.
5. Allow Claude-SearchBot and Claude-User so Claude can index your pages and fetch them on request.
6. Leave PerplexityBot alone unless you have a specific reason to block it.

Proof of Pixel is a marketing and consulting agency for businesses that live on enquiries and orders, building the website, the tracking and the follow-up as one system, with offices in Dubai, New York, London, Antwerp and Malaysia.

Proof of Pixel's position: keep every AI search and retrieval bot allowed, because blocking every bot with 'AI' in the name makes you invisible in the tool your buyers now use to find you.

What should an ecommerce or B2B site actually do?

Most B2B and ecommerce sites should disallow the training-only bots (GPTBot, ClaudeBot, and CCBot) while explicitly allowing the search and retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and Claude-User). That combination keeps your product and service pages eligible for getting cited in AI answers while opting your copy out of raw training runs. If you already tried an llms.txt file as a shortcut, skip that step and fix these three things instead, since llms.txt does not block or allow a single crawler on its own.

This is the same crawl and access audit we run inside our technical SEO work for clients moving off a default WordPress or Shopify robots.txt they never touched.

How do you check which bots are already hitting your site?

Pull your server or CDN logs and filter user agents for GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, and PerplexityBot before you touch a single line of robots.txt. On Shopify in particular, check this against the same indexing issues we cover here, since a blanket Disallow: / meant for one bot sometimes blocks all of them by accident.

Edit your robots.txt file before you close this tab

Add or confirm four lines today: allow OAI-SearchBot and ChatGPT-User, allow Claude-SearchBot and Claude-User, disallow GPTBot and ClaudeBot if you want to opt out of training, and leave PerplexityBot alone unless you have a specific reason to block it. If you want a second pair of eyes on the rest of your crawl setup, get a free audit and we'll check it alongside your indexing and schema.