GPTBot, ClaudeBot, PerplexityBot: the 7 AI crawlers deciding whether you can be cited at all

You can publish the sharpest comparison page in your category, get quoted by three review sites, and still never show up in a single ChatGPT answer — because the crawler that would have read your page never got past your robots.txt. This is the most upstream failure mode in AI visibility, and it's also the most invisible one: nothing about your site looks broken. Google still indexes you fine. Your analytics look normal. The block only shows up as an absence you can't otherwise explain. Here's exactly which crawlers matter, what each one actually does, and the one distinction almost every "block AI bots" guide gets wrong.

The 7 crawlers that feed the engines you're trying to reach

PingMyBrand's free crawlability checker reads your live robots.txt and checks it against these seven user-agents — the real, current list, not a marketing approximation:

If your robots.txt disallows any of these at / — or a wildcard User-agent: * block disallows everything and none of these are individually re-allowed — that engine cannot read your site at all. Not "ranks you lower." Cannot read it, full stop. No amount of better content fixes a crawler that never arrives.

The distinction most guides skip: training bots vs. answer bots

Here's the part that trips up almost every "how to block AI bots" post: the same company often runs two separate crawlers with opposite consequences, and blocking one does nothing to the other. OpenAI is the clearest example, and it's also the one most publishers get right by accident, because both bots are already in the table above:

• GPTBot collects content OpenAI may use to train future models. Blocking it opts your content out of training — a legitimate, common choice — but has no effect on whether ChatGPT can browse your site or cite you in an answer today.
• OAI-SearchBot is the separate crawler that powers ChatGPT's search feature. OpenAI's own documentation states plainly: sites that opt out of OAI-SearchBot will not be shown in ChatGPT's search answers. This is the one that actually governs whether you can be cited.

Block GPTBot and leave OAI-SearchBot open, and you're opted out of training but still fully citable — a perfectly reasonable stance for a lot of businesses. Block both by accident with one wildcard rule, and you've silently opted out of the thing you actually wanted.

The honest gap worth naming: Anthropic runs the same kind of split for Claude — ClaudeBot for training, and a separate Claude-SearchBot that Anthropic documents as the crawler powering Claude's search and citation quality specifically. Our crawlability checker currently checks ClaudeBot only, the training bot in this pair, not yet the citation-specific one. If you want to be thorough about Claude specifically, check both by name in your own robots.txt today rather than assuming one check covers the other — we're not going to overstate what today's version of our own tool verifies. How Claude decides which brands to recommend goes deeper on what actually earns the citation once the crawler can reach you.

Two crawlers that don't crawl at all

Google-Extended and Applebot-Extended behave differently from the other five, and it's worth knowing before you assume blocking them protects your search visibility. Neither one operates its own crawl — Google and Apple's regular crawlers (Googlebot, Applebot) do the fetching either way. These two "Extended" tokens are opt-out switches layered on top: disallowing Google-Extended tells Google not to use content it already crawled for training Gemini and for grounding (feeding live content to the model at answer time), while disallowing Applebot-Extended opts out of Apple Intelligence training the same way. Blocking either one does not remove you from regular Google or Apple search results, and — the detail worth getting exactly right — blocking Google-Extended specifically does not stop your content from appearing in Google's AI Overviews, which draw on the standard search index rather than the grounding pipeline that token controls. Allowing Googlebot does not automatically allow Google-Extended; they need separate rule blocks.

The gotcha that lives outside robots.txt entirely

A clean robots.txt isn't the whole story if your site sits behind Cloudflare. Cloudflare's dashboard ships a one-click "Block AI Bots" toggle (Security → Bots) that operates at the network edge — before your robots.txt is ever read — and it's easy to have it flipped on without realizing it, whether you enabled it yourself for a different reason or it shipped on by default for your plan. An edge block overrides a permissive robots.txt entirely: your file can say "allowed" and the crawler still never reaches your server. If your crawlability checks look clean but you still suspect you're invisible, that dashboard toggle — not your robots.txt file — is the next place to look.

Check it yourself in about a minute

You don't need to read your own robots.txt by eye and cross-reference seven user-agent strings by hand. Run the free AI crawlability checker: enter your domain, and it fetches your live robots.txt, checks it against all seven crawlers above, and tells you plainly which ones are blocked and which are open — the same upstream check we recommend before any llms.txt, JSON-LD, or content work, because none of that matters if the crawler can't get in the door.

And once you've confirmed the door is open, the real question is the one a crawlability check can't answer on its own: when an engine does reach your site, does it actually name you? Run the free scan and we'll put 25 real buyer questions to ChatGPT, Claude, Gemini and Perplexity, and show you — per question, per engine — whether you were named, whether you were cited, and the exact sentence each one returned. No signup, about a minute, and it starts with the same crawlability check this whole post just walked through.