Invisible⚡Rank
back to all guides

Technical Signals · P1.1

How to let GPTBot, ClaudeBot, and PerplexityBot into your site

Joshua Mitchellby Joshua Mitchell · updated 2026-05-25 · 2 min read

tl;dr

AI assistants use named bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) to fetch pages. If your robots.txt or CDN blocks them, you're invisible. Audit those two places, allow the bots you want, and recheck with curl.

// your move

See what AI assistants say about your site.

Free, 30 seconds, no signup. We'll check all 15 signals and tell you which ones to fix first.

Run the audit
§ 01

The bots that matter right now

  • ·GPTBot — OpenAI training and browsing
  • ·ChatGPT-User — live browsing inside ChatGPT
  • ·OAI-SearchBot — ChatGPT search results
  • ·ClaudeBot / Claude-Web — Anthropic
  • ·PerplexityBot — Perplexity citations
  • ·Google-Extended — opt-in for Gemini and AI Overviews
  • ·Applebot-Extended — Apple Intelligence
§ 02

How to check what you're blocking

  1. 01Open https://yoursite.com/robots.txt in a browser.
  2. 02Scan for User-agent: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
  3. 03Any Disallow: / under one of those means you're locked out.
  4. 04If your robots.txt looks clean, check your CDN. Cloudflare has a 'Block AI Bots' toggle that overrides robots.txt and many sites flipped it on without realizing what it does.
  5. 05Run curl -A 'GPTBot/1.0' -I https://yoursite.com from a terminal. A 200 means you're good. A 403, 429, or challenge page means a firewall is in the way.
§ 03

Open the gates

  1. 01In robots.txt, remove any Disallow rules targeting GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. Or explicitly Allow: / under each.
  2. 02In Cloudflare: Security → Bots → turn off 'Block AI Bots' (or move the categories you want from Block to Allow).
  3. 03In AWS WAF / Akamai / Fastly: check your bot management rules for the same patterns.
  4. 04Wait 24 hours and re-run the curl test. Then run the InvisibleRank audit to confirm.

“If you want to be cited and referenced, yes.”

§ 04

Should you allow all of them?

If you want to be cited and referenced, yes. There's no version of this where you stay invisible to AI assistants and still show up in their answers.

The exception is if you're a paywalled publisher who licenses content separately — then block training bots (GPTBot, ClaudeBot, Google-Extended) and allow live-browsing bots (ChatGPT-User, PerplexityBot) so users can still reach your live site.

Common questions

Does blocking GPTBot stop ChatGPT from citing me?

It stops ChatGPT from training on your content and, in many cases, from reading your live pages too. If citations matter to your business, allow it.

What's the difference between GPTBot and ChatGPT-User?

GPTBot is the training crawler. ChatGPT-User fetches pages live when a user asks ChatGPT to read a URL. Treat them as different decisions.

I run on Cloudflare. Why did this happen?

Cloudflare added a one-click 'Block AI Bots' button. It's enabled by default for some plans. Check Security → Bots.

Keep reading

// your move

See what AI assistants say about your site.

Free, 30 seconds, no signup. We'll check all 15 signals and tell you which ones to fix first.

Run the audit ⚡