Robots.txt tester for AI bots & Googlebot
Can GPTBot, ClaudeBot, PerplexityBot and Googlebot crawl your page? Test any URL against your live robots.txt, see the exact rule that applies, and try fixes before you deploy them.
Enter any URL
We fetch the robots.txt of that site and test the exact path you entered — not just the homepage.
See every bot's verdict
20 search, AI search and AI training crawlers, each with the group and rule that decides whether it's allowed.
Edit and re-test
Change the rules in the editor or add our “block training, allow AI search” snippet and watch the results update instantly.
Frequently asked questions
How does this robots.txt tester decide if a bot is blocked?
It follows RFC 9309, the robots.txt standard Google uses: a crawler obeys the group that names its user-agent (case-insensitive); if there's none it falls back to the User-agent: * group. Within that group the longest matching Allow or Disallow path wins, and Allow wins a tie. Wildcards (*) and end-of-URL anchors ($) are supported.
Should I block GPTBot, ClaudeBot and other AI crawlers?
It depends on your goals. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) collect data to train models — blocking them is a policy choice. Search and user-fetch bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) are what let AI assistants cite and link to you. Most brands block training at most, and keep AI search allowed.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls whether your content is used to train and ground Gemini models. AI Overviews and AI Mode are part of Google Search and use Googlebot — blocking Googlebot would remove you from Google entirely.
Is robots.txt enough to keep a page out of Google?
No. robots.txt stops crawling, not indexing — a blocked URL can still be indexed from links pointing to it, without a description. To keep a page out of search results, allow crawling and use a noindex meta tag or X-Robots-Tag header.
What happens if my robots.txt returns an error?
A 404 means everything may be crawled. A 5xx server error is treated by Google as “disallow everything” for a while, which can stop crawling of your whole site — so make sure robots.txt is reliably served with a 200.
Can I test changes before publishing them?
Yes. After a test, edit the robots.txt and the path in the editor — the results table updates instantly in your browser, so you can try rules safely before deploying them.