AI Crawlers and robots.txt: Should You Allow or Block GPTBot, ClaudeBot and Google-Extended?

Look at your website's server logs today and you'll likely see visits from bots with names like GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot. These are crawlers run by AI companies. Some collect content to train AI models. Others fetch pages in real time so an assistant can answer a user's question and cite sources.

For a business that wants customers, being cited by AI assistants can mean new enquiries. For a publisher worried about content reuse, blocking training bots may make sense. The key is knowing which bot does what.

The main AI crawlers (2026)

Bot names and purposes change. Check each company's official crawler documentation before relying on any list, including this one.

Important: Google-Extended doesn't affect Google Search

Blocking Google-Extended does not remove you from Google Search or, according to Google, from AI Overviews in Search, which are part of the normal search system. It only controls certain Gemini uses. Never block Googlebot unless you want to disappear from Google entirely.

What we recommend for most businesses

If you sell services or products and want to be recommended, allow search-related AI bots. Being visible in ChatGPT search, Perplexity and Copilot can bring qualified visitors. Training bots are a personal choice; for most small businesses the benefit of being known to AI models outweighs the downside.

A robots.txt that allows everything, including AI bots, while protecting private areas:

User-agent: *
Disallow: /admin/
Disallow: /crm/

Sitemap: https://example.com/sitemap.xml

A version that allows AI search but blocks AI training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot fall under the general * rule and stay allowed.

When blocking makes sense

• You publish premium or paid content.
• Your content is your product (courses, research, original data).
• Aggressive bots are overloading a small server. In that case, rate-limiting through a service like Cloudflare may be better than a full block.

Things robots.txt can't do

• It doesn't remove content already collected.
• It relies on bots choosing to obey it. Reputable AI companies say they do; unknown scrapers may not.
• It doesn't stop people from copying text from your pages manually.

How to check what's visiting your site

Your hosting control panel usually offers access logs or visitor statistics. Search for the bot names above. Cloudflare's dashboard, if you use it, also shows AI crawler activity and lets you manage it with a few clicks. Our guide to reading website logs explains where to find them.

Frequently Asked Questions

Will blocking GPTBot stop ChatGPT from mentioning my business?

Not completely. ChatGPT search uses OAI-SearchBot and other sources. But blocking search-related bots can reduce how often you're cited with a link.

Does allowing AI bots slow my website?

Usually not noticeably. Very small hosting plans can be affected by heavy crawling; a CDN or rate limits help.

Can I allow AI bots on my blog but block other sections?

Yes. Use Disallow rules for specific folders under each bot's user-agent.

Is there a legal requirement to block AI crawlers in India?

No general requirement exists. It's a business decision. If you handle personal data, make sure private pages are behind a login anyway.

Want an AI-ready robots.txt?

Joboo Web configures robots.txt, sitemaps and llms.txt for AI search visibility. See SEO services or contact us.