AI search
AI Crawlers and robots.txt for Local Business Websites — What to Allow So AI Search Can Recommend You
Which crawlers matter for search and AI answers (Googlebot, Bingbot, Google-Extended, OpenAI, Perplexity, Anthropic and others), what blocking each one does, a sensible robots.txt for local businesses, llms.txt, protecting private pages, and how crawl access affects visibility in Google, Bing, ChatGPT search and other AI assistants.
By Chamko Research Team Updated 5 min read
Quick answer
For a local business that wants to be found and recommended, allow search crawlers like Googlebot and Bingbot (Bing data also feeds some AI assistants), and generally allow AI search/answer crawlers that cite sources, so your service and location information can appear in AI answers. Some AI companies use separate user agents for training versus search or user-triggered browsing; you can allow search/answer bots while blocking training bots if you prefer. Blocking Googlebot would remove you from Google Search; blocking "Google-Extended" controls certain AI uses and doesn't remove you from Search. Keep private areas (admin, carts, internal search) disallowed.
Quick answer: For a local business that wants to be found and recommended, allow search crawlers like Googlebot and Bingbot (Bing data also feeds some AI assistants), and generally allow AI search/answer crawlers that cite sources, so your service and location information can appear in AI answers. Some AI companies use separate user agents for training versus search or user-triggered browsing; you can allow search/answer bots while blocking training bots if you prefer. Blocking Googlebot would remove you from Google Search; blocking "Google-Extended" controls certain AI uses and doesn't remove you from Search. Keep private areas (admin, carts, internal search) disallowed.
AI assistants answer questions like "best physiotherapist near Anna Nagar open Saturday". They can only use your website content if their crawlers can read it.
Key crawlers (user agents)
| Crawler | Owner | Purpose (simplified) | Typical recommendation |
|---|---|---|---|
| Googlebot | Google Search indexing | Allow | |
| Google-Extended | Controls use of content for some Google AI models (not Search ranking) | Your choice | |
| Bingbot | Microsoft | Bing Search (also used by Copilot and other services) | Allow |
| OAI-SearchBot | OpenAI | ChatGPT search results | Allow if you want visibility |
| GPTBot | OpenAI | Model training | Your choice |
| ChatGPT-User | OpenAI | User-requested page visits | Usually allow |
| PerplexityBot | Perplexity | Answer engine indexing | Allow if you want visibility |
| ClaudeBot / Claude-SearchBot / Claude-User | Anthropic | Training / search / user requests | Your choice by type |
| Applebot | Apple | Siri/Spotlight; Applebot-Extended for AI training | Allow Applebot |
| CCBot | Common Crawl | Open dataset | Your choice |
Names and behaviours change; check each provider's documentation.
A sensible robots.txt for most local businesses
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Sitemap: https://yourbusiness.in/sitemap.xml
If you want to block model training but allow search/answers:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourbusiness.in/sitemap.xml
Understand trade-offs before blocking; blocking can reduce how AI systems learn about you.
robots.txt is not security
robots.txt is a public request, not a lock. Protect private data with logins and don't publish sensitive files.
llms.txt (optional)
Some sites add an /llms.txt file — a plain-text map of important pages for AI systems. It's an emerging convention, not a standard every AI uses. It can't hurt if kept accurate.
Make content AI-friendly
- Clear service pages with prices, timings, areas (service page template).
- FAQs answering real questions.
- Consistent NAP and schema (schema, entity SEO).
- Fast, server-rendered pages (content visible without heavy JavaScript).
See Google AI Mode for local businesses and AI search.
Check your setup
- Visit yourdomain/robots.txt and read it.
- Search Console's robots.txt report and URL Inspection.
- Bing Webmaster Tools.
- Ensure staging rules ("Disallow: /") weren't left after launch (website redesign).
Example
A coaching institute's developer had added User-agent: * Disallow: / during development and never removed it. The site vanished from Google and AI tools couldn't read it. Fixing robots.txt and resubmitting the sitemap restored indexing over the next weeks.
Sources and further reading
- Google Search Central — robots.txt and Google crawlers (including Google-Extended): developers.google.com/search
- OpenAI, Anthropic, Perplexity and Apple crawler documentation pages
Decision guide: allow or block?
| Your goal | Recommended approach |
|---|---|
| Maximum visibility in Google, Bing and AI answers | Allow search and answer crawlers; allow or block training bots based on preference |
| Visibility but no model training on your content | Allow Googlebot, Bingbot, search/answer agents; block training agents like GPTBot, Google-Extended, CCBot |
| Protect paid or member-only content | Put it behind login; robots.txt alone won't protect it |
| Reduce server load from heavy bots | Rate-limit or block specific aggressive bots at hosting/CDN level |
Most local businesses want customers to find them, so the first row fits best.
How AI answers use your website
When someone asks an AI assistant about local services, the assistant may search the web, read pages and summarise. If your pages are blocked, slow, or rely on JavaScript that doesn't render, the assistant may rely on third-party sites — directories or competitors — instead of your own information. Clear, fast, crawlable pages with your services, prices, areas, hours and FAQs give AI systems accurate facts to cite (AI search).
CDN and firewall settings
Some hosting and CDN providers offer one-click "block AI bots" options. Before enabling, check whether it also blocks search or answer crawlers you want. Review firewall logs after changes to make sure Googlebot and Bingbot still reach your pages.
Key takeaways
- Never block Googlebot or Bingbot if you want to be found.
- Decide separately about training bots versus search/answer bots.
- Keep pages fast, readable and factual so AI systems describe you correctly.
Frequently asked questions
Will blocking GPTBot remove me from ChatGPT search?
OpenAI uses separate agents; check current docs — search visibility uses OAI-SearchBot.
Does Google-Extended affect my Google ranking?
Google states it doesn't affect Search inclusion or ranking.
Should small businesses block AI?
Most benefit from being visible; block training bots only if you have a specific concern.
Is llms.txt required?
No; it's optional.
How do I know if bots visit?
Check server logs or hosting analytics.
Where is my robots.txt file?
At yourdomain.com/robots.txt; most builders let you edit it in SEO settings.
Does robots.txt remove pages already indexed?
No — use noindex or removal tools for that.
Can I allow AI crawlers only on some pages?
Yes, using Allow and Disallow rules for specific paths.
Will blocking AI bots affect my Google Business Profile?
No — your profile is on Google; robots.txt affects only your website.
Should I add my sitemap to robots.txt?
Yes, a Sitemap line helps crawlers find your pages.