Your robots.txt is probably the most powerful file on your website relative to its size – a few lines of plain text that can quietly hide your entire site from Google, or delete you from ChatGPT’s answers, without a single error message. Most sites are running one copied from a template years ago, and it’s making decisions the owner never consciously made.
This is the full expansion of Step 2 of the technical SEO strategies guide on GrowWithSakib. It pairs directly with the XML sitemap guide on GrowWithSakib, since the two files work together to steer crawlers.
What Robots.txt Is – and the One Thing It Doesn’t Do
A robots.txt file lives at yourdomain.com/robots.txt and gives instructions to web crawlers about which URLs they may request. It’s the first thing most bots check before crawling. The syntax is simple – but there’s one misunderstanding that causes more damage than any other.
The Syntax: Four Directives You Need
Robots.txt is built from a handful of directives. Here they are, in plain English:
| Directive | What It Does | Example |
|---|---|---|
| User-agent | Names which bot the rules below apply to | User-agent: Googlebot |
| Disallow | Blocks a path from being crawled | Disallow: /admin/ |
| Allow | Permits a path (overrides a broader Disallow) | Allow: /admin/public/ |
| Sitemap | Points crawlers to your XML sitemap | Sitemap: https://site.com/sitemap.xml |
A few rules that trip people up: the file is case-sensitive (Disallow, not disallow; GPTBot, not gptbot), User-agent: * means “all bots not named elsewhere,” and Disallow: with nothing after it means “allow everything.” The most dangerous line in existence is Disallow: / – a single slash that blocks your entire site.
The Mistake That Hides You From Google: Blocking CSS and JavaScript
Years ago, people blocked /wp-includes/ and theme folders to “save crawl budget.” That advice is now actively harmful. Google renders your pages like a browser to understand them, and to render, it needs your CSS and JavaScript. Block those resources and Google sees a broken, contentless page – which it then ranks accordingly.
The 2026 Question: Managing AI Bots
Here’s what’s genuinely new, and where most robots.txt files are silently out of date. It is no longer just search engines crawling you – AI companies do too. And the critical insight almost every old guide misses:
The Five Bots You Must Know (and What to Do With Each)
The brief for any 2026 robots.txt comes down to five user-agents. Here’s the decision for each, with the reasoning:
| Bot | Run By | Purpose | Recommendation |
|---|---|---|---|
| Googlebot | Classic Google Search indexing | ALWAYS ALLOW. Blocking removes you from Google entirely | |
| Google-Extended | Gemini / Vertex AI training | Your call. Google confirmed blocking it has ZERO effect on Search rankings | |
| GPTBot | OpenAI | ChatGPT model training | Your call. Block to opt out of training; no effect on ChatGPT search |
| OAI-SearchBot | OpenAI | ChatGPT search index (citations) | ALLOW if you want to appear in ChatGPT’s answers |
| PerplexityBot | Perplexity | Perplexity search index (citations) | ALLOW. Perplexity doesn’t train on you; this earns citations |
The Copy-Paste 2026 Config: Block Training, Keep Citations
This is the most common posture for a business that wants AI visibility but not to feed model training. Adjust the training decisions to your own policy:
| # === Standard search engines: always allow === User-agent: Googlebot Allow: / # === AI SEARCH bots: ALLOW (this earns citations) === User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / # === AI TRAINING bots: block to opt out of training === User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / # === Everyone else: allow, but protect private paths === User-agent: * Disallow: /wp-admin/ Disallow: /checkout/ Disallow: /cart/ Allow: /wp-admin/admin-ajax.php Sitemap: https://yourdomain.com/sitemap_index.xml |
The Honest Caveat: Robots.txt Is a Request, Not a Lock
This matters, and most guides gloss over it. Robots.txt is a voluntary standard (formalised as RFC 9309). It works because well-behaved bots choose to honour it – Googlebot, GPTBot, ClaudeBot and PerplexityBot document their user-agents and comply, and you can verify that in your server logs. But:
- Some bots ignore it entirely – Bytespider (ByteDance) and, per Cloudflare’s August 2025 findings, some of Perplexity’s stealth crawlers have been documented crawling sites that disallowed them.
- It doesn’t hide sensitive data – a disallowed path is publicly readable in your robots.txt, so never use it to “hide” private URLs. It’s a signpost, not a safe.
- Real enforcement is at the server – to actually block a non-compliant bot, use server rules, a WAF, or a service like Cloudflare’s bot management, which blocks by behaviour rather than politely asking.
How to Test Your Robots.txt
Never edit robots.txt on a live site without testing – the downside is hiding your whole site. Two checks:
- Use the robots.txt report in Google Search Console (Settings, then robots.txt). It shows the file Google last fetched, flags syntax errors, and lets you request a re-fetch after edits.
- Use the URL Inspection tool to confirm a specific important page is crawlable (‘URL is available to Google’). Do this for a key page after any robots.txt change.
- Check your server logs to verify which bots are actually visiting and being allowed or blocked – the only way to confirm real behaviour rather than intended behaviour.
For the wider set of checks these tools support, see how to use Google Search Console on GrowWithSakib.
Keeping It Current
- Audit quarterly – the AI bot landscape moves fast. OAI-SearchBot arrived in 2024, Claude-SearchBot in 2025; tokens like anthropic-ai and Claude-Web are now deprecated. A stale robots.txt makes stale decisions.
- Re-check after every CMS or plugin change – SEO plugins and platform updates can rewrite robots.txt without telling you.
- Decide on purpose – your AI-bot policy should be a deliberate brand decision, written and commented in the file, not whatever your CMS shipped with.
Remember: allowing is not appearing – letting the search bots in is necessary but not sufficient. Being crawlable doesn’t guarantee a citation; for that, see how to get cited by ChatGPT and Perplexity on GrowWithSakib and the wider generative engine optimisation guide on GrowWithSakib.
Common Robots.txt Mistakes
| Mistake | Why It Hurts | Do This Instead |
|---|---|---|
| Leaving Disallow: / after launch | Hides your entire site from Google | Remove it the moment you go live |
| Using robots.txt to hide pages | It blocks crawling, not indexing | Use a noindex meta tag instead |
| Blocking CSS and JS | Google can’t render or rank the page | Never block rendering resources |
| Blocking OAI-SearchBot by accident | Deletes you from ChatGPT citations | Allow search bots; block only training bots |
| Blocking Googlebot to ‘block AI’ | Removes you from Google Search entirely | Block Google-Extended instead; never Googlebot |
| Assuming it blocks all bots | Bytespider and others ignore it | Use a WAF/server rules for real enforcement |
| Never testing after edits | One typo can hide everything | Test in Search Console before and after |
| Copying a 2023 template | It likely blocks the AI search bots | Use a current, commented config |
Frequently Asked Questions
What is a robots.txt file and what does it do?
A robots.txt file is a plain-text file at your domain root (yourdomain.com/robots.txt) that tells web crawlers which parts of your site they may access. It uses simple directives – User-agent to name a bot, Disallow to block a path, Allow to permit one, and Sitemap to point to your XML sitemap. Its crucial limitation is that it controls crawling, not indexing: a blocked page can still be indexed if other sites link to it. It’s also voluntary, so only well-behaved bots obey it.
Does robots.txt stop a page being indexed?
No, and this is the most common misconception. Robots.txt stops compliant bots from crawling a page, but it does not stop that page being indexed. If another site links to a URL you’ve disallowed, Google can still index it, often showing ‘No information is available for this page’ because it couldn’t crawl the content. To keep a page out of Google’s index, use a noindex meta tag instead – and don’t also block it in robots.txt, or Google can’t crawl the page to see the noindex directive.
Should I block AI bots in robots.txt?
It depends which ones, and this is the key 2026 decision. AI providers run separate training and search crawlers. Blocking training bots like GPTBot, Google-Extended and ClaudeBot opts your content out of model training – a legitimate choice. But blocking search bots like OAI-SearchBot and PerplexityBot removes you from AI-generated answers and citations, which is fast becoming a major discovery channel. For most businesses that want to be found, the smart posture is to block training bots if you wish, but keep the search bots allowed.
Does blocking Google-Extended hurt my Google rankings?
No. Google has explicitly confirmed that Google-Extended is entirely separate from Googlebot, and blocking it has zero effect on your Google Search rankings or how Google crawls and indexes your site for search. Google-Extended only controls whether your content is used to train Google’s generative AI models, Gemini and Vertex AI. You can safely block Google-Extended to opt out of AI training while remaining fully visible in Google Search. Just never confuse it with Googlebot, which you should never block.
What is the difference between GPTBot and OAI-SearchBot?
They’re both OpenAI’s, but they do opposite jobs. GPTBot is the training crawler – it collects content that may be used to train OpenAI’s foundation models, with no attribution or link back. OAI-SearchBot is the search crawler – it indexes pages so ChatGPT can cite them in search answers, with a link to your site. Blocking GPTBot opts you out of training but has no effect on ChatGPT search; blocking OAI-SearchBot removes you from ChatGPT’s cited answers. They’re separate user-agents needing separate rules, which is why blocking ‘OpenAI’ as one thing is a mistake.
Why shouldn’t I block CSS and JavaScript in robots.txt?
Because Google renders your pages like a browser to understand them, and rendering requires your CSS and JavaScript. If you block those resources, Google sees a broken, contentless version of your page and ranks it accordingly. Older SEO advice to block folders like /wp-includes/ to save crawl budget is now actively harmful. The problem is worse for AI search, since most AI crawlers can’t execute JavaScript at all – a page depending on blocked resources is doubly invisible. Never disallow your CSS, JS, or the folders containing them.
How do I test my robots.txt file?
Use the robots.txt report in Google Search Console, under Settings, which shows the file Google last fetched, flags syntax errors, and lets you request a re-fetch after editing. Then use the URL Inspection tool to confirm a specific important page returns ‘URL is available to Google’. Finally, check your server logs to verify which bots are actually being allowed or blocked in practice. Always test before and after any edit to a live site, because a single wrong line can hide your entire site from search.
How often should I update my robots.txt?
Audit it at least quarterly, because the AI crawler landscape changes fast – OAI-SearchBot appeared in 2024, Claude-SearchBot in 2025, and older tokens like anthropic-ai and Claude-Web are now deprecated. Also re-check it after any CMS upgrade or SEO plugin change, since those can silently rewrite the file. Most importantly, treat your AI-bot policy as a deliberate decision to revisit, not a set-and-forget file: the defaults you inherited years ago almost certainly don’t reflect the choices you’d make today.
Key Takeaways
- Robots.txt is a plain-text file at your domain root that tells crawlers which paths they may access – powerful enough that one wrong line can hide your whole site.
- It controls crawling, NOT indexing. A blocked page can still be indexed if linked elsewhere – use a noindex tag, not Disallow, to keep a page out of Google.
- The 2026 key insight: AI providers run separate training and search bots. Block one and the other is unaffected – they need separate rules.
- Block training bots (GPTBot, Google-Extended, ClaudeBot) to opt out of model training; keep search bots (OAI-SearchBot, PerplexityBot) allowed to stay citable in AI answers.
- Blocking Google-Extended has ZERO effect on Google Search rankings – Google confirmed it. But never block Googlebot, which removes you from Google entirely.
- Never block CSS or JavaScript. Google needs them to render your pages, and most AI crawlers can’t run JS at all – blocked resources make you doubly invisible.
- Robots.txt is voluntary (RFC 9309). Bots like Bytespider ignore it; real enforcement needs server rules, a WAF, or a service like Cloudflare.
- Test every change in Google Search Console before and after. The most common disaster is a leftover ‘Disallow: /’ from staging hiding a new site at launch.
