Robots.txt Explained: What It Controls, What It Cannot Do
SEO · Published 2026-11-09 · Updated 2026-11-15 · 7 min read · By Tool Magic Lab Editorial Team
Table of contents
Every website that cares about search has a robots.txt file, and a surprising number of them are wrong in ways that matter — blocking the CSS a crawler needs to render pages, or 'protecting' sensitive directories in a file that is itself public. This guide explains what robots.txt really is, so you can use it for its actual job.
What robots.txt is
Robots.txt is a plain-text file at the root of your domain (https://example.com/robots.txt) written in the Robots Exclusion Protocol. It tells well-behaved crawlers which paths they should not fetch. Googlebot, Bingbot and legitimate AI crawlers read it before crawling. Three facts follow from the design: it is a request, not an enforcement mechanism; it is publicly readable by anyone; and it controls crawling, which is not the same as indexing.
The syntax that matters
A file is groups of rules. Each group names one or more user-agents and lists Disallow and Allow paths. The most common patterns: `User-agent: *` with `Disallow: /admin/` blocks all crawlers from the admin path; an empty Disallow means 'crawl everything'; a Sitemap: line (allowed anywhere in the file) points crawlers to your sitemap index. Path matching is prefix-based and case-sensitive, and Allow can carve exceptions out of a broader Disallow. Wildcards (*) and end-anchors ($) exist but are extensions supported by the major crawlers rather than the original standard.
Generate a correct file instead of hand-typing one: the Robots.txt Generator (/seo/robots-txt-generator) assembles groups, paths and the sitemap line without syntax slips.
The three damaging misunderstandings
1. 'Disallow hides the page from search results'
A disallowed URL can still appear in results — as a bare link with no description — if other pages link to it, because the crawler learned the URL exists without fetching it. To keep a page out of the index, use a noindex directive on the page itself and let crawlers fetch it so they can see the instruction.
2. 'Robots.txt protects private content'
It does the opposite: the file is public, so listing /secret-admin/ in robots.txt advertises the path to every curious reader while stopping only the crawlers who were never the threat. Private content needs authentication, full stop.
3. 'Block everything during development, remove it at launch'
Staging files have a way of surviving launches. The classic post-launch traffic collapse is a leftover `Disallow: /`. If you must block staging, password-protect it instead, and add 'check robots.txt' to your launch checklist either way.
AI crawlers and your policy
Robots.txt is also where you state a policy for AI crawlers such as GPTBot, ClaudeBot and PerplexityBot. Blocking them opts your content out of their training/retrieval behaviour as far as they honour the protocol; allowing them keeps your content eligible to be cited in AI answers. There is no universally right answer — but make it a decision, not an accident. Tool Magic Lab's own robots.txt allows search and AI crawlers on public pages and disallows account, admin and search-result paths for everyone.
Frequently asked questions
**Where must the file live?** At the domain root, over the same protocol as the site, named exactly robots.txt. Subdirectory copies are ignored.
**Can I block a single page?** Yes, with a Disallow for its exact path — but remember that controls crawling, not indexing.
**How do I test a change?** Use the URL inspection and robots testing tools in Search Console before and after deploying; one character can change everything.
Written by the Tool Magic Lab Editorial Team. Tool Magic Lab content is produced with AI assistance and editorially reviewed for accuracy — see our AI Usage Disclosure and Editorial Policy.
