robots.txt and crawler instructions
robots.txt is the first file most crawlers request. It tells them which parts of the site they may fetch, and points them at your sitemap.
What it is
A plain text file at the root of the domain, always at /robots.txt, listing rules per user agent. It's a convention that well-behaved crawlers follow voluntarily.
It controls crawling, not indexing. A page blocked in robots.txt can still appear in results if other sites link to it, because the crawler isn't allowed to fetch the page and read the noindex instruction inside it.
Why it matters
A missing robots.txt is rarely fatal, but a wrong one is catastrophic. A stray Disallow: / left over from a staging site removes an entire website from search.
It's also where you declare your sitemap, which is the most reliable way to have new pages discovered quickly.
How crawlers read the rules
Rules are grouped by user agent. A crawler looks for the group that names it most specifically and follows only that group, ignoring the rest, including the wildcard group for everyone. So if you add a group for Googlebot, any general rules you also want it to follow have to be repeated inside that group.
Within a group, Google and Bing apply the most specific matching rule, meaning the longest path, and when an Allow and a Disallow match equally, Allow wins. Both support * as a wildcard and $ to mark the end of an address, so 'Disallow: /*.pdf$' blocks PDF files only. Paths are case-sensitive, and each rule matches the start of the address, so 'Disallow: /cart' also blocks /cartography.
What to block, and what to leave open
On a typical small business site there is very little that needs blocking. Reasonable candidates are internal search results, shopping cart and checkout pages, and administration areas, which waste crawl time and have nothing to offer a searcher. Everything a visitor might want to find should stay open, along with the CSS, JavaScript and images the pages need to render properly.
Remember that the file is public, and that blocking is not the same as hiding. Keep private material behind a login rather than listing it in robots.txt. After any change, use the robots.txt report in Google Search Console to confirm Google has fetched the new version, and test a few important addresses to make sure they are still allowed.
Testing before and after a change
Because one character can block a whole site, treat edits to robots.txt like edits to anything else that can take a business offline. Keep a copy of the current version, make the change, and test a list of important addresses against the new rules before publishing. Google Search Console's robots.txt report shows the version Google last fetched and any lines it couldn't understand.
How the file is served matters as well as what it says. A missing file, returning 404, is read as permission to crawl everything. A server error on the file is different: Google may pause crawling the whole site until it can read the rules again. If the site is behind a firewall or bot protection, make sure it doesn't challenge requests for robots.txt itself.
Some instructions people expect to work don't. Google ignores the crawl-delay directive, and treats a noindex line in robots.txt as unsupported. Very large files are cut off at 500 kilobytes, which no small business site should come near. If a rule seems to be ignored, the usual explanation is a more specific group or a longer rule that matches first.
Finally, keep robots.txt short and readable, with a comment explaining any rule that isn't obvious. The person who opens the file in two years' time, possibly during a redesign, needs to understand why each line is there before deciding whether to keep it. A file nobody understands is a file nobody dares to fix.
How to fix it
- Create a robots.txt at the root of the domain, even if it only allows everything.
- Add a Sitemap line with the absolute address of your sitemap.
- Use noindex meta tags, not robots.txt, when the goal is to keep a page out of results.
- Check it after every site migration. Blocking everything is the single most common launch-day mistake.
- Keep a dated copy of the file before every edit, so a mistake can be undone in seconds rather than rebuilt from memory.
What good looks like
User-agent: *
Allow: /
Disallow: /cart
Disallow: /admin
Sitemap: https://example.com.au/sitemap.xmlCommon mistakes
- Shipping the staging site's Disallow: / to production.
- Blocking CSS and JavaScript, which stops search engines rendering the page as a visitor sees it.
- Assuming robots.txt keeps a page private. It's a public file and a list of the directories you would rather people did not visit.
- Adding a rule for one crawler and forgetting that it now ignores every rule in the general group.
Check your own site. ppclift checks this automatically on every page it audits. Run a free audit and see every check on one page, with no account required.
Doing this for a client? This is one of 15 checks. ppclift is SEO lead generation software that audits every local business in a suburb at once, so you can find SEO clients who already need this work. Free plan, no card.