robots.txt and crawler instructions
robots.txt is the first file most crawlers request. It tells them which parts of the site they may fetch, and points them at your sitemap.
What it is
A plain text file at the root of the domain, always at /robots.txt, listing rules per user agent. It is a convention that well-behaved crawlers follow voluntarily.
It controls crawling, not indexing. A page blocked in robots.txt can still appear in results if other sites link to it, because the crawler is not allowed to fetch the page and read the noindex instruction inside it.
Why it matters
A missing robots.txt is rarely fatal, but a wrong one is catastrophic. A stray Disallow: / left over from a staging site removes an entire website from search.
It is also where you declare your sitemap, which is the most reliable way to have new pages discovered quickly.
How to fix it
- Create a robots.txt at the root of the domain, even if it only allows everything.
- Add a Sitemap line with the absolute address of your sitemap.
- Use noindex meta tags, not robots.txt, when the goal is to keep a page out of results.
- Check it after every site migration. Blocking everything is the single most common launch-day mistake.
What good looks like
User-agent: *
Allow: /
Disallow: /cart
Disallow: /admin
Sitemap: https://example.com.au/sitemap.xmlCommon mistakes
- Shipping the staging site's Disallow: / to production.
- Blocking CSS and JavaScript, which stops search engines rendering the page as a visitor sees it.
- Assuming robots.txt keeps a page private. It is a public file and a list of the directories you would rather people did not visit.
Check your own site. ppclift checks this automatically on every page it audits. Run a free audit and see every check on one page, with no account required.