What is robots.txt and how do you use it?
robots.txt is a plain text file that tells search engine crawlers which parts of your website they are allowed to crawl. It sits at a fixed address, yourdomain.com/robots.txt, and every major search engine checks it before crawling a site. The file is not a ranking factor in itself, but it shapes how Google discovers and understands your pages, so it belongs in every technical SEO checklist.
When Googlebot, Bingbot or another crawler arrives at your site it looks for robots.txt before fetching anything else. The file contains simple rules that pair a user agent (the crawler) with the paths it may or may not crawl. Most small business sites only need a few lines.
How robots.txt works
The three directives you will use most of the time are:
- User-agent: names the crawler a rule applies to. User-agent: * covers all search engines.
- Disallow: tells the crawler which paths to skip. It is matched against the start of a URL, so Disallow: /admin/ blocks everything under that folder.
- Allow: an exception that lets a crawler back into a specific path inside a disallowed folder.
A minimal file either blocks nothing or blocks only the folders nobody should see, such as admin areas, checkout steps or internal search results. You can also point crawlers to your XML sitemap with a Sitemap: line at the end. Google's own documentation at https://developers.google.com/search/docs/crawling-indexing/robots/intro is the definitive reference for the full syntax.
What robots.txt does not do
The biggest misunderstanding is that robots.txt removes pages from Google. It does not. A disallowed page can still be indexed if other pages link to it, because crawlers see those links even when they cannot fetch the page itself. If you want a page out of the search results, use a noindex meta tag instead. Robots.txt controls crawl access, and noindex controls indexing; the two are not interchangeable.
It is also not a security control. The file is public and anyone can read it, so it reveals exactly which folders you would rather hide. Anything sensitive belongs behind a login, not behind a Disallow rule.
Common mistakes
Most robots.txt problems come from a handful of recurring errors:
- Blocking CSS and JavaScript, which stops Google from rendering the page properly and can drag down how your site is assessed.
- Accidentally publishing Disallow: /, which tells crawlers the whole site is off limits.
- Disallowing pages you actually want ranked, such as product categories, then wondering why they never appear.
- Never testing changes. Google Search Console's URL inspection tool shows how Googlebot sees a page, and our free SEO report at https://victory.digital/run-a-seo-report flags crawl and technical issues across your site.
Getting it right
Keep the file small and specific. Allow what crawlers need (content, CSS and images), disallow only what serves no search purpose and use noindex for anything you want removed from the index. Treat robots.txt and your XML sitemap as a pair: the sitemap invites crawlers in, and robots.txt sets the boundaries. Our guide to sitemaps at https://victory.digital/what-is-a-sitemap covers the invitation side, and the guide to 301 redirects at https://victory.digital/what-is-a-301-redirect explains how to move pages without losing their value.
If you would like to know how Googlebot actually sees your site, our SEO team plans and runs campaigns for businesses across the UK. See the technical SEO services at https://victory.digital/technical-seo, or start with a free SEO report at https://victory.digital/run-a-seo-report.
What are your Feelings?
Share your reaction
Click a reaction to share how useful this article was.