Robots.txt Explained: What It Does and How to Write One

Category: Technical SEO

Every website has a small plain text file that most owners never look at, and it decides which parts of the site search engines and AI crawlers are allowed to visit. That file is robots.txt. Get it right and Googlebot spends its time on the pages that earn you money. Get it wrong, and a single line left over from a website rebuild can quietly stop Google from crawling your entire site. This guide explains what a robots.txt file is, how the rules work, what a good robots.txt example looks like for a typical Australian business site, how to handle AI crawlers like GPTBot and ClaudeBot, and how to test your file before it does any damage.

What Is a Robots.txt File?

A robots.txt file is a plain text file that sits at the root of your domain (for example https://example.com.au/robots.txt) and tells crawlers which URLs they may and may not request. It follows the Robots Exclusion Protocol, which started as an informal convention in 1994 and was formally published as an internet standard, RFC 9309, in September 2022.

When a well-behaved crawler arrives at your site, it requests /robots.txt first, reads the rules that apply to it, and then crawls accordingly. You can look at any site’s file by adding /robots.txt to the end of its homepage address, including your own.

Three facts about robots.txt explain most of the mistakes we see:

  • It controls crawling, not indexing. Google’s own documentation says robots.txt is not a mechanism for keeping a web page out of Google. More on this below.
  • It is a request, not a lock. The standard itself states that these rules are not a form of access authorisation. Reputable crawlers follow it; scrapers and bad bots can ignore it.
  • It is public. Anyone can read your robots.txt, so listing a “secret” folder in it simply advertises where the secret folder is.

Why Robots.txt Matters for SEO

Search engines do not have unlimited time for your site. Every request Googlebot spends on an internal search results page, a filter combination or a login screen is a request not spent on a service page, a product or a new blog post. A well-written robots.txt file keeps crawlers focused on the URLs that matter, which is the same crawl efficiency problem we covered in our guide to HTTP status codes and crawl budget.

For a small service business with a few dozen pages, robots.txt is mostly about not doing harm: making sure nothing important is blocked. For ecommerce stores with faceted navigation, sorting parameters and thousands of filter URLs, it becomes one of the main tools for controlling crawl waste.

It also now matters for AI search. The same file is where you decide whether crawlers from OpenAI, Anthropic, Google’s Gemini and Perplexity can read your content, which affects whether your business can be surfaced or cited in AI answers.

Robots.txt Syntax: The Rules Explained

A robots.txt file is made of groups. Each group starts with one or more User-agent lines naming the crawler it applies to, followed by the rules for that crawler. There are only four directives you need to know.

User-agent

Names the crawler the rules apply to. User-agent: * means every crawler. User-agent: Googlebot means only Google’s main crawler. A crawler follows the most specific group that matches its name and ignores the rest, so if you create a group for Googlebot, Googlebot will ignore everything in the * group.

Disallow

Tells the crawler not to request URLs that start with the given path. Disallow: /wp-admin/ blocks everything in that folder. Disallow: / blocks the entire site. An empty Disallow: blocks nothing.

Allow

Overrides a Disallow for a more specific path. The classic WordPress example is blocking /wp-admin/ but allowing /wp-admin/admin-ajax.php, which some themes and plugins need for front-end features.

Sitemap

Points crawlers to your XML sitemap with a full absolute URL, for example Sitemap: https://example.com.au/sitemap_index.xml. It is not tied to any user-agent group and can sit anywhere in the file. Listing the sitemap here is a simple way to make sure every crawler can find it, not just the ones you have submitted it to.

Wildcards and Matching

Google supports two pattern characters: * matches any sequence of characters and $ marks the end of a URL. So Disallow: /*?sort= blocks any URL containing a sort parameter, and Disallow: /*.pdf$ blocks URLs ending in .pdf. Paths are case-sensitive: /Blog/ and /blog/ are different rules.

When rules conflict, Google uses the most specific rule, meaning the one with the longest matching path. If an Allow and a Disallow are equally specific, Google applies the least restrictive one, which is the Allow.

Directives Google Ignores

A Robots.txt Example for a Typical Business Website

Most sites need far less in their robots.txt than people expect. Here is a sensible starting point for a WordPress business site:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com.au/sitemap_index.xml

That blocks the admin area and internal site search results (which create endless thin, duplicate URLs), keeps the AJAX endpoint open, and points crawlers to the sitemap. Everything else is crawlable.

An ecommerce store with filters might add rules for parameter URLs that create near-duplicate collection pages:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /*?sort_by=
Disallow: /*?filter.

Sitemap: https://example.com.au/sitemap.xml

Treat these as patterns, not copy-and-paste templates. Before you block any parameter, check whether those URLs earn search traffic. Some filtered pages, such as a colour or material filter on a jewellery collection, are worth keeping crawlable and building out as proper landing pages. Our collection page SEO guide covers how to make that call.

What Not to Put in Robots.txt

  • CSS and JavaScript folders. Google renders pages like a browser. Block the files it needs and it may see a broken page.
  • Pages you want removed from Google. Use noindex instead (see the next section).
  • Private or sensitive URLs. The file is public and offers no protection. Use authentication.
  • Every low-value page on the site. Thin pages are better fixed, consolidated or noindexed than hidden from crawlers.

Robots.txt vs Noindex: Which One Do You Need?

This is the most common robots.txt mistake we find in audits, and it usually makes the problem worse.

A Disallow rule stops Google from crawling a page. It does not stop Google from indexing the URL. If other pages link to a blocked URL, Google can still index it without a description, so it can appear in results as a bare link with no snippet.

A noindex meta tag (or X-Robots-Tag HTTP header) tells Google not to show the page in search results. But Google can only see that tag if it is allowed to crawl the page. Block the page in robots.txt and add noindex, and Google never reads the noindex.

The rule of thumb:

  • Want a page out of search results? Use noindex and leave it crawlable until it drops out.
  • Want to stop crawlers wasting time on a section that should never be in search? Use robots.txt.
  • Want something genuinely private? Password protect it. Neither robots.txt nor noindex is a security measure.

Robots.txt and AI Crawlers

Robots.txt is now the main lever for deciding how AI companies use your content, and the decision is more nuanced than “block AI or allow AI”. Most AI companies run separate crawlers for training models and for powering live search answers, and you can treat them differently.

  • OpenAI: GPTBot crawls content that may be used to train its models. OAI-SearchBot is used for surfacing sites in ChatGPT search. ChatGPT-User fetches pages when a user asks ChatGPT to, and OpenAI notes robots.txt rules may not apply to it.
  • Anthropic: ClaudeBot collects training data, Claude-SearchBot supports search, and Claude-User fetches pages for user questions. Anthropic states its bots honour robots.txt directives.
  • Google: Google-Extended controls whether your content can be used for training future Gemini models and for grounding. Google states it does not affect inclusion or ranking in Google Search, so blocking it does not hurt your organic rankings.
  • Perplexity: PerplexityBot surfaces sites in Perplexity’s search results and, according to Perplexity, is not used to train foundation models. Perplexity-User fetches pages on a user’s request and generally ignores robots.txt.

For most Australian businesses that want to be found, our default advice is to allow the search and user-fetch crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot), because blocking them removes you from the AI answers your customers are increasingly reading. Whether you allow the training crawlers (GPTBot, ClaudeBot, Google-Extended) is a business decision about your content, not an SEO requirement.

A file that allows AI search but opts out of model training looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com.au/sitemap_index.xml

One check people miss: robots.txt is not the only thing that can block AI crawlers. Some CDN and security settings block known AI bots at the firewall, regardless of what your robots.txt says. If AI visibility matters to you, check both. Our AI SEO and Generative Engine Optimisation service audits this as standard, and our guide to SEO vs AEO vs GEO explains why it matters.

How to Create or Edit Your Robots.txt File

How you edit the file depends on your platform. You rarely need to write one from scratch, and you rarely need a robots.txt generator: most platforms create a sensible default for you.

  • WordPress: WordPress serves a virtual robots.txt by default. SEO plugins such as Rank Math and Yoast let you edit it from the dashboard (Rank Math: General Settings, Edit robots.txt). If a physical robots.txt file exists in the site root, it overrides the virtual one and the plugin editor will not apply.
  • Shopify: Shopify generates a default file that already blocks cart, checkout and account pages. To customise it, add a robots.txt.liquid template to your theme.
  • Wix and Squarespace: both generate the file automatically. Wix lets you edit it in the SEO settings; Squarespace manages it for you with limited control.
  • Custom sites: create a UTF-8 plain text file named robots.txt and upload it to the root of the domain.

Two technical rules apply everywhere. The file must sit at the root of the host, and it only covers the host, protocol and port it is served from, so shop.example.com.au needs its own file. Google also reads only the first 500 KiB of the file and ignores anything after that, which only matters on very large, rule-heavy files.

How to Test and Check Your Robots.txt

Never publish a robots.txt change without testing it. A single misplaced slash can block your whole site.

  1. Read the live file. Open yourdomain.com.au/robots.txt in a browser and confirm it is the version you expect, on the right domain.
  2. Use the robots.txt report in Search Console. Under Settings, the robots.txt report shows the files Google has found for your property, when it last fetched them, and any errors or warnings. It is the closest thing to an official robots.txt checker. Our guide to Google Search Console covers where to find it.
  3. Inspect important URLs. Run your key pages through the URL Inspection tool and confirm crawling is allowed.
  4. Crawl the site. A site crawl that respects robots.txt will show you every URL your rules block. Review that list for anything that should be crawlable.

Keep in mind that Google generally caches robots.txt for up to 24 hours, so a fix may not be picked up immediately.

Common Robots.txt Mistakes We Find in Audits

The Staging Block That Went Live

Developers often add Disallow: / to a staging site so it does not get indexed, and the file gets copied to production at launch. The site goes live, rankings slide over the following weeks, and nobody connects it to the rebuild. Check robots.txt on launch day, every time. On WordPress, also check that “Discourage search engines from indexing this site” is unticked under Settings, Reading.

Blocking Resources Google Needs to Render

Old advice said to block /wp-includes/, theme folders or script directories. Google needs those CSS and JavaScript files to render the page, so blocking them can make it misread your layout and content.

Using Disallow to Remove Pages From Search

Covered above, and worth repeating: if a page is already indexed and you block it, Google can no longer see the noindex or the updated content, and the URL can stay in results.

A Robots.txt That Errors

How the server responds to the robots.txt request matters as much as what is in it. Google treats a 404 as if no robots.txt exists and crawls everything. A 5xx server error is more serious: Google stops crawling the site for the first 12 hours, then falls back to the last cached version. A firewall rule that returns errors on /robots.txt can pause crawling of the whole site.

Accidental Wildcards and Case Errors

Disallow: /p blocks every URL beginning with /p, including /products/ and /pricing/. Paths are case-sensitive, so a rule written for /Blog/ does nothing for /blog/. Always test new rules against real URLs.

Robots.txt Checklist

  • The file loads at /robots.txt on every hostname you use and returns a 200 status.
  • No Disallow: / under User-agent: * on the live site.
  • Admin, cart, checkout, account and internal search URLs are blocked where relevant.
  • CSS, JavaScript and image folders are crawlable.
  • Pages you want out of search use noindex, not Disallow.
  • The XML sitemap is listed with a full URL.
  • AI crawler rules reflect a deliberate decision, and your CDN or firewall is not overriding them.
  • Search Console’s robots.txt report shows no errors.

Robots.txt is one of the first things we check in a technical SEO audit, because it takes minutes to review and the cost of getting it wrong is so high.

Frequently Asked Questions

What is a robots.txt file?

A robots.txt file is a plain text file at the root of a website that tells search engine and AI crawlers which URLs they may and may not request. It manages crawler traffic so bots focus on the pages that matter. It does not control what appears in Google’s index and it is not a security measure, because crawlers follow it voluntarily and anyone can read it.

What does a robots.txt file contain?

A robots.txt file contains groups of rules, each starting with a User-agent line that names the crawler, followed by Disallow and Allow lines listing the paths that crawler may or may not visit. Most files also include a Sitemap line pointing to the XML sitemap. Comments start with a hash symbol. A typical business site needs only a handful of lines.

How do I find my robots.txt file?

You can find your robots.txt file by adding /robots.txt to the end of your homepage address, for example example.com.au/robots.txt. If nothing loads, your site does not have one, which means crawlers can access everything. In Google Search Console, the robots.txt report under Settings also shows the file Google has fetched and when it last checked it.

How do I fix a “blocked by robots.txt” error?

To fix a “blocked by robots.txt” error, first decide whether the page should be crawled. If it should, find the Disallow rule matching that URL, remove or narrow it, and confirm the change with the URL Inspection tool. If the page should not be in search at all, remove the block, add a noindex tag instead, and let Google recrawl it so it drops out properly.

Is robots.txt legally binding?

Robots.txt is not legally binding in itself: it is a voluntary standard, and the RFC that defines it states the rules are not a form of access authorisation. Reputable search engines and most major AI companies choose to follow it, but scrapers can ignore it. If content must be protected, use authentication, and take legal advice on any specific scraping or copyright concern.

Does robots.txt stop a page from appearing in Google?

Robots.txt does not reliably stop a page from appearing in Google. It blocks crawling, but Google can still index a blocked URL without a description if other pages link to it. To keep a page out of search results, use a noindex meta tag or X-Robots-Tag header and leave the page crawlable so Google can read the instruction.

Should I block AI crawlers in robots.txt?

Whether you should block AI crawlers in robots.txt depends on which crawlers you mean. Blocking search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot can remove your business from AI search answers. Blocking training crawlers like GPTBot, ClaudeBot and Google-Extended opts your content out of model training without affecting Google Search rankings. Most businesses that want visibility should allow the search crawlers.

Do I need a robots.txt file?

You do not strictly need a robots.txt file: without one, crawlers simply treat the whole site as open. Most sites still benefit from one, because it keeps crawlers out of admin, cart and internal search URLs and points them to your sitemap. Platforms like WordPress, Shopify and Wix create a sensible default automatically.

Not Sure What Your Robots.txt Is Doing?

Robots.txt is a tiny file with an outsized effect: it can protect your crawl budget, decide whether you appear in AI answers, or quietly block your best pages. If you are not sure what yours is doing, or you have just launched a new site, it is worth a five-minute check. Rank Haus reviews robots.txt, sitemaps, AI crawler access and indexing as part of every technical SEO engagement. Get in touch and we will take a look.

Related reading: What Is Technical SEO? and HTTP Status Codes for SEO.

Our Services

Similar Posts