Robots.txt Guide: Syntax, Examples and Mistakes That Kill Traffic

A robots.txt file is a plain-text file at the root of your domain (for example, https://example.com/robots.txt) that tells search engine crawlers which URLs they may and may not crawl. It controls crawling, not indexing: a blocked page can still show up in Google if other pages link to it. One wrong line in this file can remove an entire site from organic search.

Before reading on, check what your current file actually does with our free robots.txt and sitemap.xml checker. It shows whether the file is reachable, whether Googlebot or Bingbot is blocked, whether a Sitemap directive is present and how many URLs your sitemap contains.

How robots.txt works

Before crawling a site, Googlebot, Bingbot and most well-behaved bots fetch /robots.txt and follow the rules that apply to them. The practical details:

• One file per host and protocol. https://www.example.com/robots.txt does not cover https://shop.example.com or the http:// version.
• It must live at the root. A file at /folder/robots.txt is ignored.
• Status codes matter. A 404 means "no restrictions". Persistent 5xx errors, on the other hand, make Google slow down or stop crawling the site, so an unstable robots.txt endpoint is a real risk.
• Size limit. Google only processes the first 500 KiB of the file; anything beyond that is ignored.
• Caching. Google generally caches the file for up to 24 hours, so changes are not instant.
• It is a request, not a lock. Malicious bots ignore robots.txt. Protect private content with authentication.

Robots.txt syntax

The file is made of groups. Each group starts with one or more User-agent lines followed by Disallow and Allow rules.

User-agent: *
Disallow: /cart/
Allow: /cart/help

Sitemap: https://www.example.com/sitemap.xml

User-agent

Names the crawler the group applies to. * means all crawlers. Common tokens: Googlebot, Bingbot, Googlebot-Image, GPTBot. A crawler obeys only the most specific group that matches it: if a User-agent: Googlebot group exists, Googlebot ignores the * group entirely. This trips up a lot of sites.

Disallow and Allow

Disallow blocks paths that start with the given value; Allow carves out exceptions. When both match the same URL, Google applies the most specific rule (the longest path). On a tie, the least restrictive rule, Allow, wins.

Paths are case-sensitive: Disallow: /Search does not block /search.

Wildcards * and $

• * matches any sequence of characters.
• $ anchors the end of the URL.

User-agent: *
# block any URL with a sort parameter
Disallow: /*?sort=
# block all PDF files
Disallow: /*.pdf$

Sitemap

Points to the absolute URL of your XML sitemap. It can appear anywhere in the file, more than once, and is independent of any User-agent group. Both Google and Bing use it for discovery.

Directives Google ignores

• Crawl-delay: ignored by Google, supported by Bing.
• Noindex inside robots.txt: never officially supported, and Google stopped honoring it in September 2019.
• Host: a legacy Yandex directive with no effect in Google.

Ready-to-use robots.txt examples

WordPress

WordPress serves a virtual robots.txt when no physical file exists. A lean, safe version:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://www.example.com/sitemap_index.xml

Do not block /wp-content/ or /wp-includes/: Google needs your CSS and JavaScript to render the page.

E-commerce with faceted navigation and site search

Online stores generate thousands of filter combinations (color, size, price, sort order). Crawling them wastes crawl budget on near-duplicate pages.

User-agent: *
Disallow: /search
Disallow: /*?*sort=
Disallow: /*?*price=
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://www.example.com/sitemap.xml

Adapt the parameters to your platform. Shopify's default robots.txt already blocks cart and checkout, and changes go through the robots.txt.liquid template. Magento and WooCommerce stores usually need custom rules for layered navigation. Before blocking a facet, make sure it does not produce category pages you actually want to rank, such as "black leather boots".

Blocking AI crawlers

If you do not want your content collected for AI model training:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Google-Extended controls whether your content is used for Gemini models, but it does not affect Googlebot or your rankings in Search. Think before blocking everything: AI search crawlers such as OAI-SearchBot are what allow your site to be cited as a source in AI answers.

Staging environment

User-agent: *
Disallow: /

This stops crawling, but password-protecting staging is the better fix. The classic risk is this exact file being shipped to production during a deploy.

Robots.txt vs noindex vs canonical

Goal	Right tool	Note
Save crawl budget on useless URLs (facets, internal search)	robots.txt	Does not guarantee removal from the index
Remove a page from Google	noindex meta tag or X-Robots-Tag header	The page must be crawlable in robots.txt
Consolidate duplicate pages	rel="canonical"	A hint; Google may pick a different URL
Truly private content	Password or server-side block	robots.txt is public and protects nothing

The most common mix-up is combining Disallow with noindex. If Google cannot crawl the page, it never sees the noindex, and the URL may stay indexed with the snippet "No information is available for this page".

Robots.txt mistakes that kill organic traffic

1. A leftover Disallow: / after launch. It comes from staging or from WordPress's "Discourage search engines from indexing this site" setting. The site fades out of search over the following weeks.
2. Blocking CSS and JavaScript. Rules like Disallow: /assets/ or Disallow: /*.js$ break rendering, and Google may see the page as broken or not mobile-friendly.
3. Using robots.txt to deindex. As explained above, blocking hides the noindex from Google.
4. Overly broad prefixes. Disallow: /p blocks /products/, /pricing/ and /partners/ at once, because rules match by prefix.
5. A specific group that cancels the general one. Adding a User-agent: Googlebot group with a single rule makes Googlebot ignore every rule in the * group.
6. The server returning 5xx for robots.txt. A WAF, CDN or bot-protection layer that serves errors to Googlebot can stall crawling of the whole site.
7. Missing or wrong Sitemap URL. An http:// sitemap on an https:// site, a sitemap on an old domain, or one that returns 404.
8. Blocking URLs that redirect. If the old URL is disallowed, Google cannot see the 301 and cannot pass signals to the new page. After a migration, check your chains with the redirect checker.

How to test your robots.txt

1. Open the file in a browser (example.com/robots.txt) and confirm it returns a 200 status with the content you expect.
2. Run it through the Nord SEO robots.txt and sitemap checker. In a few seconds it tells you whether the file is reachable, whether Googlebot and Bingbot are blocked, whether the Sitemap directive exists and how many URLs the sitemap lists.
3. Check the robots.txt report in Google Search Console (Settings > robots.txt). It shows which version Google fetched, when, and any fetch or parsing errors. To test a specific URL, use URL Inspection.
4. In Bing Webmaster Tools, use the built-in robots.txt tester, especially if you rely on Crawl-delay.

Re-test every time the file changes, and add a robots.txt check to your deployment checklist.

FAQ

Does every website need a robots.txt file?

No. Without one (a 404 response), crawlers can access everything. Still, a simple file with a Sitemap line and, where relevant, rules for internal search and filters is good practice.

Does robots.txt remove a page from Google?

No. It only prevents crawling. To remove a page, keep it crawlable and add noindex, or return a 404/410. For an urgent, temporary removal, use the Removals tool in Search Console.

How long does Google take to pick up changes?

Google usually refreshes its cached copy within 24 hours. The robots.txt report in Search Console also lets you request a recrawl of the file.

Will blocking GPTBot hurt my Google rankings?

No. GPTBot belongs to OpenAI and has nothing to do with Googlebot. Blocking it does not affect Google Search, though it may reduce your content's presence in AI tools.

Want to know whether robots.txt is your only problem? Run the free 60-second SEO audit to see the main technical issues on your site.