robots.txt & sitemap.xml checker
Find out in seconds whether search engines can crawl your site: robots.txt rules for Googlebot and Bingbot, the Sitemap directive, and how many URLs your sitemap really contains.
Result
Full robots.txt, sitemap details and URL sample
Unlock the full report
Leave your email or WhatsApp — the full breakdown opens right away, and a specialist can send you a short comment on the priorities.
What this checker looks at
- robots.txt availability — the file must return HTTP 200 at the site root.
- Blocking rules — whether
Disallow: /shuts out Googlebot or Bingbot. - Sitemap directive — whether robots.txt points search engines to your sitemap.
- sitemap.xml health — HTTP status, valid XML, sitemap index vs. URL set, number of URLs.
Typical mistakes we see
A staging Disallow: / that survived the release; a sitemap listing redirected or
noindex URLs; a sitemap index pointing to files that return 404; CSS and JS blocked so Google
cannot render the page. Any one of these can cost months of organic traffic.
FAQ
Does robots.txt remove a page from Google?
No. It stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. Use noindex to remove a page.
How many URLs can one sitemap hold?
Up to 50,000 URLs or 50 MB uncompressed. Larger sites should use a sitemap index.
One check is a start. A full picture is better.
Run the free 60-second audit — 30 checks across on-page, technical and performance.