Skip to main content
Analyse

Free tool

Read a robots.txt file properly

A four-line file that occasionally removes an entire section of a website from search. This fetches it as served and explains what each directive actually instructs.

Enter a website address. One page is requested, and nothing is stored.

Context

The most consequential file nobody reviews

robots.txt is checked before almost anything else a crawler does. It is also, on most websites, the file least likely to have been read since it was created — frequently by a developer during a build, sometimes by a plugin, occasionally copied from a previous project along with rules that made sense for a site that no longer exists.

The consequences are asymmetric. A rule that is too permissive costs a little crawl efficiency. A rule that is too restrictive can remove a section of the website from search entirely, and because it fails silently, it can go unnoticed for months. The single most common serious instance is a staging site’s blanket disallow surviving the move to production.

This tool shows you the file as served, lists the groups and rules it declares, and explains each one. It does not tell you which rules are wrong, because that depends entirely on what lives at those paths — a judgement about your website rather than about the file.

Reference

What each directive actually does

Several of these are routinely read as meaning something stronger or weaker than they do.

robots.txt directives and their actual effect
 What it instructsWhat it does not do
User-agentOpens a group of rules that applies to the named crawler. A crawler follows the single most specific group that matches it.Rules do not accumulate across groups. A crawler matched by a specific group ignores the wildcard group entirely, which surprises people regularly.
DisallowAsks a crawler not to fetch addresses beginning with the given path.Does not remove a page from search results, does not prevent indexing by address, and provides no access control whatsoever.
AllowCarves an exception out of a broader disallow. The more specific rule wins regardless of order.Does not invite crawling of something that was never blocked. An allow rule with no corresponding disallow does nothing.
SitemapDeclares where a sitemap can be found. It applies to the whole file rather than to any one group, and may be listed more than once.Does not guarantee the sitemap is read, valid or up to date. It only makes it discoverable.
Crawl-delayRequests a pause between requests. Honoured by some crawlers, including Bing.Ignored by Google entirely. Crawl rate for Google is managed in Search Console, not here.
NoindexNothing. It is not part of the standard and is not supported in this file.Does not prevent indexing. A noindex directive belongs in a meta tag or an HTTP header, on a page that is permitted to be crawled.

Diagnosis

Patterns worth recognising

Disallow: / against the wildcard agent

Correct on a staging environment, serious on a live one. If a site has never appeared in search since launch, this is the first thing to check and remains one of the most common single causes.

Blocked CSS or JavaScript directories

Common on older sites, inherited from advice that predates rendering. Blocking these prevents a search engine seeing the page as a visitor does, which affects how it assesses layout, mobile suitability and sometimes content itself.

A specific bot group placed after the wildcard

Order does not matter, but specificity does. A group naming Googlebot replaces the wildcard group for Googlebot entirely, so anything you intended to apply to everyone must be repeated inside it.

No sitemap declared

Not a fault — sitemaps are found by other means — but declaring one here is the most reliable route, costs a single line, and is the only place a crawler is guaranteed to look before doing anything else.

Common questions

Every disallow rule is flagged. Are they all problems?
No, and this tool deliberately does not present them as problems. Most websites legitimately hold crawlers back from internal search results, baskets, account areas, filtered listing pages and print variants. Blocking those is good practice. The rules are listed so you can read them, not so you can remove them.
We disallowed a page and it still appears in Google.
That is expected behaviour and one of the most commonly misunderstood parts of the standard. A disallow rule asks a crawler not to fetch the page. It does not ask a search engine to remove it from results. Where a blocked page is linked from elsewhere, it can still be indexed by address alone, usually with no description. To remove a page from results you must let it be crawled and then serve a noindex directive — the two instructions are contradictory when combined.
Is robots.txt a security measure?
It is the opposite of one. The file is public, it is one of the first things anyone looks at, and listing a path in it advertises that the path exists. Anything genuinely sensitive needs authentication. Using robots.txt to hide an administrative area simply publishes its location.
Should we block AI crawlers?
That is a commercial decision rather than a technical one, and it deserves more thought than it usually gets. Blocking them protects content from being used in training and in generated answers; it also removes the site from consideration when those systems assemble responses about your sector. There is no universally correct answer, but choosing by default — in either direction — is worth avoiding.
Why does the file look different when we fetch it ourselves?
Some sites serve different content by user agent, by region through a CDN, or by whether the requester appears to be a recognised search engine. This tool identifies itself honestly as an IXSEO analyser, so what you see here is what a non-search-engine visitor receives. If that differs from what your team sees, that difference is itself worth understanding.

Related

Free toolSitemap CheckerConfirm the sitemap this file declares is reachable and parses correctly.ServiceTechnical SEOCrawlability and indexation assessed across the whole site rather than one file.SituationWebsite Not RankingThe small number of causes behind a site that never appears, and how to tell them apart.

If a section disappeared from search, this is rarely the only cause

Crawl directives, canonical declarations, internal linking and indexation signals interact. A full snapshot examines them together.

Analyse your websiteDiscuss a technical review