Missing robots.txt: why your site should serve one

A missing robots.txt is valid but usually signals a misconfigured host. See why a healthy site serves robots.txt with a 200 and what to put in it. Free check.

1462

audited sites affected

What it means

No robots.txt was found at the root of the host — the request returned a 404 or otherwise did not resolve. A missing robots.txt is technically valid: with no rules, crawlers assume everything is allowed. But a healthy, correctly configured site normally serves a robots.txt with a 200 status, and its absence often points to a misconfigured host or a routing gap worth checking.

Why it matters

Robots.txt is the first file a crawler requests before fetching anything else. Roughly 85% of sites return 200 for it and about 13% return 404, and the guidance is to make sure yours returns 200. A 404 is handled gracefully (Google proceeds as if all is allowed), so this is not a crawl-blocking emergency. The reason to fix it is that a missing robots.txt is frequently a symptom, not just a cosmetic gap: a host that cannot serve a static file at /robots.txt may also mishandle other root-level requests. More practically, robots.txt is where you declare your sitemap location, exclude crawl traps, and manage crawl budget — without one you forfeit those controls and lose the sitemap-discovery hint.

How to fix it

Create a plain-text file named robots.txt and serve it at the host root, e.g. https://example.com/robots.txt.
Confirm it returns HTTP 200 (not a 404, and not a soft 404 rendered by your app router).
Reference your sitemap so crawlers discover it, e.g. Sitemap: https://example.com/sitemap.xml.
Add only the disallow rules you actually need — do not block CSS, JS, or pages you want indexed. Disallow: with an empty value means allow everything.

When it's not a problem

A 404 robots.txt is valid per the specification, so this is advisory rather than a hard error. If your site is a small single-page project with nothing to exclude and no sitemap, the practical impact of a missing file is minimal and crawlers will index normally. The stronger reason to add one is diagnostic: it verifies the host serves root-level files correctly and gives you the standard place to declare your sitemap and any future crawl rules.

How common is it?

1462 audited sites in our corpus currently show this issue. The breakdowns below show which platforms, gatekeepers, verticals and countries are most exposed.