Missing robots.txt: why your site should serve one

What it means

No robots.txt was found at the root of the host — the request returned a 404 or otherwise did not resolve. A missing robots.txt is technically valid: with no rules, crawlers assume everything is allowed. But a healthy, correctly configured site normally serves a robots.txt with a 200 status, and its absence often points to a misconfigured host or a routing gap worth checking.

Why it matters

Robots.txt is the first file a crawler requests before fetching anything else. Roughly 85% of sites return 200 for it and about 13% return 404, and the guidance is to make sure yours returns 200. A 404 is handled gracefully (Google proceeds as if all is allowed), so this is not a crawl-blocking emergency. The reason to fix it is that a missing robots.txt is frequently a symptom, not just a cosmetic gap: a host that cannot serve a static file at /robots.txt may also mishandle other root-level requests. More practically, robots.txt is where you declare your sitemap location, exclude crawl traps, and manage crawl budget — without one you forfeit those controls and lose the sitemap-discovery hint.

How to fix it

1. Create a plain-text file named robots.txt and serve it at the host root, e.g. https://example.com/robots.txt.
2. Confirm it returns HTTP 200 (not a 404, and not a soft 404 rendered by your app router).
3. Reference your sitemap so crawlers discover it, e.g. Sitemap: https://example.com/sitemap.xml.
4. Add only the disallow rules you actually need — do not block CSS, JS, or pages you want indexed. Disallow: with an empty value means allow everything.

When it's not a problem

A 404 robots.txt is valid per the specification, so this is advisory rather than a hard error. If your site is a small single-page project with nothing to exclude and no sitemap, the practical impact of a missing file is minimal and crawlers will index normally. The stronger reason to add one is diagnostic: it verifies the host serves root-level files correctly and gives you the standard place to declare your sitemap and any future crawl rules.

How common is it?

8 audited sites in our corpus currently show this issue. The breakdowns below show which platforms, gatekeepers, verticals and countries are most exposed.

Related guides

• Robots.txt Guide: How Robots.txt Works