IETF and the Robots Exclusion Protocol

Groups, rules, matching, errors and caching

The Robots Exclusion Protocol uses user-agent groups and allow or disallow path rules. RFC 9309 defines how crawlers select matching groups, compare paths, resolve equivalent rules and handle the robots file under different server responses.

Standardization reduces ambiguity among compliant crawlers. It does not require every automated client to obey the file.

The most specific applicable path wins

• A crawler identifies the group matching its product token.
• If no specific group matches, the wildcard group may apply.
• Allow and disallow paths are compared against the requested URI.
• The most specific matching rule controls the result.
• Equivalent allow and disallow rules favour allow under the standard.
• The robots file itself is implicitly accessible.

The server response changes expected behaviour

The protocol distinguishes successful retrieval, redirects, unavailable resources and server errors. Operational teams should understand those differences rather than assuming a missing robots file and a 5xx failure communicate the same instruction.

Use robots.txt as one layer

1. Set the intended crawl policy.
2. Verify the public file after deployment.
3. Test the actual page response for the crawler.
4. Inspect CDN and WAF rules.
5. Use authentication for private material.
6. Monitor logs for expected and unexpected automated access.