Discovery
The service learns that a URL exists through links, sitemaps, submissions or other references.
Permission, access, processing and eligibility
Crawling and Indexing
A robots rule can permit a crawler while a firewall blocks it. A successful crawl can still end without indexing, retrieval or visibility.
Four decisions that are often merged
Access
The crawler requests the page and receives a usable response without a login, challenge or infrastructure block.
Processing
The service renders, parses and evaluates the content, directives and canonical relationships.
Indexing
The service decides whether and how to retain the material for future search or retrieval.
A cooperation protocol, not a security boundary
RFC 9309 standardizes the Robots Exclusion Protocol. It defines how compliant crawlers identify groups, match paths, resolve allow and disallow rules, handle errors and cache the file.
The standard explicitly distinguishes robots rules from access authorization. Sensitive information requires authentication and server-side protection, not a disallow line.
Why “all bots allowed” can still fail
- The CDN challenges non-browser requests with JavaScript or CAPTCHA.
- A web application firewall blocks the user agent or IP range.
- The server returns a 403, 429, 5xx response or redirect loop.
- Important content appears only after unsupported client-side execution.
- The page requires a session, cookie or geographic condition.
- The crawler receives a different or empty version of the page.
Directives must be read in context
noindex
Requests exclusion from a compliant search index; the crawler must access the page to read it.
Canonical URL
Signals the preferred version among duplicates or near-duplicates.
Redirect
Moves crawlers and users to another address and can consolidate a changed location.
Sitemap
Lists important canonical URLs and optional modification information.
IndexNow
Notifies participating engines that a URL was added, updated or removed.
Search Console tools
Provide platform-specific inspection and diagnostic evidence rather than universal status.
Test from outside the account
A file that looks correct in the repository may be altered by the host, CDN or deployment process. Verify the public URL, response code, headers, rendered content and logs after deployment.
Primary material behind this analysis
RFC 9309: Robots Exclusion Protocol
The standards-track specification for robots.txt parsing, matching, errors and caching.
Google's Guide to Optimizing for Generative AI Features on Search
Official guidance on SEO, crawlability, original content, local details and generative Search.
Publishers and Developers FAQ
Official information about OAI-SearchBot, GPTBot, referral tracking and agent accessibility.
Perplexity Crawlers
Official roles and access guidance for PerplexityBot and Perplexity-User.
IndexNow Documentation
Protocol for notifying participating search engines about added, updated and deleted URLs.
Continue through the discovery stack
IETF RFC 9309 and robots.txt
A standards-based explanation of robots.txt groups, matching, error handling, caching and why the protocol is not security or access authorization.
Google Search Central, AI Overviews and AI Mode
A source-based explanation of Google crawling, indexing, SEO, query fan-out, AI Overviews, AI Mode, structured data and site-owner controls.
OpenAI Crawlers, ChatGPT Search and Agent Access
An explanation of OAI-SearchBot, GPTBot, ChatGPT referrals, noindex, crawl access and accessible websites for agent interaction.
PerplexityBot, Perplexity-User and Search Retrieval
A practical guide to PerplexityBot, Perplexity-User, automatic crawling, live user-requested retrieval, WAF access and monitoring.