How to check whether OAI-SearchBot can reach your site

To check whether OpenAI’s search crawler can access your site, inspect the robots.txt file at your site’s root for rules affecting OAI-SearchBot, then check that your CDN, firewall, authentication, and server are not blocking its requests. Confirm actual requests in server or CDN logs and inspect their response codes. Allowing the crawler makes access possible; it does not guarantee that a page will be included or cited in ChatGPT Search.

Run this quick audit:

1. Open https://yourdomain.com/robots.txt and check the rules for OAI-SearchBot and the page path you care about.
2. Confirm that the page is publicly reachable, including after any redirects.
3. Check CDN, WAF, firewall, and server logs for OAI-SearchBot requests and their responses.
4. If access is denied, fix the specific policy or infrastructure block, then verify the change.

A permissive robots.txt is a useful first check, not proof that the crawler can fetch the page. OpenAI identifies OAI-SearchBot as its crawler for ChatGPT Search and publishes its IP ranges for sites that use network-level controls. (OpenAI crawler documentation)

1. Check your robots.txt rules

Open the robots.txt file for the exact host and protocol you’re auditing. For example, https://www.example.com/robots.txt and https://example.com/robots.txt are separate files; a file on one host does not automatically set rules for the other. Robots.txt belongs at the root of the host it controls, not in a page directory. (Google’s robots.txt guide)

Search the full file for OAI-SearchBot, Disallow, and Allow. Check the rules against the path of the page you’re testing: a rule for one directory may block that URL even if other pages are allowed. Review the complete policy, including broad rules and any crawler-specific rules, before editing it.

Example only—not a universal replacement for your existing policy:

User-agent: OAI-SearchBot
Allow: /

This rule indicates that OAI-SearchBot may access paths on the host covered by this robots.txt file, subject to your other access controls. Don’t paste it over your current file without reviewing the rest of the policy. Another path restriction, a different host, or an infrastructure rule may still prevent access.

If you manage the site through a hosting platform or CMS, robots.txt may be controlled through its search-visibility settings rather than by uploading a file yourself. After a change, reopen the public robots.txt URL and confirm it shows the intended content—not an old cached copy, an error page, or a login screen.

2. Make sure you’re checking the right OpenAI crawler

OpenAI uses different crawler names for different purposes. For automatic crawling associated with surfacing websites in ChatGPT Search, check OAI-SearchBot.

User agent	What it’s for	What to check
OAI-SearchBot	Crawling for ChatGPT Search	Its robots.txt rules and whether infrastructure permits its requests
GPTBot	Crawling content that may be used to train OpenAI foundation models	Its rules separately, if you want to manage potential training use
ChatGPT-User	Certain user-initiated page fetches	Don’t treat its presence as proof of automatic search crawling

The controls for OAI-SearchBot and GPTBot are independent. You can allow search crawling while disallowing GPTBot, or choose another combination based on your policy. Allowing GPTBot does not itself allow OAI-SearchBot. ChatGPT-User is also not the user agent to use for managing automatic ChatGPT Search crawling. (OpenAI crawler documentation)

This distinction matters when reading logs: a request from ChatGPT-User does not establish that OAI-SearchBot can crawl the page, and blocking GPTBot is not the same as blocking ChatGPT Search.

3. Check for blocks beyond robots.txt

If robots.txt appears to allow the test page, check the systems that handle the request before it reaches your content. A crawler can be permitted by robots.txt but still receive a denial or fail a human-only challenge.

Ask your engineering or hosting team to check:

• CDN, WAF, firewall, and bot-mitigation rules: Look for challenges, reputation-based blocks, rate limits, or deny rules that apply to the request’s source IP or user agent.
• Authentication and consent walls: Check whether the page requires a login, CAPTCHA, cookie, or browser interaction before serving its content.
• HTTP responses and availability: Investigate 403 (forbidden), 429 (too many requests), and 5xx (server-side error) responses. A 200 response is a useful sign that a fetch worked, but confirm it contains the intended page rather than an error or challenge screen.
• Redirects: Review the full redirect chain. The final destination should be publicly accessible and should not lead to a blocked host, login page, or error.
• The exact URL and host: Test the same scheme, hostname, and path you expect the crawler to fetch. A working www version does not prove that the non-www version works, or vice versa.

Robots.txt controls whether a crawler is asked to fetch a path; it does not grant access through authentication or override firewall and server decisions. A noindex directive is a separate consideration: for a crawler to read a page-level directive, it generally needs to fetch the page. Google’s documentation explains this distinction for Google Search; it does not guarantee a particular outcome in ChatGPT Search. (Google’s noindex documentation)

4. Verify actual crawler requests in logs

A browser check shows what your browser can access. To see whether OAI-SearchBot has actually reached your site, search your server or CDN logs for the OAI-SearchBot user-agent string. OpenAI publishes an example string and notes that its version may change; requests for robots.txt may also include a robots.txt marker. (OpenAI crawler documentation)

For each matching request, record:

• The requested host and URL path
• The timestamp and time zone
• The HTTP response code
• Whether the request was challenged, rate-limited, redirected, or served from the CDN
• The final destination and response, if there was a redirect

If you see repeated 403, 429, or 5xx responses, use the timestamps and request details to find the matching CDN, WAF, or origin-server event. If you see no requests, that alone does not identify the cause: the crawler may not have attempted that URL yet, or the request may be logged in a different layer. Check with the team that operates the CDN and origin before concluding that access is blocked.

If your infrastructure requires an IP allowlist, use the current OpenAI-published OAI-SearchBot IP ranges rather than relying only on a user-agent string. User-agent strings can be imitated, so they help identify likely requests in logs but are not, by themselves, reliable identity checks. OpenAI’s crawler documentation links to its published ranges for this purpose.

After you change a rule

After updating robots.txt or an infrastructure rule, verify the public file and the relevant CDN/WAF configuration again. OpenAI says its systems may take about 24 hours to adjust after a robots.txt update. That is an estimate for systems to reflect the change—not a promise that a particular page will be crawled or appear in Search within that time. (OpenAI crawler documentation)

A command-line request that sends an OAI-SearchBot-like user agent can help an engineer inspect how a server responds to that header. But it tests a request from that machine and network; it does not prove that OpenAI’s actual crawler can reach the site from its own published IP ranges or that a real crawl has occurred. Treat it as a diagnostic aid, then confirm with infrastructure logs.

5. What crawler access does—and doesn’t—tell you

These checks answer three different questions:

1. Robots.txt permission: Does the published policy allow OAI-SearchBot to fetch the path?
2. Technical reachability: Can the request get through your CDN, firewall, authentication, and server and receive the page?
3. Search inclusion: Does ChatGPT Search select that page for a particular query?

Passing the first two checks makes crawling possible. It does not guarantee discovery, inclusion in a summary, a citation, or a particular placement. OpenAI advises publishers not to block OAI-SearchBot if they want content to be eligible for discovery and inclusion, but crawler access is not a promise of a search result. (OpenAI’s publisher FAQ)

If you’ve confirmed the crawler is allowed and the page is reachable, but want to assess broader technical readiness, see the technical AI search readiness guide.

Troubleshooting checklist

Use this checklist yourself or send it to your engineering team:

• Check robots.txt at the root of the exact host serving the page.
• Check OAI-SearchBot rules against the page path; review the full policy before editing.
• Confirm the public robots.txt file shows the intended, current content.
• Load the target URL without a login, CAPTCHA, or browser-only challenge.
• Check redirects and final-page status codes.
• Inspect CDN, WAF, firewall, and origin logs for denials, challenges, or rate limits.
• Search logs for OAI-SearchBot requests and note their response codes.
• If IP filtering is enabled, compare it with OpenAI’s published OAI-SearchBot ranges.
• Recheck after changes; allow about 24 hours for OpenAI systems to reflect a robots.txt update.
• Treat access as eligibility to be crawled—not as a guarantee of inclusion or citation.

Watch: How Robots.txt Works

Google Search Central’s video “How Robots.txt Works” is a general explanation of robots.txt and its relationship to crawling and other directives. It provides background on robots.txt, not OpenAI-specific crawler guidance.

Frequently asked questions

Which OpenAI crawler should I allow for ChatGPT Search?

Check OAI-SearchBot. OpenAI identifies it as the crawler used to surface websites in ChatGPT Search. Also review network-level rules that could block its requests. (OpenAI crawler documentation)

Does allowing GPTBot also allow OAI-SearchBot?

No. OpenAI documents their robots.txt controls as independent. Set rules for each according to your separate search-access and potential-training policies. (OpenAI crawler documentation)

If OAI-SearchBot is allowed in robots.txt, why might it still fail to access a page?

A CDN, WAF, firewall, authentication wall, CAPTCHA, rate limit, redirect, or server error can still prevent a successful fetch. Check the actual request and response in infrastructure logs rather than relying on robots.txt alone.

How long can a robots.txt change take to affect ChatGPT Search?

OpenAI says its systems may take about 24 hours to adjust after a robots.txt update. This does not guarantee that a page will be crawled or shown after that time. (OpenAI crawler documentation)

Does allowing OAI-SearchBot guarantee that my site will be cited?

No. Permission and successful fetching make access possible, but they do not guarantee that ChatGPT Search will select or cite a page. (OpenAI’s publisher FAQ)