GPTBot
GPTBot is OpenAI's web-crawler user agent, used to collect publicly available content that may contribute to training OpenAI's generative AI foundation models. Site owners control it through robots.txt: disallowing GPTBot signals that crawled content should be excluded from model training. OpenAI publishes separate user agents for other purposes, including OAI-SearchBot, which surfaces sites in ChatGPT's search features.
The controls are independent by design. OpenAI's crawler documentation states that a site can allow OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot to keep crawled content out of training — the two settings answer different questions. OpenAI also publishes the IP ranges its crawlers operate from, so operators can verify that traffic claiming to be GPTBot actually is.
For an API producer, the robots.txt treatment of GPTBot on the docs domain is a discovery decision, distinct from the content-licensing debates that made AI crawlers controversial. Documentation exists to be read, and models that have read an API's docs are the models developers ask about that API. A blanket Disallow rule — often copied from a template written for content sites — shuts the crawler out of pages whose entire purpose is distribution.
Discry's robots.txt discovery check fetches the docs domain's robots.txt and parses its rules for three AI crawler tokens: GPTBot, ClaudeBot, and Google-Extended. The check passes when none of the three is fully blocked, earns partial credit when some are, and fails when all three are blocked or no robots.txt exists.
Checks: robots.txt AI directives · how each is scored
What does an AI agent make of your API?
Find out in about a minute — no signup.
Discry your API — free