DISCOVERY SIGNALS · LIVE FROM THE INDEX

Which APIs allow AI crawlers in robots.txt?

86%

of the APIs in the Discry Index allow AI crawlers to read their docs.

339 pass4 partial52 failof 395 scanned

What this signal is

robots.txt is where a docs site tells crawlers what they may read. GPTBot, ClaudeBot, and Google-Extended check it before fetching; a blanket disallow makes your documentation invisible to the systems that recommend and integrate APIs.

The check credits docs surfaces that allow AI crawlers — either explicitly or by not disallowing them. Blocking AI on a marketing site is a choice; blocking it on API docs removes you from the agent channel.

Definition →How to ship it →

Notable passes and misses

Most of the corpus passes — the default of an open robots.txt works in most producers’ favor. The fails are usually inherited rather than chosen: a site-wide crawler policy written for content protection that happens to cover docs. Salesforce, Linear, Figma, and Intercom currently restrict AI crawlers on their docs surfaces.

Every API, checked

395 APIs
Pass339
Show the other 309
Partial4
Fail52
Show the other 22

Does your API pass this check?

Find out in about a minute — no signup.

Discry your API — free

Check data assessed under Discry methodology 3.0.0 · Methodology