Headless browsing
Headless browsing is running a web browser without a visible interface so that software can load pages, execute JavaScript, and read the fully rendered result — the technique tools like Playwright and Puppeteer automate. For AI agents it is the expensive fallback: most agents fetch pages over plain HTTP without executing JavaScript, and reach for a headless browser only when a site yields nothing without one.
The cost asymmetry drives agent behavior. A plain HTTP fetch is one request returning text; a headless render means launching a browser, executing scripts, and waiting for a page to settle — orders of magnitude more time and compute per page. Agents reading documentation at scale default to the cheap path, and many never attempt the expensive one. Documentation that only renders with JavaScript is, to a plain-fetch reader, close to empty.
Producers rarely discover this by looking, because a human's browser always runs the JavaScript. The fix layer is well established: server-rendered docs, markdown mirrors of key pages, an llms.txt routing to machine-readable sources, and full-text bundles like llms-full.txt all give agents a path that requires no rendering at all. The docs platforms that generate these artifacts automatically exist because the plain-fetch reader is now a mainstream audience.
Discry's plain-fetch policy is the measurement counterpart of this reality: the corpus is fetched over plain HTTP with no JavaScript execution, because that is how most agents fetch, and what renders is what gets graded. The comprehension dimension includes a markdown-docs check crediting docs available as clean markdown, and a docs surface that yields essentially nothing on plain fetch publishes as docs-unreachable, with the reason and the URLs tried as receipts.
What does an AI agent make of your API?
Find out in about a minute — no signup.
Discry your API — free