Task bank
A task bank is a versioned, published set of tasks used to quiz models on an API's documentation. It covers the facts an integration depends on — authentication, errors, pagination, limits — along with request-construction tasks and trap tasks that ask for capabilities the API does not support. Publishing the bank makes the measurement reproducible and auditable rather than a private judgment.
The task bank is what turns "can an agent understand these docs" into an experiment. Models answer the bank's tasks with the fetched documentation as their only source, and each task type probes a different way integrations fail: factual tasks catch misread fundamentals, request-construction tasks catch docs a model cannot build from, and trap tasks catch docs that invite confident invention.
Versioning and publication are trust properties, on par with the tasks themselves. A benchmark whose questions are secret asks to be taken on faith; a published bank lets anyone inspect exactly what was asked of every API and confirm the same instrument graded everyone.
Every API in the Discry corpus is quizzed from the published task banks, and every answer is graded mechanically against citation-verified ground truth — each ground-truth fact must trace to a specific location in the API's own documentation. A closed-book pass first removes tasks a model could answer without the docs, so the bank measures what the documentation contributes rather than what the model remembered.
What does an AI agent make of your API?
Find out in about a minute — no signup.
Discry your API — free