Score any site or API on how well AI agents and answer engines (ChatGPT, Claude, Perplexity, and crawlers like GPTBot and ClaudeBot) can find, read, and call it, straight from your terminal, plus the exact files to fix what is wrong.
Single static binary, no third-party dependencies.
$ crwl example.com
crwl https://example.com/
88/100 grade B profile saas-api 12 page(s) 2026-06-13 10:00 UTC
Discovery ##################.. 91
Structure #################### 100
Interface ################.... 78
PASS llms.txt present [discovery]
PASS robots.txt allows AI crawlers [discovery]
WARN sitemap.xml present [discovery]
! no sitemap found
fix Publish /sitemap.xml and reference it from robots.txt.
BONUS publishes an agent-skills index [discovery]
- no agent-skills index (optional, emerging)
add Publish /.well-known/agent-skills/index.json so agents can load your skills.
...
PASS/WARN/FAIL are scored checks that move your grade. BONUS marks
optional, emerging signals: earning one lifts you toward an A, missing one never
counts against you, so it is never shown as a failure.
curl -fsSL crwl.sh/install | sh
Or with Go:
go install github.com/crwl-sh/crwl-cli/cmd/crwl@latest
crwl <url|host> [flags]
-json print the report as JSON instead of the terminal view
-o <file> also save the report to a file (.json -> JSON, else text)
-pages <n> maximum pages to crawl (0 = all discovered; 1 = homepage only)
-min <0-100> minimum score to exit 0; below this exits 2 (CI gate)
-summary print only the score, layer bars, and check counts
-timeout <dur> overall timeout for the request (default 120s)
-no-color disable ANSI colour
Examples:
crwl example.com
crwl https://example.com -o report.json # show on screen and save
crwl example.com -pages 50 # crawl up to 50 pages
crwl example.com -min 80 # fail CI if score < 80
crwl example.com -summary # just the score and layer bars
Exit codes: 0 score met the threshold, 2 score below -min, 1 usage or
request error. The -min gate drops it cleanly into CI to fail a build when a
deploy regresses your agent readiness.
crwl fix <url> writes a ready-to-commit file for each failing check, built
from what the scan actually saw: an llms.txt, a robots.txt that welcomes AI
crawlers, schema.org JSON-LD, a starter agent card, and more.
crwl fix example.com # writes into ./crwl-out
crwl fix example.com -out ./public # write straight into your site
crwl fix example.com --dry-run # list what it would generate
Three readiness layers, weighted because an agent cannot use what it cannot find:
| Layer | Weight | Question |
|---|---|---|
| Discovery | 45% | Can an agent find you? (llms.txt, agent/MCP card, robots AI policy, sitemap) |
| Structure | 30% | Can it parse you? (server-rendered content, schema.org JSON-LD, a clear H1) |
| Interface | 25% | Can it call you? (a discoverable OpenAPI spec) |
Grades: A (>=92), B (>=82), C (>=70), D (>=55), F (otherwise). Strict on purpose: an A means strong across every layer.
Apache-2.0. See LICENSE.