perf(crawlers): GraphQL metadata batch + tree-call cache (rate-limit ⑤+⑥b)

[yya007]

Summary

Implements rate-limit options ⑤ (GraphQL metadata batch) and ⑥b (tree-call cache) from docs/crawler-rate-limit-strategy.md, building on ①②③ (#63). Reference integration is the topic crawler. Built test-first via subagent-driven development; per-task reviewed plus 8 rounds of codex review.

• ⑥b — Cache the Trees call. find_skill_md_paths_cached reuses the cached {path: blob_sha} map when a repo's (pushed_at, default_branch) is unchanged, skipping the recursive Trees API call. pushed_at comes free from the ②-cached metadata. This is what makes a warm run approach zero metered calls per unchanged repo. Keyed on default_branch too, since a default-branch change re-resolves HEAD to a different tree without bumping pushed_at; empty/failed lookups are never cached (so a transient Trees failure can't pin a repo to "empty").
• ⑤ — Batch metadata via GraphQL. fetch_repo_metadata_batch fetches stars/pushedAt/defaultBranch/topics for up to 100 repos in one GraphQL POST (separate 5k-point/hr pool) instead of one REST call each. Uses GraphQL variables (no injection), tolerates partial errors, omits null repos, chunks >100. The topic crawler pre-filters discovered repos through the skip checks, then lazily batches metadata in chunks of 100 as the loop consumes them (so a small --limit doesn't over-fetch), falling back to the per-repo cached REST path for any repo GraphQL omits.

Composition: GraphQL gives pushed_at+default_branch for every repo in one call → feeds ⑥b's tree-skip → feeds ③'s content cache.

④ was dropped (deliberately)

Option ④ (date-filtered incremental topic discovery via pushed:>last-run) was implemented, then removed after codex review found it fundamentally unsound: a repo that adds a skill topic without a new commit keeps its old pushed_at, so pushed:>since filters it out and the advancing watermark makes it permanently undiscoverable — and GitHub search has no "topic-added-since" qualifier to fix it. Topic discovery now searches fully each run (correct; repo-search at 30/min was never the bottleneck — the wins are ⑤/⑥b and the ①②③ caches). ④ returns to the backlog noting incremental topic discovery needs a different mechanism (e.g. enumerate-and-diff, not a date watermark). ⑦/⑧ were also deferred (low/niche).

Verification

• pytest — 571 passed, 20 deselected; ruff check crawlers/ tests/crawlers/ clean; tests do no network/disk (verified no data/crawl_state/topic.json leak).
• TDD throughout. New tests: GraphQL parse/null-omit/partial-errors/non-200/chunking/counting; batch-used-vs-REST-fallback; batch excludes already-covered/filtered repos; lazy-chunk respects --limit; tree-cache hit/miss/stale-pushed_at/changed-default-branch/empty-not-cached.
• 8 codex-review rounds; all P1/P2 findings resolved (the unfixable one drove the ④ removal).

Risk & rollback

Moderate — touches crawlers/base.py (new helpers, no signature change to existing public fns) and the topic crawler loop. Additive; per-repo REST path remains as fallback. Only the topic crawler is wired. Rollback = revert. No index/runtime/schema change.

🤖 Generated with Claude Code