Record whether a discovered catalog was live or curated - #36
Closed
bsmi021 wants to merge 1 commit into
Closed
Conversation
`models.discovery.json` records `refreshed_at` with `mode: "apply"` for every source, so a Claude Code / Codex / Hermes / agentic entry is indistinguishable from a live Cursor or `/v1/models` fetch. But those platforms have no queryable model list -- `static_catalog.py` says so in its module docstring -- and their "discovery" only re-applies `CURATED_CATALOGS`, a list compiled into the package. The result reads as a live refresh that never happened: - `doctor` reports "catalog fresh (claude 0d)" for a hand-maintained list, and ages it toward a stale warning that tells the user to run a refresh which cannot possibly help. - Re-running `models discover --write` looks like it should surface a newly released model. It can only ever re-emit what the installed version already ships; a genuinely new model needs a package upgrade (or a hand-edited registry). Add an `origin` field to the sidecar, additive and backward compatible: - `write_discovery_meta(..., origin=...)` accepts `live` (default, so every existing call site is unchanged) or `curated`, and rejects anything else the way `mode` already does. - `discovery_origin(meta, source)` reads it back, defaulting to `live` for sidecars written before the field existed and for an unrecognized value -- no migration, nothing fails closed on a missing key. - `_discover_one_source` stamps `report["origin"]`: `curated` for hermes / agentic / claude / codex, `live` for cursor / openai / anthropic. `models discover --write` threads it into the sidecar, and `ensure_subscription_plan_catalog` stamps `curated` on the first-run auto-merge path. - `doctor` reports curated sources as "curated (compiled into this build, not queryable)" instead of ageing them, and no longer counts them toward the staleness warning. Drift and catalog-changed detection are untouched -- those are real signals whatever the origin. `refreshed_at` keeps its name and meaning; `catalog_staleness_days` still reads it, so persisted sidecars and every other consumer are unaffected. Note for review: `preflight._cached_catalog_verdict` also keys off `catalog_staleness_days`, which means a curated snapshot is treated as authoritative for one hour and then reported "catalog unverified (cached Nd old)" forever after -- both odd framings for a list that ages with the package version, not the clock. That path is left alone here because it gates dispatch; `origin` is now available to fix it separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2 tasks
Owner
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
~/.puppetmaster/models.discovery.jsonrecords the same shape for every source:That reads as "the Claude catalog was successfully refreshed from the platform 20 minutes ago." It wasn't.
static_catalog.pystates it plainly in its own module docstring:So for
claude,codex,hermes, andagentic, "discovery" re-appliesCURATED_CATALOGS— a list compiled into the installed package. Nothing is fetched. But the sidecar entry is byte-identical in shape to a live Cursor plan fetch or an/v1/modelscall, so nothing downstream can tell them apart:doctorreports it as fresh —catalog-freshness: ok, catalog fresh (claude 0d)— and then ages it towardcatalog stale (>7d): claude 9d. Run puppetmaster models discover --probe ... --write to apply. That refresh cannot help. Re-running it re-emits the same compiled-in list.models discover --writelooks like it should surface a newly released model. It can only ever produce what the installed version already ships; a genuinely new model needs a package upgrade or a hand-edited registry.The change — additive, backward compatible
refreshed_atkeeps its name and meaning.catalog_staleness_daysstill reads it, so every persisted sidecar stays readable and no other consumer moves.write_discovery_meta(..., origin=...)accepts"live"(the default, so every existing call site is unchanged) or"curated", and rejects anything else the waymodealready does.discovery_origin(meta, source)reads it back, defaulting to"live"for sidecars written before the field existed and for an unrecognized value — no migration, nothing fails closed on a missing key._discover_one_sourcestampsreport["origin"]:curatedforhermes/agentic/claude/codex,liveforcursor/openai/anthropic.models discover --writethreads it into the sidecar, andensure_subscription_plan_catalogstampscuratedon the first-run auto-merge path.doctorreports curated sources ascurated (compiled into this build, not queryable): claudeinstead of ageing them, and stops counting them toward the staleness warning. Drift and catalog-changed detection are untouched — those are real signals whatever the origin.Tests
Four, covering the field, its default, and both rendering branches:
origin: curatedlive; the default write stayslive; a bogusoriginraisesdoctordoes not drag a curated source into the stale list alongside a genuinely stale live onedoctorreports a curated source as curated, not as freshLeft for a follow-up, deliberately
preflight._cached_catalog_verdictalso keys offcatalog_staleness_days. For a curated snapshot that means it is treated as an authoritative catalog for one hour and then reportedcatalog unverified (cached Nd old)forever after — both odd framings for a list that ages with the package version, not the clock. I left that path alone here because it gates dispatch and I didn't want a labelling PR to change routing behaviour.discovery_originis now available to fix it separately if you want it.Verification
Fork CI (this repo's own
ci.yml) at this branch's head commit4066eb0, since Actions don't run on a fork PR without maintainer approval — all 4 legs green: ubuntu 3.9, ubuntu 3.12, macOS 3.12, windows 3.12.https://github.com/bsmi021/Puppetmaster/actions/runs/32195096999
Note
No
docs/CHANGELOG.mdentry: sibling PRs from the same review are open concurrently and would all conflict on the same## Unreleasedheading. Happy to add one on request.