Skip to content

Record whether a discovered catalog was live or curated - #36

Closed
bsmi021 wants to merge 1 commit into
professorpalmer:mainfrom
bsmi021:fix/discovery-meta-curated-origin
Closed

Record whether a discovered catalog was live or curated#36
bsmi021 wants to merge 1 commit into
professorpalmer:mainfrom
bsmi021:fix/discovery-meta-curated-origin

Conversation

@bsmi021

@bsmi021 bsmi021 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

The problem

~/.puppetmaster/models.discovery.json records the same shape for every source:

"claude": {
  "refreshed_at": "2026-08-17T14:40:29Z",
  "count": 7,
  "mode": "apply",
  "catalog_hash": "sha256:...",
  "last_applied_hash": "sha256:..."
}

That reads as "the Claude catalog was successfully refreshed from the platform 20 minutes ago." It wasn't. static_catalog.py states it plainly in its own module docstring:

The two CLI agent loops — Claude Code and Codex — do not offer a queryable model list, so historically their registry tiers had to be hand-maintained.

So for claude, codex, hermes, and agentic, "discovery" re-applies CURATED_CATALOGS — a list compiled into the installed package. Nothing is fetched. But the sidecar entry is byte-identical in shape to a live Cursor plan fetch or an /v1/models call, so nothing downstream can tell them apart:

  • doctor reports it as freshcatalog-freshness: ok, catalog fresh (claude 0d) — and then ages it toward catalog stale (>7d): claude 9d. Run puppetmaster models discover --probe ... --write to apply. That refresh cannot help. Re-running it re-emits the same compiled-in list.
  • Re-running models discover --write looks like it should surface a newly released model. It can only ever produce what the installed version already ships; a genuinely new model needs a package upgrade or a hand-edited registry.

The change — additive, backward compatible

refreshed_at keeps its name and meaning. catalog_staleness_days still reads it, so every persisted sidecar stays readable and no other consumer moves.

  • write_discovery_meta(..., origin=...) accepts "live" (the default, so every existing call site is unchanged) or "curated", and rejects anything else the way mode already does.
  • discovery_origin(meta, source) reads it back, defaulting to "live" for sidecars written before the field existed and for an unrecognized value — no migration, nothing fails closed on a missing key.
  • _discover_one_source stamps report["origin"]: curated for hermes / agentic / claude / codex, live for cursor / openai / anthropic. models discover --write threads it into the sidecar, and ensure_subscription_plan_catalog stamps curated on the first-run auto-merge path.
  • doctor reports curated sources as curated (compiled into this build, not queryable): claude instead of ageing them, and stops counting them toward the staleness warning. Drift and catalog-changed detection are untouched — those are real signals whatever the origin.

Tests

Four, covering the field, its default, and both rendering branches:

  • curated auto-discovery writes origin: curated
  • legacy entries (no field), unknown sources, and a garbage value all read back as live; the default write stays live; a bogus origin raises
  • doctor does not drag a curated source into the stale list alongside a genuinely stale live one
  • doctor reports a curated source as curated, not as fresh

Left for a follow-up, deliberately

preflight._cached_catalog_verdict also keys off catalog_staleness_days. For a curated snapshot that means it is treated as an authoritative catalog for one hour and then reported catalog unverified (cached Nd old) forever after — both odd framings for a list that ages with the package version, not the clock. I left that path alone here because it gates dispatch and I didn't want a labelling PR to change routing behaviour. discovery_origin is now available to fix it separately if you want it.

Verification

Fork CI (this repo's own ci.yml) at this branch's head commit 4066eb0, since Actions don't run on a fork PR without maintainer approval — all 4 legs green: ubuntu 3.9, ubuntu 3.12, macOS 3.12, windows 3.12.

https://github.com/bsmi021/Puppetmaster/actions/runs/32195096999

Note

No docs/CHANGELOG.md entry: sibling PRs from the same review are open concurrently and would all conflict on the same ## Unreleased heading. Happy to add one on request.

`models.discovery.json` records `refreshed_at` with `mode: "apply"`
for every source, so a Claude Code / Codex / Hermes / agentic entry is
indistinguishable from a live Cursor or `/v1/models` fetch. But those
platforms have no queryable model list -- `static_catalog.py` says so
in its module docstring -- and their "discovery" only re-applies
`CURATED_CATALOGS`, a list compiled into the package.

The result reads as a live refresh that never happened:

- `doctor` reports "catalog fresh (claude 0d)" for a hand-maintained
  list, and ages it toward a stale warning that tells the user to run
  a refresh which cannot possibly help.
- Re-running `models discover --write` looks like it should surface a
  newly released model. It can only ever re-emit what the installed
  version already ships; a genuinely new model needs a package upgrade
  (or a hand-edited registry).

Add an `origin` field to the sidecar, additive and backward compatible:

- `write_discovery_meta(..., origin=...)` accepts `live` (default, so
  every existing call site is unchanged) or `curated`, and rejects
  anything else the way `mode` already does.
- `discovery_origin(meta, source)` reads it back, defaulting to `live`
  for sidecars written before the field existed and for an
  unrecognized value -- no migration, nothing fails closed on a
  missing key.
- `_discover_one_source` stamps `report["origin"]`: `curated` for
  hermes / agentic / claude / codex, `live` for cursor / openai /
  anthropic. `models discover --write` threads it into the sidecar,
  and `ensure_subscription_plan_catalog` stamps `curated` on the
  first-run auto-merge path.
- `doctor` reports curated sources as
  "curated (compiled into this build, not queryable)" instead of
  ageing them, and no longer counts them toward the staleness warning.
  Drift and catalog-changed detection are untouched -- those are real
  signals whatever the origin.

`refreshed_at` keeps its name and meaning; `catalog_staleness_days`
still reads it, so persisted sidecars and every other consumer are
unaffected.

Note for review: `preflight._cached_catalog_verdict` also keys off
`catalog_staleness_days`, which means a curated snapshot is treated as
authoritative for one hour and then reported "catalog unverified
(cached Nd old)" forever after -- both odd framings for a list that
ages with the package version, not the clock. That path is left alone
here because it gates dispatch; `origin` is now available to fix it
separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@professorpalmer

Copy link
Copy Markdown
Owner

Absorbed onto dev as 79f4c93 in #38 with credit to @bsmi021.
We applied stack-wide edits (not a rubber-stamp of the fork branch) so this can ship in the next release cut. Closing this PR in favor of that landing.

professorpalmer added a commit that referenced this pull request Aug 19, 2026
…-catalog-prompt

absorb #35 #36 #37: catalog origin and prompt orientation
@professorpalmer professorpalmer mentioned this pull request Aug 19, 2026
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants