Publish model availability through fabric telemetry

[ttupper92618]

What changed

• add bounded NodeArtifactInventory readings to the existing telemetry plane
• publish cache availability from the node lifecycle—including --no-api nodes—at startup, after relevant store/runtime transitions, and every 60 seconds for repair
• contain transient scan/publication failures inside the lifetime service and retry on the normal cadence
• retain readings in TelemetryView using local receipt time, membership pruning, rejoin convergence, and two-interval staleness
• fully verify detached records once per stable filesystem fingerprint so periodic repair does not continuously rehash read-only model weights
• extend GET /store/registry with additive cache inventory coverage and explicit store_local / node_cache locations
• extend GET /store/storage diagnostics with location provenance
• keep canonical catalog/card bodies out of telemetry and retain direct verified storage reads for reconciliation
• document the API and architecture changes and add focused convergence, freshness, bounds, publication, failure-recovery, cache-invalidation, and reconciliation coverage

Why

Any healthy API node should be able to report the same eventual model-availability truth without an app-specific proxy or durable cache inventory in State. Telemetry is the existing lossy operator plane; transfer and publication correctness continue to use direct verified storage reads.

Validation

• uv run basedpyright
• uv run ruff check
• nix --extra-experimental-features 'nix-command flakes' fmt
• uv run pytest — 3,346 passed, 2 skipped, 231 deselected
• uv run python scripts/build_docs.py
• uv run python scripts/export_openapi.py

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ec7973d5cc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

This PR extends Skulk’s existing telemetry plane to include a compact, bounded per-node artifact-availability snapshot, and uses that telemetry to project fleet-wide cache availability through the store registry APIs (without moving the canonical catalog/manifests into telemetry).

Changes:

• Add NodeArtifactInventory (bounded telemetry reading) and retain it in TelemetryView using local receipt time + membership pruning.
• Publish artifact inventory from the API lifetime task (startup, debounced triggers, periodic repair) and project cache availability/coverage into GET /store/registry.
• Update OpenAPI types, tests, and docs/architecture references to reflect the new telemetry-derived availability surface.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
website/docs/architecture.md	Updates telemetry-plane description to include bounded artifact inventory and GET /store/registry coverage semantics.
website/docs/architecture-reference.md	Adds the authoritative “node artifact availability” fact-sheet entry and updates endpoint contract notes.
website/docs/api-guide.md	Documents locationKind and cacheInventory coverage for /store/registry and locationKind for /store/storage.
src/skulk/store/staging_eviction.py	Adds location_kind provenance onto staged model info for /store/storage reporting.
src/skulk/shared/types/tests/test_telemetry.py	Adds round-trip + coalescing + bounds tests for artifact-inventory telemetry.
src/skulk/shared/types/telemetry.py	Extends NodeTelemetry / TelemetryView to store NodeArtifactInventory with local receipt timestamps.
src/skulk/shared/types/artifact_inventory.py	Introduces the bounded NodeArtifactInventory + entry type and cadence/staleness constants.
src/skulk/main.py	Wires the router telemetry sender into the API so the API can publish artifact-inventory telemetry.
src/skulk/api/types/api.py	Extends /store/registry schema with cache_inventory and adds location_kind to cached locations.
src/skulk/api/types/init.py	Re-exports new API types for external import stability.
src/skulk/api/tests/test_artifact_inventory_telemetry.py	Adds focused tests for publication cadence, bounds, freshness, and /store/registry projection.
src/skulk/api/main.py	Implements inventory scanning/publication loop and registry projection (store_local synthesized + node_cache from telemetry).
CLAUDE.md	Updates architecture/reference notes to include artifact inventory on the telemetry plane and /store/registry coverage.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/skulk/api/main.py:11150

• _cache_inventory_projection() keeps _artifact_inventory_nodes_seen forever, so a node that previously published an inventory, leaves membership, and later rejoins will be treated as already-seen and the overall cacheInventory.state can incorrectly skip syncing and jump straight to degraded/unavailable until it publishes again. Trim nodes_seen to the current expected membership before computing first_reading_pending so re-joins are treated like first-time convergence.

        expected_nodes = set(self.state.topology.list_nodes())
        expected_nodes.add(self.node_id)
        now = datetime.now(tz=timezone.utc)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0259d474b6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5731d96495

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/skulk/api/main.py:11104

• GET /store/registry synthesizes store_local locations by overwriting locations_by_node[str(store_host)]. If the store host also has a node_cache copy for the same installed_identity (e.g., staging or fallback roots outside the canonical store), that node-cache location is silently dropped. This also prevents representing multiple location kinds on the same node even though location_kind was introduced for that purpose. Consider appending synthesized store_local locations instead of replacing the existing per-node entry.

                for store_host in store_hosts:
                    locations_by_node[str(store_host)] = CachedArtifactLocation(
                        node_id=str(store_host),
                        complete=True,
                        installed_identity=identity,

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/skulk/main.py:969

• _artifact_models_in_use() marks models as in-use for telemetry by iterating over all runner_to_shard entries once the instance is present on this node. For multi-node instances, that will incorrectly treat shards running on other nodes as locally in-use, which can misreport in_use in the published inventory and skew the sorting/truncation toward unrelated artifacts.

Select only this node’s runner/shard (via node_to_runner[self.node_id]) before extracting the model card + companion repos.

        for instance in self._current_artifact_inventory_state().instances.values():
            if self.node_id not in instance.shard_assignments.node_to_runner:
                continue
            for shard in instance.shard_assignments.runner_to_shard.values():
                card = shard.model_card

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f55579ae88

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/skulk/main.py:974

• _artifact_models_in_use() currently marks a model as in-use on this node by iterating all runner_to_shard entries for any instance that has at least one shard on this node. That includes shards assigned to other nodes, so a node can incorrectly report (and sort/protect) artifacts as in_use even when they are only in use elsewhere in the cluster.

        for instance in self._current_artifact_inventory_state().instances.values():
            if self.node_id not in instance.shard_assignments.node_to_runner:
                continue
            for shard in instance.shard_assignments.runner_to_shard.values():
                card = shard.model_card