feat(restore): one restore check per server, replicas as instances

.workhorse/specs/monitoring/checks.md

@@ -57,6 +57,7 @@ A name that encodes a parameter turns one configurable check into as many entrie
 Where a target has several instances of one condition, Canopy holds one state for the check, as it does for every (target, source, check).
 
 Each instance is graded through policy on its own, against its own detail, so a rule or silence written for one instance applies to only that instance.
+Where it takes more than one field to say which instance this is, the detail carries those fields joined into one as well as separately, because a rule condition matches a single variable and a silence for one instance has to pin all of them.
 The check's effective result is then the most urgent across the instances that were not skipped, and its detail carries every instance that is not passing, each with its own result, so an operator can see which ones are in trouble without opening anything else.
 Its message names those instances.
 The check recovers when no instance is left degraded.

.workhorse/specs/public-server/restore-replicas.md

@@ -355,26 +355,38 @@ Each replica's redaction outcome is presented alongside its restore health, so a
 
 A failed or overdue restore-health report raises a restore-verification check on the affected server, subject to the same monitoring and incident gates as any other of that server's checks.
 
-Restore-health is tracked independently per server, type, and intent: the affected server is the check's scope, and the type and intent name it, so one replica's failed restore does not mask or merge with another's, and the snapshot is carried in the check's detail.
-The check recovers when the next report for the same server, type, and intent is healthy.
+A server has one restore-verification check however many replicas it has.
+Each replica is an instance of that check, graded on its own and carrying its type, its intent, its declared name, and the snapshot in the check's detail (see [CHK](../monitoring/checks.md)), so a rule or silence written for one replica applies to only that replica.
+The detail names the replica by its type and intent together as well as separately, since a rule matches one variable at a time and a silence for a single replica has to pin both.
+The check reflects the most urgent of them, names the ones in trouble, and recovers when none is left degraded.
+
+A replica's state is the worse of what its latest report said and whether it has gone past its overdue bound; these are one judgement about the replica, not two competing ones.
+Canopy re-derives a server's replicas on the same periodic sweep that decides overdue, from the declarations covering the server and from the reports it holds about it.
+A declaration covering the server names one of its replicas whatever its consumer currently advertises: an intent that stops being advertised is a gap, and a finding standing against a replica does not go away because the consumer that reported it stopped offering to.
+A replica the server has a report for is one of its replicas too, for as long as a declaration still asks for that replica somewhere in the server's group — which is what a consumer needs in order to report on it at all.
+So a report about a server its declaration does not name still surfaces against that server, and a replica nothing declares any more stops being one of the server's instances and needs no separate recovery: nothing can report on it again, so a finding held against it could never recover.
+A report's effect appears on the next sweep rather than the instant it lands, which is immaterial for a warning that pages nobody.
 
 A replica is also overdue — raising the same check on a periodic sweep, rather than waiting for a report that never arrives — when it has not met its intent's health expectation within the declaration's overdue bound.
 For an intent carrying `once`, the expectation is measured against the latest snapshot: the replica is overdue when the latest snapshot has gone unverified for longer than the bound, not merely because time has passed since an earlier snapshot was verified.
 For an intent without `once`, it is measured against wall-clock time since the last healthy report.
 Overdue applies only to intents carrying `check`.
 
 A failed migration test raises a migration-test check on the affected server, under the same gates, because that server is on the upgrade path to the version that failed.
-The check is named for the type and intent, as restore-verification is, and carries the target version in its detail rather than its name.
+It is one check per server with its replicas as instances, as restore-verification is, and carries the target version in the detail rather than the name.
 A server has one candidate at a time, so there is no second version whose result the first could mask, and a name per version would spawn a catalog policy per release.
+A replica whose candidate has not been tried within its overdue bound degrades the same check: untested and failed are both "this version is not known good against this deployment's data".
+A recorded verdict raises the check whatever declares that replica now, and a later verdict is what supersedes it: what a version's migrations did to a deployment's data is a fact about the version, not something a declaration has to keep asking about for it to remain true.
 
 The check is a warning rather than a failure, and does not escalate.
 Nothing is wrong with the live server: it is running the version it always was, serving patients, and the finding is about a version it has not taken yet.
 Treating it as a failure would open an incident against a healthy deployment and put a migration problem in front of whoever is on call for outages, when the people who need it are the ones deciding whether that version ships.
 The version's readiness is where the finding does its work.
 
 A redaction that did not fully apply raises a redaction check on the affected server, under the same gates.
-The check is named for the type and intent, as restore-verification is, and carries the redaction outcome, the manifest version, and the counts of masked and skipped columns in its detail.
-It recovers when the next report for the same server, type, and intent redacts fully.
+It is one check per server with its redacting replicas as instances, as restore-verification is, each carrying the redaction outcome, the manifest version, and the counts of masked and skipped columns in the check's detail.
+A replica is an instance of it only once it has reported a redaction outcome: a declaration that redacts but has produced no replica yet has nothing unmasked to report.
+An instance recovers when the replica's next report redacts fully, and the check when none of them is left degraded.
 
 The check is a warning rather than a failure, and does not escalate.
 The deployment is healthy and its data is where it should be; the finding is that a replica made from that data is not as safe to hand out as it was declared to be.

crates/canopy-mcp/src/restore.rs

@@ -216,7 +216,7 @@ impl CanopyMcp {
 	/// exact `(server, type, intent)` (from
 	/// `BackupRestoreCheck::latest_healthy_by_key_for_group`). Does not compute
 	/// an overdue verdict — that logic lives solely in
-	/// `database::restore::sweep_overdue`, which alone owns the once-vs-check
+	/// `database::restore::sweep_restore_checks`, which alone owns the once-vs-check
 	/// semantics distinction.
 	async fn restore_replica_outs(
 		&self,

crates/database/src/backup.rs

@@ -28,7 +28,7 @@ pub async fn sweep(db: &mut AsyncPgConnection) -> Result<usize> {
 	let rows = staleness::scan_rows(db).await?;
 	let mut filed = staleness::sweep(db, &rows).await?;
 	filed += reconcile::sweep(db, &rows).await?;
-	filed += crate::restore::sweep_overdue(db).await?;
+	filed += crate::restore::sweep_restore_checks(db).await?;
 	// Not an event, but the same cadence: a plan closes once its group reports
 	// the target, and this sweep is what notices.
 	crate::upgrade_plans::close_met_plans(db).await?;

crates/database/src/backup/refs.rs

@@ -104,14 +104,22 @@ pub const PREFLIGHT_ASSUME: &str = "preflight-assume";
 /// exist are no longer protected from deletion.
 pub const PREFLIGHT_OBJECT_LOCK: &str = "preflight-object-lock";
 
-/// PGRO reported a failed/stale restorability check for one replica.
-/// Server-scoped, `Warning`; the ref carries the `(type, intent)` dimension so
-/// each replica of a server recovers independently.
+/// A restore replica reported unhealthy, or has gone past its overdue bound.
+/// Server-scoped, `Warning`. One check per server: each of its replicas is an
+/// instance carrying its own `type`, `intent`, and replica name in the detail,
+/// so an operator configures restore-verification once rather than once per
+/// `(type, intent)` pair. Filed only by `crate::restore::sweep_restore_checks`.
 pub const RESTORE_VERIFICATION: &str = "restore-verification";
+
+/// A candidate version's migrations failed against a replica of a server's
+/// data, or have not been tried within the replica's overdue bound.
+/// Server-scoped, `Warning`, one check per server with its replicas as
+/// instances.
 pub const MIGRATION_TEST: &str = "migration-test";
 
 /// The masking manifest for a redacting replica did not fully apply.
-/// Server-scoped, `Warning`, does not escalate.
+/// Server-scoped, `Warning`, does not escalate. One check per server with its
+/// redacting replicas as instances.
 pub const REDACTION: &str = "redaction";
 
 // --- shipped documentation (seeded into the catalog on first filing) ---
@@ -284,31 +292,31 @@ Restore the bucket's Object-Lock configuration to GOVERNANCE mode with at least
 
 pub const RESTORE_VERIFICATION_DOC: &str = "## Description
 
-The managed restore replica for this (server, type, intent) reported a failed or stale restorability check.
+One of this server's managed restore replicas reported a failed restorability check, or has gone past its overdue bound without a healthy one. The server has one of these checks however many replicas it has: each replica is an instance of it, named in the detail by its type, its intent, and the declaration's own name.
 
 ## Results
 
-- **warn** — the replica couldn't restore or verify the latest snapshot.
+- **warn** — a replica couldn't restore, or couldn't verify the latest snapshot within its bound.
 
 ## Solve
 
-Check the restore consumer's report detail: restore errors point at the snapshot or credentials, staleness at the consumer itself.";
+Read the detail for the replicas named: restore errors point at the snapshot or credentials, staleness at the consumer itself. To handle one replica differently from the rest, write a rule or silence against its `check.replica_key` rather than the check as a whole.";
 
 pub const MIGRATION_TEST_DOC: &str = "## Description
 
-A candidate version's schema migrations were applied to a restore replica of this server's data, and one of them failed. The server itself is unaffected: it is still running the version it was, and the finding is about a version it has not taken.
+A candidate version's schema migrations were applied to a restore replica of this server's data and one of them failed, or the candidate has gone untried past the replica's overdue bound. The server itself is unaffected: it is still running the version it was, and the finding is about a version it has not taken. The version under test is in the detail rather than the check name, so a release doesn't spawn a catalog entry of its own.
 
 ## Results
 
-- **warn**: the migrations did not complete against this deployment's data. The version carries a known issue and is held back from rollout.
+- **warn**: the migrations did not complete against this deployment's data, or have not been tried against it in time. A failure carries a known issue against the version and holds it back from rollout.
 
 ## Solve
 
 Read the failing migration named in the report detail. The fix belongs to the migration or to the deployment's data, and the version stays unready until someone resolves the known issue against it.";
 
 pub const REDACTION_DOC: &str = "## Description
 
-A replica of this server's data was declared to be served de-identified, and its masking manifest did not fully apply. The server itself is unaffected — this is about the copy, not the deployment.
+A replica of this server's data was declared to be served de-identified, and its masking manifest did not fully apply. The server itself is unaffected — this is about the copy, not the deployment. The server has one of these checks however many redacting replicas it has, each an instance named in the detail by its type, its intent, and the declaration's own name.
 
 ## Results
 

crates/database/src/backup/staleness.rs

@@ -649,6 +649,31 @@ pub(crate) async fn server_check_observed_degraded(
 	Ok(n > 0)
 }
 
+/// Every server with a currently-open, active `(canopy, ref)` issue for one of
+/// these checks.
+///
+/// A sweep that re-derives its checks from current state has to visit these
+/// servers even when it derives nothing for them: a check whose last instance is
+/// gone is recovered by being filed as passing, and a server nobody visits is a
+/// check left open with nothing that could ever clear it.
+pub(crate) async fn servers_with_open_checks(
+	db: &mut AsyncPgConnection,
+	checks: &[&str],
+) -> Result<Vec<Uuid>> {
+	use crate::schema::issues::dsl;
+	let ids: Vec<Option<Uuid>> = dsl::issues
+		.select(dsl::server_id)
+		.distinct()
+		.filter(dsl::server_id.is_not_null())
+		.filter(dsl::source.eq(refs::CANOPY_SOURCE))
+		.filter(dsl::ref_.eq_any(checks.to_vec()))
+		.filter(dsl::active.eq(true))
+		.filter(dsl::resolved_at.is_null())
+		.load(db)
+		.await?;
+	Ok(ids.into_iter().flatten().collect())
+}
+
 /// Whether a group-scoped `(canopy, ref)` issue is currently open + active.
 pub(crate) async fn open_group_issue_active(
 	db: &mut AsyncPgConnection,
@@ -670,10 +695,10 @@ pub(crate) async fn open_group_issue_active(
 
 /// How an alert message names a server: the name an operator knows it by,
 /// qualified with its host when both are known, falling back to the host
-/// alone and finally to the id. Shared with [`crate::backup::reconcile`] so
-/// every backup alert names servers the same way.
+/// alone and finally to the id. Shared across every canopy-determined check
+/// so they all name servers the same way — never interpolate a bare id.
 // spec: BKJ#alerting
-pub(super) fn server_label(server: &Server) -> String {
+pub fn server_label(server: &Server) -> String {
 	let host = server.host.as_ref().map(|h| h.0.to_string());
 	match (&server.name, host) {
 		(Some(n), Some(h)) if !n.is_empty() => format!("{n} ({h})"),