Manual incidents with MCP write access; drop manual conditions

.workhorse/specs/monitoring/checks.md

@@ -21,12 +21,12 @@ Canopy-wide checks are Canopy monitoring its own operation (see [SELF](../privat
 A source is a named reporter of checks, identified by a short string.
 Multiple sources may report on the same server, each concerned with part of the system, and each source's reports are independent: a report from one source says nothing about another source's checks.
 
-Two source names are reserved for Canopy itself: `canopy` for conditions Canopy determines on its own (reachability, backup health, key expiry, self-monitoring), and `manual` for conditions raised by operators.
-Reports arriving over the device API cannot use the reserved names.
+One source name is reserved for Canopy itself: `canopy`, for conditions Canopy determines on its own (reachability, backup health, key expiry, self-monitoring).
+Reports arriving over the device API cannot use the reserved name.
 
 ### Source policy
 
-Each source other than the reserved names carries two operator-set modes, global to the source and edited alongside the check catalog.
+Each source other than the reserved name carries two operator-set modes, global to the source and edited alongside the check catalog.
 
 Its **reachability mode** governs how the source's silence bears on its servers' reachability (see "Reachability"):
 
@@ -162,11 +162,6 @@ A resolved state that degrades again reopens: the resolution is cleared and the
 
 **Notes** attach free-form operator commentary to a state.
 
-## Manual conditions
-
-Operators can raise a condition directly against a server, under the `manual` source, with a chosen check name, result, and message, and optionally marked as escalating.
-A manual condition behaves as a reported check whose reporter is the operator: it stays active until an operator resolves it or raises it again as recovered.
-
 ## Monitoring gate
 
 Server-targeted checks on a server that is not monitored are recorded and presented for visibility but do not contribute to incidents.

.workhorse/specs/monitoring/incidents.md

@@ -49,3 +49,14 @@ Resolution cascades to the incident's open issues — each is resolved with the
 Unresolving clears the resolution record; it does not reopen the incident.
 
 Notes attach free-form operator commentary to an incident.
+
+## Manual incidents
+
+Alongside the automatic incidents above, Canopy keeps manual incidents: records of incidents the support team managed, written after the fact rather than derived from check state.
+
+A manual incident carries a title, a markdown description, when it started, and when it ended; an ended time may be absent while the incident is ongoing.
+Every manual incident names exactly one server group as the affected target, and a group with manual incidents on record cannot be removed: the record is history.
+Each records who created it and when it was created and last changed.
+
+Manual incidents are independent of the check-state model: no issue joins them, they never notify, and nothing opens, closes, or resolves them except the people editing them.
+They are created, edited, and deleted both over the MCP interface (see [MCP](../private-server/mcp.md)) and in the operator UI, where they are presented alongside automatic incidents; either way every write is attributed to the identity that made it.

.workhorse/specs/private-server/mcp.md

@@ -4,8 +4,9 @@ id: MCP
 
 # Fleet query interface
 
-A read-only query interface to the Canopy fleet, exposed for AI agents and other automated clients that operators run.
+A query interface to the Canopy fleet, exposed for AI agents and other automated clients that operators run.
 It lets such a client discover servers and groups, read their status and health, learn what Tamanu versions exist and which are deployed, and inspect backup state and problems — without granting any ability to change the fleet.
+Its one write surface is the manual incident record (see [INC](../monitoring/incidents.md), "Manual incidents"), which suitably authorised callers can create and edit.
 
 ## Why it exists
 
@@ -32,16 +33,22 @@ Each query a caller makes is attributable to its identity: the tailnet user on t
 
 Access tokens are minted, listed, and revoked by administrators on the operator surface.
 Each token has an operator-chosen name, records who minted it and when, and carries a fixed lifetime of one year from minting that cannot be extended or chosen at mint time.
+A token is read-only unless it was minted with write access, a mint-time choice that cannot be changed afterwards; a token's scope is visible wherever the token is listed.
 The token secret is shown exactly once, at minting; the system persists only a digest of it, and never logs or re-displays the secret.
 A token can be revoked at any time, taking effect immediately.
 Each token records when it was last used, so idle tokens are visible.
 
 From fifteen days before a token's expiry until that token expires or is revoked, a [self-alert](self-alerts.md) is raised, so rotation happens on schedule rather than as an outage.
 
-## Read-only
+## Reads and writes
 
 Every query in this interface only reads.
-Nothing it exposes creates, modifies, deletes, or triggers any fleet action, and no query has a side effect beyond being recorded as having happened.
+Nothing it exposes triggers any fleet action, and no query has a side effect beyond being recorded as having happened.
+
+The sole write surface is the manual incident record: the interface can create, edit, and delete manual incidents, which describe incidents the support team managed and touch nothing in the fleet or the automatic monitoring.
+On the token path, writing requires a token minted with write access; a read-only token's write attempt is refused with a message saying so.
+On the operator path, any tailnet user may write.
+Every write is attributed: the identity that made it — the token's name or the tailnet user — is recorded as the manual incident's author at creation.
 Mutating the fleet is out of scope for this interface.
 
 ## Queries
@@ -94,6 +101,23 @@ A summary or ranking of incidents should count published incidents rather than r
 
 **Get issue** takes an issue identifier and returns the issue with the incidents it is or was part of.
 
+### Manual incidents
+
+Manual incidents (see [INC](../monitoring/incidents.md), "Manual incidents") are both queried and written through this interface.
+
+**Find manual incidents** optionally narrows by group and to ongoing ones only, and returns a bounded list ordered most recently started first, each with its title, description, start and end times, affected group (with its name resolved), author, and when it was created and last changed.
+When the result is truncated to its bound, the result says so.
+
+**Get manual incident** takes an identifier and returns the full record.
+
+**Record manual incident** takes a title, a start time, and the affected group, and optionally a markdown description and an end time; it creates the record with the caller's identity as author and returns it.
+
+**Update manual incident** takes an identifier and any subset of title, description, start time, end time, and affected group, and applies them; it can also explicitly clear the end time, marking the incident ongoing again.
+
+**Delete manual incident** takes an identifier and removes the record.
+
+### Check knowledge
+
 **Get check documentation** takes a source and check name and returns the check's operator-authored markdown documentation (see [CHK](../monitoring/checks.md), "Documentation"), which by convention covers what the check observes, what each result means, and how to solve a failure.
 A client investigating an issue consults this before deriving a check's meaning from other sources.
 

.workhorse/specs/public-server/statuses.md

@@ -17,7 +17,7 @@ The payload carries:
 - **source** — the name of the reporter pushing this status.
   Transitionally optional: a push without a source is attributed to `alertd`.
   The field will become mandatory; new reporters must send it.
-  The reserved source names (see [CHK](../monitoring/checks.md), "Sources") are rejected.
+  The reserved source name (see [CHK](../monitoring/checks.md), "Sources") is rejected.
 - **health** — the source's complete set of checks: for each, the check's name, exactly one result (`passed`, `warning`, `failed`, `broken`, or `skipped`), and any further detail fields, which are recorded verbatim against the check.
   The set may be empty, meaning the source currently has no checks — which recovers every check it previously reported.
 - any further top-level fields, recorded verbatim as the status's server-wide detail.

crates/canopy-mcp/Cargo.toml

@@ -13,6 +13,7 @@ authors = [
 [dependencies]
 commons-types = { path = "../commons-types" }
 database = { path = "../database" }
+http.workspace = true
 jiff = { version = "0.2.24", features = ["serde"] }
 rmcp = { version = "1.8", features = ["server", "macros", "transport-streamable-http-server"] }
 schemars = "1"

crates/canopy-mcp/src/incidents.rs

@@ -314,7 +314,9 @@ impl CanopyMcp {
 		)
 		.await?;
 		let ids: Vec<Uuid> = incidents.iter().map(|i| i.id).collect();
-		let stats = Incident::stats_for(&self.db, &ids).await.map_err(mcp_err)?;
+		let stats = Incident::stats_for(&self.db_read, &ids)
+			.await
+			.map_err(mcp_err)?;
 		let published = SlackOutbox::delivered_open_ids(&mut conn, &ids)
 			.await
 			.map_err(mcp_err)?;

crates/canopy-mcp/src/lib.rs

@@ -1,12 +1,14 @@
-//! Read-only MCP (Model Context Protocol) query interface over the fleet.
+//! MCP (Model Context Protocol) query interface over the fleet.
 //!
 //! Spec: `.workhorse/specs/private-server/mcp.md` (id `MCP`).
 //!
 //! Mounted twice: at `/api/mcp` on the operator surface, behind the
 //! tagged-device guard and an "any tailnet user" gate (private-server's
 //! `mcp::require_tailnet_user`), and at `/mcp` on the internet-facing
 //! surface behind the bearer-token gate (public-server's `mcp` module).
-//! Every tool only reads; nothing here mutates the fleet.
+//! Every fleet tool only reads; nothing here mutates the fleet. The one
+//! write surface is the manual incident record (`manual_incidents`
+//! module), gated per caller by [`McpIdentity`].
 //!
 //! Tools call the existing `database` read functions directly and shape lean,
 //! agent-legible JSON. The one piece of logic that must NOT be reimplemented is
@@ -15,15 +17,17 @@
 //! so the verdicts match what the operator UI and the alerting sweep present.
 //!
 //! Tools are grouped into domain modules (`servers`, `groups`, `versions`,
-//! `fleet`, `backups`, `restore`, `incidents`), each contributing its own tool
-//! router (via rmcp's `#[tool_router(router = ..., vis = "pub(crate)")]`) that
-//! [`CanopyMcp::new`] combines into the single stored `ToolRouter`. `util`
-//! holds helpers shared across more than one of those modules.
+//! `fleet`, `backups`, `restore`, `incidents`, `manual_incidents`), each
+//! contributing its own tool router (via rmcp's `#[tool_router(router = ...,
+//! vis = "pub(crate)")]`) that [`CanopyMcp::new`] combines into the single
+//! stored `ToolRouter`. `util` holds helpers shared across more than one of
+//! those modules.
 
 mod backups;
 mod fleet;
 mod groups;
 mod incidents;
+mod manual_incidents;
 mod restore;
 mod servers;
 mod util;
@@ -40,28 +44,53 @@ use rmcp::{
 	},
 };
 
+/// The authenticated caller, inserted into the HTTP request's extensions by
+/// each mount's auth gate. The transport carries the request parts into the
+/// tool context, where the write tools read this to authorise and attribute
+/// the write; the read tools never look at it.
+#[derive(Clone, Debug)]
+pub struct McpIdentity {
+	/// The tailnet user's login (operator mount) or the token's name
+	/// (internet-facing mount); recorded as the author of writes.
+	pub who: String,
+	/// Whether this caller may use the write tools: always for tailnet
+	/// users, only for tokens minted with write access.
+	pub can_write: bool,
+}
+
 #[derive(Clone)]
 pub struct CanopyMcp {
-	db: database::Db,
+	db_read: database::Db,
+	db_write: database::Db,
 	tool_router: ToolRouter<CanopyMcp>,
 }
 
 impl CanopyMcp {
-	pub fn new(db: database::Db) -> Self {
+	pub fn new(db_write: database::Db, db_read: database::Db) -> Self {
 		Self {
-			db,
+			db_read,
+			db_write,
 			tool_router: Self::servers_router()
 				+ Self::groups_router()
 				+ Self::versions_router()
 				+ Self::fleet_router()
 				+ Self::backups_router()
 				+ Self::restore_router()
-				+ Self::incidents_router(),
+				+ Self::incidents_router()
+				+ Self::manual_incidents_router(),
 		}
 	}
 
 	async fn conn(&self) -> Result<impl std::ops::DerefMut<Target = AsyncPgConnection>, McpError> {
-		self.db.get().await.map_err(util::mcp_err)
+		self.db_read.get().await.map_err(util::mcp_err)
+	}
+
+	/// A connection on the primary pool, for the manual-incident write
+	/// tools; everything else reads via [`Self::conn`].
+	async fn write_conn(
+		&self,
+	) -> Result<impl std::ops::DerefMut<Target = AsyncPgConnection>, McpError> {
+		self.db_write.get().await.map_err(util::mcp_err)
 	}
 }
 
@@ -70,9 +99,13 @@ impl ServerHandler for CanopyMcp {
 	fn get_info(&self) -> ServerInfo {
 		let mut info = ServerInfo::default();
 		info.instructions = Some(
-			"Read-only access to the Canopy fleet: servers, groups, health/status, Tamanu \
-			 versions, backups, and incidents/issues. All data is live. Use find_* to locate \
-			 entities and get_* for detail; fleet_summary and find_backup_problems for triage.\n\n\
+			"Access to the Canopy fleet: servers, groups, health/status, Tamanu versions, \
+			 backups, and incidents/issues. All data is live. Use find_* to locate entities and \
+			 get_* for detail; fleet_summary and find_backup_problems for triage. Everything is \
+			 read-only except manual incidents: support-recorded incident records \
+			 (record/update/delete_manual_incident), which touch nothing else in the fleet. On \
+			 the token-authenticated surface those write tools need a token minted with write \
+			 access.\n\n\
 			 Incidents: an incident groups the issues active for a group over a span of time. \
 			 find_incidents returns everything open in the window, including heavy sub-grace \
 			 flapping that was recorded but never surfaced. When summarizing or ranking, count \
@@ -93,8 +126,13 @@ impl ServerHandler for CanopyMcp {
 
 /// Build the tower service nested into an axum router (`/api/mcp` on the
 /// operator surface, `/mcp` on the internet-facing one). Auth is the mount's
-/// business, not this service's.
-pub fn service(db: database::Db) -> StreamableHttpService<CanopyMcp, LocalSessionManager> {
+/// business, not this service's: the mount's gate must insert an
+/// [`McpIdentity`] into the request extensions for the write tools to work.
+/// Reads go to `db_read`; only the manual-incident writes touch `db_write`.
+pub fn service(
+	db_write: database::Db,
+	db_read: database::Db,
+) -> StreamableHttpService<CanopyMcp, LocalSessionManager> {
 	let mut config = StreamableHttpServerConfig::default();
 	// Stateless: each request is self-contained, with no server-side session.
 	// The default stateful mode keeps sessions in process memory and 404s
@@ -129,7 +167,7 @@ pub fn service(db: database::Db) -> StreamableHttpService<CanopyMcp, LocalSessio
 	}
 
 	StreamableHttpService::new(
-		move || Ok(CanopyMcp::new(db.clone())),
+		move || Ok(CanopyMcp::new(db_write.clone(), db_read.clone())),
 		LocalSessionManager::default().into(),
 		config,
 	)