diff --git a/packages/core/doc/spec/watchdogs-and-execution-contracts.md b/packages/core/doc/spec/watchdogs-and-execution-contracts.md new file mode 100644 index 00000000..0cbc561e --- /dev/null +++ b/packages/core/doc/spec/watchdogs-and-execution-contracts.md @@ -0,0 +1,309 @@ +# Watchdogs And Execution Contracts + +Status: Draft +Date: 2026-05-28 +Audience: Product + Engineering +Scope: Run accountability, deliverable guarantees, deviation detection, and +controlled multi-runtime fallback + +## 1. Why this exists + +Paperclip already has strong runtime primitives: + +- `heartbeat_runs` for execution state +- `heartbeat_run_events` for evidence +- `heartbeat_run_watchdog_decisions` for bounded overrides +- `deliverables` for proof of work +- `requestWakeup(s)` for controlled actuation +- recovery services for silent runs, stranded issues, and retry storms + +What is still missing is the contract layer that answers: + +1. What was this task expected to produce? +2. Which runtime/profile was allowed to execute it? +3. What counts as a deviation? +4. Which watchdog subscribes to that deviation? +5. What is the bounded automatic response? + +This spec defines that abstraction. + +## 2. Design goals + +1. No silent failures. +2. No invisible retries or invisible cost burn. +3. Every successful task must have proof, validation, or escalation. +4. Multiple runtime profiles can coexist intentionally: + - Claude stable + - Claude preview/alternate version + - Codex backup +5. Watchdog actions must respect existing host controls: + - budgets + - wakeup policy + - assignment ownership + - pause holds + - recovery issue lineage + +## 3. Core concepts + +### 3.1 Execution profile + +An execution profile is a named runtime policy for how a task is attempted. + +Examples: + +- `claude-stable` +- `claude-preview` +- `claude-heavy-research` +- `codex-backup` + +Each profile defines: + +- adapter family (`claude_local`, `codex_local`, etc.) +- model +- optional channel/version hint +- retry / fallback policy +- metadata for governance and UI + +Profiles are reusable. Tasks should point at profiles, not raw provider strings. + +### 3.2 Execution contract + +An execution contract binds an expected outcome to a scope. + +Scopes may include: + +- issue +- agent +- workflow step +- goal + +The contract declares: + +- allowed execution profiles +- fallback profile, if any +- expected deliverables +- freshness/latency SLA +- validators +- escalation owner + wake reason +- automatic retry ceiling + +The contract is the source of truth for “what good looks like.” + +### 3.3 Watchdog signal + +A watchdog signal is normalized evidence that something notable happened or did +not happen. + +Examples: + +- run is silent too long +- run failed +- run succeeded but expected deliverable is missing +- deliverable exists but validation failed +- budget threshold crossed +- cost slope deviated from baseline + +Signals should be normalized enough that both core-runtime and WaveX-specific +policy can reason over them. + +### 3.4 Watchdog rule + +A watchdog rule subscribes to one or more signal types and decides whether to +emit one or more bounded actions. + +Rules should not directly mutate arbitrary task content. They should produce +decisions such as: + +- create evaluation issue +- request issue wakeup +- switch execution profile +- pause agent +- pause fleet +- escalate to owner role +- snooze + +### 3.5 Watchdog decision + +A watchdog decision is the durable output of a rule evaluation. + +It should be explainable: + +- which rule fired +- which contract applied +- what evidence triggered it +- what action was chosen +- whether a human or agent overrode/snoozed it later + +This is where `heartbeat_run_watchdog_decisions` already gives us a strong +precedent for run-level cases. + +## 4. Architecture shape + +The abstraction sits above existing runtime services: + +1. existing runtime emits run state and run evidence +2. contract resolver finds the applicable execution contract +3. watchdog registry selects rules subscribed to the signal +4. rules emit normalized actions +5. host-owned services execute those actions through safe pathways + +Important: the watchdog layer is decisioning, not direct process control. +Actual execution still goes through existing host services. + +## 5. Event channels + +Use both: + +### 5.1 Event-driven channels + +For fast reaction: + +- `agent.run.failed` +- `activity.logged` +- `budget.incident.opened` +- `cost_event.created` +- deliverable/activity events + +### 5.2 Periodic scans + +For absence-based failures: + +- silent active runs +- issue completed with no deliverable +- validation overdue +- fallback rate anomaly +- conversation/context growth anomaly + +The existing active-run watchdog is the model here: not every failure has a +single event edge. + +## 6. Default rule families + +### 6.1 Run liveness watchdog + +Purpose: + +- detect active runs that have stopped producing useful output + +Typical actions: + +- create evaluation issue +- request recovery owner wakeup +- escalate when critical silence crosses the higher threshold + +This should build on the existing silent-run watchdog, not replace it. + +### 6.2 Deliverable missing watchdog + +Purpose: + +- detect “run says success” or “issue says done” but expected proof is missing + +Typical actions: + +- create evaluation issue +- wake original assignee or validator +- mark contract as violated + +This is the main bridge between runtime correctness and artifact correctness. + +### 6.3 Deliverable validation watchdog + +Purpose: + +- detect artifacts that exist but fail schema/tests/review/semantic validators + +Typical actions: + +- open validator/recovery issue +- request assignment wakeup for the producing issue +- block dependent approvals + +### 6.4 Quota and cost watchdog + +Purpose: + +- prevent runaway burn, invisible fallback churn, or exhausted-provider storms + +Typical actions: + +- switch to backup execution profile when policy allows +- pause agent or fleet on hard ceilings +- open operator-facing issue when fallback surge exceeds threshold + +This is where Claude-primary / Codex-backup governance lives. + +### 6.5 Context-growth watchdog + +Purpose: + +- catch conversations/issues whose context cost keeps compounding + +Typical actions: + +- request summarization/compression step +- switch to cheaper profile when contract allows +- escalate when cost slope exceeds budget + +## 7. Multi-runtime policy + +This abstraction is explicitly meant to support intentional overlap between +multiple Claude variants and Codex backup. + +Recommended shape: + +- primary profile: + `claude-stable` +- alternate Claude profile: + `claude-preview` +- backup profile: + `codex-backup` + +Recommended governance: + +- fallback should be opt-in per contract +- backup profile activation must be attributable +- provider fallback should be queryable by issue/run/session +- hard budget incidents may pause work instead of falling back if the contract + forbids model/provider changes + +## 8. Safety rules + +1. Watchdogs should fail closed on observability loss when the action is + safety-critical. +2. Watchdogs must never create infinite retry loops. +3. Watchdogs must not silently rewrite deliverables. +4. Automatic actions must be bounded and reversible. +5. Every automatic action should leave durable evidence. + +## 9. Implementation plan + +Phase 1: + +- add typed execution-profile, execution-contract, signal, rule, and decision + scaffolding +- add a registry + contract resolver +- add default rule shapes: + - run liveness + - deliverable missing + - quota/cost + +Phase 2: + +- wire the registry into existing runtime/recovery event sources +- persist rule evaluations +- add UI surfaces for contract + watchdog state + +Phase 3: + +- integrate validator pipelines and conversation/context optimization +- add company-configurable watchdog policies + +## 10. Non-goals for this slice + +- full runtime integration +- UI implementation +- new autonomous remediation behavior beyond existing safe host actions +- provider-specific streaming chunk accounting changes + +This slice should establish the abstraction, not finish the whole runtime. diff --git a/packages/core/server/src/routes/github-ci-webhook.ts b/packages/core/server/src/routes/github-ci-webhook.ts index cb3e2043..19b4b510 100644 --- a/packages/core/server/src/routes/github-ci-webhook.ts +++ b/packages/core/server/src/routes/github-ci-webhook.ts @@ -48,7 +48,11 @@ export function githubCiWebhookRoutes(db: Db) { // Secret: value of GITHUB_WEBHOOK_SECRET env var // Events: Workflow runs router.post("/:companyId", async (req: Request, res: Response) => { - const { companyId } = req.params; + const companyId = firstParamValue(req.params.companyId); + if (!companyId) { + res.status(400).json({ error: "missing companyId" }); + return; + } const event = req.headers["x-github-event"] as string | undefined; const secret = process.env.GITHUB_WEBHOOK_SECRET; @@ -120,7 +124,7 @@ async function resolveUserId(db: Db, companyId: string, githubLogin: string | un // Look for a company member whose name or email matches the GitHub login. const members = await db - .select({ userId: companyMemberships.userId }) + .select({ userId: companyMemberships.principalId }) .from(companyMemberships) .where(eq(companyMemberships.companyId, companyId)) .limit(50); @@ -170,3 +174,9 @@ interface GitHubWorkflowRunPayload { }; repository?: { full_name: string }; } + +function firstParamValue(value: string | string[] | undefined): string | null { + if (typeof value === "string" && value.length > 0) return value; + if (Array.isArray(value)) return value.find((item) => item.length > 0) ?? null; + return null; +} diff --git a/packages/core/server/src/services/index.ts b/packages/core/server/src/services/index.ts index 6d2d530f..f93cdb1f 100644 --- a/packages/core/server/src/services/index.ts +++ b/packages/core/server/src/services/index.ts @@ -53,4 +53,20 @@ export { logActivity, type LogActivityInput } from "./activity-log.js"; export { notifyHireApproved, type NotifyHireApprovedInput } from "./hire-hook.js"; export { publishLiveEvent, subscribeCompanyLiveEvents } from "./live-events.js"; export { reconcilePersistedRuntimeServicesOnStartup, restartDesiredRuntimeServicesOnStartup } from "./workspace-runtime.js"; +export { + WatchdogRegistry, + createWatchdogRegistry, + snapshotWatchdogRegistry, + defaultWatchdogRules, + createRunLivenessWatchdogRule, + createDeliverableMissingWatchdogRule, + createQuotaAndCostWatchdogRule, + type ExecutionProfile, + type ExecutionProfileTarget, + type ExecutionContract, + type DeliverableExpectation, + type WatchdogSignal, + type WatchdogDecision, + type WatchdogRule, +} from "./watchdogs/index.js"; export { createStorageServiceFromConfig, getStorageService } from "../storage/index.js"; diff --git a/packages/core/server/src/services/watchdogs/defaults.ts b/packages/core/server/src/services/watchdogs/defaults.ts new file mode 100644 index 00000000..66b92d24 --- /dev/null +++ b/packages/core/server/src/services/watchdogs/defaults.ts @@ -0,0 +1,191 @@ +import type { + BudgetThresholdCrossedSignal, + DeliverableMissingSignal, + ExecutionProfile, + RunSilentSignal, + WatchdogDecision, + WatchdogRule, +} from "./types.js"; + +function fallbackDecisionReason(profile: ExecutionProfile | null): string { + if (!profile) return "Fallback requested but no fallback profile is configured"; + return `Switch execution to fallback profile ${profile.id}`; +} + +function buildRunLivenessDecision(signal: RunSilentSignal): WatchdogDecision { + const critical = signal.silenceAgeMs >= signal.criticalThresholdMs; + return { + ruleId: "run-liveness", + severity: critical ? "crit" : "warn", + reason: critical + ? `Run ${signal.runId ?? "unknown"} has been silent for ${signal.silenceAgeMs}ms, above the critical threshold` + : `Run ${signal.runId ?? "unknown"} has been silent for ${signal.silenceAgeMs}ms`, + actions: [ + { + type: "create_evaluation_issue", + payload: { + runId: signal.runId ?? null, + issueId: signal.issueId ?? null, + agentId: signal.agentId ?? null, + silenceAgeMs: signal.silenceAgeMs, + }, + }, + ...(critical + ? [{ + type: "request_issue_wakeup" as const, + payload: { + issueId: signal.issueId ?? null, + reason: "watchdog_run_silent", + }, + }] + : []), + ], + evidence: { + silenceAgeMs: signal.silenceAgeMs, + suspicionThresholdMs: signal.suspicionThresholdMs, + criticalThresholdMs: signal.criticalThresholdMs, + }, + }; +} + +function buildDeliverableMissingDecision(signal: DeliverableMissingSignal, contractId: string | null): WatchdogDecision { + return { + ruleId: "deliverable-missing", + severity: "warn", + contractId, + reason: `Expected deliverables are missing for issue ${signal.issueId ?? "unknown"}`, + actions: [ + { + type: "create_evaluation_issue", + payload: { + issueId: signal.issueId ?? null, + runId: signal.runId ?? null, + missingKinds: signal.missingKinds, + }, + }, + { + type: "request_issue_wakeup", + payload: { + issueId: signal.issueId ?? null, + reason: "watchdog_deliverable_gap", + }, + }, + ], + evidence: { + missingKinds: signal.missingKinds, + }, + }; +} + +function buildQuotaAndCostDecision( + signal: BudgetThresholdCrossedSignal, + fallbackProfile: ExecutionProfile | null, + contractId: string | null, +): WatchdogDecision { + if (signal.threshold === "soft" && fallbackProfile) { + return { + ruleId: "quota-and-cost", + severity: "warn", + contractId, + reason: fallbackDecisionReason(fallbackProfile), + actions: [ + { + type: "switch_execution_profile", + payload: { + issueId: signal.issueId ?? null, + runId: signal.runId ?? null, + profileId: fallbackProfile.id, + provider: signal.provider ?? null, + }, + }, + ], + evidence: { + threshold: signal.threshold, + utilizationPct: signal.utilizationPct, + }, + }; + } + + return { + ruleId: "quota-and-cost", + severity: "crit", + contractId, + reason: `Budget threshold crossed for provider ${signal.provider ?? "unknown"}`, + actions: [ + { + type: signal.issueId || signal.agentId ? "pause_agent" : "pause_fleet", + payload: { + issueId: signal.issueId ?? null, + agentId: signal.agentId ?? null, + provider: signal.provider ?? null, + threshold: signal.threshold, + utilizationPct: signal.utilizationPct, + }, + }, + { + type: "create_evaluation_issue", + payload: { + issueId: signal.issueId ?? null, + runId: signal.runId ?? null, + provider: signal.provider ?? null, + threshold: signal.threshold, + }, + }, + ], + evidence: { + threshold: signal.threshold, + utilizationPct: signal.utilizationPct, + fallbackProfileId: fallbackProfile?.id ?? null, + }, + }; +} + +export function createRunLivenessWatchdogRule(): WatchdogRule { + return { + id: "run-liveness", + displayName: "Run Liveness Watchdog", + description: "Detect active runs that have gone silent and create bounded recovery work.", + subscribesTo: ["run.silent"], + severity: "warn", + evaluate: ({ signal }) => signal.type === "run.silent" ? [buildRunLivenessDecision(signal)] : [], + }; +} + +export function createDeliverableMissingWatchdogRule(): WatchdogRule { + return { + id: "deliverable-missing", + displayName: "Deliverable Missing Watchdog", + description: "Catch completed work that lacks the proof artifact required by its execution contract.", + subscribesTo: ["deliverable.missing"], + severity: "warn", + evaluate: ({ signal, contract }) => + signal.type === "deliverable.missing" + ? [buildDeliverableMissingDecision(signal, contract?.id ?? null)] + : [], + }; +} + +export function createQuotaAndCostWatchdogRule(): WatchdogRule { + return { + id: "quota-and-cost", + displayName: "Quota And Cost Watchdog", + description: "Switch to backup profiles or pause work when quota or burn pressure breaches policy.", + subscribesTo: ["budget.threshold_crossed"], + severity: "crit", + evaluate: ({ signal, contract, profiles }) => { + if (signal.type !== "budget.threshold_crossed") return []; + const fallbackProfile = contract?.fallbackProfileId + ? profiles.get(contract.fallbackProfileId) ?? null + : null; + return [buildQuotaAndCostDecision(signal, fallbackProfile, contract?.id ?? null)]; + }, + }; +} + +export function defaultWatchdogRules(): WatchdogRule[] { + return [ + createRunLivenessWatchdogRule(), + createDeliverableMissingWatchdogRule(), + createQuotaAndCostWatchdogRule(), + ]; +} diff --git a/packages/core/server/src/services/watchdogs/index.ts b/packages/core/server/src/services/watchdogs/index.ts new file mode 100644 index 00000000..4ea9161d --- /dev/null +++ b/packages/core/server/src/services/watchdogs/index.ts @@ -0,0 +1,37 @@ +export { + defaultWatchdogRules, + createDeliverableMissingWatchdogRule, + createQuotaAndCostWatchdogRule, + createRunLivenessWatchdogRule, +} from "./defaults.js"; +export { + WatchdogRegistry, + createWatchdogRegistry, + snapshotWatchdogRegistry, + type EvaluateWatchdogSignalOptions, +} from "./registry.js"; +export type { + BudgetThresholdCrossedSignal, + CostAnomalySignal, + DeliverableExpectation, + DeliverableMissingSignal, + DeliverableValidationFailedSignal, + ExecutionContract, + ExecutionContractScope, + ExecutionContractScopeKind, + ExecutionEscalationPolicy, + ExecutionProfile, + ExecutionProfileFallbackPolicy, + ExecutionProfileTarget, + RunCompletedSignal, + RunFailedSignal, + RunSilentSignal, + WatchdogAction, + WatchdogActionType, + WatchdogDecision, + WatchdogEvaluationContext, + WatchdogRule, + WatchdogSeverity, + WatchdogSignal, + WatchdogSignalType, +} from "./types.js"; diff --git a/packages/core/server/src/services/watchdogs/registry.ts b/packages/core/server/src/services/watchdogs/registry.ts new file mode 100644 index 00000000..83bf6e76 --- /dev/null +++ b/packages/core/server/src/services/watchdogs/registry.ts @@ -0,0 +1,151 @@ +import type { + CostAnomalySignal, + ExecutionContract, + ExecutionProfile, + ExecutionProfileTarget, + WatchdogDecision, + WatchdogEvaluationContext, + WatchdogRule, + WatchdogSignal, +} from "./types.js"; + +function uniqueById(items: Iterable): Map { + const out = new Map(); + for (const item of items) out.set(item.id, item); + return out; +} + +function readSignalMetadataId(signal: WatchdogSignal, ...keys: string[]): string | null { + for (const key of keys) { + const value = signal.metadata?.[key]; + if (typeof value === "string" && value.length > 0) return value; + } + return null; +} + +function matchesCostAnomalyScope(contract: ExecutionContract, signal: CostAnomalySignal): boolean { + return contract.scope.kind === signal.scopeKind && contract.scope.refId === signal.scopeRef; +} + +function contractMatchesSignal(contract: ExecutionContract, signal: WatchdogSignal): boolean { + if (signal.type === "cost.anomaly" && matchesCostAnomalyScope(contract, signal)) return true; + switch (contract.scope.kind) { + case "issue": + return contract.scope.refId === signal.issueId; + case "agent": + return contract.scope.refId === signal.agentId; + case "workflow_step": + return contract.scope.refId === readSignalMetadataId(signal, "workflowStepId", "workflow_step_id"); + case "goal": + return contract.scope.refId === readSignalMetadataId(signal, "goalId", "goal_id"); + case "conversation": + return contract.scope.refId === signal.conversationId; + default: + return false; + } +} + +export interface EvaluateWatchdogSignalOptions { + now?: Date; +} + +export class WatchdogRegistry { + private readonly profiles = new Map(); + private readonly contracts = new Map(); + private readonly rules = new Map(); + + registerProfile(profile: ExecutionProfile): this { + this.profiles.set(profile.id, profile); + return this; + } + + registerProfiles(profiles: Iterable): this { + for (const profile of profiles) this.registerProfile(profile); + return this; + } + + registerContract(contract: ExecutionContract): this { + this.contracts.set(contract.id, contract); + return this; + } + + registerContracts(contracts: Iterable): this { + for (const contract of contracts) this.registerContract(contract); + return this; + } + + registerRule(rule: WatchdogRule): this { + this.rules.set(rule.id, rule); + return this; + } + + registerRules(rules: Iterable): this { + for (const rule of rules) this.registerRule(rule); + return this; + } + + listProfiles(): ExecutionProfile[] { + return [...this.profiles.values()]; + } + + listContracts(): ExecutionContract[] { + return [...this.contracts.values()]; + } + + listRules(): WatchdogRule[] { + return [...this.rules.values()]; + } + + resolveContract(signal: WatchdogSignal): ExecutionContract | null { + const exact = this.listContracts().find((contract) => contractMatchesSignal(contract, signal)); + if (exact) return exact; + return null; + } + + resolveProfile(profileId: string | null | undefined): ExecutionProfile | null { + if (!profileId) return null; + return this.profiles.get(profileId) ?? null; + } + + resolveFallbackTarget(contract: ExecutionContract | null): ExecutionProfileTarget | null { + if (!contract?.fallbackProfileId) return null; + const profile = this.resolveProfile(contract.fallbackProfileId); + return profile?.targets[0] ?? null; + } + + evaluateSignal(signal: WatchdogSignal, options?: EvaluateWatchdogSignalOptions): WatchdogDecision[] { + const contract = this.resolveContract(signal); + const ctx: WatchdogEvaluationContext = { + now: options?.now ?? new Date(), + signal, + contract, + profiles: this.profiles, + }; + const decisions: WatchdogDecision[] = []; + for (const rule of this.rules.values()) { + if (!rule.subscribesTo.includes(signal.type)) continue; + decisions.push(...rule.evaluate(ctx)); + } + return decisions; + } +} + +export function createWatchdogRegistry(input?: { + profiles?: Iterable; + contracts?: Iterable; + rules?: Iterable; +}): WatchdogRegistry { + const registry = new WatchdogRegistry(); + if (input?.profiles) registry.registerProfiles(input.profiles); + if (input?.contracts) registry.registerContracts(input.contracts); + if (input?.rules) registry.registerRules(input.rules); + return registry; +} + +export function snapshotWatchdogRegistry(registry: WatchdogRegistry) { + return { + profiles: uniqueById(registry.listProfiles()).size, + contracts: uniqueById(registry.listContracts()).size, + rules: uniqueById(registry.listRules()).size, + }; +} diff --git a/packages/core/server/src/services/watchdogs/types.ts b/packages/core/server/src/services/watchdogs/types.ts new file mode 100644 index 00000000..97d6eb0d --- /dev/null +++ b/packages/core/server/src/services/watchdogs/types.ts @@ -0,0 +1,182 @@ +export type ExecutionProfileFallbackPolicy = + | "manual" + | "on_quota" + | "on_provider_error" + | "on_watchdog_trigger"; + +export interface ExecutionProfileTarget { + adapterType: string; + model: string; + cliChannel?: string | null; + versionTag?: string | null; + maxAttempts?: number | null; + metadata?: Record; +} + +export interface ExecutionProfile { + id: string; + displayName: string; + description?: string; + targets: ExecutionProfileTarget[]; + fallbackPolicy: ExecutionProfileFallbackPolicy; + tags?: string[]; + metadata?: Record; +} + +export type ExecutionContractScopeKind = + | "issue" + | "agent" + | "workflow_step" + | "goal" + | "conversation"; + +export interface ExecutionContractScope { + kind: ExecutionContractScopeKind; + refId: string; +} + +export interface DeliverableExpectation { + kind: string; + required: boolean; + minCount?: number; + validatorIds?: string[]; + description?: string; +} + +export interface ExecutionEscalationPolicy { + ownerRole?: string | null; + wakeReason: string; + maxAutomaticRetries: number; + createEvaluationIssue: boolean; +} + +export interface ExecutionContract { + id: string; + companyId?: string | null; + scope: ExecutionContractScope; + allowedProfileIds: string[]; + fallbackProfileId?: string | null; + expectedDeliverables: DeliverableExpectation[]; + freshnessSlaMs?: number | null; + validatorIds?: string[]; + escalation: ExecutionEscalationPolicy; + metadata?: Record; +} + +export type WatchdogSeverity = "info" | "warn" | "crit"; + +export type WatchdogSignalType = + | "run.silent" + | "run.completed" + | "run.failed" + | "deliverable.missing" + | "deliverable.validation_failed" + | "budget.threshold_crossed" + | "cost.anomaly"; + +interface WatchdogSignalBase { + type: WatchdogSignalType; + companyId: string; + occurredAt: Date; + issueId?: string | null; + agentId?: string | null; + runId?: string | null; + conversationId?: string | null; + profileId?: string | null; + provider?: string | null; + metadata?: Record; +} + +export interface RunSilentSignal extends WatchdogSignalBase { + type: "run.silent"; + silenceAgeMs: number; + suspicionThresholdMs: number; + criticalThresholdMs: number; +} + +export interface RunCompletedSignal extends WatchdogSignalBase { + type: "run.completed"; + outcome: "succeeded" | "failed" | "cancelled" | "timed_out"; + producedDeliverableKinds?: string[]; +} + +export interface RunFailedSignal extends WatchdogSignalBase { + type: "run.failed"; + errorCode?: string | null; + errorMessage?: string | null; +} + +export interface DeliverableMissingSignal extends WatchdogSignalBase { + type: "deliverable.missing"; + missingKinds: string[]; +} + +export interface DeliverableValidationFailedSignal extends WatchdogSignalBase { + type: "deliverable.validation_failed"; + deliverableKind: string; + validatorId?: string | null; + failureReason: string; +} + +export interface BudgetThresholdCrossedSignal extends WatchdogSignalBase { + type: "budget.threshold_crossed"; + threshold: "soft" | "hard"; + utilizationPct: number; +} + +export interface CostAnomalySignal extends WatchdogSignalBase { + type: "cost.anomaly"; + scopeKind: ExecutionContractScopeKind | "fleet"; + scopeRef: string; + observedCostCents: number; + baselineCostCents: number; + deviationPct: number; +} + +export type WatchdogSignal = + | RunSilentSignal + | RunCompletedSignal + | RunFailedSignal + | DeliverableMissingSignal + | DeliverableValidationFailedSignal + | BudgetThresholdCrossedSignal + | CostAnomalySignal; + +export type WatchdogActionType = + | "create_evaluation_issue" + | "request_issue_wakeup" + | "switch_execution_profile" + | "pause_agent" + | "pause_fleet" + | "escalate_to_role" + | "snooze_rule"; + +export interface WatchdogAction { + type: WatchdogActionType; + payload?: Record; +} + +export interface WatchdogDecision { + ruleId: string; + severity: WatchdogSeverity; + reason: string; + contractId?: string | null; + actions: WatchdogAction[]; + evidence?: Record; +} + +export interface WatchdogEvaluationContext { + now: Date; + signal: WatchdogSignal; + contract: ExecutionContract | null; + profiles: ReadonlyMap; +} + +export interface WatchdogRule { + id: string; + displayName: string; + description: string; + subscribesTo: WatchdogSignalType[]; + severity: WatchdogSeverity; + evaluate(ctx: WatchdogEvaluationContext): WatchdogDecision[]; +} diff --git a/packages/core/server/src/services/watchdogs/watchdogs.test.ts b/packages/core/server/src/services/watchdogs/watchdogs.test.ts new file mode 100644 index 00000000..6efa55af --- /dev/null +++ b/packages/core/server/src/services/watchdogs/watchdogs.test.ts @@ -0,0 +1,236 @@ +import { describe, expect, it } from "vitest"; +import { + createDeliverableMissingWatchdogRule, + createQuotaAndCostWatchdogRule, + createRunLivenessWatchdogRule, +} from "./defaults.js"; +import { createWatchdogRegistry, snapshotWatchdogRegistry } from "./registry.js"; +import type { BudgetThresholdCrossedSignal, ExecutionContract, ExecutionProfile, RunSilentSignal, WatchdogSignal } from "./types.js"; + +const primaryProfile: ExecutionProfile = { + id: "claude-stable", + displayName: "Claude Stable", + targets: [{ adapterType: "claude_local", model: "sonnet", cliChannel: "stable" }], + fallbackPolicy: "manual", +}; + +const fallbackProfile: ExecutionProfile = { + id: "codex-backup", + displayName: "Codex Backup", + targets: [{ adapterType: "codex_local", model: "codex" }], + fallbackPolicy: "on_quota", +}; + +function makeContract(overrides?: Partial): ExecutionContract { + return { + id: "contract-1", + scope: { kind: "issue", refId: "issue-1" }, + allowedProfileIds: [primaryProfile.id], + fallbackProfileId: fallbackProfile.id, + expectedDeliverables: [{ kind: "patch", required: true, minCount: 1 }], + escalation: { + wakeReason: "watchdog_issue", + maxAutomaticRetries: 1, + createEvaluationIssue: true, + }, + ...overrides, + }; +} + +function makeRunSilentSignal(overrides?: Partial): RunSilentSignal { + return { + type: "run.silent", + companyId: "company-1", + occurredAt: new Date("2026-05-28T12:00:00Z"), + issueId: "issue-1", + agentId: "agent-1", + runId: "run-1", + silenceAgeMs: 31_000, + suspicionThresholdMs: 30_000, + criticalThresholdMs: 60_000, + ...overrides, + }; +} + +function makeBudgetSignal(overrides?: Partial): BudgetThresholdCrossedSignal { + return { + type: "budget.threshold_crossed", + companyId: "company-1", + occurredAt: new Date("2026-05-28T12:00:00Z"), + issueId: "issue-1", + agentId: "agent-1", + runId: "run-1", + provider: "anthropic", + threshold: "soft", + utilizationPct: 92, + ...overrides, + }; +} + +describe("watchdog registry", () => { + it("resolves workflow step contracts from signal metadata", () => { + const registry = createWatchdogRegistry({ + contracts: [makeContract({ scope: { kind: "workflow_step", refId: "step-7" } })], + }); + + const contract = registry.resolveContract({ + type: "deliverable.missing", + companyId: "company-1", + occurredAt: new Date("2026-05-28T12:00:00Z"), + metadata: { workflowStepId: "step-7" }, + missingKinds: ["patch"], + }); + + expect(contract?.id).toBe("contract-1"); + }); + + it("resolves goal contracts from signal metadata", () => { + const registry = createWatchdogRegistry({ + contracts: [makeContract({ scope: { kind: "goal", refId: "goal-9" } })], + }); + + const contract = registry.resolveContract({ + type: "deliverable.missing", + companyId: "company-1", + occurredAt: new Date("2026-05-28T12:00:00Z"), + metadata: { goal_id: "goal-9" }, + missingKinds: ["report"], + }); + + expect(contract?.id).toBe("contract-1"); + }); + + it("resolves cost anomaly contracts from explicit scope references", () => { + const registry = createWatchdogRegistry({ + contracts: [makeContract({ scope: { kind: "conversation", refId: "conv-4" } })], + }); + + const contract = registry.resolveContract({ + type: "cost.anomaly", + companyId: "company-1", + occurredAt: new Date("2026-05-28T12:00:00Z"), + scopeKind: "conversation", + scopeRef: "conv-4", + observedCostCents: 4500, + baselineCostCents: 900, + deviationPct: 400, + }); + + expect(contract?.id).toBe("contract-1"); + }); + + it("captures snapshot counts for configured registry state", () => { + const registry = createWatchdogRegistry({ + profiles: [primaryProfile, fallbackProfile], + contracts: [makeContract()], + rules: [createRunLivenessWatchdogRule()], + }); + + expect(snapshotWatchdogRegistry(registry)).toEqual({ + profiles: 2, + contracts: 1, + rules: 1, + }); + }); +}); + +describe("default watchdog rules", () => { + it("emits a wakeup for critical silent runs", () => { + const rule = createRunLivenessWatchdogRule(); + + const [decision] = rule.evaluate({ + now: new Date("2026-05-28T12:05:00Z"), + signal: makeRunSilentSignal({ silenceAgeMs: 65_000 }), + contract: null, + profiles: new Map(), + }); + + expect(decision.severity).toBe("crit"); + expect(decision.actions.map((action) => action.type)).toEqual([ + "create_evaluation_issue", + "request_issue_wakeup", + ]); + }); + + it("includes the resolved contract id for deliverable gaps", () => { + const rule = createDeliverableMissingWatchdogRule(); + const contract = makeContract(); + + const [decision] = rule.evaluate({ + now: new Date("2026-05-28T12:05:00Z"), + signal: { + type: "deliverable.missing", + companyId: "company-1", + occurredAt: new Date("2026-05-28T12:00:00Z"), + issueId: "issue-1", + runId: "run-1", + missingKinds: ["patch"], + }, + contract, + profiles: new Map(), + }); + + expect(decision.contractId).toBe("contract-1"); + expect(decision.actions.map((action) => action.type)).toEqual([ + "create_evaluation_issue", + "request_issue_wakeup", + ]); + }); + + it("switches to the fallback profile on a soft budget threshold", () => { + const registry = createWatchdogRegistry({ + profiles: [primaryProfile, fallbackProfile], + contracts: [makeContract()], + rules: [createQuotaAndCostWatchdogRule()], + }); + + const [decision] = registry.evaluateSignal(makeBudgetSignal()); + + expect(decision.reason).toContain("codex-backup"); + expect(decision.actions).toEqual([ + { + type: "switch_execution_profile", + payload: { + issueId: "issue-1", + runId: "run-1", + profileId: "codex-backup", + provider: "anthropic", + }, + }, + ]); + }); + + it("pauses the agent and creates recovery work on a hard threshold without fallback", () => { + const registry = createWatchdogRegistry({ + profiles: [primaryProfile], + contracts: [makeContract({ fallbackProfileId: null })], + rules: [createQuotaAndCostWatchdogRule()], + }); + + const [decision] = registry.evaluateSignal(makeBudgetSignal({ threshold: "hard" })); + + expect(decision.severity).toBe("crit"); + expect(decision.actions.map((action) => action.type)).toEqual([ + "pause_agent", + "create_evaluation_issue", + ]); + }); + + it("ignores unrelated signals for rules that do not subscribe", () => { + const rule = createQuotaAndCostWatchdogRule(); + + expect( + rule.evaluate({ + now: new Date("2026-05-28T12:05:00Z"), + signal: { + type: "run.failed", + companyId: "company-1", + occurredAt: new Date("2026-05-28T12:00:00Z"), + errorCode: "boom", + } satisfies WatchdogSignal, + contract: null, + profiles: new Map(), + }), + ).toEqual([]); + }); +});