Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion docs/architecture/ownership/cells/harness-eval.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ code_anchors:
- packages/api/src/infrastructure/harness-eval/harness-ledger-snapshot-provider.ts
- packages/api/src/infrastructure/harness-eval/segment-judgment-engine.ts
- packages/api/src/infrastructure/harness-eval/guard-threshold-escalation.ts
- packages/api/src/infrastructure/harness-eval/objective-registry.ts
- packages/api/src/routes/callback-docs-routes.ts
- packages/mcp-server/src/tools/list-objectives-tool.ts
- packages/api/src/infrastructure/harness-eval/publish-verdict/harness-ledger-generator-adapter.ts
- packages/api/src/domains/prompt-hooks/InjectionTraceStore.ts
- packages/api/src/domains/prompt-hooks/SegmentJudgmentCache.ts
Expand All @@ -52,15 +55,17 @@ doc_anchors:
- docs/features/F245-friction-signal-eval.md
- docs/features/F257-harness-ledger.md
- docs/features/assets/F257/
- docs/harness-feedback/objectives/registry.yaml
- docs/harness-feedback/eval-domains/eval-harness-ledger.yaml
- docs/harness-feedback/
- feature-discussions/2026-05-21-f192-phase-e-eval-hub-kickoff/README.md
- sop-definitions/README.md
static_scan_hints: [harness-eval, VerdictHandoffPacket, eval-domain, reeval, harness-fit-digest, Eval Hub, SopDefinition, sop-definitions, predicate, friction, paw-feel, FrictionSignal, harness-ledger, GuardRejectionEvent, SegmentJudgment, segment-lifeline, denominatorKind]
static_scan_hints: [harness-eval, VerdictHandoffPacket, eval-domain, reeval, harness-fit-digest, Eval Hub, SopDefinition, sop-definitions, predicate, friction, paw-feel, FrictionSignal, harness-ledger, GuardRejectionEvent, SegmentJudgment, segment-lifeline, denominatorKind, ObjectiveRegistry, objective-registry, list_objectives, objectiveId]
cited_by:
- F192 Phase E-pilot
- F245 Phase A (paw-feel friction collector) + Phase B (cancel/user-feedback/eval-domain adapters + aggregator + clusterer + rollup input; domain registration + rollup sink land in Phase C)
- F257 Phase A (observation + snapshot) + Phase C (eval:harness-ledger + SegmentJudgment) + Phase D (segment lifeline + governance operations)
- F257 #3 (objective registry definition layer + list_objectives discovery; canonical objectiveId source for report_harness_signal)
---

# Harness Eval Control Plane
Expand Down Expand Up @@ -88,6 +93,7 @@ F192 owns the socio-technical harness evaluation contract: harnesses declare exp
- Require dry-run evidence before disabling or redirecting legacy scheduled tasks.
- Keep F257 observation and judgment joins on the frozen three-key window contract unless a later schema version explicitly upgrades correlation.
- Add new ledger evidence sources through the existing snapshot/judgment path instead of creating a second eval control plane.
- F257 #3 objective registry (`objective-registry.ts` loader + `docs/harness-feedback/objectives/registry.yaml` + `list_objectives` MCP discovery) is the **canonical objective DEFINITION layer** of this same control plane — the read-only `id + statement` source `report_harness_signal`'s `objectiveId` and (V2) eval_models reference. It is NOT a parallel registry: unit→objective membership stays in the versioned `UnitEvaluationManifest` (redesign §4.8), and the loader fail-closes rather than emit an empty catalog, so one authority governs discovery.

## Do NOT Unify With

Expand Down
32 changes: 32 additions & 0 deletions docs/harness-feedback/objectives/registry.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
# F257 #3 — Objective registry (definition layer). Read-only discovery source
# for report_harness_signal's `objectiveId`, so cats stop doing archaeology to
# find valid objectives (修复清单 #3: "三次上报三次考古"). Surfaced at call time
# by the `cat_cafe_list_objectives` MCP tool + report_harness_signal description
# sync.
#
# KD-3 (F257 spec): registry 定义层用 YAML;运行时 stats 拆到 Redis/eval,Console
# 读时 join. OQ-1: YAML registry + runtime stats split. This file is the canonical
# objective DEFINITION layer only (id / statement). Unit→objective membership
# (挂靠段) is NOT authored here — per frozen redesign §4.8 it belongs to the
# versioned UnitEvaluationManifest.objectives keyed by unitRef; a derived
# read-model can expose attachments in V2 (2a R1 P1-1: single authority, no dual
# writable source). eval_models (metrics/conditions) are likewise V2.
#
# SCOPE (slice 2a): objectives are enumerated as they are CANONIZED. The redesign
# (assets/F257/objective-driven-redesign-v1.md §2) names 8 objectives (OBJ-1..8)
# but only canonizes ids for the two that are in use / imminent. The other six
# (OBJ-2/4/5/6/7/8) are Chinese-name-only there; their canonical ids + eval_models
# land with redesign V2. Registering invented ids now would fabricate a contract
# V2 depends on — so this registry starts with the canonized set and grows as
# objectives are canonized. The discovery tool always reflects the current set.
registryVersion: 1
objectives:
- id: obj-routing-delivery
# OBJ-1 (redesign §2). In use: V1 T-A RoutingDecisionFact / report_harness_signal.
statement: 球权经 @ 路由准确送达目标猫,不掉地、不假接
- id: obj-identity-integrity
# OBJ-3 (redesign §2). Target objective for #4's signature-miss ledger closure
# (deferred to after PR #47 merge). Registered now so #4's closure has a
# canonical home instead of a free-string objectiveId.
statement: 签名/身份/能力边界始终正确
130 changes: 130 additions & 0 deletions packages/api/src/infrastructure/harness-eval/objective-registry.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
/**
* F257 修复清单 #3 — Objective registry loader (definition layer).
*
* Reads `docs/harness-feedback/objectives/registry.yaml` — the read-only
* discovery source for `report_harness_signal`'s `objectiveId`, so cats stop
* doing archaeology to find valid objectives ("三次上报三次考古"). KD-3:
* registry 定义层用 YAML;运行时 stats 拆到 Redis/eval.
*
* This is the canonical objective DEFINITION layer only: `id + statement`.
* Unit→objective membership (which segments an objective is evaluated over) is
* NOT authored here — per the frozen redesign §4.8 it lives in the versioned
* `UnitEvaluationManifest.objectives` keyed by typed unitRef, and a derived
* read-model can expose attachments in V2. Keeping a second writable authority
* out of this file avoids the dual-source drift sol flagged (2a R1 P1-1).
*
* Fail-closed (2a R1 P1-2): a load/parse/validation failure is reported as an
* explicit error, NEVER collapsed to a valid-but-empty catalog — a silent empty
* would recreate the archaeology gap it exists to close.
*/

import { readFile } from 'node:fs/promises';
import { parse as parseYaml } from 'yaml';

export interface ObjectiveDefinition {
id: string;
statement: string;
}

export interface ObjectiveRegistry {
registryVersion: number;
objectives: ObjectiveDefinition[];
}

/** Discriminated result: a valid registry, or an explicit failure reason. */
export type ObjectiveRegistryResult = { ok: true; registry: ObjectiveRegistry } | { ok: false; error: string };

/** Canonical objective id shape: `obj-` + kebab-case (lowercase alnum groups). */
const OBJECTIVE_ID_RE = /^obj-[a-z0-9]+(?:-[a-z0-9]+)*$/;

/** The only registry schema version this loader implements (2a R3 P2-1). */
const SUPPORTED_REGISTRY_VERSION = 1;

function fail(error: string): ObjectiveRegistryResult {
return { ok: false, error };
}

/** Validate a single objective row → definition, or an error string. */
function validateObjective(entry: unknown, index: number, seen: Set<string>): ObjectiveDefinition | string {
if (!entry || typeof entry !== 'object') return `objectives[${index}] is not a mapping`;
// 2a R2 P2-1: fail-closed on unknown keys (registryVersion=1 allows only id/statement).
// Rejecting rather than stripping keeps the file the single authority — a stray
// `segments`/typo/未版本化 field can't silently reappear and mislead a human reader.
for (const key of Object.keys(entry)) {
if (key !== 'id' && key !== 'statement') {
const hint =
key === 'segments'
? ' — unit→objective membership belongs to the versioned UnitEvaluationManifest (§4.8), not this definition registry'
: ' (registryVersion=1 allows only id/statement; bump the schema to add fields)';
return `objectives[${index}] has unsupported key "${key}"${hint}`;
}
}
const o = entry as { id?: unknown; statement?: unknown };
if (typeof o.id !== 'string' || o.id.trim() !== o.id || o.id.length === 0) {
return `objectives[${index}].id must be a trimmed non-empty string`;
}
if (!OBJECTIVE_ID_RE.test(o.id)) return `objectives[${index}].id "${o.id}" must match ${OBJECTIVE_ID_RE.source}`;
if (typeof o.statement !== 'string' || o.statement.trim().length === 0) {
return `objectives[${o.id}].statement must be a non-empty string`;
}
if (seen.has(o.id)) return `duplicate objective id "${o.id}"`;
seen.add(o.id);
return { id: o.id, statement: o.statement.trim() };
}

/**
* Parse + strictly validate the objective registry YAML. Pure (no I/O) so it is
* unit-testable. Returns an explicit failure (ok:false) on malformed YAML, a
* non-mapping root, a non-positive-integer version, a missing/non-array
* `objectives`, any invalid row, or a duplicate id — never a silent empty.
*/
export function parseObjectiveRegistry(rawYaml: string): ObjectiveRegistryResult {
let doc: unknown;
try {
doc = parseYaml(rawYaml);
} catch (err) {
return fail(`malformed registry YAML: ${err instanceof Error ? err.message : String(err)}`);
}
if (!doc || typeof doc !== 'object') return fail('registry root must be a mapping');
// 2a R2 P2-1: fail-closed on unknown root keys (registryVersion=1 allows only
// registryVersion/objectives). Future fields must bump the schema, not slip through.
for (const key of Object.keys(doc)) {
if (key !== 'registryVersion' && key !== 'objectives') {
return fail(`unknown registry key "${key}" (registryVersion=1 allows only registryVersion/objectives)`);
}
}

const record = doc as { registryVersion?: unknown; objectives?: unknown };
const version = record.registryVersion;
// 2a R3 P2-1: this loader implements ONLY v1 semantics, so accept exactly v1. A future
// schema must ship a versioned parser + bump this — advertising an unimplemented version
// (2, 999, …) as supported to discovery clients is the inconsistency being closed.
if (version !== SUPPORTED_REGISTRY_VERSION) {
return fail(`registryVersion must be exactly ${SUPPORTED_REGISTRY_VERSION} (this loader implements only v1)`);
}
if (!Array.isArray(record.objectives)) return fail('registry `objectives` must be an array');

const objectives: ObjectiveDefinition[] = [];
const seen = new Set<string>();
for (let i = 0; i < record.objectives.length; i++) {
const result = validateObjective(record.objectives[i], i, seen);
if (typeof result === 'string') return fail(result);
objectives.push(result);
}
return { ok: true, registry: { registryVersion: version, objectives } };
}

/**
* Load + validate the objective registry from disk. Returns an explicit failure
* (ok:false) when the file is unreadable — the caller (route/tool) must surface
* it (503/error), not present it as an empty catalog.
*/
export async function loadObjectiveRegistry(registryPath: string): Promise<ObjectiveRegistryResult> {
let raw: string;
try {
raw = await readFile(registryPath, 'utf-8');
} catch (err) {
return fail(`registry unreadable at ${registryPath}: ${err instanceof Error ? err.message : String(err)}`);
}
return parseObjectiveRegistry(raw);
}
33 changes: 32 additions & 1 deletion packages/api/src/routes/callback-docs-routes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import { dirname, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import type { FastifyPluginAsync } from 'fastify';
import { RICH_BLOCK_RULES } from '../domains/cats/services/context/rich-block-rules.js';
import { loadObjectiveRegistry } from '../infrastructure/harness-eval/objective-registry.js';

const __dirname = dirname(fileURLToPath(import.meta.url));

Expand All @@ -27,17 +28,47 @@ function refsPath(fileName: string): string {
return resolve(__dirname, '..', '..', '..', '..', 'cat-cafe-skills', 'refs', fileName);
}

/** F257 #3: resolve the objective registry YAML (docs/harness-feedback/objectives/). */
function objectiveRegistryPath(): string {
return resolve(__dirname, '..', '..', '..', '..', 'docs', 'harness-feedback', 'objectives', 'registry.yaml');
}

export interface CallbackDocsRoutesOptions {
/** Test seam: override the objective registry path (defaults to the shipped location). */
objectiveRegistryPath?: string;
}

/**
* Register documentation endpoints (fallback for Skills system).
* No auth required — these return static reference text.
*/
export const registerCallbackDocsRoutes: FastifyPluginAsync = async (app) => {
export const registerCallbackDocsRoutes: FastifyPluginAsync<CallbackDocsRoutesOptions> = async (app, opts) => {
const registryPath = opts.objectiveRegistryPath ?? objectiveRegistryPath();
// Rich block usage rules
app.get('/api/callbacks/rich-block-rules', async (_request, reply) => {
reply.header('cache-control', 'public, max-age=3600');
return { rules: RICH_BLOCK_RULES };
});

// F257 #3: objective registry — read-only discovery for report_harness_signal
// objectiveId (so cats stop doing archaeology). Definition layer (id/statement).
// Fail-closed (2a R1 P1-2): an unreadable/malformed/invalid catalog returns 503,
// never a cacheable empty list that would masquerade as "no objectives".
app.get('/api/callbacks/objectives', async (request, reply) => {
const result = await loadObjectiveRegistry(registryPath);
if (!result.ok) {
// 2a R2 P2-1: this endpoint is UNAUTHENTICATED. The loader's reason contains the
// registry path + fs errno — log it server-side, but return a stable, path-free 503
// so a caller (and the MCP tool that forwards response.text()) never learns the
// install path / layout. The MCP tool still only needs to recognize the 503.
request.log.error({ reason: result.error }, '[F257] objective registry unavailable');
reply.code(503);
return { error: 'Objective registry unavailable' };
}
reply.header('cache-control', 'public, max-age=3600');
return { registryVersion: result.registry.registryVersion, objectives: result.registry.objectives };
});

// MCP callback instructions — reads refs file (SOT moved from skill to refs/)
app.get('/api/callbacks/instructions', async (_request, reply) => {
try {
Expand Down
44 changes: 42 additions & 2 deletions packages/api/test/callback-docs-route.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ import { describe, test } from 'node:test';
import Fastify from 'fastify';

describe('Callback Docs Routes', () => {
async function createApp() {
async function createApp(opts = {}) {
const { registerCallbackDocsRoutes } = await import('../dist/routes/callback-docs-routes.js');
const app = Fastify();
await app.register(registerCallbackDocsRoutes);
await app.register(registerCallbackDocsRoutes, opts);
await app.ready();
return app;
}
Expand Down Expand Up @@ -43,4 +43,44 @@ describe('Callback Docs Routes', () => {
await app.close();
}
});

// F257 #3: objective registry discovery route — serves the shipped registry.yaml
// so cat_cafe_list_objectives can surface valid objectiveIds (no archaeology).
test('GET /api/callbacks/objectives returns 200 with canonized objectives', async () => {
const app = await createApp();
try {
const response = await app.inject({ method: 'GET', url: '/api/callbacks/objectives' });
assert.equal(response.statusCode, 200);
const body = response.json();
assert.ok(Array.isArray(body.objectives), 'response should have objectives array');
const ids = body.objectives.map((o) => o.id);
assert.ok(ids.includes('obj-routing-delivery'), 'obj-routing-delivery served');
assert.ok(ids.includes('obj-identity-integrity'), 'obj-identity-integrity served');
for (const o of body.objectives) {
assert.ok(o.id && o.statement, 'each objective has id + statement');
assert.equal('segments' in o, false, 'no segments authority in served objective');
}
} finally {
await app.close();
}
});

// 2a R1 P1-2: an unreadable/invalid registry must fail-closed (503), never a
// cacheable 200 empty list that masquerades as "no objectives".
// 2a R2 P2-1: the unauthenticated 503 must NOT leak the internal path / fs errno.
test('GET /api/callbacks/objectives returns a path-free 503 when registry unreadable', async () => {
const secretPath = '/private/secret-install/objectives-registry.yaml';
const app = await createApp({ objectiveRegistryPath: secretPath });
try {
const response = await app.inject({ method: 'GET', url: '/api/callbacks/objectives' });
assert.equal(response.statusCode, 503);
const body = response.json();
assert.match(body.error, /unavailable/i, 'surfaces an explicit unavailability error');
assert.doesNotMatch(body.error, /secret-install/, 'must not leak the install path');
assert.doesNotMatch(body.error, /ENOENT|errno|no such file/i, 'must not leak fs errno');
assert.equal(response.headers['cache-control'], undefined, 'failure is not cached');
} finally {
await app.close();
}
});
});
Loading