feat(ingest): add telemetry ingestion endpoint for governance cycles

[fagemx]

Context

Thyra 的 governance cycle 會產生 per-operation telemetry（fagemx/thyra#232）：每步耗時、token 用量、成本。這些數據應該存到 Edda 做趨勢分析。

What Needs to Happen

• 新 endpoint：POST /api/telemetry（或擴展 /api/decisions）
• 接收 CycleTelemetry 格式（operations[], total_duration, cost）
• 存儲 + 可查詢
• 支持聚合查詢：「過去 7 天平均 cycle duration」「最慢的操作是什麼」

References

• Cross-repo: feat(chief): add per-operation telemetry for governance cycles thyra#232 (per-operation telemetry)
• Cross-repo: feat(market): summary generator queries Edda for historical insights thyra#224 (summary 趨勢分析)

---

Midnight Market: operational intelligence accumulation

[fagemx]

Research: Telemetry Ingestion Endpoint (GH-317)

1. Problem Statement

Thyra's governance cycles produce per-operation telemetry (fagemx/thyra#232): per-step duration, token usage, cost. This data needs to flow into Edda for trend analysis and operational intelligence.

Requirements from issue:

• New endpoint: POST /api/telemetry
• Accept CycleTelemetry format: operations[], total_duration, cost
• Storage + queryability
• Aggregation queries: "average cycle duration over 7 days", "slowest operation"

2. Existing Architecture

2.1 Event Model (edda-core)

All data in Edda flows through the Event model (crates/edda-core/src/types.rs:255):

pub struct Event {
    pub event_id: String,        // evt_<ulid>
    pub ts: String,              // RFC 3339
    pub event_type: String,      // e.g. "note", "execution_event"
    pub branch: String,
    pub parent_hash: Option<String>,
    pub hash: String,
    pub payload: serde_json::Value,  // flexible JSON
    pub refs: Refs,
    pub schema_version: u32,
    pub digests: Vec<Digest>,
    pub event_family: Option<String>,
    pub event_level: Option<String>,
}

Events are classified by event_type into families/levels (types.rs:49-77). The payload is a flexible serde_json::Value -- no rigid schema enforcement.

2.2 Closest Existing Pattern: Karvi Execution Events

The POST /api/events/karvi endpoint (edda-serve/src/lib.rs:1763) is the most relevant precedent:

• Accepts a KarviEventBody struct with version, event_id, event_type, occurred_at, usage, result, etc.
• Stores it as an execution_event type via new_execution_event()
• Uses idempotent append (dedup by event_id)
• Already tracks usage.cost_usd in the payload

Key detail: The rollup/aggregate layer already extracts payload.usage.cost_usd from execution_event types for cost trends (edda-aggregate/src/aggregate.rs:277-282).

2.3 Ingestion Flow

HTTP POST -> validate body -> open_ledger -> acquire WorkspaceLock
  -> create Event (event_type, payload) -> append_event -> return event_id

2.4 Aggregation Layer (edda-aggregate)

• rollup_metrics_by_date() does single-pass event scanning per project
• Already extracts cost_usd, execution_count, success_count from execution_events
• edda-serve exposes /api/metrics/trends (daily/weekly/monthly) and /api/metrics/overview

2.5 Query Layer (edda-ask, edda-ledger)

• Ledger::iter_events_by_type(event_type) -- filter at SQL level
• Ledger::iter_events_filtered(branch, event_type, keyword, after, before, limit) -- full SQL pushdown
• /api/decisions supports ?after=&before=&tags=&q= query params

3. Key Design Questions

Q1: New event_type or extend execution_event?

Option A: New telemetry event_type -- Clean separation, dedicated classify_event_type entry, independent aggregation logic.

Option B: Reuse execution_event with a sub-type -- The existing execution_event already has usage/cost fields. Telemetry is a different granularity though (governance cycle vs. single task execution).

Recommendation: New event_type cycle_telemetry -- governance cycle telemetry has different semantics (operations array, total_duration, per-operation breakdown) than a single execution event.

Q2: New crate or extend edda-serve?

The endpoint handler goes in edda-serve. The question is whether the telemetry data model and aggregation logic needs a new crate.

Recommendation: No new crate. Types can live in edda-core (if reused) or inline in edda-serve. Aggregation extensions go in edda-aggregate. YAGNI.

Q3: How to expose aggregation queries?

Option A: Dedicated /api/telemetry/stats endpoint -- Purpose-built for cycle duration averages, slowest operations.

Option B: Extend existing /api/metrics/trends -- Add telemetry metrics alongside existing cost/execution metrics.

Recommendation: Dedicated query endpoint /api/telemetry with GET for queries and POST for ingestion. The aggregation semantics (per-operation breakdown, duration percentiles) are different enough to warrant a separate endpoint.

Q4: Storage -- same ledger or separate table?

Events go through the standard ledger (append-only, hash-chained). No separate table needed. The event_type discriminator + SQL filtering is sufficient.

4. Gaps and Risks

1. CycleTelemetry format undefined in Edda -- The Thyra side (feat(chief): add per-operation telemetry for governance cycles thyra#232) defines the producer format. We need to define the ingestion contract.

2. Aggregation performance -- Current rollup scans all events. Adding telemetry events (potentially high volume per governance cycle) could slow down rollups. Mitigation: filter by event_type at SQL level.

3. No existing "per-operation breakdown" aggregation -- The rollup system is date-bucketed. Per-operation analysis (slowest op, avg duration per op type) requires a new aggregation function.

4. classify_event_type needs update -- New event type must be added to the taxonomy in edda-core/src/types.rs:49.

5. Related Code Locations

File	Lines	What
crates/edda-serve/src/lib.rs	1728-1833	Karvi event ingestion (closest pattern)
crates/edda-core/src/types.rs	49-77	Event type taxonomy
crates/edda-core/src/event.rs	622-668	new_execution_event() constructor
crates/edda-aggregate/src/aggregate.rs	277-282	Cost extraction from execution_events
crates/edda-aggregate/src/rollup.rs	131-160	Rollup computation
crates/edda-serve/src/lib.rs	156-318	Router definitions (both with/without auth)
crates/edda-ledger/src/ledger.rs	81-128	Append + query API

[fagemx]

Innovation: Telemetry Ingestion Endpoint (GH-317)

Solution Approaches

Approach A: Lean Event-Only (Recommended)

Core idea: Store telemetry as a new cycle_telemetry event type through the existing ledger, with a thin query endpoint that computes aggregations on-the-fly from event data.

Ingestion: POST /api/telemetry accepts a typed CycleTelemetryBody:

#[derive(Deserialize)]
struct CycleTelemetryBody {
    cycle_id: String,                    // unique cycle identifier
    source: String,                      // "thyra" or other governance engine
    started_at: String,                  // RFC 3339
    total_duration_ms: u64,
    operations: Vec<OperationTelemetry>,
    cost: Option<CostTelemetry>,
    #[serde(default)]
    tags: Vec<String>,
    #[serde(default)]
    metadata: Option<serde_json::Value>, // extensible
}

#[derive(Deserialize)]
struct OperationTelemetry {
    name: String,                 // "probe", "evaluate", "decide", etc.
    duration_ms: u64,
    token_usage: Option<TokenUsage>,
    status: String,               // "ok", "error", "skipped"
}

#[derive(Deserialize)]
struct TokenUsage {
    input_tokens: u64,
    output_tokens: u64,
}

#[derive(Deserialize)]
struct CostTelemetry {
    total_usd: f64,
    breakdown: Option<Vec<CostBreakdown>>,
}

Query: GET /api/telemetry?after=&before=&source=&limit= returns raw events. A separate GET /api/telemetry/stats?days=7 computes aggregations.

Aggregation logic lives in edda-aggregate as a new function:

pub struct TelemetryStats {
    pub cycle_count: usize,
    pub avg_duration_ms: f64,
    pub p95_duration_ms: f64,
    pub total_cost_usd: f64,
    pub slowest_operations: Vec<OpStat>,  // top N by avg duration
    pub error_rate: f64,
}

Pros:

• Follows existing patterns exactly (Karvi event ingestion as template)
• No new crates, no new tables
• Aggregation is simple: scan cycle_telemetry events, extract payload fields
• Cost data flows into existing rollup naturally (extend cost extraction)

Cons:

• On-the-fly aggregation could be slow with many telemetry events
• No pre-computed materialized views

Approach B: Dedicated Telemetry Table

Core idea: In addition to the event ledger, maintain a separate SQLite table optimized for telemetry queries (indexed by timestamp, operation name, etc.).

Storage: Dual-write -- event goes to ledger (audit trail), and a denormalized row goes to a telemetry table:

CREATE TABLE telemetry (
    cycle_id TEXT PRIMARY KEY,
    source TEXT NOT NULL,
    started_at TEXT NOT NULL,
    total_duration_ms INTEGER NOT NULL,
    total_cost_usd REAL,
    operation_count INTEGER,
    event_id TEXT REFERENCES events(event_id)
);

CREATE TABLE telemetry_ops (
    cycle_id TEXT REFERENCES telemetry(cycle_id),
    name TEXT NOT NULL,
    duration_ms INTEGER NOT NULL,
    status TEXT,
    input_tokens INTEGER,
    output_tokens INTEGER
);

Pros:

• Fast SQL queries without JSON parsing
• Can do GROUP BY name ORDER BY AVG(duration_ms) DESC natively
• Scales better for high-volume telemetry

Cons:

• Dual-write complexity (ledger + table must stay in sync)
• New migration infrastructure needed in edda-ledger
• Violates current architecture (ledger is the single source of truth)
• YAGNI -- we don't know if volume will warrant this

Approach C: Extend Existing Execution Events

Core idea: Reuse execution_event type with a telemetry_type: "cycle" sub-discriminator. No new event type.

Pros:

• Zero changes to event taxonomy
• Cost extraction already works

Cons:

• Muddies the semantics -- execution events are per-task, telemetry is per-governance-cycle
• Operations array doesn't fit the existing usage/result pattern
• Query filtering becomes awkward (need to distinguish cycle telemetry from task executions)

Trade-off Analysis

Criterion	A (Lean Event)	B (Dedicated Table)	C (Extend Execution)
Implementation effort	Low	High	Low
Query performance	Medium	High	Medium
Architectural consistency	High	Low	Medium
Future extensibility	Medium	High	Low
YAGNI compliance	High	Low	Medium

Recommendation: Approach A (Lean Event-Only)

Rationale:

1. Follows YAGNI -- start simple, add complexity only when volume demands it
2. Consistent with project architecture -- ledger as single source of truth
3. Proven pattern -- mirrors Karvi event ingestion exactly
4. Natural cost integration -- extend the existing rollup cost extraction to also scan cycle_telemetry events
5. Migration path -- if query performance becomes a problem, Approach B can be added later as an optimization layer on top, without changing the ingestion contract

Key innovation: The stats endpoint computes percentiles and per-operation breakdowns by scanning filtered events. For the expected volume (governance cycles are not high-frequency), this is perfectly adequate. The iter_events_by_type("cycle_telemetry") SQL filter keeps the scan scoped.

[fagemx]

Plan: Telemetry Ingestion Endpoint (GH-317)

Approach: Lean Event-Only (Approach A)

Store governance cycle telemetry as cycle_telemetry events in the existing ledger. Provide ingestion via POST /api/telemetry and query via GET /api/telemetry + GET /api/telemetry/stats.

---

Step 1: Add cycle_telemetry event type to taxonomy

File: crates/edda-core/src/types.rs

Changes:

1. Add "cycle_telemetry" to classify_event_type() (line ~49):
   • Family: signal
   • Level: info
2. Add test case in the existing classify_all_known_event_types table test

Rationale: Telemetry is operational signal data, not governance or milestone.

---

Step 2: Add new_telemetry_event() constructor

File: crates/edda-core/src/event.rs

Changes:

1. Add a new_telemetry_event() function following the new_execution_event() pattern:
   • Accepts branch, parent_hash, cycle_id (used as event_id for idempotency), occurred_at, payload
   • Sets event_type = "cycle_telemetry"
   • Calls finalize()

Why not just inline in serve? The event constructor pattern is established -- all event types have constructors in event.rs.

---

Step 3: Add ingestion endpoint POST /api/telemetry

File: crates/edda-serve/src/lib.rs

Changes:

1. Define request/response structs:

#[derive(Deserialize)]
struct TelemetryBody {
    cycle_id: String,
    source: String,
    started_at: String,
    total_duration_ms: u64,
    operations: Vec<TelemetryOp>,
    #[serde(default)]
    cost: Option<TelemetryCost>,
    #[serde(default)]
    tags: Vec<String>,
    #[serde(default)]
    metadata: Option<serde_json::Value>,
}

#[derive(Deserialize, Serialize)]
struct TelemetryOp {
    name: String,
    duration_ms: u64,
    #[serde(default)]
    token_usage: Option<TelemetryTokenUsage>,
    #[serde(default)]
    status: Option<String>,
}

#[derive(Deserialize, Serialize)]
struct TelemetryTokenUsage {
    input_tokens: u64,
    output_tokens: u64,
}

#[derive(Deserialize, Serialize)]
struct TelemetryCost {
    total_usd: f64,
    #[serde(default)]
    breakdown: Option<Vec<serde_json::Value>>,
}

2. Implement post_telemetry() handler:

   • Validate started_at is valid ISO 8601
   • Open ledger, acquire workspace lock
   • Serialize full body as payload JSON
   • Call new_telemetry_event()
   • Use append_event_idempotent() (dedup by cycle_id)
   • Return { event_id, status: "created"|"duplicate" }

3. Add route: .route("/api/telemetry", post(post_telemetry).get(get_telemetry))

   • In both router functions (with-auth and without-auth)

---

Step 4: Add query endpoint GET /api/telemetry

File: crates/edda-serve/src/lib.rs

Changes:

1. Define query params:

#[derive(Deserialize)]
struct TelemetryQuery {
    #[serde(default)]
    after: Option<String>,
    #[serde(default)]
    before: Option<String>,
    #[serde(default)]
    source: Option<String>,
    #[serde(default)]
    limit: Option<usize>,
}

2. Implement get_telemetry() handler:
   • Use ledger.iter_events_filtered(branch, Some("cycle_telemetry"), keyword, after, before, limit)
   • If source is provided, post-filter by payload.source
   • Return array of telemetry event payloads

---

Step 5: Add stats endpoint GET /api/telemetry/stats

File: crates/edda-serve/src/lib.rs + crates/edda-aggregate/src/aggregate.rs

Changes in edda-aggregate:

1. Add telemetry_stats() function:

pub struct TelemetryStats {
    pub cycle_count: usize,
    pub avg_duration_ms: f64,
    pub p95_duration_ms: f64,
    pub total_cost_usd: f64,
    pub slowest_operations: Vec<OpStat>,
    pub error_rate: f64,
}

pub struct OpStat {
    pub name: String,
    pub avg_duration_ms: f64,
    pub max_duration_ms: u64,
    pub count: usize,
}

2. Implement: scan cycle_telemetry events, extract durations and operations, compute stats.

Changes in edda-serve:

1. Define query params: TelemetryStatsQuery { days: Option<u32>, source: Option<String> }

2. Implement get_telemetry_stats() handler:

   • Default days = 7
   • Call telemetry_stats() from edda-aggregate
   • Return JSON response

3. Add route: .route("/api/telemetry/stats", get(get_telemetry_stats))

---

Step 6: Extend rollup cost extraction

File: crates/edda-aggregate/src/aggregate.rs

Changes:

• In rollup_metrics_by_date(), extend the cost extraction block (~line 277) to also check cycle_telemetry events for payload.cost.total_usd
• This makes governance cycle costs appear in the existing /api/metrics/trends dashboard

---

Step 7: Tests

Unit tests (edda-core):

• classify_event_type("cycle_telemetry") returns correct family/level
• new_telemetry_event() produces valid event with correct type and hash

Integration tests (edda-serve):

Following existing patterns (e.g., post_note_creates_event at line 3780):

1. post_telemetry_creates_event -- POST valid body, verify 201 + event_id returned
2. post_telemetry_deduplicates -- POST same cycle_id twice, verify second returns "duplicate"
3. post_telemetry_rejects_bad_json -- POST invalid body, verify 422
4. get_telemetry_returns_events -- POST 2 events, GET with date filter, verify results
5. get_telemetry_stats_computes_averages -- POST 3 events with known durations, verify avg/p95/slowest

Aggregation tests (edda-aggregate):

1. telemetry_stats_computes_correctly -- Create events in temp ledger, verify stats output
2. rollup_includes_telemetry_cost -- Verify cycle_telemetry cost flows into rollup metrics

---

File Change Summary

File	Change Type	Description
crates/edda-core/src/types.rs	Modify	Add cycle_telemetry to taxonomy
crates/edda-core/src/event.rs	Modify	Add new_telemetry_event() constructor
crates/edda-serve/src/lib.rs	Modify	Add POST/GET /api/telemetry, GET /api/telemetry/stats
crates/edda-aggregate/src/aggregate.rs	Modify	Add telemetry_stats(), extend rollup cost extraction
crates/edda-serve/Cargo.toml	No change	Already depends on edda-aggregate

Total files changed: 4 (all modifications, no new files)

---

Execution Order

1. Step 1 + 2 (edda-core changes) -- independent, can be done first
2. Step 3 (POST endpoint) -- depends on step 2
3. Step 4 (GET query) -- depends on step 3
4. Step 5 (stats endpoint) -- depends on step 3, parallel with step 4
5. Step 6 (rollup integration) -- independent of steps 3-5
6. Step 7 (tests) -- after each step, but listed last for clarity

Risk Mitigation

• Contract stability: The TelemetryBody struct is the ingestion contract. Once Thyra starts sending data, changing it is costly. Design it slightly extensible (optional metadata field, optional status on operations).
• Volume: If telemetry volume proves high, add a limit default to the GET endpoint and consider adding rollup-style pre-computation later.
• Cross-repo coordination: The Thyra side (feat(chief): add per-operation telemetry for governance cycles thyra#232) needs to produce data matching this contract. The source field allows multiple producers without conflict.

[fagemx]

Work completed. PR created: #358

All CI checks passing (2 pre-existing test failures in edda-bridge-claude and edda-core are unrelated).