LLM worker panics: AgentClassifierUnknownCount/MixedCount not registered (stage.rs vs processor.rs metric drift) — silently stops capture

[vaderyang]

Summary

The production LLM pipeline worker panics ~intermittently with:

thread 'tokio-rt-worker' panicked at server/h-common/src/internal_metrics.rs:348:
metric AgentClassifierUnknownCount not registered for worker llm.2-15
ERROR heron: pipeline stage 'local.llm.2' exited abnormally

This kills the llm stage → the whole capture pipeline drains and stops. (Post-#80 it no longer crashes the process — it "parks" with pipelines[].running=false, i.e. /api/health still says ready but heron is silently not capturing, which is arguably worse.) This is the real root cause behind the recurring prod capture outages that #79/#80 only addressed the shutdown symptom of.

Root cause

The production LLM worker registration in server/h-llm/src/stage.rs:69-83 lists its metrics explicitly:

let worker_metrics = metrics_sys.register_worker(
    &format!("llm.{i}"),
    &[
        Metric::WireDetected, Metric::WireIgnored,
        Metric::LlmCallsWithAgent, Metric::LlmCallsWithoutAgent,
        Metric::LlmGenericToolIdCanonicalized, Metric::LlmGenericSessionIdSynthFailed,
        Metric::LlmHeartbeatsReceived, Metric::LlmTokensEstimated,
        Metric::MetricsHeartbeatsDropped, Metric::TurnHeartbeatsDropped,
        // MISSING: AgentClassifierUnknownCount, AgentClassifierMixedCount
    ],
);

But LlmProcessor increments two metrics that are not in that list:

• server/h-llm/src/processor.rs:376 → .counter(Metric::AgentClassifierUnknownCount) when tool_surface == Unknown
• server/h-llm/src/processor.rs:381 → .counter(Metric::AgentClassifierMixedCount) when tool_surface == Mixed

MetricsWorker::counter() (internal_metrics.rs:346-352) panics when a metric isn't registered for the worker. So the first real LLM call that classifies as Unknown (or Mixed) panics the worker — data-dependent, hence intermittent (~hourly in prod).

Why no test caught it

processor.rs:422 test_metrics() registers a worker with the FULL metric set — including the two Agent* counters — so unit tests pass. The production list in stage.rs and the test list in processor.rs drifted: the test helper masks the gap instead of mirroring prod.

Proposed fix

1. Add Metric::AgentClassifierUnknownCount and Metric::AgentClassifierMixedCount to the register_worker list in stage.rs:71-82.
2. Kill the drift class: make the LLM worker's metric set a single shared source of truth (e.g. a pub const LLM_WORKER_METRICS: &[Metric] or a fn llm_worker_metrics() in h-llm) used by BOTH stage.rs and processor.rs::test_metrics(), so a metric the processor increments can never be absent from the registration the test exercises.

Acceptance criteria

• A real tool_surface == Unknown (and Mixed) call driven through the production worker registration path increments the counter without panicking.
• A deterministic test fails if any metric LlmProcessor increments is missing from the production registration list (i.e. the test uses the same shared metric set as stage.rs, not a hand-maintained superset).
• cargo test --workspace green.

Severity

High — silently stops prod capture (parked, health-green) on a data-dependent trigger; recurs ~hourly.

[github-actions]

🤖 Triage: do — scope: h-llm crate metric registration fix

Add 2 missing metrics to stage.rs worker registration, extract shared metric set to prevent future drift. ~50 LOC, 2 files, deterministic test.

Auto-labeled agent:try. wiwi will pick this up shortly.

[vaderyang]

Fast-tracking this directly — fix is in #82 (prod is actively affected). Removed the agent labels so wiwi doesn't duplicate the work.