[mara] prod incident: prod-heron-down

[vaderyang]

🤖 mara detected a production heron incident.

• Signature: prod-heron-down
• What: heron /api/health unreachable or non-200 (HTTP 000)
• Health URL: http://:4500/api/health
• HTTP: 000
• Observed (UTC): 2026-06-01T18:31:02Z

Health response

000

Log context

2026-06-01T17:30:36.170893106Z  WARN h_storage::pair_sweeper: pair-sweeper: sweep failed; continuing error=storage error: query pair candidates: FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
2026-06-01T17:30:36.646871439Z  INFO h_storage_duckdb: reopened all DuckDB connections after FATAL invalidation pool_size=4
2026-06-01T18:30:35.961890713Z  WARN h_storage::pair_sweeper: pair-sweeper: post-sweep CHECKPOINT failed error=storage error: CHECKPOINT failed: FATAL Error: Failed to create checkpoint because of error: INTERNAL Error: Failed to remove all rows while merging checkpoint deltas - this signifies a bug or broken index
2026-06-01T18:30:35.964932129Z  WARN h_storage::retention: retention: sweep failed error=storage error: checkpoint failed: FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
2026-06-01T18:30:36.527998897Z ERROR h_storage::buffer: flush failed; batch dropped entity="http_exchanges" batch_len=1 trigger="time" error=storage error: failed to flush exchanges: Failed to append: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
2026-06-01T18:30:36.52813451Z ERROR h_storage::buffer: flush failed; batch dropped entity="calls" batch_len=1 trigger="time" error=storage error: failed to flush calls: Failed to append: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
2026-06-01T18:30:37.965417536Z  WARN h_storage::pair_sweeper: pair-sweeper: sweep failed; continuing error=storage error: query pair candidates: FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
2026-06-01T18:30:38.537330123Z  INFO h_storage_duckdb: reopened all DuckDB connections after FATAL invalidation pool_size=4
---- tail ----
/home/<user>/TokenScope/server/target/release/heron(+0xf14deb) [0x55b1f9eefdeb]
/home/<user>/TokenScope/server/target/release/heron(+0xef92a1) [0x55b1f9ed42a1]
/home/<user>/TokenScope/server/target/release/heron(+0xf87b77) [0x55b1f9f62b77]
/home/<user>/TokenScope/server/target/release/heron(+0x2f8fbf8) [0x55b1fbf6abf8]
/home/<user>/TokenScope/server/target/release/heron(+0x2f8d812) [0x55b1fbf68812]
/home/<user>/TokenScope/server/target/release/heron(+0x310690f) [0x55b1fc0e190f]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f0d3a71aac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x1268d0) [0x7f0d3a7ac8d0]
"
2026-06-01T18:30:38.537330123Z  INFO h_storage_duckdb: reopened all DuckDB connections after FATAL invalidation pool_size=4
2026-06-01T18:30:38.537415969Z  INFO h_storage::pair_sweeper: pair-sweeper: reopened all DuckDB connections after invalidation
2026-06-01T18:30:38.819695963Z DEBUG h_protocol::tcp: tcp.push: SYN → set client_side, synced=true flow=[any] <ip>:9008 <-> <ip>:60630 dir=BtoA seq=1643540997 synced_before=false
2026-06-01T18:30:38.819762252Z DEBUG h_protocol::tcp: tcp.push: SYN-ACK → Established flow=[any] <ip>:9008 <-> <ip>:60630 dir=AtoB seq=2501088844
2026-06-01T18:30:38.819779257Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:9008 <-> <ip>:60630 dir=BtoA seq=1643540998 len=191 is_client=true waiting_for_resp=false diff=Some(0)
2026-06-01T18:30:38.819875101Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:9008 <-> <ip>:60630 dir=BtoA seq=1643541189 len=587 is_client=true waiting_for_resp=false diff=Some(0)
2026-06-01T18:30:38.81989362Z DEBUG h_protocol::tcp: tcp.push: parse-result flow=[any] <ip>:9008 <-> <ip>:60630 dir=BtoA seq=1643541189 resync=false new_events=1
2026-06-01T18:30:39.092824782Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:9008 <-> <ip>:60630 dir=AtoB seq=2501088845 len=778 is_client=false waiting_for_resp=true diff=Some(0)
2026-06-01T18:30:39.092902374Z DEBUG h_protocol::tcp: tcp.push: parse-result flow=[any] <ip>:9008 <-> <ip>:60630 dir=AtoB seq=2501088845 resync=false new_events=1
2026-06-01T18:30:39.552066832Z DEBUG h_storage::buffer: flushed entity="http_exchanges" batch_len=1 trigger="time" flush_ms=25
2026-06-01T18:30:39.567926309Z DEBUG h_storage::buffer: flushed entity="calls" batch_len=1 trigger="time" flush_ms=41

---

Filed automatically by mara (prod observer). Dedup window: 6h.
Add agent:assess to route this to the triage → wiwi loop once confirmed actionable.

[vaderyang]

Resolved — prod heron was SIGSEGV-crashed after a DuckDB 102GB 'merging checkpoint deltas / broken index' FATAL (the #50/PR#52 class) and stayed down (~5.7h, no Restart= on the unit). Recovered by moving the uncommitted .wal aside (preserved as .wal.crash-20260602; all 102GB of history kept) and restarting — stable since, capture flowing.

mara filed this automatically + deduped correctly — the observer loop works end-to-end in prod. The underlying 102GB-DB + recurring checkpoint-delta FATAL is the real root cause and is tracked separately.