[mara] prod incident: prod-heron-down

[vaderyang]

🤖 mara detected a production heron incident.

• Signature: prod-heron-down
• What: heron /api/health unreachable or non-200 (HTTP 000)
• Health URL: http://:4599/api/health
• HTTP: 000
• Observed (UTC): 2026-06-01T10:25:25Z

Health response

000

Log context

2026-06-01T09:30:36.024536391Z  WARN h_storage::retention: retention: sweep failed error=storage error: failed to delete llm_finish_metrics[1m]: FATAL Error: Invalid Input Error: Failed to delete all rows from index. Only deleted 158 out of 235 rows.
2026-06-01T09:30:36.289204512Z  WARN h_storage::pair_sweeper: pair-sweeper: sweep failed; continuing error=storage error: query pair candidates: FATAL Error: Failed: database has been invalidated because of a previous fatal error. The database must be restarted prior to being used again.
2026-06-01T09:30:36.915199722Z  INFO h_storage_duckdb: reopened all DuckDB connections after FATAL invalidation pool_size=4
---- tail ----
2026-06-01T10:25:24.850578975Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174079 len=153 is_client=false waiting_for_resp=true diff=Some(-153)
2026-06-01T10:25:24.892347747Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174232 len=160 is_client=false waiting_for_resp=true diff=Some(0)
2026-06-01T10:25:24.892416196Z DEBUG h_protocol::tcp: tcp.push: parse-result flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174232 resync=false new_events=1
2026-06-01T10:25:24.892435297Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174232 len=160 is_client=false waiting_for_resp=true diff=Some(-160)
2026-06-01T10:25:24.936135154Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174392 len=145 is_client=false waiting_for_resp=true diff=Some(0)
2026-06-01T10:25:24.936205479Z DEBUG h_protocol::tcp: tcp.push: parse-result flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174392 resync=false new_events=1
2026-06-01T10:25:24.936222964Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174392 len=145 is_client=false waiting_for_resp=true diff=Some(-145)
2026-06-01T10:25:24.980715595Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174537 len=151 is_client=false waiting_for_resp=true diff=Some(0)
2026-06-01T10:25:24.980786138Z DEBUG h_protocol::tcp: tcp.push: parse-result flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174537 resync=false new_events=1
2026-06-01T10:25:24.980804053Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174537 len=151 is_client=false waiting_for_resp=true diff=Some(-151)
2026-06-01T10:25:25.014605756Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:56202 <-> <ip>:9000 dir=BtoA seq=1247851226 len=932 is_client=false waiting_for_resp=false diff=Some(0)
2026-06-01T10:25:25.01466823Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:56202 <-> <ip>:9000 dir=BtoA seq=1247851226 len=932 is_client=false waiting_for_resp=false diff=Some(-932)
2026-06-01T10:25:25.014686887Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:56202 <-> <ip>:9000 dir=BtoA seq=1247851226 len=932 is_client=false waiting_for_resp=false diff=Some(0)
2026-06-01T10:25:25.01469924Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:56202 <-> <ip>:9000 dir=BtoA seq=1247851226 len=932 is_client=false waiting_for_resp=false diff=Some(-932)
2026-06-01T10:25:25.023964635Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174688 len=142 is_client=false waiting_for_resp=true diff=Some(0)
2026-06-01T10:25:25.02403346Z DEBUG h_protocol::tcp: tcp.push: parse-result flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174688 resync=false new_events=1
2026-06-01T10:25:25.024055356Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174688 len=142 is_client=false waiting_for_resp=true diff=Some(-142)
2026-06-01T10:25:25.066259721Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174830 len=169 is_client=false waiting_for_resp=true diff=Some(0)
2026-06-01T10:25:25.066331931Z DEBUG h_protocol::tcp: tcp.push: parse-result flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174830 resync=false new_events=1
2026-06-01T10:25:25.066349428Z DEBUG h_protocol::tcp: tcp.push: SYNCED+APPEND → append+parse flow=[any] <ip>:60650 <-> <ip>:30000 dir=BtoA seq=2693174830 len=169 is_client=false waiting_for_resp=true diff=Some(-169)

---

Filed automatically by mara (prod observer). Dedup window: 6h.
Add agent:assess to route this to the triage → wiwi loop once confirmed actionable.

[vaderyang]

Controlled mara end-to-end test — closing. Prod is healthy.