Summary
heron panics during graceful shutdown with JoinHandle polled after completion (tokio), exiting with code 101. On a long-running capture deployment this took the process down and left its systemd unit in failed state. The crash is in the shutdown path, not the capture/parse path — capture was healthy right up to the drain (pkts_received high, pkts_dropped_kernel=0).
Observed crash
INFO heron: waiting for pipeline (incl. storage sink) to drain...
thread 'main' panicked at .../tokio-1.51.1/src/runtime/task/core.rs:422:22:
JoinHandle polled after completion
Exit status 101; preceded by a normal-looking pipeline teardown (protocol worker stopping: downstream closed, pcap-live: channel closed, stopping, ... pipeline stages ... ).
Root cause
In server/app/heron/src/main.rs, the supervisor task is a single JoinHandle:
- spawned at main.rs:731 —
let mut supervisor = tokio::spawn(Pipeline::supervise(stage_handles));
- polled in the steady-state
select! arm at ~main.rs:742 — res = &mut supervisor => { ... }
- polled again in the drain block at ~main.rs:781 —
tokio::time::timeout(PIPELINE_DRAIN_TIMEOUT, &mut supervisor).await
When a pipeline stage exits first (downstream-closed cascade, e.g. the capture source closes its channel and the stages drain), the select! completes via the supervisor arm at ~742. That drives the JoinHandle to completion. Execution then falls through to the drain block at ~781, which polls the already-completed &mut supervisor a second time → panic.
(The other path — select! won by the "all capture sources finished" arm — leaves supervisor still pending, so the drain poll at ~781 is its first completion and does NOT panic. That's why this only bites when a stage exits before capture is explicitly stopped.)
Impact
- Every shutdown that takes this branch exits 101, so the unit always lands in
failed instead of a clean stop — and here it caused an unplanned outage of the capture process.
- Because the "clean stop" looks like a failure (exit 101),
Restart=on-failure can't safely be added to the unit until this is fixed (auto-restart would fight an intentional stop).
Proposed direction
Poll the supervisor JoinHandle to completion exactly once. Options:
- Capture the result the first time it completes (in the
select! arm) into an Option/flag (supervisor_done), and gate the drain timeout(..., &mut supervisor) on !supervisor_done; when already done, skip straight to "pipeline drained".
- Or fuse the handle (e.g. only ever await it in one place) so the select! arm doesn't consume it before drain.
Either way the invariant is: a completed JoinHandle is never polled again.
Acceptance criteria
Repro sketch
Run heron against a live/replayed capture, then stop capture in a way that makes a downstream stage close first (channel close cascade) rather than the capture-source-finished arm. The drain log line waiting for pipeline (incl. storage sink) to drain... is immediately followed by the panic.
Summary
heronpanics during graceful shutdown withJoinHandle polled after completion(tokio), exiting with code 101. On a long-running capture deployment this took the process down and left its systemd unit infailedstate. The crash is in the shutdown path, not the capture/parse path — capture was healthy right up to the drain (pkts_receivedhigh,pkts_dropped_kernel=0).Observed crash
Exit status 101; preceded by a normal-looking pipeline teardown (
protocol worker stopping: downstream closed,pcap-live: channel closed, stopping,... pipeline stages ...).Root cause
In
server/app/heron/src/main.rs, thesupervisortask is a singleJoinHandle:let mut supervisor = tokio::spawn(Pipeline::supervise(stage_handles));select!arm at ~main.rs:742 —res = &mut supervisor => { ... }tokio::time::timeout(PIPELINE_DRAIN_TIMEOUT, &mut supervisor).awaitWhen a pipeline stage exits first (downstream-closed cascade, e.g. the capture source closes its channel and the stages drain), the
select!completes via the supervisor arm at ~742. That drives theJoinHandleto completion. Execution then falls through to the drain block at ~781, which polls the already-completed&mut supervisora second time → panic.(The other path —
select!won by the "all capture sources finished" arm — leavessupervisorstill pending, so the drain poll at ~781 is its first completion and does NOT panic. That's why this only bites when a stage exits before capture is explicitly stopped.)Impact
failedinstead of a clean stop — and here it caused an unplanned outage of the capture process.Restart=on-failurecan't safely be added to the unit until this is fixed (auto-restart would fight an intentional stop).Proposed direction
Poll the
supervisorJoinHandleto completion exactly once. Options:select!arm) into anOption/flag (supervisor_done), and gate the draintimeout(..., &mut supervisor)on!supervisor_done; when already done, skip straight to "pipeline drained".Either way the invariant is: a completed
JoinHandleis never polled again.Acceptance criteria
Okresult).cargo test --workspacestays green.Repro sketch
Run
heronagainst a live/replayed capture, then stop capture in a way that makes a downstream stage close first (channel close cascade) rather than the capture-source-finished arm. The drain log linewaiting for pipeline (incl. storage sink) to drain...is immediately followed by the panic.