feat: restart accounting on the Linux runtime

[passcod]

Implements the plan from #133.

Seedling now records every container restart and derives crash_loop from the recorded rate. systemd still actions restarts on Linux; seedling keeps the books. Sub-threshold flapping — a container that crashes a couple of times a day forever, never exhausting StartLimitBurst inside its window — is no longer silent.

The two open decisions

The plan left two open; both are recorded in its ## Decided section (in history, since the plan is removed by this PR).

• Threshold and window are operator-settable, not compiled in. Stored in restart_settings, read and changed through /restarts/settings/{get,set}, and re-read by the reconciler each tick so a change takes effect without a restart. Default is 5 recovery restarts within 30 minutes; threshold floors at 2, window_secs at 60.
• Restart history is its own surface, /restarts/list, plus a per-instance summary folded into app.describe so the count is visible without going looking.

Cause, not actor

Following review: a restart record's cause is recovery or deliberate — why the restart happened, not which component performed it. On Linux the two coincide, since systemd actions recovery and seedling actions deploys, but only as a property of this platform. A runtime with no service supervisor performs both kinds, so a field naming the actor would classify everything there identically, leave the rate counting nothing, and kill the primary crash-loop trigger on the very runtime that made this requirement portable.

Spec first

docs/spec/runtime.md: new autonomous.restart.record, autonomous.restart.rate, autonomous.restart.rate.settings, gc.restarts; autonomous.restart.backoff becomes the Linux pacing mechanism rather than the definition of a crash loop; autonomous.restart.start-limit-hit is demoted to a secondary trigger; fault.crash-loop gains both triggers and must say which fired.

docs/spec/interface.md: restart.record, restart.list, restart.settings, and the app.describe amendment. docs/spec/web.md: routes.restarts.

All four specs are at 100% implementation coverage with zero stale references.

How restarts are observed

Polling container state cannot see a restart that completes inside one observe interval — the unit is active at both ends. So the runtime reads systemd's own NRestarts, which lives on org.freedesktop.systemd1.Service rather than the Unit interface already proxied, alongside ExecMainCode/ExecMainStatus for the last exit. Two extra property reads on a path already fetching ActiveState and SubState; failure is non-fatal, since the interface only exists on .service units.

Each tick diffs the counter against a stored per-instance baseline:

• No baseline — first sighting. Adopt the counter without recording; those restarts happened at times seedling cannot know, and inventing timestamps would corrupt the rate.
• Counter decreased — a reset (reset-failed, or a recreated unit), not a negative delta. Re-baseline, recording whatever accrued after the reset.
• Counter increased — record the delta, with the known exit attached to the most recent of the batch.

A start the reconciler issues for an instance it has already run is recorded as deliberate and re-baselines to zero, so a rolling update never reads as a crash burst — and the zeroed counter on the next tick is expected state rather than a reset.

Fault behaviour

crash_loop is filed on the rate reaching the threshold, or on start-limit-hit, and the description names which. Auto-restart suppression is re-derived from the fault table each tick, so it now covers rate-derived loops, survives a daemon restart, and lifts on the next tick when an operator clears the fault.

Retention

Bounded per instance (50 records) on write rather than by rate-limiting recording: a crash loop produces rows fastest exactly when the per-attempt exit codes are the diagnostic. The GC pass drops records and counters for instances that have left the registry.

Operator surfaces

seedling-ctl restarts list|settings|set-settings, and a /restarts web route with an app filter, an instance filter reachable from a per-instance chip on an app's resource table, and the threshold/window editable in minutes.

One incidental fix: useOiQuery kept its effect off the params, so any view that narrowed its query never refetched. It is now keyed on the params' serialisation, which the cache path already computed — object-identity churn from a fresh literal each render still does not refetch.

Note on the migration

v54 is edited in place by the cause rename rather than superseded by a v55. AGENTS.md forbids editing a shipped migration because its schema_version row exists in real databases; v54 is introduced in this unmerged branch and has never been on main, so no such row exists. Anyone who has run this branch locally will need to drop their dev database.

Tests

43 tests covering all five the plan listed: counter deltas, bursts, resets and re-baselining, deliberate-restart exclusion, threshold crossing and the operator setting, the rate trigger filing and observed-healthy clearing the fault, start-limit-hit filing below the rate threshold, per-instance cap under sustained crash-looping, GC of orphans, the observer fact end to end against a stub whose counter moves behind its back, the OI endpoints, and the web route.

cargo clippy --workspace --all-targets is clean, cargo fmt --check passes, tracey validates with 0 errors and 0 stale references, and all 47 frontend test files (319 tests) pass.

Five oi::server tests fail in the dev sandbox with Address family not supported by protocol — they need QUIC/IPv6. Verified as pre-existing by running them at the base commit, where they fail identically; they pass in CI.

🤖 Generated with Claude Code

[github-code-quality]

Code Coverage Overview

Languages: TypeScript, Rust

TypeScript / code-coverage/vitest

The overall coverage in commit 840013b in the claude/pr-133-implem... branch is 66%. The coverage in commit 9983b79 in the claude/pr-107-review... branch is 65%.

Show a code coverage summary of the most impacted files.

File	claude/pr-107-review... 9983b79	claude/pr-133-implem... 840013b	+/-
src/routes/AppDetail.tsx	74%	74%	0%
src/hooks/useOi.ts	97%	97%	0%
src/routes/Restarts.tsx	0%	91%	+91%

Rust / code-coverage/rust

The overall coverage in commit 840013b in the claude/pr-133-implem... branch is 58%. The coverage in commit 9983b79 in the claude/pr-107-review... branch is 57%.

Show a code coverage summary of the most impacted files.

File	claude/pr-107-review... 9983b79	claude/pr-133-implem... 840013b	+/-
crates/core/src/runtime/gc.rs	91%	90%	-1%
crates/core/src...handler/apps.rs	70%	69%	-1%
crates/core/src...em/reconcile.rs	15%	15%	0%
crates/core/src...stem/systemd.rs	17%	20%	+3%
crates/core/src...ncile/faults.rs	0%	4%	+4%
crates/core/src/system/stub.rs	25%	43%	+18%
crates/core/src...tem/observer.rs	0%	62%	+62%
crates/core/src...ile/restarts.rs	0%	75%	+75%
crates/core/src...ime/restarts.rs	0%	96%	+96%
crates/core/src...ler/restarts.rs	0%	100%	+100%

---

_{Updated August 01, 2026 08:30 UTC}

[passcod]

Reviewing this from the Windows runtime side (#107), since the portable rules added here are what that runtime will have to conform to. The implementation follows the plan closely and the spec rules read better than the ones the plan sketched. One thing is worth changing before merge, because it is operator-visible and gets expensive afterwards.

initiator splits on actor where the meaning is cause

i[restart.record] defines initiator as "supervisor" or "runtime", and r[autonomous.restart.rate] counts supervisor-actioned restarts toward the threshold, excluding runtime-initiated ones so rolling updates do not read as crash bursts.

On Linux that is exactly right, and the two meanings line up perfectly: systemd actions crash restarts, seedlingd actions deliberate ones. They line up because of a Linux implementation detail, not because actor and cause are the same thing.

The Windows container runtime has no platform supervisor actioning restarts. containerd has no restart policy and the runhcs shim does not restart tasks, so seedlingd actions every restart there — that is why wcr[shim.ownership] has to drop its restart clause. Under this taxonomy that means:

• every Windows restart is initiator: "runtime"
• r[autonomous.restart.rate] counts only supervisor-actioned restarts
• the rate never reaches the threshold, and the primary crash-loop trigger is dead

The secondary trigger does not cover for it either: r[autonomous.restart.start-limit-hit] keys on systemd's failed/start-limit-hit, which has no Windows equivalent. So the second runtime would end up with no crash-loop detection at all — the precise blind spot this PR exists to close.

The alternative is for Windows to report crash restarts as "supervisor" when no supervisor was involved, which empties the field of meaning.

Suggested change

Split on cause rather than actor: recovery from an unexpected exit, versus a deliberate runtime action. Something like cause: "recovery" | "deliberate" in place of initiator: "supervisor" | "runtime", with the rate counting recovery restarts.

On Linux this is a pure rename — systemd-actioned becomes recovery, deploys and replacements become deliberate, identical records and identical numbers. On Windows it works, because seedlingd is both the actor and the thing that knows why it is restarting something.

Worth noting that r[autonomous.restart.backoff] was already generalised in this PR to "where the platform's supervisor actions restarts", so the PR does anticipate a runtime where it does not. The initiator field simply did not get the same treatment, which reads as an oversight rather than a decision.

Why now

Pre-merge this is a field rename across i[restart.record], r[autonomous.restart.record], r[autonomous.restart.rate], i[restart.settings]'s wording, a DB column, a CLI column and a web column. Post-merge it is a breaking change to a published operator-interface field.

Nothing else blocking from here — the NRestarts baseline handling (adopt on first sighting, re-baseline on decrease, exclude runtime-initiated starts) is the part I would most have expected to get subtly wrong, and it looks right.

---

Generated by Claude Code

[passcod]

Agreed, and done in 840013b. You're right that it's an oversight rather than a decision — autonomous.restart.backoff got generalised and the field didn't, which is exactly the shape of one.

The failure mode you describe is the decisive part: not that the taxonomy reads oddly on Windows, but that it leaves that runtime with no working crash-loop trigger. initiator: "runtime" on everything, a rate that counts nothing, and no failed/start-limit-hit to fall back on — reintroducing precisely the blind spot this exists to close, on the runtime that was the reason for making the requirement portable in the first place.

cause: "recovery" | "deliberate" throughout, with the rate counting recovery. On Linux it is a pure rename: identical records, identical numbers, all 39 tests pass with only the enum swapped.

Both specs now say the split is about why rather than who, so the next reader does not have to rediscover this:

The cause is a statement about why the restart happened, not about which component performed it. Who actions a restart is a platform detail — a platform with a service supervisor leaves recovery to it, and one without leaves the runtime to perform both kinds — so recording the actor would make the record mean different things on different platforms.

Renamed across i[restart.record], r[autonomous.restart.record], r[autonomous.restart.rate], the i[restart.settings] and i[app.describe] wording, w[routes.restarts], the instance_restarts.cause column, Cause in runtime::restarts, the OI field, the CLI help, and the web column and chip.

Two things worth flagging:

The v54 migration is edited in place rather than superseded. AGENTS.md forbids editing a shipped migration because its schema_version row exists in real databases; v54 is introduced in this unmerged branch and has never been on main, so no such row exists and the hash check has nothing to disagree with. Anyone who has run this branch locally will need to drop their dev database. Say the word if you'd rather have a v55 doing the rename.

The plan is deleted in 9263ff6, per the convention that it goes at the end of its implementation PR. That takes the "What Windows inherits" section with it — the wcr[shim.ownership] restart clause, folding the exit observation into history before containerd reaps the task, and stating the daemon-down gap as a property. It survives in #133's description and in git history, but nothing in-tree carries it now. Happy to reproduce it as a comment on #107 if that's a more useful home.

I also filled in the two tests the plan listed that I'd missed: the rate trigger filing and observed-healthy clearing the fault, and start-limit-hit filing below the rate threshold. Both needed the same DB-side extraction I'd already done for the counter reconciliation.

---

Generated by Claude Code

[Copilot]

Pull request overview

Adds durable restart accounting to the Linux runtime (observing systemd’s NRestarts/last exit), derives crash_loop from a configurable restart-rate threshold/window, and exposes the resulting history + settings through OI, CLI, and the web UI.

Changes:

• Persist restart records and per-instance baselines in the DB, compute rate-derived crash loops, and GC retained restart history.
• Extend reconciliation to observe systemd restart counters and file crash_loop faults from either restart-rate or start-limit-hit triggers (with suppression derived from the fault table each tick).
• Add operator surfaces: /restarts/* OI endpoints, seedling-ctl restarts …, and a /restarts web route plus per-instance restart summary in app.describe.

Reviewed changes

Copilot reviewed 38 out of 38 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
docs/spec/web.md	Specifies the new /restarts web route requirements.
docs/spec/runtime.md	Defines restart recording, rate-based crash-loop detection, settings, and GC expectations.
docs/spec/interface.md	Defines restart record/list/settings interfaces and adds restarts summary to app.describe.
docs/runtime-overview.md	Updates conceptual model to include “Restart History” as a fourth persistent history.
crates/web/frontend/src/routes/Restarts.tsx	Implements the /restarts UI: list + filters + editable rate settings.
crates/web/frontend/src/routes/Restarts.test.tsx	Frontend tests for the restarts route behaviour and settings mutation payload.
crates/web/frontend/src/routes/AppDetail.tsx	Adds per-instance restart summary chip linking to /restarts?instance=….
crates/web/frontend/src/routes/AppDetail.resources.test.tsx	Tests restart chip rendering/linking behaviour in AppDetail resources table.
crates/web/frontend/src/lib/types.ts	Adds frontend types for restart records/settings/summary.
crates/web/frontend/src/hooks/useOi.ts	Fixes useOiQuery to refetch when params change (value-based keying).
crates/web/frontend/src/components/Navbar.tsx	Adds navbar link/icon to /restarts.
crates/web/frontend/src/App.tsx	Registers the new restarts route.
crates/ctl/src/restarts.rs	Adds `seedling-ctl restarts list
crates/ctl/src/main.rs	Wires the new restarts subcommand into the CLI.
crates/core/src/system/types.rs	Extends unit state and observation facts to carry restart counter + last exit.
crates/core/src/system/systemd.rs	Adds systemd Service proxy reads for NRestarts/ExecMain* and parsing.
crates/core/src/system/stub.rs	Extends stub process manager to simulate restarts/counter resets and last exits.
crates/core/src/system/reconcile/restarts/tests.rs	Unit tests for counter delta logic, resets, deliberate exclusion, threshold, and GC.
crates/core/src/system/reconcile/restarts.rs	Implements baseline reconciliation, recording, rate evaluation, and restart GC.
crates/core/src/system/reconcile/pods.rs	Threads restart counters through pods phase and adds crash-loop cause modelling.
crates/core/src/system/reconcile/phases.rs	Plumbs crash_looped suppression set into pods phase execution.
crates/core/src/system/reconcile/faults/crash_loop_tests.rs	Tests crash-loop fault filing/clearing for both triggers and deduping.
crates/core/src/system/reconcile/faults.rs	Updates crash-loop fault handling to support multiple triggers and clearer descriptions.
crates/core/src/system/reconcile.rs	Adds per-tick derivation of crash_looped from the fault table and runs bookkeeping.
crates/core/src/system/observer.rs	Emits UnitRestartCounter observation facts for deployment/job units.
crates/core/src/system/breadcrumb.rs	Adds tracey annotation for breadcrumb emission requirement.
crates/core/src/runtime/restarts/tests.rs	Tests DB restart recording, filtering, rate-window logic, caps, summaries, and settings validation.
crates/core/src/runtime/restarts.rs	Implements DB schema operations for restart records, summaries, baselines, and settings.
crates/core/src/runtime/gc.rs	Hooks restart-record GC into the periodic GC cycle.
crates/core/src/runtime/db/tests.rs	Bumps expected schema version to 54.
crates/core/src/runtime/db/migrations/v54.sql	Adds tables for restart records, baselines, and restart-rate settings.
crates/core/src/runtime/db.rs	Registers migration v54.
crates/core/src/runtime/barrier/replay.rs	Adds tracey annotation for breadcrumb replay-boundary behaviour.
crates/core/src/runtime.rs	Exposes the new runtime::restarts module.
crates/core/src/oi/handler/restarts/tests.rs	Tests OI restart list/settings endpoints end-to-end.
crates/core/src/oi/handler/restarts.rs	Adds OI handlers for restart list and settings get/set.
crates/core/src/oi/handler/apps.rs	Adds per-instance restart summary into app.describe responses.
crates/core/src/oi/handler.rs	Registers /restarts/* routes in the OI dispatcher.

Suppressed comments (1)

crates/core/src/oi/handler/restarts.rs:61

• As above in list_restarts, get_settings maps DB failures to ErrorCode::NotFound. This should be ErrorCode::Internal so clients can distinguish missing resources from server/database failures.

        .map_err(|e| OiError::new(ErrorCode::NotFound, format!("db query: {e}")))?;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.