agent: scheduled verify and durability-refresh cadences#172
Merged
Conversation
Close the two standing automation-loop gaps (friction log F32/F33) so a correctly configured household keeps its offsite evidence fresh with zero typed commands. - verify_every: a per-destination cadence (content-addressed/packed only) plus an [agent]-level default, driving the same pass as `squirrel verify` and recording an `audit` run exactly like the manual command. - pull_durability_every: a per-node cadence that gives relayed durability evidence its own clock, independent of sync activity. A receive-only node (the reference htpc) now refreshes its offload-gate evidence unattended. Both surface in the scheduler log with the existing kicked/finished/error discipline and respect the marker/init boundary: verify is read-only, the durability pull never rewinds a watermark (allowRewind is always false), and neither ever bootstraps a destination — the agent does not escalate. Plumbing: the scheduler gains two non-volume evaluation phases keyed on destinations and peer nodes, with in-memory cadence watermarks (cheap, idempotent, read-only work re-run once after a restart is harmless and, for evidence freshness, desirable). The verify/pull operations are injected as VerifyRunner/DurabilityPuller closures so the agent package keeps no edge to sync. Durability-pull runs are recorded as volume-less `audit` runs (like BeginRemoteVerifyRun), keeping them out of the per-volume drift-audit reads while still appearing in `squirrel runs`. No schema change: kind='audit' and free-text runs_audit transitions already exist. Docs and the reference setup are updated to configure and describe both cadences. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
There was a problem hiding this comment.
Pull request overview
Adds two new agent scheduler cadences to close automation gaps: periodic remote verification for content-addressed/packed destinations (verify_every) and periodic peer durability refresh (pull_durability_every) so evidence freshness doesn’t depend on manual commands or sync activity.
Changes:
- Config: add
verify_every(per-destination and[agent]default) andpull_durability_every(per-node) with validation and tests. - Agent/scheduler: add destination-scoped verify and (volume,peer)-scoped durability-pull evaluation phases, including deterministic logging and in-memory cadence watermarks.
- Store/docs/CLI wiring: record durability pulls as volume-less
auditruns with a newpull-durabilityruns_audit transition; wire agent closures tosync.VerifyRemoteandsync.PullDurability; document new knobs and update reference setup.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| store/runs.go | Factor out a shared helper for volume-less audit runs; add BeginDurabilityPullRun. |
| store/runs_audit.go | Add TransitionPullDurability audit transition constant. |
| store/remote_objects_test.go | Add test asserting durability-pull runs are volume-less and excluded from per-volume audit reads. |
| docs/src/content/docs/reference/configuration.md | Document verify_every (dest + agent default) and pull_durability_every (node). |
| docs/src/content/docs/guides/agent.md | Document scheduled verify and scheduled durability pull behavior. |
| design/reference-setup.md | Update reference household config/examples to exercise both new cadences. |
| config/nodes.go | Parse and expose pull_durability_every on nodes. |
| config/destinations.go | Parse/validate per-destination verify_every and reject it for non-verifiable layouts. |
| config/config.go | Add Destination.VerifyEvery to resolved config. |
| config/config_test.go | Add tests for destination verify cadence, agent default verify cadence, and node pull cadence parsing/errors. |
| config/agent.go | Parse and expose [agent] verify_every default cadence. |
| cmd/squirrel/agent.go | Wire scheduler closures for verify/durability-pull; broaden rclone setup to verify-only schedules; add run bookkeeping for durability pulls. |
| cmd/squirrel/agent_scheduler_test.go | Add unit tests for mapping/status helpers and wiring predicates. |
| agent/serve.go | Start scheduler loop when any scheduled work exists (volume, verify, or pull). |
| agent/scheduler.go | Add verify and durability-pull cadence evaluation phases with in-memory watermarks and deterministic ordering. |
| agent/scheduler_test.go | Update scheduler fixture to initialize new cadence fields/watermarks. |
| agent/scheduler_cadence_test.go | Add cadence-walk tests for verify and durability pulls (due/skip/refire/no-op/error/rewind/nil-runner/start-gate). |
| agent/agent.go | Extend agent config and public types for injected verify runner and durability puller. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- store: BeginDurabilityPullRun now labels its run "pull-durability", matching the runs_audit transition constant and the scheduler's log kind so the run, its note, and the scheduler line all correlate under one name (was "durability-pull"). - cmd/squirrel/agent: finishDurabilityPullRun joins the pull error with the AppendRunAudit/FinishRun bookkeeping errors (errors.Join) instead of only surfacing a bookkeeping error when the pull itself succeeded. A failure to record or finish the audit run — which would otherwise strand a 'running' row — now always reaches the scheduler error log. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
Make offload reachable against packed/CA offsites by fixing the two friction-log defects that together closed every gate path (F13 + F29). F13 — the pending-artifact gate was per-run. certifyPacked held the vector only when *this run's* packs were unverified, so a later run that packed nothing advanced the whole volume past an earlier still-pending pack, and once `squirrel verify` filled the fingerprints nothing re-advanced. The advance is now gated on zero pending artifacts for the whole (volume, destination) pair (CountVolumeContentsPendingFingerprint), and `squirrel verify` re-attempts the advance itself once a clean pass completes (UpgradeDestinationVectorToFingerprintVerified). F29 — the component method was never upgraded past presence+size, so the fail-closed gate refused relayed offsites forever. Capture and verify now upgrade a fully fingerprint-verified pair to a new `fingerprint-verified` verify_method (a free-text value; no schema change, no CHECK constraint). It relays over the durability pull like any other method. Decision 1b — a verified provider-fingerprint chain counts as content-verified for the gate, but only while a scheduled verify cadence (#154) keeps re-confirming it. The responder relays its effective verify cadence per destination (DurabilityComponent.VerifyEveryNs); the pull bakes a relayed fingerprint-verified component down to presence+size when that cadence is absent (older peer, or a destination the responder runs on no cadence), failing closed. The gate accepts a fingerprint-verified component when relayed (cadence proven at pull time) or when the local destination carries a live cadence; a local component without a cadence falls back to the per-object/pack scan-back, since the node re-reads its own fingerprints — the coupling binds the relayed path, where the evidence cannot be re-checked locally. Tests: both F13 sides (per-state hold, verify-then-advance) for packed and content-addressed, the capture-time method upgrade, the relayed accept / relayed-absent refuse cadence coupling, the local cadence/fallback/refuse cases, the pull-time baking, and the responder relaying its cadence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
The comment implied the fingerprint pending/advance gate covers the placement map and manifest segment, but CountVolumeContentsPendingFingerprint only considers the volume's present file contents (verified via remote_objects / remote_packs). Reword to state exactly what the gate covers — packs and per-hash objects, the artifacts that carry content bytes — and note that the placement map and manifest segment are re-derivable metadata whose landing is confirmed upstream in push (presence+size, before the run is promoted to success), so a run whose map or segment failed to land never reaches certifyPacked. Comment-only; no behaviour change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
The scheduler's pull-durability finished line logged fetched/applied/ dropped but omitted the rewinds count, leaving DurabilityPullReport.Rewinds set (cmd wiring) yet never read — an unused field on a public type, and an observability gap: a refused rewind surfaced only as status=partial with no count. Refused rewinds are a first-class "scary moment" (ux-principle 4) and automation is never invisible (ux-principle 5), so the count belongs in the finished line alongside dropped. Test now asserts rewinds=1 appears. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
Textual conflicts: - cmd/squirrel/agent.go: keep this branch's agent.Config wiring (Nodes, VerifyEvery, VerifyRunner, DurabilityPuller) on top of main's `version` var, which replaced this branch's local `agentVersion` const. - cmd/squirrel/offload.go: both sides appended a different helper at the same spot and happened to share a loop-closing tail — keep this branch's verifyCadencedTargets and main's relayed-capability pre-check helpers (#145), each with its own tail. - agent/durability_test.go: git interleaved two independent endpoint tests that share the request boilerplate — this branch's relayed verify-cadence assertions and main's capability advertisement. Split back into two functions. Semantic conflict: both sides added a Destinations field to agent.Config, for different readers — main's durability endpoint advertising gating capability, this branch's scheduler finding verify cadences. Same name, same type, same resolved map; collapsed into one field whose comment documents both consumers. go vet, go test ./..., and gofmt are clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQUzRmQHqbP2kV9kFrbBHL
main now carries #171 and the listener-less agent mode (#175), which is where this branch's two new cadences had to be reconciled. Textual conflicts: - store/runs_audit.go, agent/agent.go: both sides appended at the same spot — keep this branch's pull-durability transition and its VerifyRunner/DurabilityPuller types alongside main's reset-destination transition; main rewrote the Config doc comment for listener-less mode, so its version of that comment stands. - docs reference: both sides added to the [agent] section. Kept main's listener-less paragraph and this branch's key tables, and corrected the table to match — `listen` is no longer required, and `auth.token` is required only alongside it. Left as-is the two would have contradicted each other in adjacent paragraphs. Semantic conflict: main's listener-less start gate (cmd's agentHasWork) duplicated the scheduler's volume-cadence logic and predated this branch's verify_every / pull_durability_every, so it would have refused to start a listener-less agent whose only work is one of those cadences — exactly the receive-only htpc in design/reference-setup.md that F33 exists to serve. Replaced the duplicate with an exported agent.ScheduledWorkInConfig, built on the same resolvers the scheduler's own anyScheduledWork gate uses, so the two cannot drift again; the refusal message now names every cadence. Covered by a table test including the receive-only pull-only config. go vet, go test ./..., and gofmt are clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQUzRmQHqbP2kV9kFrbBHL
mbertschler
pushed a commit
that referenced
this pull request
Jul 25, 2026
main now carries #172's verify and durability-refresh cadences. Textual conflicts: - agent/scheduler.go, agent/scheduler_test.go: this branch replaced the scheduler's syncRun field with the dispatcher; main added the verify/pull cadence fields beside it. Keep the dispatcher and graft the cadence fields on. - cmd/squirrel/agent.go: keep the startup reaper and drop agentHasWork — main replaced it with the shared agent.ScheduledWorkInConfig this file already calls. In resolveSchedulerRclone, keep the stall-timeout guard and put main's sync-only gate around the version preflight (a verify-only schedule reads provider checksums and needs no `--hash blake3` check, but every automatic transfer still wants the guard). - store/runs_audit.go: both sides added transitions; keep all of them. Semantic conflict: #172 gave loadGate a `cadenced` argument, which this branch's offload.Readiness (added for the status surface) did not pass — the two files merged cleanly and stopped compiling. Readiness must feed the gate exactly what a real offload feeds it: `cadenced` widens the gate for locally-advanced fingerprint-verified components, so omitting it would make "N GB offloadable now" under-report against what an offload would actually move — the precise contract Readiness documents. The set is now needed in two packages, so the resolver moved out of cmd/squirrel to config.Config.VerifyCadencedTargets, beside the CanEverGateOffload predicate it complements; the offload command and the status readiness query now share one implementation. go vet, go test ./..., and gofmt are clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SQUzRmQHqbP2kV9kFrbBHL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the two standing automation-loop gaps from the friction log (F32, F33)
so a correctly configured household keeps its offsite evidence fresh with zero
typed commands — a prerequisite for the fingerprint-verified offload upgrade
that stacks on this (#155).
verify_everycadence per content-addressed /packed destination, plus an
[agent]-level default, drives the same passas
squirrel verifyand records it as anauditrun exactly like the manualcommand. Offsite bitrot detection no longer depends on anyone typing it.
pull_durability_everycadence per[nodes.X]gives relayed durability evidence its own clock, independent ofsync activity. A receive-only node (the reference htpc) now refreshes its
offload-gate evidence unattended, and any node keeps refreshing even when
nothing changes — exactly when
offload_max_evidence_agestarts counting.Both surface in the scheduler log with the existing
kicked/finished/errordiscipline and respect the marker/init boundary: verify is read-only, the
durability pull never rewinds a watermark (
allowRewindis alwaysfalse), andneither ever bootstraps a destination — the agent does not escalate.
Changes
verify_everyon destinations (rejected on non-verifiablelayouts) and on
[agent](fleet default);pull_durability_everyon[nodes.X]. All reuse the existing cadence-floor parsing.destinations and peer nodes, with in-memory cadence watermarks. The start
gate is broadened (
anyScheduledWork) so a receive-only node with only apull cadence still runs the scheduler. Verify/pull are injected as
VerifyRunner/DurabilityPullerclosures so the agent package keeps noimport edge to
sync.sync.VerifyRemoteandsync.PullDurability; rclone is resolved for a verify-only schedule too.BeginDurabilityPullRun(shares a helper withBeginRemoteVerifyRun) records the pull as a volume-lessauditrun with anew
pull-durabilityruns_audit transition — so it appears insquirrel runsyet stays out of the per-volume drift-audit reads.exercise both cadences; configuration reference and agent guide document them.
No schema change —
kind='audit'and free-textruns_audittransitionsalready exist (issue #157 owns the v25 bump in parallel).
Design decision to double-check
Durability-pull runs are recorded with
volume_idNULL (mirroringBeginRemoteVerifyRun), which keeps them out ofListAuditRunsSince(thedrift-since-last-sync handshake) and the TUI's per-volume "last audit" — the
pulled volume/peer live in the runs_audit note instead. The cadence watermarks
are in-memory rather than persisted: both operations are cheap, idempotent, and
read-only w.r.t. content, so re-running once after an agent restart is harmless
and (for evidence freshness) desirable.
Testing
go vet ./...— cleango test ./...— all packages passgofmt -l— cleanverify/durability cadence walks (due/skip/re-fire, no-op, error-consumes-
cadence, stake filter, partial-on-rewind, nil-runner inert, broadened start
gate);
BeginDurabilityPullRunshape + handshake-isolation; CLI statusmappers and wiring predicates.
golangci-lintcould not run locally (the sandbox's linter binary is builtwith go1.25 and refuses this module's
go 1.26.1directive); CI will run it.Unblocks #155.
Closes #154
🤖 Generated with Claude Code
https://claude.ai/code/session_014Skm2Qm6wWUB7dD4CpPTz7
Generated by Claude Code