Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions context/knowledge/gotchas/orchestration.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,14 @@ Hera orchestration + task-creation invariants that caused bugs when violated.
- **The seed prompt requires zero follow-up tool calls from the recycled session by construction, not by convention.** `BuildRecycleSeedPrompt` concatenates the role's stored mission prompt + a plan-DAG section (sibling role names/statuses) + the `handoff_note` meta value (if any) into one static string assembled server-side, handed to the new session as its literal opening prompt (`task.Prompt`). Only the role lookup is fatal; a missing plan-DAG or absent `handoff_note` degrades to an empty section rather than blocking the recycle.
- **The `B` rail key now bounces a worker/freelance selection instead of no-op'ing (add-worker-bounce); only an empty selection stays a true no-op.** `page.go`'s `case 'B'` switches on `sel.IsCoordinator()` (unchanged: confirm modal → `RecycleCoord(..., RecycleHumanForced)` directly, restarting mid-turn without waiting — the intended behavior for a session the self-service path can't reach because it's wedged) vs `sel.IsWorkerOrFreelance()` (new: confirm modal → `heraDoBounceWorker` → `sess.WriteInputSystem` sends a plain instruction asking the role to call `hera_status(handoff_note=..., request_recycle=true)` ITSELF — no direct kill/restart, no new `RecycleTrigger`; the now-widened self-service pipeline above completes the recycle once the role goes idle and makes that call). Nothing polls for a response and nothing times out if the role never calls it (D6, explicit scope decision) — the human can just press `B` again.

## Hera plan-DAG hygiene (add-hera-plan-hygiene)

Three independently-verified data-hygiene bugs, found live in `~/.argus/data.sql` — deliberately NOT unified under one root cause; two got code fixes, one did not.

- **A planned node outliving its archived/nuked orchestrator needs BOTH a write-time cascade AND a read-time filter — neither alone is sufficient.** `ArchiveHeraOrchestrator`/`NukeHeraOrchestrator` (`internal/db/hera.go`) cascade-cancel (`cancelled_at`) their still-planned (never-materialized) worker-kind child roles via `cancelStillPlannedChildRoles`, so a FUTURE archive/nuke never orphans a pollable dead node. But that alone does not repair rows that predate the fix (a real live bug: roles 343/358 retried `heragater` materialization every ~60s tick for over a month behind orchestrators archived+nuked a month earlier, spamming `hold: no coordinator to ping: <nil>`). `ListHeraPlannedNodes` (`internal/db/hera_plan.go`) therefore ALSO joins `hera_orchestrators` and requires `archived_at IS NULL AND nuked_at IS NULL` on the parent, independent of the node's own `cancelled_at` — this is what makes the fix retroactive with zero data migration. Both exist for the same reason the nuked-orchestrator check is duplicated in `internal/tui/hera/model.go`'s `BuildModel` even though the DB list query already excludes nuked rows: list queries are this codebase's belt-and-braces layer, not just the write path.
- **`heragater.materializeNode`'s "stays planned, retry next tick" failure path had no escalation — a permanently-broken node (e.g. blank `argus_project`) retried in total silence forever.** Fixed by a bounded consecutive-failure counter (`Watcher.materializeFailures`/`escalatedMaterializeFailures`, `materializeFailureEscalationTicks=5`) that sends ONE coordinator notice on crossing the threshold, mirroring `holdAndPing`'s ping-once-per-condition dedup shape (NOT `agent.EscalateParkedSelection`'s 8-tick threshold — that guards isolated torn reads on a sub-second polling loop, a noise source that doesn't exist for a synchronous, deterministic materialize call). Escalation is advisory ONLY: it never auto-cancels or reconfigures the node — a node with zero remaining blockers is presumptively genuine pending work, and only a human can say what's actually wrong with it. Both maps are swept every `Tick()` for any node id no longer in the planned set (`sweepMaterializeFailures`, mirroring `rearmHeldPings`'s cleanup of `heldPings`) — an unbounded per-node-forever map would itself be exactly the class of hygiene defect this fix exists to close.
- **A `kind=freelance` role stuck in the Hera rail's flat top-level Freelance section with an intact role→orchestrator→binding→task chain is a data-hygiene gap, not a display bug.** `internal/tui/hera/model.go`'s rule (`role.Kind == HeraKindFreelance && role.ArchivedAt == nil && o.ArchivedAt == nil` → flat section) is intentional and working as designed; a `kind=worker` role nests under its orchestrator unconditionally regardless of archived state, but a freelance role escapes the flat section only by being archived. Two live rows (roles 813/814) simply were never archived once their work finished (tasks already `in_review`/`complete`) — the identical "task finished, binding still open" shape is the NORMAL, common case for ~150 other historical `kind=worker` roles that render fine. No `Clean`/`Prune`/`Sweep`-named function touching hera data exists anywhere in this codebase; whatever produced this pair's un-archived state was very likely a manual/ad hoc action outside argus. Resolution is a one-time data cleanup (`ArchiveHeraRole` + `EndHeraBinding` on the two rows), NOT a code change — auto-archiving a freelance role on binding-end would be a genuine, separately-scoped behavior change with its own design surface (when exactly does "binding ended" count?), deliberately not smuggled into a three-bug hygiene fix.

## Hera subtree TLDR roll-up (M5)

- **The subtree (TLDR roll-up) and the orchestration tree share the SAME nesting graph now.** `SubtreeOrchIDs` walks the orchestrator *nesting* graph (multi-binding bridges) for message roll-up; `heraTreeNodes` (the Details-pane graph) walks the SAME nesting in-memory for rendering. Both derive from role bindings — there is no longer a separate `depends_on` dependency graph to conflate them with (it was retired). A message roll-up is still a distinct concern from the rendered tree, but they no longer use orthogonal data sources.
Expand Down
2 changes: 1 addition & 1 deletion context/knowledge/index.md

Large diffs are not rendered by default.

50 changes: 46 additions & 4 deletions internal/db/hera.go
Original file line number Diff line number Diff line change
Expand Up @@ -360,9 +360,21 @@ func (d *DB) ListHeraOrchestrators(includeArchived bool) ([]*HeraOrchestrator, e
// ArchiveHeraOrchestrator stamps archived_at (current time) and CLEARS
// pinned_at — pin and archive are mutually exclusive. Idempotent: re-archiving
// preserves the original archived_at. Returns ErrHeraNotFound if no row matches.
//
// Cascade-cancels the orchestrator's still-planned (never-materialized) child
// roles (add-hera-plan-hygiene Bug A) — a planned node whose parent just ended
// would otherwise retry materialization forever with no coordinator to ping.
// The cascade is best-effort: a failure is logged, never returned, so archive's
// existing error contract (nil / ErrHeraNotFound) is unchanged.
func (d *DB) ArchiveHeraOrchestrator(id int64) error {
return d.heraSetFlag(`UPDATE hera_orchestrators SET archived_at=?, pinned_at=NULL WHERE id=? AND archived_at IS NULL`,
heraOrchExistsProbe, id, formatTime(time.Now()))
if err := d.heraSetFlag(`UPDATE hera_orchestrators SET archived_at=?, pinned_at=NULL WHERE id=? AND archived_at IS NULL`,
heraOrchExistsProbe, id, formatTime(time.Now())); err != nil {
return err
}
if err := d.cancelStillPlannedChildRoles(id); err != nil {
slog.Warn("ArchiveHeraOrchestrator: cascade-cancel planned children failed", "orchestrator_id", id, "err", err)
}
return nil
}

// UnarchiveHeraOrchestrator clears archived_at. Idempotent. Returns
Expand Down Expand Up @@ -395,11 +407,41 @@ func (d *DB) UnpinHeraOrchestrator(id int64) error {
// rail-feeding lists (ListHeraOrchestrators) but its row is retained and still
// returned by id (HeraOrchestrator) for DB-only recovery. NEVER a hard delete.
// Returns ErrHeraNotFound if no row matches id.
//
// Cascade-cancels still-planned child roles exactly like ArchiveHeraOrchestrator
// (add-hera-plan-hygiene Bug A) — same best-effort, log-only-on-failure contract.
func (d *DB) NukeHeraOrchestrator(id int64) error {
now := formatTime(time.Now())
return d.heraSetFlag(
if err := d.heraSetFlag(
`UPDATE hera_orchestrators SET nuked_at=COALESCE(nuked_at, ?), archived_at=COALESCE(archived_at, ?), pinned_at=NULL WHERE id=?`,
heraOrchExistsProbe, id, now, now)
heraOrchExistsProbe, id, now, now); err != nil {
return err
}
if err := d.cancelStillPlannedChildRoles(id); err != nil {
slog.Warn("NukeHeraOrchestrator: cascade-cancel planned children failed", "orchestrator_id", id, "err", err)
}
return nil
}

// cancelStillPlannedChildRoles stamps cancelled_at on every still-planned
// (never-materialized) worker-kind child role of an orchestrator — mirrors
// ListHeraPlannedNodes's own definition of a planned node exactly (kind=worker,
// not archived, not cancelled, no binding ever) so the cascade only ever
// touches rows that query would otherwise keep surfacing forever. Idempotent
// (COALESCE) and scoped to one orchestrator; a materialized (bound) child is
// never touched.
func (d *DB) cancelStillPlannedChildRoles(orchID int64) error {
d.mu.Lock()
defer d.mu.Unlock()
_, err := d.conn.Exec(
`UPDATE hera_roles SET cancelled_at=COALESCE(cancelled_at, ?)
WHERE orchestrator_id=? AND kind=? AND archived_at IS NULL AND cancelled_at IS NULL
AND NOT EXISTS (SELECT 1 FROM hera_bindings b WHERE b.role_id = hera_roles.id)`,
formatTime(time.Now()), orchID, string(HeraKindWorker))
if err != nil {
return fmt.Errorf("cancel still-planned child roles: %w", err)
}
return nil
}

// SetHeraOrchestratorKanbanStatus sets the orchestrator's independent kanban
Expand Down
9 changes: 9 additions & 0 deletions internal/db/hera_plan.go
Original file line number Diff line number Diff line change
Expand Up @@ -356,14 +356,23 @@ type HeraPlannedNode struct {
// materialized (it has a binding row, live or ended) is NOT a planned node, even
// if its binding has since ended; the gater never re-materializes. Archived
// roles are excluded.
//
// The parent orchestrator's own archived_at/nuked_at is ALSO checked (JOIN,
// not just the node's own columns) — a defensive filter independent of
// ArchiveHeraOrchestrator/NukeHeraOrchestrator's cascade-cancel of still-planned
// children. The cascade prevents the state going forward; this filter tolerates
// it regardless of cause, including rows that predate the cascade existing at
// all, with no data migration required (add-hera-plan-hygiene Bug A).
func (d *DB) ListHeraPlannedNodes() ([]*HeraRole, error) {
d.mu.Lock()
defer d.mu.Unlock()
rows, err := d.conn.Query(
`SELECT r.id, r.orchestrator_id, r.name, r.kind, r.argus_project, r.prompt,
r.created_at, r.archived_at, r.pinned_at, r.nuked_at, r.node_kind, r.cancelled_at, r.archetype
FROM hera_roles r
JOIN hera_orchestrators o ON o.id = r.orchestrator_id
WHERE r.kind=? AND r.archived_at IS NULL AND r.cancelled_at IS NULL
AND o.archived_at IS NULL AND o.nuked_at IS NULL
AND NOT EXISTS (SELECT 1 FROM hera_bindings b WHERE b.role_id = r.id)
ORDER BY r.id ASC`,
string(HeraKindWorker))
Expand Down
55 changes: 55 additions & 0 deletions internal/db/hera_plan_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,61 @@ func TestListHeraPlannedNodes_StaysPlannedAfterBindingEnds(t *testing.T) {
_ = role
}

func TestListHeraPlannedNodes_ExcludesArchivedOrNukedOrchestrator(t *testing.T) {
// add-hera-plan-hygiene Bug A: a planned node whose PARENT orchestrator has
// ended must not keep surfacing forever, even if the node's own
// cancelled_at was never stamped (the defensive read-path filter must work
// independent of ArchiveHeraOrchestrator/NukeHeraOrchestrator's cascade —
// this is what makes the fix retroactive for rows that predate it).
t.Run("archived orchestrator", func(t *testing.T) {
d := testDB(t)
orch := planTestOrch(t, d, "archived-orch")
node := plannedRole(t, d, orch, "node")

// Stamp archived_at directly, bypassing ArchiveHeraOrchestrator's own
// cascade, to prove the list query itself excludes the node.
_, err := d.conn.Exec(`UPDATE hera_orchestrators SET archived_at=? WHERE id=?`, "2026-06-20T00:00:00Z", orch)
testutil.NoError(t, err)

got, err := d.ListHeraPlannedNodes()
testutil.NoError(t, err)
for _, r := range got {
if r.ID == node.ID {
t.Fatalf("planned node %d under an archived orchestrator must not be listed", node.ID)
}
}
})

t.Run("nuked orchestrator", func(t *testing.T) {
d := testDB(t)
orch := planTestOrch(t, d, "nuked-orch")
node := plannedRole(t, d, orch, "node")

_, err := d.conn.Exec(`UPDATE hera_orchestrators SET nuked_at=?, archived_at=? WHERE id=?`,
"2026-06-20T00:00:00Z", "2026-06-20T00:00:00Z", orch)
testutil.NoError(t, err)

got, err := d.ListHeraPlannedNodes()
testutil.NoError(t, err)
for _, r := range got {
if r.ID == node.ID {
t.Fatalf("planned node %d under a nuked orchestrator must not be listed", node.ID)
}
}
})

t.Run("active orchestrator's planned node is unaffected", func(t *testing.T) {
d := testDB(t)
orch := planTestOrch(t, d, "active-orch")
node := plannedRole(t, d, orch, "node")

got, err := d.ListHeraPlannedNodes()
testutil.NoError(t, err)
testutil.Equal(t, len(got), 1)
testutil.Equal(t, got[0].ID, node.ID)
})
}

func TestPlannedNode_ShortIDIsStableAcrossPlanEdits(t *testing.T) {
// The planner-assigned short-id-prefixed name is a durable handle: it is
// persisted verbatim and never recomputed by a plan operation (adding or
Expand Down
63 changes: 63 additions & 0 deletions internal/db/hera_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1063,6 +1063,69 @@ func TestNukeHeraOrchestrator(t *testing.T) {
})
}

// TestArchiveNukeOrchestrator_CascadeCancelPlannedChildren pins the
// add-hera-plan-hygiene Bug A cascade: archiving or nuking an orchestrator
// stamps cancelled_at on its still-planned (never-materialized) worker-kind
// children, so they stop being retried by the gater forever, while an
// already-materialized (bound) child is left untouched.
func TestArchiveNukeOrchestrator_CascadeCancelPlannedChildren(t *testing.T) {
t.Run("archive cancels a still-planned child", func(t *testing.T) {
d := heraTestDB(t)
o := mkOrch(t, d, "alpha")
planned := mkRole(t, d, o.ID, "planned", HeraKindWorker)

testutil.NoError(t, d.ArchiveHeraOrchestrator(o.ID))

got, err := d.HeraRole(planned.ID)
testutil.NoError(t, err)
if got.CancelledAt == nil {
t.Fatal("expected the still-planned child role to be cancelled")
}
})

t.Run("nuke cancels a still-planned child", func(t *testing.T) {
d := heraTestDB(t)
o := mkOrch(t, d, "beta")
planned := mkRole(t, d, o.ID, "planned", HeraKindWorker)

testutil.NoError(t, d.NukeHeraOrchestrator(o.ID))

got, err := d.HeraRole(planned.ID)
testutil.NoError(t, err)
if got.CancelledAt == nil {
t.Fatal("expected the still-planned child role to be cancelled")
}
})

t.Run("a materialized (bound) child is left untouched", func(t *testing.T) {
d := heraTestDB(t)
o := mkOrch(t, d, "gamma")
bound, _, err := d.CreateHeraRoleWithBinding(CreateHeraRoleInput{
OrchestratorID: o.ID, Name: "bound", Kind: HeraKindWorker, ArgusProject: "proj",
}, "task-1", "/wt/bound")
testutil.NoError(t, err)

testutil.NoError(t, d.ArchiveHeraOrchestrator(o.ID))

got, err := d.HeraRole(bound.ID)
testutil.NoError(t, err)
testutil.Nil(t, got.CancelledAt)
})

t.Run("an already-cancelled or already-archived child is left as-is", func(t *testing.T) {
d := heraTestDB(t)
o := mkOrch(t, d, "delta")
archived := mkRole(t, d, o.ID, "archived", HeraKindWorker)
testutil.NoError(t, d.ArchiveHeraRole(archived.ID))

testutil.NoError(t, d.ArchiveHeraOrchestrator(o.ID))

got, err := d.HeraRole(archived.ID)
testutil.NoError(t, err)
testutil.Nil(t, got.CancelledAt) // archived, not cancelled — cascade skips it
})
}

// TestHeraOrchestratorKanbanStatus pins the add-hera-kanban-status axis: default
// value, round-tripping through Set + both read paths, independence from
// pin/archive, and the missing-row error.
Expand Down
Loading
Loading