fix(supervisor): return a deny decision instead of crashing at max escalation depth - #3557
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
PR Review Summary
Verdict: AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims. |
|
🔴 Contributor Check: HIGH
Automated check by AGT Contributor Check. |
There was a problem hiding this comment.
Pull request overview
TL;DR: 1 blocker, 1 warning. Fix the new regression test assertions and this ships.
| # | Sev | Issue | Where |
|---|---|---|---|
| 1 | Block | Regression test should assert the specific authority value (and avoid is True/False assertions) |
test_supervisor_escalation_depth.py |
| 2 | Warn | Inline comment is misleading now that escalate() can return early at the depth cap |
supervisor.py |
Changes:
- Fix
SupervisorHierarchy.escalate()to return a well-formed denyTrustDecisionat max escalation depth (instead of raisingTypeError). - Add regression tests covering above-cap and below-cap escalation behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
agent-governance-python/agent-os/src/agent_os/supervisor.py |
Fixes the depth-cap decision construction to match TrustDecision fields. |
agent-governance-python/agent-os/tests/test_supervisor_escalation_depth.py |
Adds regression tests for the max escalation depth guard and normal escalation flow. |
Suppressed comments (2)
agent-governance-python/agent-os/tests/test_supervisor_escalation_depth.py:58
- Use a truthiness assertion (
assert decision.allowed) instead ofis Trueto avoid boolean identity comparisons and keep tests consistent with the rest of the suite’s style.
assert decision.allowed is True
agent-governance-python/agent-os/src/agent_os/supervisor.py:149
- This comment is now misleading because
escalate()can return early with a depth-cap deny decision (i.e., the final decision does not always come from the trust root). Update it to reflect the depth-cap behavior.
authority="supervisor",
)
# Final decision always comes from the deterministic trust root
return self.trust_root.validate_action(action)
| assert isinstance(decision, TrustDecision) | ||
| assert decision.allowed is False | ||
| assert "Max escalation depth" in decision.reason | ||
| assert decision.authority # a non-empty authority is set | ||
| assert decision.deterministic is True |
c00117e to
fa74164
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (1)
agent-governance-python/agent-os/src/agent_os/supervisor.py:148
- The inline comment says the final decision always comes from the trust root, but
escalate()can now return early when the escalation-depth cap is exceeded. This comment is misleading and should be updated to reflect that the trust root is only reached when the depth cap is not hit.
# Final decision always comes from the deterministic trust root
fa74164 to
79609c0
Compare
79609c0 to
2947c22
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (1)
agent-governance-python/agent-os/src/agent_os/supervisor.py:146
- The
escalate()docstring still states the return value always comes from the trust root, but this method can now return early with a supervisor-authored deny when the escalation-depth cap is exceeded. Updating theReturns:section will keep the documentation accurate for callers.
return TrustDecision(
allowed=False,
reason="Max escalation depth exceeded",
authority="supervisor",
)
…calation depth At the escalation-depth cap, escalate() built TrustDecision with a policy_name keyword the dataclass does not define, and omitted the required authority field, so the guard raised TypeError instead of returning a deny. Depending on the caller's error handling that is a crash or, if an outer layer swallows the exception, a fail-open at exactly the point the limiter should engage. Construct the decision with the fields TrustDecision defines (authority "supervisor"), so the cap returns a well-formed deny. Adds a regression test that drives an escalation chain past max_escalation_depth and asserts a well-formed limiting decision, plus a below-cap case. Refs microsoft#3539 Signed-off-by: AlgoVoi <chopmob@gmail.com>
2947c22 to
de89b71
Compare
|
Duplicate of #3569, which fixes the same bug (#3539) in the same place. Yours came first by a day and the diagnosis is identical and equally well written, so this is not a quality call. The two differ in exactly one thing: you set One thing your PR has that the other does not, and it should not be lost: you also corrected the trailing comment to say "Below the escalation-depth cap, the final decision comes from the deterministic trust root". The original read "Final decision always comes from the deterministic trust root", which stopped being true the moment the cap branch started returning its own decision. That is worth carrying over. Also worth saying: #3659 and #3660 are both yours, both green, and both close real fail-open paths. I have reviewed and endorsed both. This one going to the other PR is a coin-flip on a naming detail, not a signal about your work. Leaving this open rather than closing it, since a maintainer may prefer your version or want the comment fix. |
|
Thanks for the careful read, and no disagreement on the call. The one piece worth not losing is the trailing comment. Once the cap branch returns its own decision, "Final decision always comes from the deterministic trust root" is no longer strictly true, so I changed it to "Below the escalation-depth cap, the final decision comes from the deterministic trust root". That is a one line change and easy to fold into #3569 if you would like it carried over. Whichever way is least friction for you. Fine to close this in favour of #3569 whenever suits. |
Summary
At the escalation-depth cap in
SupervisorHierarchy.escalate(), the guard built its decision as:but
TrustDecision(trust_root.py) is a dataclass defining onlyallowed,reason,authority,deterministic. So the call raisesTypeErrortwo ways: an unexpectedpolicy_namekeyword, and a missing requiredauthority. The branch meant to cap runaway escalation instead crashes the caller, and if an outer layer swallows the exception it becomes a fail-open at exactly the point the limiter should engage.Change
Construct the decision with the fields the dataclass defines, using
authority="supervisor"(the depth cap is enforced bySupervisorHierarchy; the only other authority literal in the tree is"native-runtime"). The cap now returns a well-formed deny.Tests
Adds
tests/test_supervisor_escalation_depth.py: one test drives an escalation chain pastmax_escalation_depthand asserts a well-formed limitingTrustDecision(allowed=False, ...)with noTypeError; one asserts below-cap escalation still flows to the trust root. Both pass, and the existingtest_trust_root,test_escalation, andtest_escalation_and_reversibilitysuites still pass.Refs #3539