Skip to content

feat(validate): add explicit Tier 2 and Tier 3 gating controls - #23

Merged
rng1995 merged 3 commits into
naren/tier3-provider-preflightfrom
naren/profile-aware-gating
Aug 12, 2026
Merged

feat(validate): add explicit Tier 2 and Tier 3 gating controls#23
rng1995 merged 3 commits into
naren/tier3-provider-preflightfrom
naren/profile-aware-gating

Conversation

@rng1995

@rng1995 rng1995 commented Jul 26, 2026

Copy link
Copy Markdown
Collaborator

Landing status

This PR merged into the staging branch naren/tier3-provider-preflight, not into main. Its base PR #19 is closed and was intentionally not merged, so the flags and reporting contract below are not available from the current GitHub release line.

Follow-up #49 replays the reviewed gating commits onto current main, adds the current Fern and repository documentation, and supersedes this PR for the release line. Review and merge #49 rather than attempting to retarget this already-merged stacked PR.

Summary

Adds explicit Tier 2 and Tier 3 gating controls and makes CLI, JSON, Markdown, and HTML consume the same finalized gating policy.

Source and dependency

Public-compatible behavior

  • Tier 1 always blocks.
  • Tier 2 remains blocking by default; --no-block-on-dedup makes it advisory.
  • Tier 3 remains advisory by default; --block-on-agent-eval makes it blocking.
  • Validates selected YAML/JSON or native Harbor task sources before live execution.
  • Records per-tier gating, Tier 3 applicability, public profile metadata, and a source-independent policy digest.

Adaptations and exclusions

  • Does not port internal/external audience switching or internal profile defaults.
  • Preserves the existing public exit-code behavior unless users opt into/out of the new controls.
  • Adds no managed runtime, internal provider, private endpoint, Milvus, or server behavior.

Validation

  • Complete exact-lockfile suite: 3338 passed, 8 skipped, 3 deselected.
  • Focused flag, source-preflight, policy-digest, and reporter-consistency tests: passed.
  • Ruff, CLI golden, git diff --check, packaging, and OSS-boundary checks: passed.

Risk controls

  • Reports and process exit use the same result objects after policy application.
  • Missing/invalid Tier 3 sources are advisory unless explicitly configured to block.
  • Network/runtime failures remain visible even when advisory.

rng1995 added 2 commits July 25, 2026 10:41
Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
…e-gating

Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>
@rng1995
rng1995 marked this pull request as ready for review July 27, 2026 21:28

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the explicit Tier 2/Tier 3 gate selection, source preflight, policy digest, and JSON/HTML/Markdown reporting consistency. I found no blocking code issues. The focused command, source, reporter, policy, and evaluation tests passed locally (86 tests), and the GitHub check matrix is green.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review found one blocking reporter/exit consistency issue; details inline. The stacked branch is otherwise current with its declared base and has no existing unresolved threads.

Comment thread src/skillevaluator/reporting/base.py Outdated
Signed-off-by: Narendran Raghavan <nraghavan@nvidia.com>

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready for an independent maintainer approval. All actionable review feedback is resolved. Final head 4f67972 is conflict-free against the exact declared base naren/tier3-provider-preflight at d407bd2: GitHub reports MERGEABLE/CLEAN, the branch is 0 behind / 3 ahead, and local git merge-tree produced 1589d694503e9e78384b38389c8394b61203f74f without conflicts. Every final-SHA CI/security/package check is green. Local verification: 3,339 passed, 8 skipped, 3 deselected with 79% coverage; focused gating/source/reporting matrix 87 passed; ruff, OSS boundary, diff check, and wheel/sdist build passed. GitHub rejected my APPROVE submission because this authenticated account is the PR author and cannot approve its own pull request.

@rng1995
rng1995 merged commit ff4aa5e into naren/tier3-provider-preflight Aug 12, 2026
12 checks passed
@rng1995
rng1995 deleted the naren/profile-aware-gating branch August 12, 2026 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant