Skip to content

fix: make generated VAP deterministic to stop reconcile loop#4640

Open
speer wants to merge 1 commit into
open-policy-agent:masterfrom
speer:fix/vap-reconcile-loop-sort-namespaces
Open

fix: make generated VAP deterministic to stop reconcile loop#4640
speer wants to merge 1 commit into
open-policy-agent:masterfrom
speer:fix/vap-reconcile-loop-sort-namespaces

Conversation

@speer

@speer speer commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

GetExcludedNamespaces and GetAllExemptedNamespacesWithWildcard built their returned slices by ranging over maps, so the order was nondeterministic.

These slices are rendered (via sync-vap-enforcement-scope) into the generated ValidatingAdmissionPolicy's CEL match conditions. Because the order varied each reconcile, the freshly-built VAP never matched the stored one, so the ConstraintTemplate controller issued a VAP Update every reconcile. The VAP change re-enqueued its owner ConstraintTemplate, producing a self-sustaining reconcile loop — constant VAP updates and ConstraintTemplate status rewrites (resourceVersion climbing with no spec change), and the resulting etcd write pressure.

Fix: sort both slices before returning, so VAP generation is byte-stable across reconciles. With stable ordering no Update is issued, and the loop stops.

Added regression tests asserting both functions return sorted output.

The diff in the screenshot shows the issue. The dumps were taken just seconds apart:
image

Copilot AI review requested due to automatic review settings June 24, 2026 19:28
@speer speer requested a review from a team as a code owner June 24, 2026 19:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a nondeterministic ordering bug in two functions that produce namespace lists consumed by the constraint template controller when generating a ValidatingAdmissionPolicy's CEL match conditions. Because both functions built their slices by ranging over Go maps (whose iteration order is unstable), the generated VAP differed between reconciles, causing the controller to repeatedly re-patch the VAP and churn the template's status. Sorting the slices before returning makes the output stable.

Changes:

  • Sort the result of GetAllExemptedNamespacesWithWildcard (pkg/webhook/namespacelabel.go) before returning.
  • Sort the result of Excluder.GetExcludedNamespaces (pkg/controller/config/process/excluder.go) before returning.
  • Add regression tests that repeatedly invoke each function and assert the output is sorted and matches the expected order.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
pkg/webhook/namespacelabel.go Adds sort import and sorts the combined exempt-namespace/prefix/suffix slice before returning.
pkg/webhook/namespacelabel_test.go Adds a regression test asserting deterministic, sorted output across repeated calls.
pkg/controller/config/process/excluder.go Adds sort import and sorts the excluded-namespace slice before returning.
pkg/controller/config/process/excluder_test.go Adds a regression test asserting deterministic, sorted output across repeated calls.

@codecov-commenter

codecov-commenter commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 44.85%. Comparing base (3350319) to head (edd6ab9).
⚠️ Report is 773 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (3350319) and HEAD (edd6ab9). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (3350319) HEAD (edd6ab9)
unittests 2 1
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4640      +/-   ##
==========================================
- Coverage   54.49%   44.85%   -9.65%     
==========================================
  Files         134      284     +150     
  Lines       12329    21045    +8716     
==========================================
+ Hits         6719     9439    +2720     
- Misses       5116    10797    +5681     
- Partials      494      809     +315     
Flag Coverage Δ
unittests 44.85% <100.00%> (-9.65%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JaydipGabani JaydipGabani left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we also make the operation list generated in convertWebhookRulesToResourceRules deterministic?

This PR fixes the namespace-slice source of VAP churn, but there is another map-backed conversion in the same generated VAP path -

func convertWebhookRulesToResourceRules(rules []admissionregistrationv1beta1.RuleWithOperations, ctOps []admissionregistrationv1beta1.OperationType) ([]admissionregistrationv1beta1.NamedRuleWithOperations, error) {
:

intersection := ruleOpsSet.Intersection(ctOpsSet)
...
operations = intersection.UnsortedList()

operations is written into spec.matchConstraints.resourceRules[*].operations. Since manageVAP compares the current and generated VAP with reflect.DeepEqual before updating, semantically equivalent operation sets can still look different if UnsortedList() returns a different order between reconciles.

For example, with syncVAPEnforcementScope=true, webhook ops ["CREATE", "UPDATE", "DELETE", "CONNECT"] and CT ops ["CREATE", "UPDATE"] could generate either ["CREATE", "UPDATE"] or ["UPDATE", "CREATE"]. That would still trigger a VAP update even though the policy is unchanged.

I think this should use a stable order, probably the existing canonical allOps order (CREATE, UPDATE, DELETE, CONNECT), rather than UnsortedList(). It would also be good to add a test that asserts exact operation order; the current tests compare operations as sets, so they would miss this churn path.

Two map/set-backed conversions in the VAP generation path had unstable
ordering between reconciles: the exempt/excluded namespace slices (built
by ranging over maps) and the resource-rule operations (from
sets.Set.UnsortedList()). Since the controller compares VAPs with
reflect.DeepEqual before updating, an unchanged policy compared unequal
and triggered a VAP update every reconcile, which re-enqueued its owner
ConstraintTemplate into a self-sustaining loop churning VAP/CT status.

Sort the namespace slices at their source and emit operations in
canonical order. Add regression tests.

Signed-off-by: Stefan Peer <stefan@peerweb.it>
Copilot AI review requested due to automatic review settings July 7, 2026 05:59
@speer speer force-pushed the fix/vap-reconcile-loop-sort-namespaces branch from b1e78a5 to edd6ab9 Compare July 7, 2026 05:59
@speer speer changed the title fix: sort exempt/excluded namespaces for deterministic ordering fix: make generated VAP deterministic to stop reconcile loop Jul 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

@speer

speer commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

@JaydipGabani thanks for the review, changed as requested

@speer speer requested a review from JaydipGabani July 7, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants