fix: make generated VAP deterministic to stop reconcile loop#4640
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes a nondeterministic ordering bug in two functions that produce namespace lists consumed by the constraint template controller when generating a ValidatingAdmissionPolicy's CEL match conditions. Because both functions built their slices by ranging over Go maps (whose iteration order is unstable), the generated VAP differed between reconciles, causing the controller to repeatedly re-patch the VAP and churn the template's status. Sorting the slices before returning makes the output stable.
Changes:
- Sort the result of
GetAllExemptedNamespacesWithWildcard(pkg/webhook/namespacelabel.go) before returning. - Sort the result of
Excluder.GetExcludedNamespaces(pkg/controller/config/process/excluder.go) before returning. - Add regression tests that repeatedly invoke each function and assert the output is sorted and matches the expected order.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
pkg/webhook/namespacelabel.go |
Adds sort import and sorts the combined exempt-namespace/prefix/suffix slice before returning. |
pkg/webhook/namespacelabel_test.go |
Adds a regression test asserting deterministic, sorted output across repeated calls. |
pkg/controller/config/process/excluder.go |
Adds sort import and sorts the excluded-namespace slice before returning. |
pkg/controller/config/process/excluder_test.go |
Adds a regression test asserting deterministic, sorted output across repeated calls. |
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #4640 +/- ##
==========================================
- Coverage 54.49% 44.85% -9.65%
==========================================
Files 134 284 +150
Lines 12329 21045 +8716
==========================================
+ Hits 6719 9439 +2720
- Misses 5116 10797 +5681
- Partials 494 809 +315
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
JaydipGabani
left a comment
There was a problem hiding this comment.
Could we also make the operation list generated in convertWebhookRulesToResourceRules deterministic?
This PR fixes the namespace-slice source of VAP churn, but there is another map-backed conversion in the same generated VAP path -
:intersection := ruleOpsSet.Intersection(ctOpsSet)
...
operations = intersection.UnsortedList()operations is written into spec.matchConstraints.resourceRules[*].operations. Since manageVAP compares the current and generated VAP with reflect.DeepEqual before updating, semantically equivalent operation sets can still look different if UnsortedList() returns a different order between reconciles.
For example, with syncVAPEnforcementScope=true, webhook ops ["CREATE", "UPDATE", "DELETE", "CONNECT"] and CT ops ["CREATE", "UPDATE"] could generate either ["CREATE", "UPDATE"] or ["UPDATE", "CREATE"]. That would still trigger a VAP update even though the policy is unchanged.
I think this should use a stable order, probably the existing canonical allOps order (CREATE, UPDATE, DELETE, CONNECT), rather than UnsortedList(). It would also be good to add a test that asserts exact operation order; the current tests compare operations as sets, so they would miss this churn path.
Two map/set-backed conversions in the VAP generation path had unstable ordering between reconciles: the exempt/excluded namespace slices (built by ranging over maps) and the resource-rule operations (from sets.Set.UnsortedList()). Since the controller compares VAPs with reflect.DeepEqual before updating, an unchanged policy compared unequal and triggered a VAP update every reconcile, which re-enqueued its owner ConstraintTemplate into a self-sustaining loop churning VAP/CT status. Sort the namespace slices at their source and emit operations in canonical order. Add regression tests. Signed-off-by: Stefan Peer <stefan@peerweb.it>
b1e78a5 to
edd6ab9
Compare
|
@JaydipGabani thanks for the review, changed as requested |
What this PR does / why we need it:
GetExcludedNamespaces and GetAllExemptedNamespacesWithWildcard built their returned slices by ranging over maps, so the order was nondeterministic.
These slices are rendered (via sync-vap-enforcement-scope) into the generated ValidatingAdmissionPolicy's CEL match conditions. Because the order varied each reconcile, the freshly-built VAP never matched the stored one, so the ConstraintTemplate controller issued a VAP Update every reconcile. The VAP change re-enqueued its owner ConstraintTemplate, producing a self-sustaining reconcile loop — constant VAP updates and ConstraintTemplate status rewrites (resourceVersion climbing with no spec change), and the resulting etcd write pressure.
Fix: sort both slices before returning, so VAP generation is byte-stable across reconciles. With stable ordering no Update is issued, and the loop stops.
Added regression tests asserting both functions return sorted output.
The diff in the screenshot shows the issue. The dumps were taken just seconds apart:
