Add EKS Auto Mode AZ impairment experiment - #36
Open
aamantle-aws wants to merge 2 commits into
Open
Conversation
Author
|
@JTipecska Would you mind reviewing this? (apologies don't have access to add you explicitly as a reviewer) It's the EKS Auto Mode AZ impairment template. TIA |
…ment Validated this experiment end-to-end against a live EKS Auto Mode cluster and fixed three issues found during testing. Reject built-in NodePools. NodePoolName defaulted to general-purpose, which AWS documents as unmodifiable: the Kubernetes API accepts the patch, then EKS reconciles it away. In testing, the AZ exclusion was reverted ~4 minutes into a 15-minute impairment while the experiment still reported success — a false pass that tests almost nothing. ValidateAndGetCluster now fails fast on general-purpose/system and the default is empty. Against a custom NodePool the exclusion held for the full 15 minutes. Don't destroy customer NodePool config on early failure. RestoreNodePool read PatchResult from a step that may never have run, so an early CordonNodesInAZ failure routed to restore with no patch record and deleted any pre-existing zone requirement. Restore now runs only on a confirmed NODEPOOL_PATCHED record, skips if the requirement was modified externally, and reports what it did instead of assuming success. Correct the README. It documented pod-delete and packet-loss targets, a wait-before-pod-delete action, and four placeholders that no file uses. Replaced with how pods are actually selected (nodes by zone label, then pods by nodeName, all non-system namespaces), and documented that rebalancing after recovery depends on topology constraints and resource requests — with soft constraints Auto Mode consolidates into one AZ and the impairment becomes invisible. Also dropped unused eks:AccessKubernetesApi from the FIS role policy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FIS experiment template for simulating AZ failure in EKS Auto Mode clusters. Uses FIS network disruption (subnet-level NACLs) combined with an SSM automation that cordons nodes, patches the NodePool to exclude the target AZ, deletes pods, and restores on cleanup.