Widen retry window for salt-minion service re-registration race - #70014
Open
twangboy wants to merge 1 commit into
Open
Widen retry window for salt-minion service re-registration race#70014twangboy wants to merge 1 commit into
twangboy wants to merge 1 commit into
Conversation
The NSIS stress tests still hit intermittent installer Abort (exit code 2) after the previous retry fix, on both 3006.x and 3007.x. Timing recovered from the CI logs shows the SCM held the salt-minion service key for 25s+ before the uninstall side's own wait_svc_deleted loop and the test harness's post-uninstall wait both gave up, leaving only ~10s of retry budget on the install side before it aborted -- not enough headroom for the observed delay. Wait for the salt-minion service registry key to disappear before even attempting "ssm install" (CreateService), instead of only reacting after CreateService fails. This is a no-op on a normal install, since the key was never present. Also widen the existing retry budgets: the install-side CreateService retry from 5x2s to 10x2s, and the uninstall-side wait_svc_deleted from 10s to 15s. Add diagnostics so future occurrences don't require a fresh repro: print the tail of the relevant %TEMP%\SaltInstaller\*.log directly into the pytest failure output on any non-zero exit or timeout, and upload the full log directory as a CI artifact from both the Logic Tests and Stress Tests jobs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The NSIS stress tests still hit intermittent installer Abort (exit code 2) after the previous retry fix, on both 3006.x and 3007.x. Timing recovered from the CI logs shows the SCM held the salt-minion service key for 25s+ before the uninstall side's own wait_svc_deleted loop and the test harness's post-uninstall wait both gave up, leaving only ~10s of retry budget on the install side before it aborted -- not enough headroom for the observed delay. Wait for the salt-minion service registry key to disappear before even attempting "ssm install" (CreateService), instead of only reacting after CreateService fails. This is a no-op on a normal install, since the key was never present. Also widen the existing retry budgets: the install-side CreateService retry from 5x2s to 10x2s, and the uninstall-side wait_svc_deleted from 10s to 15s.
Add diagnostics so future occurrences don't require a fresh repro: print the tail of the relevant %TEMP%\SaltInstaller*.log directly into the pytest failure output on any non-zero exit or timeout, and upload the full log directory as a CI artifact from both the Logic Tests and Stress Tests jobs.
Merge requirements satisfied?
[NOTICE] Bug fixes or features added to Salt require tests.
Commits signed with GPG?
Yes/No