Fix state.apply ignoring saltenv when autoloading dynamic modules - #69981
Open
matthewdva wants to merge 1 commit into
Open
Fix state.apply ignoring saltenv when autoloading dynamic modules#69981matthewdva wants to merge 1 commit into
matthewdva wants to merge 1 commit into
Conversation
`state.apply saltenv=<env>` (and `state.highstate saltenv=<env>`) could sync
custom modules from saltenvs other than the one requested, and the extra
saltenv would silently overwrite the requested one.
`BaseHighState.load_dynamic()` passed every saltenv in the top file match dict
to `saltutil.sync_all`:
syncd = self.state.functions["saltutil.sync_all"](list(matches), refresh=False)
`matches` is not limited to the requested saltenv. Extra saltenvs get in via
`_master_tops()` results -- which `top_matches()` merged with no saltenv
filter, even though top file sections are filtered against `opts["saltenv"]`
about 25 lines earlier -- and via cross-saltenv `- <saltenv>: <sls>` entries in
a top file.
That alone would be harmless if each saltenv had its own destination, but
`salt.utils.extmods.sync()` copies every saltenv into one flat
`extension_modules/<form>/` directory. There is no per-saltenv separation, so
the last saltenv in the list wins -- and stays winning for subsequent runs.
Reported in saltstack#67069, where `state.apply saltenv=dev` logs the `dev` copy of a
custom state being written and then the `base` copy overwriting it:
Copying '.../files/dev/_states/sideboard.py' to '.../extmods/states/sideboard.py'
Syncing states for environment 'base'
Copying '.../files/base/_states/sideboard.py' to '.../extmods/states/sideboard.py'
This also contradicts the documented behavior in
doc/topics/development/modules/index.rst: "When dynamic modules are autoloaded
via states, only the modules defined in the same saltenv as the states
currently being run are synced."
Changes:
* `load_dynamic()` syncs from exactly `opts["saltenv"]` when a saltenv is
pinned. With no explicit saltenv the previous behavior is kept and every
matched saltenv is synced -- `state.highstate` leaves `opts["saltenv"]` as
`None`, so that path stays reachable.
* `top_matches()` skips `master_tops` data for saltenvs other than the
requested one, matching the filter already applied to top file sections.
* `salt.utils.extmods.sync()` warns when a module name is present in more than
one of the saltenvs being synced, naming the saltenv whose copy wins. The
flat `extension_modules` directory and its last-one-wins precedence are left
as they are; this only makes the collision visible instead of silent.
* Documents the saltenv-pinned behavior and the shared `extension_modules`
directory. The sentence this builds on was an incomplete sentence on this
branch ("only the modules defined in the same saltenvs as the states
currently being run."); it is completed here to match the wording already
present on 3007.x, so the change merges forward cleanly.
Fixes saltstack#67069
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Makes a saltenv-pinned state run sync dynamic modules (
_modules,_states,_grains, ...) from that saltenv and nothing else.BaseHighState.load_dynamic()passed every saltenv present in the top file match dict tosaltutil.sync_all:matchesis not limited to the requested saltenv. Extra saltenvs get in two ways:_master_tops()results, whichtop_matches()merged with no saltenv filter — even though the top file sections ~25 lines earlier are filtered againstopts["saltenv"];- <saltenv>: <sls>entries in a top file.That would be harmless if each saltenv had its own destination, but
salt.utils.extmods.sync()copies every saltenv into one flatextension_modules/<form>/directory. There is no per-saltenv separation, so the last saltenv in the list overwrites the earlier ones — silently, and it stays overwritten for subsequent runs.This also contradicts what
doc/topics/development/modules/index.rstalready documents:What issues does this PR fix or reference?
Fixes #67069
Previous Behavior
From #67069 —
state.apply saltenv=devwrites thedevcopy of a custom state, then overwrites it with thebasecopy:The net effect, with a custom module
foo.pythat differs betweenbaseandqa:The targeted form works only because
state.slsdoes not autoload; it just uses whatever was cached.New Behavior
state.apply saltenv=qasyncs dynamic modules fromqaonly.salt/state.py—load_dynamic(): whenopts["saltenv"]is set, sync from exactly that saltenv. With no explicit saltenv the old behavior is kept and every matched saltenv is synced.state.highstatewith no saltenv leavesopts["saltenv"]asNone, so that path stays reachable and unchanged.salt/state.py—top_matches(): skipmaster_topsdata for saltenvs other than the requested one, matching the filter already applied to top file sections. Please look at this hunk specifically — it is the one behavior change beyond module syncing. If an ENC /master_topsreturnsbasestates and an operator wants those in asaltenv=qarun, this drops them. It is self-contained and can be dropped if you would rather not change that; theload_dynamic()hunk alone fixes the reported bug.salt/utils/extmods.py—sync(): warn when a module name exists in more than one synced saltenv, naming the winner. Pure logging — the flatextension_modulesdirectory and its last-one-wins precedence are deliberately left alone, since making it per-saltenv would be a much larger change touching the loader. This just makes the collision visible instead of silent.Merge requirements satisfied?
doc/topics/development/modules/index.rstdocuments the saltenv-pinned behavior and the sharedextension_modulesdirectorychangelog/67069.fixed.mdtests/pytests/unit/state/test_load_dynamic.py(6) andtests/pytests/unit/utils/test_extmods.py(4)The tests were verified to be genuine regression tests: with
salt/state.pyandsalt/utils/extmods.pyreverted, 4 of the 10 fail; with the fix, all 10 pass. The other 6 cover the unchanged no-explicit-saltenv path, guarding against over-correcting it.Full
tests/pytests/unit/suite was run with and without the patch on this branch; the failure sets are identical, so nothing here regresses. (The residual failures are pre-existing on a clean checkout and environment-specific — local pygit2 version, macOS-only tests, zeromq timing.)Notes for reviewers
3006.xper CONTRIBUTING.rst ("the oldest supported branch where the bug exists"). All three code sites are byte-identical on3006.x,3007.x,3008.xandmaster, and the reporter in [BUG] salt state.apply syncs base modules irrespective of saltenv during salt runs #67069 was running 3006.9.3006.x— "only the modules defined in the same saltenvs as the states currently being run." It is completed here to match the wording already on3007.x, so the doc change merges forward without conflict.Commits signed with GPG?
No