[BUG] salt state.apply syncs base modules irrespective of saltenv during salt runs

[EliAndrewC]

I have a custom salt state sideboard.py (and a custom sideboard.py module which the custom states make use of) which provides the custom state function sideboard.config.

I'm using gitfs with the following config:

gitfs_disable_saltenv_mapping: false
gitfs_base: master
gitfs_update_interval: 10
gitfs_global_lock: False

gitfs_remotes:
  - file:///srv/salt/salt-states:
     - root: salt/states
     - base: master
     - saltenv:
        - prod:
           - ref: master
        - dev:
           - ref: dev

So I have a dev branch, and I'm able to push changes to that branch and then sync them to my local minion:

# salt-call saltutil.sync_all saltenv=dev
...
local:
    ----------
    beacons:
    clouds:
    engines:
    executors:
    grains:
    log_handlers:
    matchers:
    modules:
        - modules.sideboard
    output:
    proxymodules:
    renderers:
    returners:
    sdb:
    serializers:
    states:
        - states.sideboard
    thorium:
    utils:

So far so good. However, when I run

# salt-call state.apply saltenv=dev
...
[INFO    ] Caching directory '_states/' for environment 'dev'
[INFO    ] Copying '/var/cache/salt/minion/files/dev/_states/sideboard.py' to '/var/cache/salt/minion/extmods/states/sideboard.py'
[INFO    ] Syncing states for environment 'base'
[INFO    ] Loading cache from salt://_states, for base
[INFO    ] Caching directory '_states/' for environment 'base'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_states/sideboard.py' to '/var/cache/salt/minion/extmods/states/sideboard.py'
...

What seems to be happening is that the dev branch is downloaded, and then the base is also downloaded, and overwrites whatever was just downloaded from dev. This matches my observations, that the changes in the dev branch are not used in the run even though I specified saltenv=dev, and when I run

# cat /var/cache/salt/minion/files/dev/_states/sideboard.py

I can see that the master branch version is present rather than the dev branch. There doesn't seem to be a way to specify at the command line or in config to use the saltenv I provide as the version of the custom state. This used to work in prior versions of salt - I first noticed this on 3006.9 though I can't say for sure this is when this behavior started.

Versions Report

# salt-call --versions-report
Salt Version:
          Salt: 3006.9
 
Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: rocky 9.4 Blue Onyx
        locale: utf-8
       machine: x86_64
       release: 5.14.0-427.35.1.c9.x86_64
        system: Linux
       version: Rocky Linux 9.4 Blue Onyx

This seems to be related to #58362 and #55437 in that the same underlying issue may to be causing all 3 issues.

[matthewdva]

Root cause, for anyone triaging this:

BaseHighState.load_dynamic() passes every saltenv present in the top file match dict to saltutil.sync_all, rather than the saltenv the run was pinned to:

syncd = self.state.functions["saltutil.sync_all"](list(matches), refresh=False)

matches is not limited to the requested saltenv — extra saltenvs get in from master_tops data (which top_matches() merged without a saltenv filter, even though top file sections are filtered a few lines earlier) and from cross-saltenv - <saltenv>: <sls> entries in a top file.

That would be harmless if each saltenv had its own destination, but salt.utils.extmods.sync() copies every synced saltenv into the same flat extension_modules/<form>/ directory. There is no per-saltenv separation, so the last saltenv synced wins. That is exactly the sequence in the log above — dev is written, then base overwrites it at the same path — and it stays overwritten for subsequent runs, which is why saltutil.sync_all saltenv=dev followed by state.apply saltenv=dev behaves differently from state.apply saltenv=dev <sls>.

Opened #69981 against 3006.x with a fix and regression tests. A saltenv-pinned run now syncs dynamic modules from that saltenv only; runs with no explicit saltenv keep the previous behavior.

@EliAndrewC if you still have that gitfs setup around, a confirmation against your dev/master branches would be helpful.