PKI index optimization using memory-mapped hash table

changelog/68936.added.md

@@ -0,0 +1 @@
+Implemented an O(1) memory-mapped PKI index to optimize minion public key lookups. This optimization substantially reduces master disk I/O and publication overhead in large-scale environments by replacing linear directory scans with constant-time hash table lookups. The feature is opt-in via the `pki_index_enabled` master configuration setting.

doc/contents.rst

@@ -36,6 +36,7 @@ Salt Table of Contents
     topics/api
     topics/topology/index
     topics/cache/index
+    topics/performance/index
     topics/slots/index
     topics/windows/index
     topics/development/index

doc/ref/configuration/master.rst

@@ -207,6 +207,76 @@ following the Filesystem Hierarchy Standard (FHS) might set it to
     pki_dir: /etc/salt/pki/master
 
 
+.. conf_master:: pki_index_enabled
+
+``pki_index_enabled``
+---------------------
+
+.. versionadded:: 3009.0
+
+Default: ``False``
+
+Enable the O(1) PKI index optimization. This uses a memory-mapped hash table
+to speed up minion public key lookups, which can substantially decrease
+master publish times and authentication overhead in large environments.
+
+.. code-block:: yaml
+
+    pki_index_enabled: True
+
+.. conf_master:: pki_index_size
+
+``pki_index_size``
+------------------
+
+.. versionadded:: 3009.0
+
+Default: ``1000000``
+
+The number of slots in the PKI index. For best performance and minimal
+collisions, this should be set to approximately 2x your total minion count.
+This value applies to each shard if sharding is enabled.
+
+.. code-block:: yaml
+
+    pki_index_size: 1000000
+
+.. conf_master:: pki_index_shards
+
+``pki_index_shards``
+--------------------
+
+.. versionadded:: 3009.0
+
+Default: ``1``
+
+The number of shards to split the PKI index across. Sharding allows the index
+to span multiple memory-mapped files, which can improve concurrency and
+performance in extremely large environments or on filesystems with specific
+locking characteristics.
+
+.. code-block:: yaml
+
+    pki_index_shards: 1
+
+.. conf_master:: pki_index_slot_size
+
+``pki_index_slot_size``
+-----------------------
+
+.. versionadded:: 3009.0
+
+Default: ``128``
+
+The size in bytes of each slot in the PKI index. This must be large enough
+to hold your longest minion ID plus approximately 10 bytes of internal
+overhead (state information and separators).
+
+.. code-block:: yaml
+
+    pki_index_slot_size: 128
+
+
 .. conf_master:: cluster_id
 
 ``cluster_id``

doc/ref/runners/all/index.rst

@@ -26,6 +26,7 @@ runner modules
     net
     network
     pillar
+    pki
     queue
     reactor
     salt

doc/ref/runners/all/salt.runners.pki.rst

@@ -0,0 +1,9 @@
+.. _all-salt.runners.pki:
+
+================
+salt.runners.pki
+================
+
+.. automodule:: salt.runners.pki
+    :members:
+    :undoc-members:

doc/topics/performance/index.rst

@@ -0,0 +1,13 @@
+.. _performance:
+
+===========
+Performance
+===========
+
+This section covers various performance optimizations and scaling considerations
+for Salt.
+
+.. toctree::
+    :maxdepth: 1
+
+    pki_index

doc/topics/performance/pki_index.rst

@@ -0,0 +1,52 @@
+.. _pki_index:
+
+====================
+PKI Index Operations
+====================
+
+The PKI index is an optional, high-performance optimization designed for Salt
+environments with a large number of minions.
+
+Overview
+========
+
+By default, the Salt Master performs linear directory scans to find minion
+public keys during authentication and job publication. As the number of minions
+grows into the thousands, these disk I/O operations can become a significant
+bottleneck.
+
+The PKI index replaces these linear scans with a constant-time O(1) lookup
+using a memory-mapped hash table. This substantially reduces disk I/O and
+improves Master responsiveness.
+
+Enabling the Index
+==================
+
+To enable the PKI index, add the following to your Master configuration file:
+
+.. code-block:: yaml
+
+    pki_index_enabled: True
+
+Configuration
+=============
+
+While the default settings work for most environments, you can tune the index
+using these options:
+
+* :conf_master:`pki_index_size`: The number of slots in the hash table (default: 1,000,000).
+* :conf_master:`pki_index_slot_size`: The size of each slot in bytes (default: 128).
+
+Monitoring and Management
+=========================
+
+You can check the status of your PKI index or force a manual rebuild using the
+:ref:`PKI runner <all-salt.runners.pki>`:
+
+.. code-block:: bash
+
+    # Check index status and load factor
+    salt-run pki.status
+
+    # Manually rebuild the index from the filesystem
+    salt-run pki.rebuild_index

salt/cache/__init__.py

@@ -260,6 +260,36 @@ def list(self, bank):
         fun = f"{self.driver}.list"
         return self.modules[fun](bank, **self.kwargs)
 
+    def list_all(self, bank, include_data=False):
+        """
+        Lists all entries with their data from the specified bank.
+        This is more efficient than calling list() + fetch() for each entry.
+
+        :param bank:
+            The name of the location inside the cache which will hold the key
+            and its associated data.
+
+        :param include_data:
+            Whether to include the full data for each entry. For some drivers
+            (like localfs_key), setting this to False avoids expensive disk reads.
+
+        :return:
+            A dict of {key: data} for all entries in the bank. Returns an empty
+            dict if the bank doesn't exist or the driver doesn't support list_all.
+
+        :raises SaltCacheError:
+            Raises an exception if cache driver detected an error accessing data
+            in the cache backend (auth, permissions, etc).
+        """
+        fun = f"{self.driver}.list_all"
+        if fun in self.modules:
+            return self.modules[fun](bank, include_data=include_data, **self.kwargs)
+        else:
+            # Fallback for drivers that don't implement list_all
+            raise AttributeError(
+                f"Cache driver '{self.driver}' does not implement list_all"
+            )
+
     def contains(self, bank, key=None):
         """
         Checks if the specified bank contains the specified key.