What happened?
We have several salt masters, running on Ubuntu 22.04. They all had version 3006.13 as it was stable.
We tried to upgrade them, first the test one, then all others, but we couldn't find a newer stable version until 3006.27.
We tried 3006.16 and 3006.19 and even on test salt master with just a few salt minion there was a memory leak in salt-api.
And finally 3006.27 seemed good, no memory leaks in salt-api, so we upgraded all our salt masters to 3006.27.
All of salt-masters are doing well except one. The only difference with the other salt masters is that this one has a lot of minions that are not available at the moment.
Memory utilization looks like this:
When it gets to the peak:
Some time before salt-master goes completely crazy it had the errors like this (several times):
2026-07-27 07:31:56,432 [salt.master :1956][ERROR ][2425] Error in function _return:
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/master.py", line 1950, in run_func
ret = getattr(self, func)(load)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/master.py", line 1743, in _return
salt.utils.job.store_job(
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/job.py", line 31, in store_job
return _store_job(opts, load, event, mminion, endtime=endtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/job.py", line 91, in _store_job
event.fire_event(
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/event.py", line 883, in fire_event
self.pusher.send(msg)
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 153, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 169, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
yielded = next(result)
^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/transport/ipc.py", line 445, in send
yield self.stream.write(pack)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/iostream.py", line 409, in write
self._add_io_state(self.io_loop.WRITE)
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/iostream.py", line 975, in _add_io_state
self.io_loop.update_handler(self.fileno(), self._state)
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/ioloop.py", line 734, in update_handler
self._impl.modify(fd, events | self.ERROR)
FileNotFoundError: [Errno 2] No such file or directory
Then this one:
2026-07-27 08:00:59,982 [salt.client :1916][ERROR ][648013] Message timed out
2026-07-27 08:01:00,024 [salt._logging.impl:1101][ERROR ][648013] An un-handled exception was caught by Salt's global exception handler:
SaltClientError: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can
then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 1914, in pub
payload = channel.send(payload_kwargs, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 153, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/utils/asynchronous.py", line 169, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/channel/client.py", line 380, in send
ret = yield self._uncrypted_transfer(load, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/channel/client.py", line 354, in _uncrypted_transfer
ret = yield self.transport.send(self._package_load(load), timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/transport/zeromq.py", line 1404, in send
ret = yield self.message_client.send(load, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/transport/zeromq.py", line 923, in send
recv = yield future
^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
salt.exceptions.SaltReqTimeoutError: Message timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 386, in run_job
pub_data = self.pub(
^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 1917, in pub
raise SaltReqTimeoutError(
salt.exceptions.SaltReqTimeoutError: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/salt", line 12, in <module>
sys.exit(salt_main())
^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/scripts.py", line 552, in salt_main
client.run()
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/cli/salt.py", line 55, in run
self._run_batch()
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/cli/salt.py", line 291, in _run_batch
for res, job_retcode in batch.run():
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/cli/batch.py", line 239, in run
part = next(queue)
^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 986, in cmd_iter_no_block
for fn_ret in self.get_iter_returns(
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 1270, in get_iter_returns
jinfo = self.gather_job_info(
^^^^^^^^^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 294, in gather_job_info
pub_data = self.run_job(
^^^^^^^^^^^^^
File "/opt/saltstack/salt/lib/python3.11/site-packages/salt/client/__init__.py", line 408, in run_job
raise SaltClientError(general_exception)
salt.exceptions.SaltClientError: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.
Salt master gets tasks to do salt deployments from salt-api, the queue grows and grows until it eats all the memory.
Type of salt install
Official deb
Major version
3006.x
What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)
ubuntu-22.04
salt --versions-report output
Salt Version:
Salt: 3006.27
Python Version:
Python: 3.11.15 (main, Jun 29 2026, 22:21:49) [GCC 11.2.0]
Dependency Versions:
cffi: 2.0.0
cherrypy: 18.10.0
cryptography: 47.0.0
dateutil: 2.9.0.post0
docker-py: Not Installed
gitdb: 4.0.12
gitpython: 3.1.50
Jinja2: 3.1.6
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.1.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 24.0
pycparser: 3.00
pycrypto: Not Installed
pycryptodome: 3.23.0
pygit2: Not Installed
python-gnupg: 0.5.6
PyYAML: 6.0.3
PyZMQ: 27.1.0
relenv: 0.22.16
smmap: 5.0.2
timelib: 0.3.0
Tornado: 6.5.5
ZMQ: 4.3.5
System Versions:
dist: ubuntu 22.04.5 jammy
locale: utf-8
machine: x86_64
release: 6.8.0-124-generic
system: Linux
version: Ubuntu 22.04.5 jammy
What happened?
We have several salt masters, running on Ubuntu 22.04. They all had version 3006.13 as it was stable.
We tried to upgrade them, first the test one, then all others, but we couldn't find a newer stable version until 3006.27.
We tried 3006.16 and 3006.19 and even on test salt master with just a few salt minion there was a memory leak in salt-api.
And finally 3006.27 seemed good, no memory leaks in salt-api, so we upgraded all our salt masters to 3006.27.
All of salt-masters are doing well except one. The only difference with the other salt masters is that this one has a lot of minions that are not available at the moment.
Memory utilization looks like this:
When it gets to the peak:
Some time before salt-master goes completely crazy it had the errors like this (several times):
Then this one:
Salt master gets tasks to do salt deployments from salt-api, the queue grows and grows until it eats all the memory.
Type of salt install
Official deb
Major version
3006.x
What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)
ubuntu-22.04
salt --versions-report output
Salt Version: Salt: 3006.27 Python Version: Python: 3.11.15 (main, Jun 29 2026, 22:21:49) [GCC 11.2.0] Dependency Versions: cffi: 2.0.0 cherrypy: 18.10.0 cryptography: 47.0.0 dateutil: 2.9.0.post0 docker-py: Not Installed gitdb: 4.0.12 gitpython: 3.1.50 Jinja2: 3.1.6 libgit2: Not Installed looseversion: 1.3.0 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.1.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 24.0 pycparser: 3.00 pycrypto: Not Installed pycryptodome: 3.23.0 pygit2: Not Installed python-gnupg: 0.5.6 PyYAML: 6.0.3 PyZMQ: 27.1.0 relenv: 0.22.16 smmap: 5.0.2 timelib: 0.3.0 Tornado: 6.5.5 ZMQ: 4.3.5 System Versions: dist: ubuntu 22.04.5 jammy locale: utf-8 machine: x86_64 release: 6.8.0-124-generic system: Linux version: Ubuntu 22.04.5 jammy