Spikes in CPU usage

[Minyan910]

Hi developers,

My institution recently rolled out Arbiter2 on our production login nodes. However, we had an issue where users' CPU usage could spike even after constraints were applied. A screenshot of the utilization plot was attached for your reference.

Oddly we didn't see this behavior on the dev nodes, which should be exact clones of the production ones. On the dev nodes, the CPU usage plot is more like a standard rectangle (attached below). And the spikes only happen for CPU but not for memory.

We are wondering if your team has ever seen this behavior before? Our current guess is that it might be due to how cgroup CPU limiting works through a quota/period scheduling mechanism, e.g. process can burst to 100% CPU utilization at the start of a period, exhaust its quota, and then get throttled for the remainder. Or it might be due to the fact that there are a lot more users and processes on the production nodes, therefore, it takes longer for Arbiter to process those and throttle them.

Any thoughts would be greatly appreciated!

Thanks,
Minyan

[jay-mckay]

Hello Minyan,

We have not seen this issue before. I have a couple of hunches on what might be happening, but would need some information from you to find out. If you don't mind, could you post your configuration file with sensitive information redacted? I am mostly interested in the status section. Additionally, it might help if there are logs associated with a violation where the limits should have been set.

My feeling is the normal status is either not configured or not applied correctly.

Thanks,
Jay

[Minyan910]

Hi Jay,

Thanks for your response. Our config.toml is attached below for your reference. Our login nodes are VMs with 8 cores and 8 GB of RAM each. And all of them are sharing the same sync_group and using a centralized MariaDB on another server.

[general]
# Whether or not to actually email users and set quotas
debug_mode = false
# Arbiter will evaluate users for violations every 30s, with 3 points of information
arbiter_refresh = 30
history_per_refresh = 3
min_uid = 1000  # i.e. non-service accounts

[self]
# Arbiter's groupname (used for sudoers permissions and the --exit-file flag)
groupname = "arbiter"

[badness]
# Change these to match your policy
max_history_kept = 180  # arbiter_refresh / history_per_refresh * 180 seconds of history for plots
cpu_badness_threshold = 0.75
mem_badness_threshold = 0.75
time_to_max_bad = 600  # 10 minutes
time_to_min_bad = 1800  # 30 minutes

[email]
email_domain = 'REDACTED'  # Used for a generic email addr integration in integrations.py
from_email = 'REDACTED'
admin_emails = ['REDACTED']
mail_server = 'REDACTED'
keep_plots = false
plot_location = '../logs/%H/plots'  # %H means hostname, see special values. Make sure you create this directory before arbiter starts

[ldap]
ldap_server = "REDACTED"
auth_ldap_bind_dn = "REDACTED"
# BEGIN ANSIBLE MANAGED - LDAP readonly password
auth_ldap_bind_password = ""
# END ANSIBLE MANAGED - LDAP readonly password
people_ou = "REDACTED"

[database]
log_location = "../logs/%H"    # default
log_rotate_period = 7 # default 7 days
# BEGIN ANSIBLE MANAGED - Database URL
statusdb_url = ""
# END ANSIBLE MANAGED - Database URL
statusdb_sync_group = "login"  # instances with the same attribute will use the same db

[processes]
memsw = true
pss = true  # Requires CAP_SYS_PTRACE capability! See the install guide for info.
# whitelist = ['python3']
# whitelist_file = '../etc/whitelist.txt'

[status]
# A status group is selected by picking the first listed here that the user's uids or gids match.
order = ['invincible', 'admin', 'normal']
# If a user doesn't match into the status groups above, they land in the fallback status group
fallback_status = 'normal'
div_cpu_quotas_by_threads_per_core = true

  [status.normal]
  cpu_quota = 400  # 4 virtual cores
  mem_quota = 4  # Gigabyte

  [status.admin]
  # uids = [1000]
  # gids = [gid_of_admin, ]
  cpu_quota = 800  # 8 virtual cores
  mem_quota = 6   # Gigabyte
  whitelist = ['bash', 'python', 'python3']

  [status.invincible]
  # If arbiter2.service has Slice=user-$ARBUID.slice (for turning on cgroups acct),
  # arbiter will be automatically tracked. You can tell arbiter to effectively
  # ignore itself by making itself invincible here.
  # uids = [$ARBUID]
  uids = [REDACTED]
  cpu_quota = 1e5
  mem_quota = 1e5   # Gigabyte

  [status.penalty]
  relative_quotas = true  # Note the relative_quotas, makes the *_quota less than 1
  order = ['penalty1', 'penalty2', 'penalty3']
  occur_timeout = 10800  # 3 hours

    [status.penalty.penalty1]
    timeout = 1800  # 30 mins
    cpu_quota = 0.8
    mem_quota = 0.8
    expression = 'new'

    [status.penalty.penalty2]
    timeout = 3600  # 1 hour 
    cpu_quota = 0.5 
    mem_quota = 0.5
    expression = 'repeated'

    [status.penalty.penalty3]
    timeout = 7200  # 2 hours 
    cpu_quota = 0.3
    mem_quota = 0.3
    expression = 'severe'

[high_usage_watcher]
high_usage_watcher = true
cpu_usage_threshold = 0.9
mem_usage_threshold = 0.5
timeout = 600  # warn at most every 10 minutes
div_cpu_thresholds_by_threads_per_core = true
user_count = 8

Attaching the log for one penalty for your reference.

Feb 24 11:03:18 hostname python3.12[2506141]: 26-02-24T11:03:18 - arbiter.triggers - DEBUG: 2008301 (REDACTED) has spent: 1789 seconds in penalty of a required 1800
Feb 24 11:03:49 hostname python3.12[2506141]: 26-02-24T11:03:49 - arbiter.triggers - DEBUG: 2008301 (REDACTED) has status: Status(penalty1/normal, occur=1, ts=2026-02-24T10:33:29, occur_ts=2026-02-24T10:33:29, authority=hostname)
Feb 24 11:03:49 hostname python3.12[2506141]: 26-02-24T11:03:49 - arbiter.triggers - INFO: Decreasing the penalty status of 2008301 (REDACTED)
Feb 24 11:16:39 hostname python3.12[2506141]: 26-02-24T11:16:39 - arbiter.triggers - INFO: Increasing the penalty status of 2008301 (REDACTED)
Feb 24 11:16:58 hostname python3.12[2506141]: 26-02-24T11:16:58 - arbiter.actions - DEBUG: Sent penalty2 email to 2008301 (REDACTED), where their top processes were: ['pigz (1)', 'gzip (1)', 'conda (1)', 'find (1)', 'other processes§ (1)', 'bash (2-4)', 'sshd (2-3)', 'srun (2)', 'acompile (1)', 'sshd* (2-3)', 'salloc (1)', 'sort (1)']
Feb 24 11:17:30 hostname python3.12[2506141]: 26-02-24T11:17:30 - arbiter.triggers - DEBUG: 2008301 (REDACTED) has status: Status(penalty2/normal, occur=2, ts=2026-02-24T11:16:39, occur_ts=2026-02-24T11:16:39, authority=hostname)
Feb 24 11:17:30 hostname python3.12[2506141]: 26-02-24T11:17:30 - arbiter.triggers - DEBUG: 2008301 (REDACTED) has spent: 51 seconds in penalty of a required 3600
Feb 24 11:18:04 hostname python3.12[2506141]: 26-02-24T11:18:04 - arbiter.triggers - DEBUG: 2008301 (REDACTED) has status: Status(penalty2/normal, occur=2, ts=2026-02-24T11:16:39, occur_ts=2026-02-24T11:16:39, authority=hostname)

Please let me know if you need anything else. Thanks!

Best,
Minyan

[jay-mckay]

Minyan,

Looking at the logs, it seems we are missing the actual setting of the limits. An example from our systems:

26-02-18T21:15:31 - arbiter.triggers - DEBUG: uid (username) has spent: 1769 seconds in penalty of a required 1800
26-02-18T21:16:22 - arbiter.triggers - DEBUG: uid (username) has status: Status(penalty1/normal, occur=1, ts=2026-02-18T20:46:02, occur_ts=2026-02-18T20:46:02, authority=hostname)
26-02-18T21:16:22 - arbiter.triggers - INFO: Decreasing the penalty status of uid (username)
26-02-18T21:16:22 - arbiter.actions - DEBUG: Applying limits for uid
26-02-18T21:16:22 - arbiter.actions - DEBUG: Successfully set the memory quota of user-{uid}.slice to 8.5%
26-02-18T21:16:22 - arbiter.actions - DEBUG: Successfully set the CPU quota of uid (username) to 400.0%

The last 3 lines seems to be missing in your logs, which leads me to believe that the limits are not being correctly set.

In particular, say we are downgrading (or upgrading) a penalty, arbiter calls actions.set_quotas(user_obj):

arbiter2/arbiter/actions.py

Lines 414 to 458 in ac8902a

	def set_quotas(user_obj):
	"""
	Applies the quotas of the new status group to user. If the quotas are
	already set, then nothing happens.
	user_obj: user.User()
	A user to update the status of.
	"""
	uid = user_obj.uid
	memsw = cfg.processes.memsw
	status = user_obj.status
	cgroup = user_obj.cgroup
	cpu_quota, mem_quota = status.quotas()
	default_cpu_quota, default_mem_quota = status.quotas(default=True)
	try:
	# mostly_eq() because we can't set a lower memory limit (e.g. putting
	# someone in penalty) than a cgroup already has, and our subsequent
	# attempts to set this limit lower will fail quite often. It's enough
	# to call it done if it's mostly equal
	eq_cpu_quota = mostly_eq(cpu_quota, cgroup.cpu_quota())
	eq_mem_quota = mostly_eq(mem_quota, sysinfo.bytes_to_pct(cgroup.mem_quota(memsw)))
	except FileNotFoundError:
	return
	except OSError as err:
	logger.debug("Failed to get quotas because %s", err)
	return
	if eq_cpu_quota and eq_mem_quota:
	return
	logger.debug("Applying limits for %s", uid)
	if not eq_cpu_quota:
	cpu_thread = threading.Thread(
	target=limit_user,
	args=(user_obj, "cpu", cpu_quota, default_cpu_quota)
	)
	cpu_thread.start()
	if not eq_mem_quota:
	mem_thread = threading.Thread(
	target=limit_user,
	args=(user_obj, "mem", mem_quota, default_mem_quota, memsw)
	)
	mem_thread.start()

We do not see a message like Applying limits for uid (note, this message would not be included in your output if you only grepped for a username instead of the uid), which makes me think that we encounter the error before, probably on line 434 or 435 leading to a FileNotFoundError.

If this is indeed the error, that means that Arbiter cannot set limits because the cgroup controllers cannot be found.

Can you check to see if the cgroup limits exist? For example, for the uid you included in the logs:
ls /sys/fs/cgroup/cpuacct/user.slice/user-2008301.slice/cpuacct.cfs_quota_us
Note, the user must be logged in to see these limits.

Please let me know if you have any questions or need clarifications :)

Thanks,
Jay