25.11.7 - #83
Open
itkovian wants to merge 7324 commits into
Open
Conversation
Assign directly via FORWARD_INITIALIZER macro instead of calling forward_init(). Cherry-picked: 47ae128 Ticket: 24122
Track forward threads and wait for all of the threads to finish. Ticket: 24122
Shutdown conmgr before any cleanup to avoid race conditions that could require plugins. Cherry-picked: d08a37b Ticket: 24122
Call conmgr_request_shutdown() to start the shutdown before calling http_fini() and http_switch_fini() which should only cleanup after all connections are cleaned up to avoid a NULL dereference. Changelog: slurmctld - Avoid possible race condition during shutdown that could cause a crash in the HTTP handling logic. Ticket: 24122
Changelog: slurmctld - Avoid race condition during shutdown that could cause a crash due to tree forwarding. Cherry-picked: fe90add Ticket: 24122
Changelog: slurmd - Avoid race condition during shutdown that could cause a crash due to tree forwarding. Cherry-picked: 9c74c7b Ticket: 24122
Changelog: slurmstepd - Avoid race condition during shutdown that could cause a crash due to tree forwarding. Cherry-picked: 2a563a3 Ticket: 24122
Changelog: srun - Avoid race condition during shutdown that could cause a crash due to tree forwarding. Cherry-picked: 666a620 Ticket: 24122
Changelog: slurmdbd - Avoid race condition during shutdown that could cause a crash due to tree forwarding. Cherry-picked: adf10ef Ticket: 24122
Wait for slurmd process to be moved to the new cgroup before writing on cgroup.subtree_control. This caused EBUSY errors on some slow systems. Changelog: Fix race condition with cgroups not migrating slurmd process quickly, which caused EBUSY errors on startup. Ticket: 24722 Cherry-picked: e4096b7
Commit 0f9ff8c tried to fix a race with systemd where it logged EBUSY errors when slurmd was reconfigured, as it was living in the top cgroup of the unit and this caused a conflict. After that commit slurmd was always moved into a subcgroup but on reconfigure this caused it to try to move in a nested hierarchy. This condition caused reconfigure to fail. Changelog: Fix slurmd reconfigure failure with cgroup/v2. Ticket: 24722 Cherry-picked: 58c2657
Call auth_g_get_reconfig_fd() before env_array_copy() so that the SACK_RECONFIG_FD environment variable set by auth_p_get_reconfig_fd() is included in the child's environment during reconfigure. This fixes a regression added by commit eb1b69e in 25.05.0. Previously, child_env was copied from environ before auth_g_get_reconfig_fd() was called, meaning SACK_RECONFIG_FD was never passed to the new slurmctld process. This caused the child to always take the fresh socket creation path in init_sack_conmgr(), leaving the inherited old sack socket FD leaked and un-served. Clients that connected to sack.socket during the reconfigure window would block indefinitely in safe_read() waiting for a response, causing slurmctld to appear fully locked up until sack.socket was manually deleted. This matches the ordering already used in slurmd's _try_to_reconfig(). Changelog: Fix a regression added in 25.05.0 concerning how the slurmctld inherits /run/slurmctld/sack.socket when using AuthType=auth/slurm to prevent clients that connected during a reconfigure from hanging indefinitely. Ticket: 24863 Cherry-picked: e1c31d8
See merge request SchedMD/dev/slurm!3262
See merge request SchedMD/dev/slurm!3261
See merge request SchedMD/dev/slurm!3258
Changelog: slurmctld - Wait for forwarding threads to complete before shutdown to avoid crashing due to NULL dereferences or using unloaded plugins. Cherry-picked: 7a965c6 Ticket: 24122
Changelog: Avoid failure for spank options that do not require arguments. Ticket: 22914 Cherry-picked: addb13b
See merge request SchedMD/dev/slurm!3284
See merge request SchedMD/dev/slurm!3268
See merge request SchedMD/dev/slurm!3286
See merge request SchedMD/dev/slurm!3280
Changelog: namespace/linux - fix memory leak in slurmstepd when namespace_p_recv_stepd() fails. Cherry-picked: d7e9a3d
sem_destroy() on an uninitialized semaphore may not result in a clean teardown. Changelog: namespace/linux - Fix potential crash on failure if mmap() or sem_init() fails during namespace construction. Cherry-picked: 44fb6ec
This removes some extraneous cleanup code from before calling execvp(). Cherry-picked: 9839bb3
Use a shared memory flag to propagate the child's setup result back to the parent. This ensures that the a failure is caught and avoids potential races between the exit of the process and what would otherwise be a waitpid() call. Changelog: namespace/linux - fix failure to detect namespace setup problems when launching a job. Cherry-picked: 7cd3d83
See merge request SchedMD/dev/slurm!3290
_fill_jobs_statistics() dereferences j->part_ptr->name without a NULL check. When a partition is deleted, finished jobs within MinJobAge have part_ptr set to NULL, crashing slurmctld on the next metrics scrape. Fall back to j->partition when part_ptr is NULL. Changelog: Fix slurmctld crash when querying the metrics endpoint after a partition is deleted with finished jobs still present. Ticket: 24948 Cherry-picked: b6ac156
When reconfiguring with auth/slurm and AuthInfo=disable_sack enabled, an xassert failure could occur in auth_p_get_reconfig_fd() due to the check asserting fd >= 0 before attempting to apply F_SETFD via fcntl(). On production systems built without DEBUG, this scenario would instead cause fcntl() to return EBADF and log an error. This change avoids both the assertion fail and the erroneous fcntl() call. Changelog: Fix potential crash when reconfiguring with auth/slurm. Cherry-picked: 4e70e15 Ticket: 24222
See merge request SchedMD/dev/slurm!3812
If the slurmctld and slurmdbd's view of the QOS are out of sync and a job with a qos list has all of their qos removed, a stale qos_ptr could be left in the job. This prevents that by NULLing the stale qos_ptr. Ticket: 25273 Cherry-picked: 59581d1
Do not allow job->qos_ptr and job_ptr->qos_id to be mismatched. When they are mismatched, then the _remove_qos() function can leave stale qos_ptrs in job record, leading to a crash. Ticket: 25273 Cherry-picked: a4dcdfb
The saved qos_ptr may no longer be available to the job or may even be a stale pointer if the assoc_mgr updated the qos list. This is intentionally using pointer comparisons in order not to dereference the potentially invalid saved pointer. Ticket: 25273 Cherry-picked: c621780
When the assoc_mgr_*_lists are wiped out and replaced, make sure to remove all the defunct pointers of the assoc, qos, tres, etc. Ticket: 25273 Cherry-picked: e58999c
If the slurmdbd connection ends part way through updating the assoc_mgr's variables, then originally the _assoc_cache_mgr() thread would just keep looping without updating the jobs, partitions, or burst_buffer even if their references to those assoc_mgr list elements where now invalid. Now if the assoc_mgr partially updates is variables the slurmctld will always update the jobs, partitions, and burst_buffers references. Ticket: 25273 Cherry-picked: eeec0fe
resv_ptr->qos_list holds pointers managed by assoc_mgr_qos_list. After _refresh_assoc_mgr_qos_list() swaps the qos list every cached pointer in every reservation's qos_list is dangling, but _assoc_mgr_post_update_all() does not update the invalid pointers. Add resv_cache_update_qos_list() that rebuilds resv_ptr->qos_list from resv_ptr->qos string, and call it from _assoc_mgr_post_update_all(). Ticket: 25273 Cherry-picked: fd64af4
When the assoc_mgr's SLURMDB_REMOVE_QOS handling triggers init_setup.remove_qos_notify which is _remove_qos() in the slurmctld, the qos record will be freed once FREE_NULL_LIST() is called on remove_list after _remove_qos() returns. The function _remove_qos was only clearing the soon to be defunct pointers from partitions and jobs, leaving reservations still referencing the defunct qos pointers in its qos_list. Add _foreach_resv_remove_qos() that walks resv_list, drops the matching pointer from resv_ptr->qos_list, and strips the qos name from resv_ptr->qos so a later rebuild from the string stays in sync. Ticket: 25273 Cherry-picked: 92e7698
A partition's allow_accts_list and deny_accts_list cache slurmdb_assoc_rec_t pointers owned by assoc_mgr_assoc_list. When an association was removed, assoc_mgr_update_assocs() freed the record (via list_delete_all() on remove_list) before init_setup.update_resvs() ran, leaving the partition lists referencing freed memory until part_update_assoc_lists() rebuilt them. Change the list_delete_all() back to list_for_each() like before commit c99d4bf but deferring the FREE_NULL_LIST() past rebuild time. Ticket: 25273 Cherry-picked: eb40a0f
When the assoc_mgr_user_list is replaced, it frees every old slurmdb_user_rec_t, but slurmdb_assoc_rec_t->user_rec is a soft reference into the user list. This updates the associations with the new user record before freeing the assoc_mgr_user_list. Ticket: 25273 Cherry-picked: 0df4e43
Changelog: slurmctld - Prevent segfaulting from dereferencing stale QOS, association, or user pointers after updating the assoc_mgr. Ticket: 25273 Cherry-picked: ee3c4d2
See merge request SchedMD/dev/slurm!3864
PMIXP_KVP_CREATE() loads the regex into the pmix_info_t via PMIX_INFO_LOAD() which copies the data into the value. The original buffer was then dropped by setting the pointer to NULL without freeing it. Changelog: Fix memory leak in slurmstepd's PMIx startup. Ticket: 25376 Cherry-picked: 235d39e
Starting with PMIx 3.2.0 PMIx_generate_regex()/PMIx_generate_ppn() may return a compressed blob with embedded NULL bytes once the generated map exceeds the library's compression threshold. Such a blob must be passed as a PMIX_REGEX by Slurm's PMIx plugin, but we were passing a PMIX_STRING. This will cause jobs with a large number of tasks to fail. Changelog: Fix pmix jobs failing on startup when requesting a large number of tasks. Ticket: 25376 Cherry-picked: b8b2e33
See merge request SchedMD/dev/slurm!3878
Ticket: 24784 Cherry-picked: edcb58f
When testing a pending job against multiple partitions, set the active partition to the candidate before helpers read it from the job record. Previously the delayed-start helpers read job_ptr->part_ptr, which for multi-partition jobs is the primary partition rather than the candidate being evaluated. As a result the delayed-start estimate and the queue filter used totals and a partition pointer that did not match the partition select_g_job_test() actually succeeded on. Restore the original pointer after iteration so the existing-job --test-only path does not leave the job record changed. Changelog: Make --test-only evaluate multi-partition jobs against the candidate partition being tested. Ticket: 24784 Cherry-picked: 717d736
Skip candidate partitions with no node bitmap, nodes, or CPUs before allocating temporary will-run bitmaps. This closes a SIGFPE in slurmctld where REQUEST_JOB_WILL_RUN against multiple partitions (e.g. srun --test-only -p empty,valid) reached _delayed_job_start_time() with an empty partition and divided by part_cpu_cnt == 0. Returning success from the callback keeps multi-partition requests moving to later candidates. Requests where no partition can provide resources still return an error. Changelog: Prevent slurmctld from crashing with a divide-by-zero when --test-only uses an empty partition before a usable one. Ticket: 24784 Cherry-picked: f391e8f
Return before normalizing delayed-start estimates when the active partition has no nodes or CPUs. The current will-run path rejects those partitions before reaching this helper. Keep the guard here as a local invariant so future callers, or future changes to the caller preconditions, cannot reintroduce the same divide-by-zero risk in the delayed-start estimate. Ticket: 24784 Cherry-picked: 06f3f2f
See merge request SchedMD/dev/slurm!3903
The job's hres_select structure was not re-created in job_array_split(). Once one of the jobs is purged, the remaining array jobs will have a dangling pointer to the hres_select structure. This will result in a double-free when another job is purged. Changelog: Fix slurmctld crashing when using job arrays that request mode 3 HRES. Issue: 50445 Cherry-picked: 5437cdc
See merge request SchedMD/dev/slurm!3924
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.