fix(run_slurm): correct slinky HCA list + add vendor IB tuning vars

[Johnsonms]

Summary

Two independent run_slurm.sh misconfigurations were suppressing inter-node NCCL bandwidth on the flapping-airplanes (slinky) B200 cluster. Fixing both brings 4-node/32-GPU collective performance back in line with the May 2026 historical records (to ~1%), all configs 0 wrong.

1. NCCL_IB_HCA pointed at degraded rails

mlx5_5 has flipped to Ethernet and mlx5_9/10/11 trained down to 100G HDR since the May baseline. The hard-coded list caught those instead of the 8 healthy 400G NDR IB rails (mlx5_1,2,3,4,6,7,12,13) → every collective capped at ~192 GB/s. Repo memory project_nccl_working_config.md shows the old list was healthy in May, so this is a fabric regression, not a static bug.

2. Missing vendor NCCL IB tuning vars

Even on healthy rails, ring/NVLS ran ~20–25% low (4n ring 307 vs 384 GB/s). Restored by NCCL_IB_QPS_PER_CONNECTION=2, NCCL_IB_AR_THRESHOLD=0, NCCL_IB_SPLIT_DATA_ON_QPS=0, NCCL_IB_PCI_RELAXED_ORDERING=1, NCCL_IGNORE_CPU_AFFINITY=1. Isolated to the tuning vars alone (independent of -g launch style). SHARP is unaffected.

Results — 4n/32-GPU peak busBW @8GiB (GB/s)

Collective	Config	Buggy	Fixed (both)	Historical (May)
all_reduce	ring	192.8	384.0	389.3 ✓
all_reduce	collnet_sharp	320.0	554.4	550.6 ✓
all_gather	ring	192.7	376.2	376.9 ✓✓
all_gather	nvls	193.0	376.3	377.0 ✓✓
reduce_scatter	ring	192.6	384.1	—

Full analysis, before/after tables, and reproducibility (3 HCA-fix runs + tuned run, all 0 wrong) in baselines/flapping-airplanes/2026-07-11_nccl_collective_scaling_report.md.

Action item (fabric, not this PR)

mlx5_5/9/10/11 were 400G NDR IB in May and are now Ethernet/100G — worth a hardware/cabling/switch-port check on all 4 nodes.

🤖 Generated with Claude Code

[broly-code-security-scanner]

Broly Security Scan

Note

Baseline snapshot is missing for this repo. Broly is running in PR-only fallback mode until the first scheduled baseline completes. This does not block the PR.

Note

Summary

18 actionable finding(s) in this PR

• 🟠 4 high
• 🟡 12 medium
• 🔵 2 low

9 highest-priority actionable rows in the table below (critical/high first, then top medium).

Severity	Scanner	Issue	Location	Dismiss	Verdict
🟠 HIGH	SAST	Command injection via unsanitized user input interpolated into a shell script executed by sbatch.	benchmarks/run_nccl_native.sh:84	d5	🔺 TRUE_POSITIVE · Confidence: HIGH
🟠 HIGH	SAST	Command injection via unsanitized user input interpolated into a shell heredoc that is executed b...	benchmarks/run_nccl_test.sh:90	d6	🔺 TRUE_POSITIVE · Confidence: HIGH
🟠 HIGH	SAST	Command injection via unvalidated command-line arguments passed to shell execution.	tools/docker_build.sh:42	d34	🔺 TRUE_POSITIVE · Confidence: HIGH
🟠 HIGH	SAST	YAML injection via unsanitized user input in Kubernetes Job manifest generation. Multiple user-co...	benchmarks/run_nccl_k8s.sh:136	d4	🔺 TRUE_POSITIVE · Confidence: HIGH
🟡 MEDIUM	DOCKERFILE	Container runs as root with no USER directive specified.	Dockerfile:74	d26	Needs review
🟡 MEDIUM	SAST	Command injection via awk in print_summary(). The variables $current and $baseline are de...	benchmarks/lib/common.sh:211	d37	🔺 TRUE_POSITIVE · Confidence: HIGH
🟡 MEDIUM	SAST	Command injection via the --exclude argument. The args.exclude value is passed to `subprocess...	stragglers/find_stragglers.py:78	d32	🔺 TRUE_POSITIVE · Confidence: HIGH
🟡 MEDIUM	SAST	Command injection via unsanitized command-line arguments (--nodes and --exclude) that are int...	benchmarks/B200/collective-scaling/submit_all.sh:59	d35	🔺 TRUE_POSITIVE · Confidence: HIGH
🟡 MEDIUM	SAST	Command injection via unsanitized command-line arguments interpolated into a generated sbatch scr...	benchmarks/B200/collective-scaling/submit_per4_groups.sh:48	d1	🔺 TRUE_POSITIVE · Confidence: HIGH

Fix Suggestions

🔧 Command injection via unsanitized user input interpolated into a shell script executed by `sbatch`. — benchmarks/run_nccl_native.sh:84

Validate all user-supplied arguments against strict allowlists or escape them with printf '%q' before interpolating into the heredoc, and prefer passing values via environment variables or a temporary file with fixed formatting rather than direct string interpolation.

# Validate all numeric args and sanitize string args before heredoc interpolation:
for v in NODES GPUS_PER_NODE ITERS MIN_BYTES MAX_BYTES; do
[[ "$v" =~ ^[0-9]+$ ]] || { echo "Invalid $v: $v" >&2; exit 1; }
done
PARTITION=$(printf '%q' "$PARTITION")
JOB_NAME=$(printf '%q' "$JOB_NAME")
OUTPUT_FILE=$(printf '%q' "$OUTPUT_FILE")
TIME_LIMIT=$(printf '%q' "$TIME_LIMIT")

🔧 Command injection via unsanitized user input interpolated into a shell heredoc that is executed b... — benchmarks/run_nccl_test.sh:90

Validate all user-supplied parameters against strict allowlists or escape shell metacharacters before interpolating them into the heredoc, and prefer passing values via sbatch --export or environment variables rather than text interpolation.

# Validate numeric parameters
for var in NODES GPUS_PER_NODE MIN_BYTES MAX_BYTES ITERS WARMUP; do
if [[ -n "${!var}" ]] && ! [[ "${!var}" =~ ^[0-9]+$ ]]; then
echo "Error: $var must be a positive integer" >&2; exit 1
fi
done
# Validate string parameters against safe character set
for var in TEST PARTITION CONTAINER_IMAGE OUTPUT_DIR TIME_LIMIT EXTRA_ARGS; do

🔧 Command injection via unvalidated command-line arguments passed to shell execution. — tools/docker_build.sh:42

Replace the string-based BUILD_CMD with a bash array and use "${array[@]}" for execution, or validate each argument against a strict regex pattern before interpolation.

BUILD_CMD=(docker build \
--build-arg "CUDA_VERSION=${CUDA_VERSION}" \
--build-arg "UBUNTU_VERSION=${UBUNTU_VERSION}" \
--build-arg "NCCL_TESTS_VERSION=${NCCL_TESTS_VERSION}" \
${NO_CACHE} \
-t "${FULL_IMAGE}" \
"${REPO_ROOT}")
if [[ $DRY_RUN -eq 1 ]]; then

🔧 YAML injection via unsanitized user input in Kubernetes Job manifest generation. Multiple user-co... — benchmarks/run_nccl_k8s.sh:136

Validate all user-supplied variables against strict allowlist patterns (e.g., alphanumeric, dash, dot for names and namespaces) before interpolating them into the YAML heredoc, or use a YAML generator library that handles escaping properly.

# Add validation before YAML generation (insert after line 101, before derived values):
validate_name() { [[ "$1" =~ ^[a-zA-Z0-9]([-a-zA-Z0-9.]*[a-zA-Z0-9])?$ ]] || { echo "ERROR: Invalid value: $1"; exit 1; }; }
validate_int() { [[ "$1" =~ ^[0-9]+$ ]] || { echo "ERROR: Expected integer: $1"; exit 1; }; }
validate_size() { [[ "$1" =~ ^[0-9]+[KMGT]?$ ]] || { echo "ERROR: Invalid size: $1"; exit 1; }; }
[[ -n "$JOB_NAME" ]] && validate_name "$JOB_NAME"
validate_name "$NAMESPACE"
validate_name "$TEST"
validate_int "$GPUS_PER_NODE"; validate_int "$STEP_FACTOR"; validate_int "$ITERS"; validate_int "$WARMUP"

Dismiss false positives

Each dismissable row has a Dismiss key (d1, d2, …) in the table above. Reply to this comment (or post on the PR) with /broly dismiss d2: your reason to mark a false positive, or /broly undismiss d2 to reverse it. Broly records every dismissal in the section below this comment (updated in place) and suppresses the finding on the next scan.

Key	Finding
d4	🟠 HIGH · YAML injection via unsanitized user input in Kubernetes Job manifest generati... · benchmarks/run_nccl_k8s.sh:136
d5	🟠 HIGH · Command injection via unsanitized user input interpolated into a shell script... · benchmarks/run_nccl_native.sh:84
d6	🟠 HIGH · Command injection via unsanitized user input interpolated into a shell heredo... · benchmarks/run_nccl_test.sh:90
d34	🟠 HIGH · Command injection via unvalidated command-line arguments passed to shell exec... · tools/docker_build.sh:42
d8	🟡 MEDIUM · Path traversal via unsanitized command-line argument allows arbitrary file ov... · .claude/install.sh:21
d26	🟡 MEDIUM · Container runs as root with no USER directive specified. · Dockerfile:74
d35	🟡 MEDIUM · Command injection via unsanitized command-line arguments (--nodes and `--ex... · benchmarks/B200/collective-scaling/submit_all.sh:59
d1	🟡 MEDIUM · Command injection via unsanitized command-line arguments interpolated into a ... · benchmarks/B200/collective-scaling/submit_per4_groups.sh:48
d36	🟡 MEDIUM · Command injection via unsanitized command-line argument · benchmarks/fa4-bench/run.sh:70
d37	🟡 MEDIUM · Command injection via awk in print_summary(). The variables $current an... · benchmarks/lib/common.sh:211
d23	🟡 MEDIUM · YAML injection via unsanitized user input in Kubernetes Job manifest generati... · benchmarks/run_k8s.sh:130
d38	🟡 MEDIUM · Command injection via unsanitized user input passed to eval · benchmarks/run_mpi.sh:166
d39	🟡 MEDIUM · Command injection via unsanitized user input interpolated into a generated sb... · benchmarks/run_slurm.sh:198
d40	🟡 MEDIUM · Command injection via unsanitized user input in git clone and cp commands · benchmarks/setup_and_build.sh:67
d15	🟡 MEDIUM · Command injection via unsanitized command-line arguments written into a gener... · benchmarks/submit_scaleout.sh:73
d32	🟡 MEDIUM · Command injection via the --exclude argument. The args.exclude value is p... · stragglers/find_stragglers.py:78
d33	🔵 LOW · Command injection via unsanitized command-line argument (NNODES) passed to ... · benchmarks/gpu-health/run.sh:45
d20	🔵 LOW · Command injection via unsanitized command-line arguments passed to shell comm... · tools/containerd_import.sh:40

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · GH Actions (zizmor) · Containers · SBOM · Powered by Together AI

[github-advanced-security]

Broly found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.