Skip to content

Fix GPU CI: disable codecov safe.directory write (.gitconfig lock race)#67

Merged
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:fix-gpu-codecov-gitconfig-race
Jun 23, 2026
Merged

Fix GPU CI: disable codecov safe.directory write (.gitconfig lock race)#67
ChrisRackauckas merged 1 commit into
SciML:mainfrom
ChrisRackauckas-Claude:fix-gpu-codecov-gitconfig-race

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor

Problem

The GPU Tests workflow on main is intermittently red. Its two matrix jobs — CUDA Core and CUDA Applications — run concurrently on the same self-hosted GPU runner host, sharing a single /home/chrisrackauckas/.gitconfig.

The codecov/codecov-action@v7 post-test step runs git config --global --add safe.directory .... When both matrix jobs reach that step at overlapping times, they race on git's config-file lock and one job dies with:

error: could not lock config file /home/chrisrackauckas/.gitconfig: File exists
##[error]Process completed with exit code 255.

This happens after ComplementaritySolve tests passed is printed, so the tests themselves are fine — it is purely the codecov action's git-config setup racing. Which of the two jobs loses the race varies run-to-run (it hit CUDA Applications on the latest main run and CUDA Core on a recent PR run), confirming a concurrency race rather than a code/test issue.

fail_ci_if_error: false is already set, but it does not help here: the failure is in the action's safe.directory setup step, which exits 255 before that flag is consulted.

Fix

Set disable_safe_directory: true on the codecov step. This skips the git config --global --add safe.directory write, which is the only operation racing on the shared ~/.gitconfig. The write is unnecessary on these self-hosted runners (the workspace is already owned by the runner user), so disabling it removes the race without masking any failure. The test step (julia-runtest) still fully gates the job, and coverage still uploads.

Notes

  • Workflow-only change; no .jl files touched (no Runic formatting needed).
  • This is a separate concern from the Downgrade (QA) precompile-segfault fix (Fix Downgrade (QA): remove PATH solve from precompile workload (intermittent libpath50 segfault) #66); kept as its own focused PR.
  • I confirmed from the CI logs that the failing step is codecov-action's safe.directory setup and that the tests pass before it; this is a self-hosted-runner-environment race, so it cannot be reproduced on a non-GPU dev box.

Please ignore until reviewed by @ChrisRackauckas

The two GPU matrix jobs (CUDA Core / CUDA Applications) run concurrently
on the same self-hosted runner host sharing a single ~/.gitconfig. The
codecov-action@v7 setup runs `git config --global --add safe.directory`,
and the two jobs race on git's config lock, failing one job with
`error: could not lock config file .../.gitconfig: File exists` (exit 255)
even though the tests themselves pass. This is the dominant cause of the
intermittent CUDA Core / CUDA Applications reds on main.

Setting `disable_safe_directory: true` skips that unnecessary global-git
write (the workspace is already owned by the runner user), removing the
race without masking any failure. The tests still fully gate the job and
coverage still uploads.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas ChrisRackauckas marked this pull request as ready for review June 23, 2026 01:20
@ChrisRackauckas ChrisRackauckas merged commit 97aaa92 into SciML:main Jun 23, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants