`sh_binary` fails on Windows with `#!/usr/bin/env bash` shebang when WSL is installed

[peakschris]

On Windows with MSYS2 and WSL, sh_binary targets fail at runtime when the source script uses #!/usr/bin/env bash.

Environment: Windows, MSYS2 (BAZEL_SH=C:\msys64\usr\bin\bash.exe), WSL installed, Bazel 9.x, rules_shell 0.8.0

Reproduction

git clone https://github.com/bazelbuild/rules_jvm_external.git
cd rules_jvm_external
echo 9.1.0 > .bazelversion
bazel run @duplicate_version_warning//:pin

/bin/bash: D:/.../external/+maven+duplicate_version_warning/pin: No such file or directory

Any sh_binary with a #!/usr/bin/env bash shebang will reproduce this.

Cause

The .exe launcher invokes bash.exe -c "path/to/script args...". MSYS bash execs the script, and the MSYS runtime reads the shebang. #!/usr/bin/env bash causes env to search the MSYS PATH, where /c/WINDOWS/system32/bash (WSL) appears before /usr/bin/bash (MSYS):

$ C:\msys64\usr\bin\bash.exe -c "which -a bash"
/c/WINDOWS/system32/bash    ← WSL (found first)
/usr/bin/bash                ← MSYS (correct, but second)

WSL bash cannot resolve Windows-style paths → "No such file or directory".

Both use_bash_launcher code paths are affected:

• False (default): source is symlinked as-is, preserving the shebang.
• True: wrapper has shebang = "" but calls exec "$(rlocation ...)" "$@", which triggers shebang processing of the underlying script.

Why Bazel 9 only

sh_binary.bzl uses getattr(native, "sh_binary", _sh_binary). In Bazel 8, the native implementation was used (no .exe launcher). In Bazel 9, native rules were removed (#23043), so the Starlark implementation creates the .exe launcher, exposing this interaction.

Suggested fix

The shebang is redundant on Windows — the .exe launcher already resolves bash via bash_bin_path from the sh_toolchain.

1. use_bash_launcher=False on Windows: Instead of symlinking, copy the script and strip #!/usr/bin/env shebangs.
2. use_bash_launcher=True on Windows: Replace exec "$(rlocation ...)" "$@" with . "$(rlocation ...)" or bash "$(rlocation ...)" "$@" to avoid shebang re-processing.

Alternatively, changing bash_launcher.cc to invoke bash script args instead of bash -c "script args" would fix this at the Bazel core level (bash doesn't process shebangs for file arguments).

Workaround

Change shebangs from #!/usr/bin/env bash to #!/bin/bash. MSYS2 resolves /bin/bash directly to C:\msys64\usr\bin\bash.exe via its filesystem mount, bypassing PATH.

[fmeum]

Could you elaborate why this worked on Bazel 8? Windows builds should have always used the .exe launcher, even with Bazel 8. Any change in this area during Starlarkification wasn't intentional and should be reverted.

[peakschris]

I left github copilot & claude sonnet thinking with access to bazel codebase for some time and it generated this which looks plausible:

The regression is caused by a single commit in Bazel's Windows .exe launcher: e37ec56 — "bash launcher: append rather than prepend the bash bin dir" (bazelbuild/bazel#26612).

What changed

In src/tools/launcher/bash_launcher.cc, the launcher modifies PATH before invoking bash scripts. The ordering was flipped:

Bazel 8.x (source):

path_env = bash_bin_dir + L";" + path_env;   // MSYS dir PREPENDED

Bazel 9.x (source):

path_env = path_env + L";" + bash_bin_dir;   // MSYS dir APPENDED

This was done to avoid conflicts between link.exe from MSYS coreutils and link.exe from Visual Studio (comment in commit).

Why it breaks

On Windows machines with WSL installed, C:\Windows\System32\bash.exe (WSL) is on the system PATH before C:\msys64\usr\bin\bash.exe (MSYS):

> where.exe bash.exe
C:\Windows\System32\bash.exe          ← WSL (found first)
C:\msys64\usr\bin\bash.exe            ← MSYS (found second)

The .exe launcher calls MSYS bash directly (via the embedded bash_bin_path), but when MSYS bash processes a script with #!/usr/bin/env bash, it runs /usr/bin/env which searches PATH for bash. With MSYS appended (Bazel 9), env finds WSL's bash.exe first. WSL bash then tries to execute the script using a Windows-style execroot path (D:/udu/b/.../pin) which doesn't exist in the WSL filesystem:

/bin/bash: D:/.../external/+maven+duplicate_version_warning/pin: No such file or directory

With MSYS prepended (Bazel 8), env finds MSYS bash first, which handles Windows paths correctly.

Verified empirically

Version	Shebang	PATH order	Result
Bazel 8.6.0	#!/usr/bin/env bash	MSYS prepended	✅ Works
Bazel 9.1.0	#!/usr/bin/env bash	MSYS appended	❌ Fails
Bazel 9.1.0	#!/bin/bash	(irrelevant — no PATH search)	✅ Works

Workaround

Using #!/bin/bash instead of #!/usr/bin/env bash avoids the issue entirely because MSYS resolves /bin/bash directly via its filesystem mount table (/bin → C:\msys64\usr\bin), bypassing PATH lookup altogether.

[fmeum]

cc @novas0x2a @meisterT

I think that it's best to revert the breaking commit for Bazel 10. I don't know what the impact of doing so for 9 would be.

@peakschris Do you want to send a PR?

[novas0x2a]

The reason I had to make the change is the link.exe issue, which is the inverse of this problem; with the commit reverted, builds with visual studio would fail because it would call link.exe from coreutils rather than link.exe from visual studio.

There's actually kind of a chain of suboptimal decisions in Windows bazel, and maybe it's necessary to unwind them. I don't have the code in front of me currently, but off the top of my head there's at least the PATH manipulation in the launcher, and the kind of hacky way strict_action_env is emulated for Windows.

The Linux invariant for the bash location is the first one in the path; this also implies that BAZEL_SH is a problem, since it only exists to break the invariant.

[peakschris]

I've opened an issue against bazel bazelbuild/bazel#29637 for tracking

[peakschris]

The reason I had to make the change is the link.exe issue, which is the inverse of this problem; with the commit reverted, builds with visual studio would fail because it would call link.exe from coreutils rather than link.exe from visual studio.

I don't see this. For me, bazel autoconfigures the cc toolchain and resolves link.exe to the correct MSVC path and writes that path into the cc toolchain repo. Even if you are seeing this issue, would it be possible to fix it with a point solution in the compiler autoconfiguration code rather than in every launcher?

[novas0x2a]

Note that the autoconfiguration code only runs if you are using a nonhermetic host-installed toolchain; the actual toolchain API does not have a reliable way to do path manipulation. You can only override the path, not prepend to it. We have some hacks for this in our toolchain that only work if you're using rules_foreign_cc, but can't be generalized without the ability to manipulate the path (we use $PATH in the variable, which only expands in an rfcc context, and only works if you have no more than one toolchain attempting to do this).

I think a correct solution is going to involve undoing the windows path manipulation everywhere, then seeing where we are and trying to refix it from base principles; reverting my commit just moves the problem around.