ci: run the suite on windows-latest - #272
Merged
Merged
Conversation
jq's native Windows build opens stdout in text mode, so every `\n` it writes arrives as `\r\n`. Both shipped recipes read that output as paths, and neither reader drops the CR: `read -r` keeps it by definition, and `tr '\n' '\0'` converts only the newline. The carriage return then rides on the end of every path. The judge recipe reports `0/N verdicts present` because `[ -s "$response_path" ]` matches nothing, and the parallel-dispatch recipe hands every task a corrupted `eval_root`, `dispatch_prompt_path`, and `outputs_dir`. Git Bash with `jq` is the setup the tooling requirement documents, so this is the supported Windows path rather than an exotic one. Pipe each jq call through `tr -d '\r'`: a no-op wherever jq already writes LF, and `tr` is a declared requirement alongside `jq` itself. The regression test defines a CRLF-emitting `jq` as a shell function in front of the recipe rather than shimming `PATH`, so the failure mode is covered on every host instead of only where a Windows jq is installed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Releases attach a Windows binary and the README documents a PowerShell installer, but both jobs ran on ubuntu-latest, so every Windows fix so far was found by hand rather than by the pipeline. Matrix the test job across ubuntu-latest and windows-latest with `fail-fast: false`, since a Linux failure cancelling the Windows leg would lose the result exactly when it is worth having. Clippy earns its place on both: the `#[cfg(windows)]` arms in `core::fs` and `command_check` are lint-invisible on Ubuntu. `EVAL_MAGIC_REQUIRE_POSIX_TOOLS` applies to both runners, so the Windows host is provisioned for the gated capabilities rather than exempted from them: `jq`, which Git for Windows does not bundle, and Developer Mode for symlink creation. Long paths need nothing — task repositories carry their own `core.longpaths`, and `LongPathsEnabled` stays unset deliberately so those tests keep proving what a default box does. `EVAL_MAGIC_SH` stays unset for the same reason: discovering the shell from the Git install root is what a Windows user hits. The guard test pins all of it together, because a matrix entry whose enforcement variable went missing would report green while covering six fewer tests than it appears to. Also corrects both contributor docs, which still described two gated capabilities before long-path staging became the third. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The long-path tests padded from `std::env::temp_dir()`, which can hand back an 8.3 short name: a GitHub runner's `%TEMP%` lives under `RUNNER~1`, while git expands that to `runneradmin` before it measures. Three characters no length computed here could see. That was enough to move the deepest test from one side of a ceiling to the other. It passed locally with `.git/config` at 257 and failed on the runner with the same target at 260, which reads as a Windows-only product failure and is nothing of the sort. Canonicalise the base first, so the padding measures the spelling git will report. Then step the root back to 244 and assert the window it has to sit in, because git's long-path awareness turns out to be per-operation: creating `.git/objects/pack` survives well past the budget, `git init` writing `.git/config` stops exactly at it, and the `git config --local` that follows gives up two characters earlier still. The old literal sat one character inside the tightest of those, with nothing recording that it did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #249.
Both CI jobs ran on
ubuntu-latestwhile releases attach a Windows binary, so every Windows fix in#250, #251, #267, #268, #269 and #271 was found by hand rather than by the pipeline. This adds the
job that would have caught them.
The job caught one before it ever reached a runner
Installing
jqlocally to exercise the threejudge_recipe_*tests — which had never executed onany host, here or on the #267 branch — turned them red rather than green, and not for a test-shaped
reason. jq's native Windows build opens stdout in text mode, so every
\nit writes arrives as\r\n. Both shipped recipes read that output as paths, and neither reader drops the carriagereturn:
read -rkeeps it by definition, andtr '\n' '\0'converts only the newline.So the CR rides on the end of every path. The judge recipe reports
0/2 verdicts presentwhere itshould say
1/2, and the parallel-dispatch recipe hands every task a corruptedeval_root,dispatch_prompt_pathandoutputs_dir. This is on the setupPOSIX_TOOLING_REQUIREMENTdocuments — Git Bash with
jq— and it is in 0.9.0.The fix is
| tr -d '\r'after each of the four jq calls: a no-op wherever jq already writes LF, andtris already declared alongsidejq. Its regression test defines a CRLF-emittingjqas a shellfunction in front of the recipe rather than shimming
PATH, so the Ubuntu runner covers the Windowsfailure mode too, and no executable bit has to be set for it to work.
The job itself
The matrix runs
fail-fast: false, since a Linux failure cancelling the Windows leg would lose theresult exactly when it is worth having. Clippy earns its place on both hosts — the
#[cfg(windows)]arms in
core::fsandcommand_checkare lint-invisible on Ubuntu.EVAL_MAGIC_REQUIRE_POSIX_TOOLSapplies to both runners, so Windows is provisioned for the gatedcapabilities rather than exempted from them:
choco install jq, which Git for Windows does notbundle, and Developer Mode for symlink creation. Two deliberate non-actions:
LongPathsEnabledstays unset, because those tests are only meaningful while they pass on a default box, and
EVAL_MAGIC_SHstays unset so the run exercises the Git install-root discovery a Windows useractually hits.
Verification
cargo fmt --check,cargo test --all-targetsandcargo clippy --all-targets -- -D warningsallpass on Windows 11 (1138 tests, 0 failures). Separately, under
EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1soa skip becomes a failure: the three
judge_recipe_*tests pass with realjq, and the threetask_repository_*long-path tests pass with no provisioning, confirming #271's verbatim-pathreasoning on a real Windows host rather than inferring it.
Two things only the first run can settle: whether the runner's token permits symlink creation, and
whether
shresolves through the Git install root.Needs a repo-settings change to merge
The ruleset requires a status check named
Test suite. The matrix renames it toTest suite (ubuntu-latest)/Test suite (windows-latest), so the contexts have to be updated orthis PR — and every one after it — blocks on a check that no longer reports.
🤖 Generated with Claude Code