12 tests fail on Windows with GIX_TEST_IGNORE_ARCHIVES=1

[Sebastian Thiel (Byron)]

Note

Except the title, this has not been updated to reflect that the performance test seems now to fail only rarely and never on CI, and that since #1871 the two gix-pathspec::pathspec search are passing. The automatically checked list of expected failures, and output of test-fixtures-windows CI runs on the main branch, have full current details.

Current behavior 😯

Running tests on Windows in a Git Bash environment (similar to the environment in which they run in Windows on CI, where bash is Git Bash), all tests are able to pass normally. However, this apparently relies on the use of generated archives.

When the tests are run with the environment variable GIX_TEST_IGNORE_ARCHIVES set to 1 rather than unset, 15 tests fail. The failing tests, as quoted from the end of the test run, are:

     Summary [ 903.419s] 2363 tests run: 2348 passed (17 slow, 1 leaky), 15 failed, 9 skipped
        FAIL [   7.416s] gix-glob::glob pattern::matching::compare_baseline_with_ours
        FAIL [   3.324s] gix-pathspec::pathspec parse::baseline
        FAIL [   0.015s] gix-pathspec::pathspec parse::valid::glob_negations_are_always_literal
        FAIL [   0.018s] gix-pathspec::pathspec parse::valid::whitespace_in_pathspec
        FAIL [   1.749s] gix-pathspec::pathspec search::files
        FAIL [   3.855s] gix-pathspec::pathspec search::prefixes_are_always_case_sensitive
        FAIL [ 181.270s] gix-ref-tests::refs packed::iter::performance
        FAIL [   4.803s] gix-submodule::submodule file::baseline::common_values_and_names_by_path
        FAIL [  30.106s] gix-submodule::submodule file::is_active_platform::pathspecs_matter_even_if_they_do_not_match
        FAIL [  35.376s] gix-submodule::submodule file::is_active_platform::submodules_with_active_config_are_considered_active_or_inactive
        FAIL [  22.003s] gix-submodule::submodule file::is_active_platform::submodules_with_active_config_override_pathspecs
        FAIL [  15.882s] gix-submodule::submodule file::is_active_platform::without_any_additional_settings_all_are_inactive_if_they_have_a_url
        FAIL [   9.165s] gix-submodule::submodule file::is_active_platform::without_submodule_in_index
        FAIL [   0.193s] gix::gix revision::spec::from_bytes::regex::find_youngest_matching_commit::regex_matches
        FAIL [   0.086s] gix::gix revision::spec::from_bytes::regex::with_known_revision::contained_string_matches_in_unanchored_regex_and_disambiguates_automatically
error: test run failed

The full output is available in this gist, showing a test run at c2753b8, which has the fixes in #1444.

Before those fixes, one other test had been reported as failing here. See #1358 (comment) and the old gist if interested. The discussion in #1345 is still relevant, though it links to this even older gist.

As noted in comments in #1345, the failure in compare_baseline_with_ours seems particularly interesting, since it involves an unexpected effect of .gitignore pattern matching that is different on Windows.

Expected behavior 🤔

All tests should pass, even when suppressing the use of generated archives by setting GIX_TEST_IGNORE_ARCHIVES=1. Differences between Windows and other platforms should be accounted for when intentional and desirable, or fixed otherwise.

Git behavior

Not fully applicable, since this is about a failure of multiple gitoxide tests when run in a certain way.

However, some discrepancies--intended or unintended--between gitoxide and Git may turn out to be related to some of the failures. So this section may be expanded in the future, or perhaps new issues will be split out from this one.

Steps to reproduce 🕹

I ran the tests on Windows 10.0.19045 (x64) with developer mode enabled so that symlink creation is permitted even without UAC elevation, with git version 2.45.2.windows.1.

I used the current tip of the main branch, which at this time is c2753b8. When I opened this issue originally, that did not exist, but all experiments described here have been performed again, and the reported results have been updated. I used the latest stable Rust toolchain and cargo-nextest, though this does not seem to affect the results.

Local development environments and CI sometimes differ in relevant ways, so I also verified that the tests all pass when GIX_TEST_IGNORE_ARCHIVES is not set. This may be considered an optional step, but is beneficial because it checks that all needed dependencies are installed and working, and that failures really can be attributed to the effect of GIX_TEST_IGNORE_ARCHIVES. To run the tests that way, I run this in Git Bash:

cargo nextest run --all --no-fail-fast

Then I did a full clean:

gix clean -xde

Then, also in Git Bash, I ran this command, which produced the test output and failures described above:

GIX_TEST_IGNORE_ARCHIVES=1 cargo nextest run --all --no-fail-fast

The reason the tests must be run in Git Bash is that there is a separate issue where many test failures occur when they are run from a typical PowerShell environment (#1359).

I also ran the tests, with and without GIX_TEST_IGNORE_ARCHIVES=1, on Ubuntu 22.04 LTS (x64) and macOS 14.5 (M1). As expected, all tests passed on those systems, confirming that the failures are Windows-specific.

---

Originally tracked by Eliah Kagan (@EliahKagan) in issue #1358.

[Sebastian Thiel (Byron)]

Will now just be attempted, and the rest of it (if not everything can be fixed), can be moved back here. Meantime, closing this.