ValueError errors during CI checks

[leehart]

These recent CI errors are different to the segfault errors (issue #835) and also aren't reproduced locally.

I expect the segfault error is still an issue, but I guess the tests are hitting this issue before it gets to a segfault.

36 passed, 865 errors

There seem to be a lot of:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

I suspect that this might be caused by some unhandled duplicated rows in sourced data files, but that's just an early guess.

Whatever it is, this appears to be very fundamental, affecting many areas, e.g.:

• test_aim_variants:
• test_aim_calls
• test_plot_aim_heatmap
• test_config
• test_releases
• test_client_location
• test_sample_sets_default
• test_sample_sets_release
• test_lookup_release
• test_prep_sample_sets_param
• test_lookup_study
• test_open_cnv_hmm
• test_snp_frq

Example from https://github.com/malariagen/malariagen-data-python/actions/runs/21252008375/job/61159587031

=========================== short test summary info ============================
ERROR tests/anoph/test_aim_data.py::test_aim_variants[gambcolu_vs_arab] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_aim_data.py::test_aim_variants[gamb_vs_colu] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_aim_data.py::test_aim_calls[gambcolu_vs_arab] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_aim_data.py::test_aim_calls[gamb_vs_colu] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_aim_data.py::test_aim_calls_errors - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_aim_data.py::test_plot_aim_heatmap[gambcolu_vs_arab] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_aim_data.py::test_plot_aim_heatmap[gamb_vs_colu] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_config[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_config[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_config[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_config[amin1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_releases[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_releases[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_releases[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_releases[amin1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_client_location[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_client_location[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_client_location[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_client_location[amin1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_sample_sets_default[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_sample_sets_default[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_sample_sets_default[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_sample_sets_default[amin1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_sample_sets_release[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_sample_sets_release[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_sample_sets_release[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_sample_sets_release[amin1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_lookup_release[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_lookup_release[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_lookup_release[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_lookup_release[amin1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_prep_sample_sets_param - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_lookup_study[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_lookup_study[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_lookup_study[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_base.py::test_lookup_study[amin1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_cnv_data.py::test_open_cnv_hmm[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_cnv_data.py::test_open_cnv_hmm[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_sample_query_options[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_sample_query_options[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_min_cohort_size[ag3_sim-0] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_min_cohort_size[ag3_sim-10] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_min_cohort_size[ag3_sim-100] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_min_cohort_size[af1_sim-0] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_min_cohort_size[af1_sim-10] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_min_cohort_size[af1_sim-100] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_min_cohort_size[adir1_sim-0] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_min_cohort_size[adir1_sim-10] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_min_cohort_size[adir1_sim-100] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_variant_query[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_variant_query[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_variant_query[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_variant_query[amin1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_nobs_mode[ag3_sim-called] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_nobs_mode[ag3_sim-fixed] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_nobs_mode[af1_sim-called] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_nobs_mode[af1_sim-fixed] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_nobs_mode[adir1_sim-called] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_nobs_mode[adir1_sim-fixed] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_nobs_mode[amin1_sim-called] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_nobs_mode[amin1_sim-fixed] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_dup_samples[ag3_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_dup_samples[af1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_dup_samples[adir1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
ERROR tests/anoph/test_snp_frq.py::test_allele_frequencies_advanced_with_dup_samples[amin1_sim] - ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
================== 36 passed, 865 errors in 175.83s (0:02:55) ==================

[leehart]

This example extract from the logs might be pointing to a problem with zarr's chunk encoding, during create_dataset in tests/anoph/conftest.py, relating to numcodecs:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

numcodecs/vlen.pyx:101: ValueError
______________ ERROR at setup of test_aim_calls[gambcolu_vs_arab] ______________

fixture_dir = PosixPath('/home/runner/work/malariagen-data-python/malariagen-data-python/tests/anoph/fixture')

    @pytest.fixture(scope="session")
    def ag3_sim_fixture(fixture_dir):
>       return Ag3Simulator(fixture_dir=fixture_dir)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests/anoph/conftest.py:3083: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/anoph/conftest.py:1096: in __init__
    super().__init__(
tests/anoph/conftest.py:1003: in __init__
    self.init_haplotypes()
tests/anoph/conftest.py:1592: in init_haplotypes
    root.create_dataset(name="samples", data=samples, dtype=str)
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/hierarchy.py:1111: in create_dataset
    return self._write_op(self._create_dataset_nosync, name, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/hierarchy.py:952: in _write_op
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/hierarchy.py:1126: in _create_dataset_nosync
    a = array(data, store=self._store, path=path, chunk_store=self._chunk_store, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/creation.py:444: in array
    z[...] = data
    ^^^^^^
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/core.py:1451: in __setitem__
    self.set_basic_selection(pure_selection, value, fields=fields)
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/core.py:1547: in set_basic_selection
    return self._set_basic_selection_nd(selection, value, fields=fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/core.py:1937: in _set_basic_selection_nd
    self._set_selection(indexer, value, fields=fields)
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/core.py:1990: in _set_selection
    self._chunk_setitem(chunk_coords, chunk_selection, chunk_value, fields=fields)
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/core.py:2263: in _chunk_setitem
    self._chunk_setitem_nosync(chunk_coords, chunk_selection, value, fields=fields)
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/core.py:2273: in _chunk_setitem_nosync
    self.chunk_store[ckey] = self._encode_chunk(cdata)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/zarr/core.py:2389: in _encode_chunk
    chunk = f.encode(chunk)
            ^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

[leehart]

Even if our Zarr data had inexplicably changed, I'm not sure how it would cause this. Maybe there are missing data files or badly chunked zarrs, that have been added/removed recently? I don't know whether discontinuation of some S3 services could have disrupted things.

The pyproject.toml currently has zarr, numcodecs and numpy pinned:

[tool.poetry.dependencies]
python = ">=3.10,<3.13"
numpy = "<2.2"
numba = ">=0.60.0"
llvmlite = "*"
scipy = "*"
pandas = "*"
zarr = ">=2.11, <3.0.0"
# zarr<3.0.0 is not compatible with numcodecs 0.16.0
# https://github.com/zarr-developers/zarr-python/issues/2963
numcodecs = "<0.16"

It appears the zarr issue mentioned there was closed 09 Apr 2025: zarr-developers/zarr-python#2963

I'll try to determine what the versions of zarr, numcodecs and numpy were when things were working, versus what they are now.

[leehart]

Perhaps the last CI checks that passed were during PR #844, which got merged in today (containing apparently innocuous updates to the doc switcher.json and the main README.md):

• https://github.com/malariagen/malariagen-data-python/actions/runs/21061927397/job/60574852161

The logs for those passed tests using Python 3.11 and "numpy~=2.0" appear to show the following versions used:

• Downloading numpy-2.4.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (6.6 kB)
• Using cached numcodecs-0.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.9 kB)
• Using cached zarr-2.18.7-py3-none-any.whl.metadata (5.8 kB)

The logs for more recent failing tests (also using Python 3.11 and "numpy~=2.0") appear to show the following versions used:

• Downloading numpy-2.4.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (6.6 kB)
• Using cached numcodecs-0.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.9 kB)
• Using cached zarr-2.18.7-py3-none-any.whl.metadata (5.8 kB)

So this doesn't yet look like it's due to some version difference in those dependencies, since they look identical... 🤔

[leehart]

I notice that the tests appear to be designed to allow numpy versions from the "2.0 series", e.g. from tests.yml:

      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        numpy-version:
          - "numpy==1.26.4"  # current colab version
          - "numpy~=2.0"  # 2.0 series
          # numba (0.60.0) does not yet support numpy 2.1, disable this for now
          # - "numpy~=2.1"  # 2.1 series

However, the tests clearly use numpy 2.4.1, so that isn't working as it seems to be intended, especially since the pyproject.toml also intends to pin the version of numpy "<2.2".

I suspect the code "numpy~=2.0" actually means ">= 2.0, < 3.0" rather than what seems to be expected, i.e. ">= 2.0, < 2.1".

[adilraza99]

Hi @jonbrenas @ahernank,

I see that #847 is currently blocking CI across multiple PRs.
I’ll try reproducing this locally from the current main branch and narrow down the failing tests to help isolate the root cause.

I’ll share any findings or logs here.