NumPy 2 bringup

[jakirkham]

NumPy is currently working on a NumPy 2.0 release, which is planned to come out later this year. Here are the current (draft) release notes. Also here's the upstream tracking issue ( numpy/numpy#24300 ), and ecosystem compatibility tracker.

Some questions worth discussing:

• How do we handle tightening loose numpy pins in packages ( Hot-fix packages using NumPy to require <2 ? conda-forge-repodata-patches-feedstock#516 )?
• Do we want to keep building for NumPy 1 & 2 at the same time?
• What timeline do we want to use for adding NumPy 2?
• What timeline do we want to use for dropping NumPy 1?
• How should this be coordinated around other major migrations (like Python 3.12 or a new Boost)?
• Numpy now builds for the "oldest-available-C-API" by default, i.e. we could build for our current 1.22 C-API using numpy 1.25; we need to adapt to this in any case, at the latest when our lower bound becomes 1.25 (see Numpy pinning going forward conda-forge-pinning-feedstock#4816)?
• Anything else we should discuss?

Todos:

• Fix run-export for numpy 2.0 numpy-feedstock#313
• more granular tightening for numpy 2.0 conda-forge-repodata-patches-feedstock#728
• Piggyback migrator to drop {{ pin_compatible("numpy") }} conda-forge-bot#2469
• add mini-piggyback migrator for removing pin_compatible("numpy"...) conda-forge-bot#2470
• BUG: smithy ignores keys in migrator when affected by zipping logic conda-smithy#1911
• add _numpy_rc staged-recipes#26188
• Add runtime _numpy_rc dependency for rc builds and publish to main numpy-feedstock#314
• numpy 2.0.0rc migration conda-forge-pinning-feedstock#5790
• also migrate python 3.12 for numpy 2.0; "soft-close" 3.12 migration conda-forge-pinning-feedstock#5851
• Drop {{ pin_compatible("numpy") }} from NumPy docs #2156

cc @conda-forge/core

[h-vetinari]

See also conda-forge/conda-forge-pinning-feedstock#4816

[ocefpaf]

Can we merge these two issues just to make it easier to track them?

[jakirkham]

The issue Axel raises seems like a subpoint of this issue (depending on what we decide). Namely do we want to opt-in to this newer/slimmer ABI and how does that fit into NumPy 2

[ocefpaf]

Sure. IMO Axel issue is a subset of this one. I don't have strong opinions on which one to keep, or if you want to keep both, but I also don't want to get lost on two mega-threads 😬

[jakirkham]

Added Axel's item to the list above

[h-vetinari]

Handling the ABI is the key point here (that and current packages missing a <2). I updated the added item because the summary was not accurate.

Normally I'd say we do a dual migration (keep 1.x; add 2.0), but numpy has around 5000** dependents in conda-forge, so that would be a pretty substantial CI impact, especially if it takes a while to drop 1.x.

**

>mamba repoquery whoneeds numpy -c conda-forge -p linux-64 > tmp
># edit to remove header
>python
>>> q = open("tmp", "r").readlines()
>>> p = {x.strip().split(" ")[0] for x in q} - {""}
>>> len(p)
4898

Obviously not all of them are compiling against numpy, but still...

[jakirkham]

I updated the added item because the summary was not accurate.

Thanks Axel! 🙏

Anyone should feel free to update the issue as needed 🙂

[jakirkham]

Following up on our discussion earlier about improving the visibility of NPY_FEATURE_VERSION, started this NumPy PR ( numpy/numpy#24861 ) to message how the value is set

Also include a note about one approach we might take to ensure that value is embedded in built binaries. Though maybe there are better approaches for that portion

[jakirkham]

There should now be a string baked into binaries built with NumPy to notate what kind of NumPy compatibility they have

numpy/numpy#25948

[jakirkham]

It is worth noting that thanks to Axel and others we now have NumPy 2.0.0rc1 packages: conda-forge/numpy-feedstock#311

Also ecosystem support of NumPy 2 is being tracked in this issue Ralf opened: numpy/numpy#26191

We are now in a good spot to start testing building packages with NumPy 2

[h-vetinari]

I discussed this with @rgommers recently and one important point that he brought up is the situation with pin_compatible, which we'll have to fix as part of any migration effort, probably with a piggyback migrator, since we'll need to rewrite the recipes.

In particular, since numpy isn't separated into a library and run-time component, we don't have a run-export, and so feedstocks use pin_compatible under run:. However this will be doubly incorrect in the new setup - for one as our NPY_FEATURE_VERSION (which forms the lower bound) will be lower than the one at build time, and second because the upper bound should be something like <2.{{ numpy.split(".")[1] | int + 3 }} (for a project that's free of deprecation warnings; anything else might be deprecated in 2.{N + 1} and removed after two releases in 2.{N + 3}).