BLD: migrate build backend to meson-python

[neutrinoceros]

This is based off #511
close #516
very much a work in progress.

When attempting a build with

uvx meson setup build
uvx meson compile -C build

I get 2 errors about malloc and free being undefined. Here's the first one:

bottleneck/src/reduce_template.c:819:5: error: call to undeclared library function 'malloc' with type 'void *(unsigned long)'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  819 |     BUFFER_NEW(float64, LENGTH)
      |     ^
bottleneck/src/reduce_template.c:802:31: note: expanded from macro 'BUFFER_NEW'
  802 |         npy_##dtype *buffer = malloc(length * sizeof(npy_##dtype));
      |                               ^
bottleneck/src/reduce_template.c:819:5: note: include the header <stdlib.h> or explicitly provide a declaration for 'malloc'

I don't understand how this happens, #include <stdlib.h> is already present in bottleneck/src/reduce_template.c.

If I run uv build instead, I get

../meson.build:25:3: ERROR: File bottleneck/src/reduce.c does not exist.

so clearly I must be doing something wrong, but I'm not too surprised because this is the first time I attempt to hook python scripts into a meson build.

[neutrinoceros]

I reached a somewhat functional state where at least some jobs are passing.
What's broken:

• docs: many references to setup.py (critical)
• mnt: Makefile and bn_setup.sh should be either removed or updated (tbd)
• warnings-as-errors in tests (very important)
• dynamic config from setuptools (possibly not needed, but I need to understand what I'm giving up)
• dynamic version number (nice to have, but I'm not personally motivated to preserve this)

[neutrinoceros]

looks like this is what's failing windows-arm64 jobs: astral-sh/uv#12906

[neutrinoceros]

got a bit further using the workaround described in the issue I just linked. Now meson correctly detects a native Python interpreter, but then hit

C:\a\_temp\uv-python-dir\cpython-3.11.15-windows-aarch64-none\python311.dll:
      file not recognized: file format not recognized

which I don't understand yet

[rgommers]

I had a quick looks; this looks like a good start to me.

[rgommers]

If you define a generator, this can be a one-liner per file, same as in for example:

https://github.com/scipy/scipy/blob/20c4b5e75727ae1d3dc19a2bb8c65c43f1366592/scipy/sparse/meson.build#L7-L8

[neutrinoceros]

a generator is how I initially tried to write this but I struggled to define a template for the output filename, because it's obtained by subtraction rather than addition.

[rgommers]

Could rename the source files? They're not valid C code, so .c.in or .c.template seems more accurate anyway

[neutrinoceros]

Yes, I was thinking doing just that as a cleanup step after I got CI stable again. Though I prefer extensions like .template.c, as it allows syntax highlighting to sort-of work instead of not at all.

[neutrinoceros]

actually went for it tonight because I had limited bandwidth and this didn't require much. I ended up using your suggestion (.c.template) because I couldn't work out how to implement .template.c using available substitutions .

[neutrinoceros]

I'm hesitant to just drop windows-arm64 as a normal CI target. It works in the context of cibw but I'm struggling to set it up using actions/setup-python and/or astral-sh/setup-uv.

[neutrinoceros]

I got lost as I tried to get everything working at the same time. I've disabled limited-api builds for now because I found it hard to setup correctly with 3.14t also being a target. I'm also disabling failing targets as I go right now, with the goal to reveal the subset that does work already, before I tackle failures one by one again.

[neutrinoceros]

I was able to re-enable 2/3 windows targets. The remaining failing one is x86, where meson apparently struggles to find Python.h

  Run-time dependency python found: NO  (tried pkgconfig and sysconfig)
  
  ..\meson.build:44:3: ERROR: Python dependency not found

[neutrinoceros]

phew, I was able to fix it using the wisdom of this comment mesonbuild/meson-python#610 (comment)

I think this is ready for review now.
A couple notes on the current state:

• dynamic version is still broken (I didn't try to fix it). I have no clue how to implement it with meson, and as I mentioned earlier I'm not really motivated to restore it.
• win-arm64 is still removed from regular CI, but works in the context of cibw, which I think is the important part
• I disabled abi3 builds for now, as it's hard to setup with 3.14t also in the mix. I guess this means this could go in instead of WHL: enable Limited API-compliant (abi3) wheels #511 rather than after it, as I initially planned

TBD !

[neutrinoceros]

abi3 builds are re-enabled !

[rgommers]

dynamic version is still broken (I didn't try to fix it). I have no clue how to implement it with meson, and as I mentioned earlier I'm not really motivated to restore it.

I don't care about it either way (versioneer is really terrible), but if we want it then here are docs: mesonbuild/meson-python#845

win-arm64 is still removed from regular CI, but works in the context of cibw, which I think is the important part

There shouldn't be any hiccups as long as it's a native run on arm64 runners I believe. On x86-64 runners, meson will treat it by default as a cross compile.

I think this is ready for review now.

I'll note that I'm on holiday until the end of next week and mostly camping, so I might not get to this until after that.

[neutrinoceros]

thanks for the headsup. And no worries, this has been on the back of my mind for years, I can definitely wait a week longer 😄
Enjoy your vacation !

[neutrinoceros]

win-arm64 is still removed from regular CI, but works in the context of cibw, which I think is the important part

note to self: I forgot that we didn't run the test suite in cibw, so, even though building works, I cannot confirm that nothing breaks yet.