Adjust to run with newer dependencies

[ximion]

Hi!

I needed a newer Dask version and a newer numpy and xarray (as well as a more recent ffmpeg for better AV1 support) for some custom preprocessing on our data before running Minian, which sent me down an insane rabbit hole to update the Minian code to work with newer dependencies.
So, this is what I ended up with. The generated output for the sample video looks the same as before (but this patch only contains minimal modifications to the Jupyter notebook), and this code works fine with my data.
I also apparently solved #214 somehow (I had this one once before, and after updating Holoviews it never showed up again) and this change also definitely fixes #152, the bane of my existence (dealing with sparse/dense xarrays was a pain as not every OPs are allowed with the same parameters on either, so I do densify arrays explicitly in at some points now for some operations, and sparsify arrays that were previously dense - all of this can result in higher peak memory usage in my testing, so to run Minian on larger datasets (like my 15min movies) you really need to have 16GB of RAM or more memory/swap space.

Some deprecation warnings are still left, but some of those also existed before and could be resolved at a later time (and I wanted to keep this patch small).
This patch also switches Numba's threading layer to OpenMP, because often it switches to TBB which is not threadsafe and causes all kind of errors when combined with dask (and also thousands of warning messages), and in addition to running without random crashes, the OMP backend was also running a lot faster during my testing.

Minian also runs on Python 3.9+ now.

Thanks for considering this patch!
Cheers,
Matthias

[ximion]

So, the Windows and macOS Conda environments don't seem to resolve on your CI, and ideally demand Python 3.10 - since that's the latest release anyway, I updated the dependencies again to work with the most recent Python.

Of course, this combination isn't tested as much as the previous one. On my initial tests I didn't find any issues, but I do want to run this on larger datasets and maybe make some tweaks before claiming it's ready (so please don't merge this PR just yet).

[sneakers-the-rat]

thanks for this, just getting some maintainance to this package after awhile - we're working on this over in #301 and have arrived at most of the changes you have done here, but we'll double check to see if there's anything you caught but we didn't and add you as a co-author in those commits. thanks for this :)

[ximion]

Oh nice, thank you! It's awesome to see that Minian is alive - it's a very nice project and I enjoyed using it. It is also still in use at my lab, although I crammed it into an Apptainer (Singularity) container at some point as part of a bigger processing pipeline, so students and colleagues could use it easier. Maybe with these changes, that won't be needed for much longer!

Thank you for crediting me, let me know if you need anything from me :-)

[daharoni]

Hey @ximion! How have you been! Do you know @sneakers-the-rat? I think you both have a ton in common.

[ximion]

Hey @daharoni! I should have written more, it would have been so nice to meet you at a conference again! (I will be in the US this month, but likely only in NYC, not California...).

I finally finished my PhD at the beginning of the year (some stuff delayed it a bit), and I am now a consultant (working on open source maintenance and resilience) for a short period, until I have figured out the next steps.

The Syntalos DAQ system turned out to be an incredibly useful tool for researchers, and I am even doing some consulting for it. That project kinda blew past its expectations, and I am quite happy about it (its technical foundation is rock-solid, but I am looking for solutions keep it sustainable, which is very hard to scientific open-source software). My (former) lab is also still using Miniscopes, and we are now really leaning into customizing parts a lot. It's awesome to see! (but the method got insanely complex, I guess I am partly to blame for that 😅).

I actually looked at @sneakers-the-rat profile and website, and literally thought "cool, one like me!" followed by "ah, the AutoPiLot person!" followed by "cool, they're at UCLA". Somehow it didn't click for me immediately that they would be in your group, but it probably should have 😆

Never thought we'd meet again in a 4-year-old PR! 😄

[sneakers-the-rat]

nice to meet you :) the world can be small in a lovely way, sometimes. I am flattered that you had heard of the project <3