ci: remove cache steps in nix jobs

[Alizter]

Change

We remove the cache steps in the nix jobs in the CI. We replace them with a cachix cache hosted at https://ocaml-dune.cachix.org.

The flake is configured so that it consults this cache by default. This should speed up setting up local development too.

I've added instructions on how to warm nix cache which has to happen manually between flake updates.

Motivation

The main motivation for this change was to give some relief to the GitHub actions cache which was evicting itself often leading to CI times suffering.

Notes

We are currently caching a subset of all the nix jobs, I haven't for instance added the full oxcaml nix job because we override dune with the current scope which invalidates cache. If we can work out how to do this without, it should be no problem to add.

I have not cached anything for macos, because I don't currently know how to do this.

p.s.

I can give permissions to push to this cache as needed from other maintainers.

cc @shonfeder @anmonteiro @rgrinberg

[rgrinberg]

Manually pushing to this cache does not seem like an attractive option. Can't we push at then of a successful CI run?

[Alizter]

@rgrinberg We only have to do this every time we update the nix flake. The dune that we build here doesn't affect the packages we pull in (that's how we have it set up currently).

We could push every PR but I would have to add some secrets to this repo in order to allow it. Most of the time nothing changes, so they would be noops. When somebody updates a flake, it would invalidate the cache.

Cachix has a 5GB limit at the moment in the free tier, our nix jobs should fit nicely in that for now since there is a lot of sharing. When the limit is reached they will evict LRU.

[shonfeder]

We only have to do this every time we update the nix flake.

Could we have a CI job that is only run when the flake file changes that pushes updates to cache?

[rgrinberg]

There's a github secrets thing to prevent the key from being committed.

I am hoping that this push step would be a no-op when the flake hasn't changed anyway.

[Alizter]

We only have to do this every time we update the nix flake.

Could we have a CI job that is only run when the flake file changes that pushes updates to cache?

I am hoping that this push step would be a no-op when the flake hasn't changed anyway.

Yes, the re-push step is a no-op if the key already exists. So pushing every CI run is cheap. We just need to make sure we aren't needlessly creating new cache entries, like we would be for the ox dev shell at the moment (tho that's not being cached here).

There's a github secrets thing to prevent the key from being committed.

Yes, but we will need to coordinate the permissions here. The reason I went for manual first is practical so we can have more time to faff about and set up the secret without having to delay the cache being deployed.

[rgrinberg]

Why does it matter if we create new cache entries? Is it because of the size limit?

I don't think having any of the maintainers spend time to keep the cache update is a good use of time though. So we might as well just do the extra work upfront. Using github secrets is simple enough.

[Alizter]

Why does it matter if we create new cache entries? Is it because of the size limit?

Yes, I will get an email every time we get close.

I don't think having any of the maintainers spend time to keep the cache update is a good use of time though. So we might as well just do the extra work upfront. Using github secrets is simple enough.

I'm not suggesting this is a permanent solution, but it could end up being one so maybe best to do it now.

[rgrinberg]

What's the advantage of this approach over the existing cache? Is it just a way to get around not being able to pay for a larger github cache?

[Alizter]

@rgrinberg We also get cached dev setups too this way. We will still have to pay for a larger github cache, this doesn't address the size of the opam job caches.

[Alizter]

Replaced by #15327