opam-ci jobs getting stuck

[jmid]

There's something off currently with opam-ci.

This simple PR with a 1-line change to a not-so-popular package ocaml/opam-repository#27489 I opened 4 days ago
yet I'm still waiting to the 2(!) reverse dependencies to get a machine resource and complete it, without having (force) pushed any additional changes:
https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/0c3833ec790dccac0c74b91d4c8004e673420389

• gluon.0.0.8 (active) https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/0c3833ec790dccac0c74b91d4c8004e673420389/variant/compilers,5.3,bytestring.0.0.8,revdeps,gluon.0.0.8
• gluon.0.0.9 (active) https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/0c3833ec790dccac0c74b91d4c8004e673420389/variant/compilers,5.3,bytestring.0.0.8,revdeps,gluon.0.0.9

I've had to wait before, e.g., when being unlucky and publishing on top of a dune or a JaneStreet release, but even then, the waiting time has been ~1 day or so.

The problem is not restricted to the above PR. I see a couple of other PRs seemingly stuck and waiting to complete:

• Add missing ocaml dependency to unisim_archisec opam-repository#27484 - stuck for 5 days
• release ocamlbuild 0.16.1 opam-repository#27455 - also seems stuck for ~5 days

[mtelvers]

There are quite a few dependencies on ocaml/opam-repository#27486

[jmid]

Fair enough.

However I don't get why PRs that have been opened since, e.g. in the past couple of days, have been allocated CI resources and apparently completed their jobs: 🤔

• Package bogue.20250224 opam-repository#27497 opened 18 hours ago
• [new release] ocamlmod (0.1.0) opam-repository#27494 opened yesterday
• Package libbinaryen.116.0.0 opam-repository#27493 opened yesterday
• ...

I can't even access a CI status page ATM however.
https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/eeecc8475eb6340ccfe77ec47f02a628553c5fce and
https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/0def47e7b511023db693d6098b7284568d1f56f0
just hang on my end... 🤷

[mtelvers]

There seems to be a stack overflow in opam-repo-ci when listing ocaml/opam-repository#27486

[mtelvers]

Increasing the stack size allows the web GUI to display the progress. There are ~200K pending jobs for this PR. Typical jobs are ~30K.

[shonfeder]

This is recurring https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/09d7830dd4a7cd4cfc1725bd69ec5b222eae677d on ocaml/opam-repository#29451 (not surprising given the size of the diff).

@mtelvers Do you think we can we just keep increasing the stack size? Or do you suppose we should re-open ocurrent/opam-repo-ci#425 to fix the logic to prevent blowing the stack?