Add hook HookQueueStateCount + read middleware as hooks and vice versa

[brandur]

Here, add a new hook called HookQueueStateCount which gets invoked to
produce job queue count statistics. We do this by adding a new
maintenance service which like other maintenance services, runs only on
the leader, so we only have one client performing counts at any given
time. Furthermore, in order to not introduce a potential operational
problem without opt-in from River users, the counts only run if a
HookQueueStateCount hook/middleware has been added to the client.

The reason we do all this to to implement a feature requested by one of
users: for otelriver in contrib to be able to emit queue count
metrics, which seems like a pretty reasonable ask for the package to be
able to do, and something that every River user would likely want access
to in their ops charts.

A slight oddity, but which I think is probably okay, is that the new
hook ideally stays a hook, but the existing otelriver middleware is a
middleware. It'd be nice not to have to put otelriver.Middleware into
both a client's Hooks and Middleware configuration, so we modify
client to allow for hooks that middleware and middleware which are
hooks. This lets otelriver.Middleware continue doing what it was
already doing, but also to start producing new counts as a hook.

[brandur]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f2024369e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bgentry]

It should probably be straightforward to update riverui to work with these changes, but we should make sure before merging it since it breaks an API it uses.

[bgentry]

This is a breaking change for the UI. It has a few callers in https://github.com/riverqueue/riverui/blob/fa202780c19504cb8510d0b2c8d92f8ff31d8dc9/handler_api_endpoint.go

I was going to hold off on asking about this, but in light of this break, I'm wondering if we should give any thought to whether or not the UI runs its own queries for this purpose or somehow reuses logic in River so that we don't have multiple redundant queriers. But I think it probably is going to be too hard to try and share that state among backend nodes and UI frontend nodes.

[bgentry]

Also I believe this might break the prior query's efficient usage of the prioritized fetch index; I'm running an agent w/ benchmarks now to get some more details on whether this is a significant perf regression.

[bgentry]

Yeah so I opened #1211 with a perf benchmark showing this is a huge regression. Essentially the old split format was able to effectively leverage the index whereas this variant does not:

GPT 5.4 high explanation:

The new query is inefficient for the current schema because it filters on queue and groups on (queue, state), but the important existing btree on river_job is keyed (state, queue, priority, scheduled_at, id). PostgreSQL btree indexes are most useful when the query constrains the leftmost columns first, and this query no longer does that. Because state is not fixed in the predicate, the planner cannot use the index as a selective access path in the same way, so it tends to fall back to a parallel sequential scan and aggregate after reading a large slice of the table.

The old query shape was more appropriate for the existing index because it split the work into separate branches with constant state predicates, specifically state = 'available' and state = 'running'. That matches the left edge of the (state, queue, ...) index, and once Postgres has a fixed state, it can efficiently narrow by queue within the same index. In practice that gave us index-only or bitmap/index-driven plans instead of scanning the table broadly, which is why the old query performed much better on selective queue lists.

If we want to fix this without adding indexes, the main option is to reshape the new query so it restores a constant-state predicate per branch. Concretely, that means counting each state in its own grouped subquery, then UNION ALLing those results together or joining them back to the requested queue list. That is more verbose than the current grouped query, but it realigns the SQL with the index we already have and gives the planner a much better chance of using it efficiently.

The benchmark is probably useful regardless of how we proceed on this PR but I'll let you decide how to proceed on it ✌️