Enforce cluster-level WLM request throttling (shared_limit)

[dzane17]

Adds cluster-level request throttling to Workload Management via a new shared_limit on a workload group. Builds on the node-level (node_limit) tier: a request is admitted locally first, and only on local exhaustion does it consult a shared, cluster-wide pool.

Total ceiling = node_limit × N + shared_limit.

How it works

• Each throttle bucket has one authoritative in-flight counter on an owner node chosen by a consistent-hash ring over data nodes.
• The coordinator acquires a shared permit from the owner (async round-trip; no hop when it owns the bucket) before running the search, and releases on completion.
• TTL leases reclaim permits from crashed coordinators or lost releases.
• Fails open (admits) on unreachable owner, timeout, or empty ring, so a network blip can't cause a cluster-wide rejection storm.

Tests

• Unit: shared pool (concurrency, TTL, sweep/release races), consistent-hash ring (determinism, rebalance, version filtering), and the throttle service.
• Integration: WlmClusterThrottlingIT verifies the cluster-wide ceiling holds across multiple coordinators.
• Also verified on a live 3-node cluster.

Latency benchmark

Measured on a local 3-node cluster (match_all searches round-robined across all 3 coordinators, tagged via the workloadGroupId header, attribute=group). Limits set far above peak concurrency so nothing is rejected — this isolates the admission-path overhead on admitted requests (0 throttled / 0 rejected across 16k completions). Median of 3 runs; latencies in ms.

config	profile	mean	p50	p90	p99
none	steady	4.07	3.83	5.18	8.77
node-only	steady	4.05	3.87	5.14	8.49
shared-only	steady	4.70	4.41	6.23	9.00
none	burst	5.23	5.02	7.14	10.01
node-only	burst	5.15	4.98	7.02	9.74
shared-only	burst	6.34	5.81	8.89	12.71

Node-only throttling is effectively free (within noise); shared-only adds ~0.6ms mean / ~2.7ms p99-under-burst for the cross-node acquire round-trip.

Caveat: the 3 nodes run on one machine, so the shared-tier acquire RPC crosses a real process boundary but travels over loopback — the measured overhead includes serialization, transport-thread scheduling, and queueing, but not true inter-node network RTT. On a real multi-host cluster the shared-only overhead would be higher (roughly + one physical RTT for the fraction of requests landing on a non-owner coordinator). Read the shared-only numbers as a floor. Node-only is unaffected since that path never leaves the coordinator.

[dzane17]

Change to latest version when merging to main

[LilyCaroline17]

One other thing, for the SharedThrottleTracker, we have leasesByBucket which an outer map (per bucket) of inner maps (leases per bucket) so the inner map is bounded by shared_limit per bucket, but the outer map isn't bounded by anything so it could potentially become really big. For example, if the throttling is done by username, and there's a large number of simultaneous requests from users, the owner node will have a large number of entries in that outer map all on that one node. Would it make sense to create a bound per buckets per owner, tracking this information, or figuring out a way to spread that load?

[dzane17]

One other thing, for the SharedThrottleTracker, we have leasesByBucket which an outer map (per bucket) of inner maps (leases per bucket) so the inner map is bounded by shared_limit per bucket, but the outer map isn't bounded by anything so it could potentially become really big. For example, if the throttling is done by username, and there's a large number of simultaneous requests from users, the owner node will have a large number of entries in that outer map all on that one node. Would it make sense to create a bound per buckets per owner, tracking this information, or figuring out a way to spread that load?

You're right that SharedThrottleTracker doesn't cap the number of buckets (keys) in the outer map — a bucket can exist for every (WLM group + username/role) combination that has an ACTIVE search.

In the worst case, a domain could have up to 100 WLM groups (current limit) and 100,000 unique users, which is 100 × 100,000 = 10,000,000 possible buckets. That sounds like a lot, but the real constraint is how many of those buckets have in-flight searches at any given moment — only those hold an entry in SharedThrottleTracker.

In-flight searches are bounded by the search thread pool + queue (~1050 per node), so a node's map can't exceed its in-flight capacity no matter how many users/groups exist. On top of that, buckets are distributed across all data nodes by a consistent-hash ring: bucket X's cluster-level permits live on a single owner node Y (while nodes are stable), so each node tracks just its ~1/N share, not every bucket.