Sharding implementation

[osvaldoandrade]

Context

codeQ currently runs on a single KVRocks instance. All queue state — tasks, results, leases, pending/in-progress/delayed/DLQ structures — lives on one node. The API servers are stateless and scale horizontally, but KVRocks is the bottleneck for both throughput and availability.

Decision Drivers

1. Single point of failure: If KVRocks goes down, the entire system stops. No automated failover exists today.
2. No leader election: KVRocks supports slaveof (async replication) but requires manual intervention or Sentinel for failover — and async replication can lose recent writes.
3. Throughput ceiling: A single KVRocks node saturates around 2,500 claims/sec. Beyond that, vertical scaling hits diminishing returns.
4. Cost: Multi-master replication doesn't exist in KVRocks. Running N masters + N replicas (2N nodes) for per-tenant sharding is too expensive.

Decision

We will adopt kvrocks-controller (Apache KVRocks RAFT-based cluster manager) to provide:

• Automated leader election via RAFT consensus (3-node groups)
• Strong consistency — writes acknowledged only after quorum commit
• Future sharding — hash-slot model (16384 slots) for horizontal throughput scaling

Phase 1 (Immediate): Single RAFT Group

• 3 KVRocks nodes, 1 RAFT group, all 16384 slots on one group
• kvrocks-controller with etcd backend for controller HA
• Zero codeQ code changes — point redisAddr at the controller proxy
• Gives us leader election without slot migration risk

Phase 2 (When throughput demands): Multi-Shard

• Add hash tags to queue keys: codeq:q:{command:tenantID}:pending:0
• Shard the global hashes (codeq:tasks, codeq:results) by command+tenant
• Implement ShardSupplier interface (already designed in docs/02-domain-model.md)
• Assign hash slots to multiple RAFT groups

Risks Accepted

Risk	Severity	Mitigation
kvrocks-controller is not production-grade	High	Start with single RAFT group (simplest topology); aggressive monitoring; budget for infra debugging
Lua RPOP + SADD claim script crosses slots in multi-shard	Medium	Phase 1 is single-shard (no CROSSSLOT); Phase 2 adds hash tags before adding shards
Split-brain during network partitions	Medium	RAFT quorum prevents split-brain in normal operation; monitor group health
Slot migration may lose keys	Low (Phase 2 only)	Defer slot migration until absolutely necessary; plan capacity upfront

Alternatives Considered

Alternative	Why rejected
Sentinel + async replicas	Doesn't solve write-loss on failover; still single-master per shard
Redis Cluster (instead of KVRocks)	Loses SSD-cost advantage that motivated KVRocks
Multi-master KVRocks	Doesn't exist in KVRocks's replication model
Infra-level sharding (isolated pairs)	Solves throughput but multiplies SPOFs; no automated failover
Do nothing	Unacceptable — single node SPOF is a production risk

References

• Parent issue: Design and implement queue sharding #31
• HLD sub-issue: Draft High-Level Design (HLD) and RFC for queue sharding implementation #118
• docs/06-sharding.md — current placeholder
• docs/02-domain-model.md — ShardSupplier interface sketch
• docs/08-consistency.md — consistency model and failure windows
• apache/kvrocks-controller

[osvaldoandrade]

The tone of HLD misses the target. The readers already know the basics, the doc does not need to explain that KVRocks does not implement RAFT.

We chose a design that combines sharding + RAFT as we push for the highest standards and that's the reason sentinel is not acceptable here.

Last thing, the doc also misses we can evolve the plugin architecture so persistence is not tied to KVRocks. The plugin interfaces should separate concerns, at minimum: auth and persistence. With that split, we can plug in Cassandra, HBase, or even TiKV (Raft & Rocks backed ) as persistence backends, without rewriting the core.