Commit fcccb0c
committed
feat(queue/sql): add subscriber with partition leasing and offset tracking
Implements distributed SQL queue subscriber with partition-based load balancing,
offset tracking, and retry/DLQ support.
## Core Features
- **Partition Leasing**: Workers acquire exclusive 30s leases on partitions,
auto-renewed every 10s. Failed workers release via expiry for automatic rebalancing.
- **Offset Tracking**: Per-partition offsets with optimistic locking. Ack deletes
message + updates offset atomically. Prevents duplicate processing on restarts.
- **Retry/DLQ**: Persistent retry counter, visibility timeout for crash recovery,
automatic DLQ move when MaxAttempts exceeded.
- **Event-Driven Architecture**: One goroutine per topic with dual timers
(poll: 100ms, lease renewal: 10s). Dynamic partition discovery and rebalancing.
## Implementation
**Store Interfaces** (in `extensions/queue/sql/store/`):
- `MessageStore`: FetchByOffset, MoveToDLQ, SetVisibilityTimeout
- `OffsetStore`: Initialize, GetAckedOffset, UpdateAckedOffset, AckMessage
- `PartitionLeaseStore`: DiscoverAndAcquirePartitions, RenewLease, ReleaseLease
**Subscriber Design**:
- `managePartitions()` event loop: poll partitions, renew leases, handle shutdown
- Buffered delivery channel (BatchSize*2) for throughput
- `sqlDelivery` implements Ack/Nack/ExtendVisibilityTimeout
- Standardized metrics with partition-level tags (topic + partition_key)
**Polling Flow**:
1. Discover and acquire new partitions
2. Fetch messages from owned partitions (beyond last acked offset)
3. Check retry limit → DLQ if exceeded
4. Send to delivery channel for consumer processing
## Breaking Changes
- `SetVisibilityTimeout`: Changed from `time.Duration` to `int64` (milliseconds)
- Created `store/` subpackage for interfaces (breaks circular dependency with mocks)
- Added `MessageRow` struct for fetched message representation
## Known Limitations
1. Blocking channel send prevents lease renewal if consumer slow (can cause duplicates)
2. Worker lifetime tied to Subscribe() context (dies if context cancelled)
3. No backpressure metrics or channel depth monitoring
4. Sequential partition processing (one slow partition blocks others)
Follow-up PRs will add timeout-based sends, channel metrics, and concurrent polling.
## Testing
- Unit tests with gomock for all store interfaces
- External test package (`sql_test`) avoids circular dependencies
- Tests: subscription lifecycle, multiple topics, graceful shutdown, close idempotency
## Verification
```bash
bazel test //extensions/queue/sql:sql_test # PASSED
```1 parent 331f173 commit fcccb0c
8 files changed
Lines changed: 1023 additions & 12 deletions
File tree
- extensions/queue/sql
- mocks
- store
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
| 16 | + | |
| 17 | + | |
15 | 18 | | |
16 | 19 | | |
17 | 20 | | |
| |||
22 | 25 | | |
23 | 26 | | |
24 | 27 | | |
| 28 | + | |
25 | 29 | | |
26 | | - | |
27 | 30 | | |
28 | 31 | | |
29 | 32 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
10 | 11 | | |
11 | 12 | | |
12 | 13 | | |
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
0 commit comments