Skip to content

architecture.md

Latest commit

 

History

History
169 lines (134 loc) · 6.56 KB

architecture.md

File metadata and controls

169 lines (134 loc) · 6.56 KB

Architecture

Overview

DQueue is a distributed task queue with strong consistency guarantees, built on the Raft consensus algorithm. It provides reliable task processing with automatic leader election, fault tolerance, and at-least-once delivery semantics.

System Diagram

                                    ┌─────────────────────────────────────────┐
                                    │              DQueue Cluster             │
                                    │                                         │
  ┌──────────┐     HTTP API         │  ┌─────────┐  gRPC   ┌─────────┐       │
  │  Client  │ ──────────────────▶  │  │  Node 1 │ ◀─────▶ │  Node 2 │       │
  │  (SDK)   │                      │  │ (Leader)│         │(Follower)│       │
  └──────────┘                      │  └────┬────┘         └────┬────┘       │
                                    │       │                   │            │
                                    │       │      gRPC         │            │
  ┌──────────┐     HTTP API         │       │    ┌──────────────┘            │
  │  Worker  │ ──────────────────▶  │       │    │                           │
  │          │                      │       ▼    ▼                           │
  └──────────┘                      │  ┌─────────────┐                       │
                                    │  │   Node 3    │                       │
                                    │  │  (Follower) │                       │
                                    │  └─────────────┘                       │
                                    └─────────────────────────────────────────┘

Components

Queue Node (cmd/dqueue)

The main server process that:

  • Participates in Raft consensus
  • Stores the task queue state
  • Exposes HTTP API for clients
  • Exposes gRPC API for inter-node communication

Worker (cmd/dqueue-worker)

A task processor that:

  • Polls the leader for available tasks
  • Executes task handlers
  • Reports success (ACK) or failure (NACK)

CLI (cmd/dqueue-cli)

Command-line tool for:

  • Enqueueing tasks
  • Checking task status
  • Viewing statistics

Data Flow

Enqueue Flow

1. Client sends POST /tasks to any node
2. If follower → returns 503 with leader hint
3. If leader:
   a. Create ENQUEUE command
   b. Append to Raft log
   c. Replicate to followers
   d. Wait for majority acknowledgment
   e. Commit and apply to state machine
   f. Return task ID to client

Lease Flow (Visibility Timeout)

1. Worker sends POST /tasks/lease to leader
2. Leader finds next pending task
3. Leader creates leader-local lease (NOT replicated)
4. Leader returns task to worker
5. Task has "visibility timeout" - if not ACKed, returns to pending

ACK/NACK Flow

1. Worker sends POST /tasks/{id}/ack or /nack
2. Leader validates worker holds the lease
3. Leader creates ACK/NACK command
4. Command replicated via Raft
5. On commit: task marked completed or scheduled for retry

State Machine

The task queue state is derived from the Raft log via event sourcing:

┌─────────────┐
│   Raft Log  │
│  (Commands) │
└──────┬──────┘
       │
       ▼ Apply committed entries
┌─────────────┐
│ Queue State │
│  (Derived)  │
└─────────────┘

Commands in the Raft log:

  • ENQUEUE - Add new task
  • ACK - Mark task completed
  • NACK - Mark task failed (triggers retry or dead-letter)
  • DEAD_LETTER - Move task to dead-letter queue

Task Lifecycle

                    ┌─────────┐
                    │ PENDING │◀──────────────────┐
                    └────┬────┘                   │
                         │ Lease                  │ Retry (with backoff)
                         ▼                        │
                    ┌─────────┐                   │
                    │ LEASED  │───────────────────┤
                    └────┬────┘  Lease expires    │
                         │                        │
              ┌──────────┴──────────┐             │
              │ ACK                 │ NACK        │
              ▼                     ▼             │
        ┌───────────┐         ┌─────────┐         │
        │ COMPLETED │         │ FAILED  │─────────┘
        └───────────┘         └────┬────┘
                                   │ Max retries exceeded
                                   ▼
                             ┌─────────────┐
                             │ DEAD_LETTER │
                             └─────────────┘

Consistency Model

Strong Consistency for State Changes

All state-changing operations (ENQUEUE, ACK, NACK) go through Raft consensus:

  • Linearizable writes
  • Durability guaranteed after majority acknowledgment
  • State survives any minority of node failures

Leader-Local Leases (Performance Optimization)

Leases (visibility timeouts) are NOT replicated:

  • Avoids consensus overhead for every lease
  • If leader fails, leases expire and tasks become available
  • Trade-off: Tasks may be processed multiple times (at-least-once)

This is the same approach used by production systems like Amazon SQS.

Persistence

Unified Log Architecture

The Raft log serves as the write-ahead log (WAL):

  • Single source of truth
  • All state derived by replaying committed entries
  • No separate WAL for queue events

Snapshots

Periodic snapshots compress the log:

  • Full queue state serialized
  • Log entries before snapshot deleted
  • New nodes catch up via snapshot + recent log