feat(webhook): add persistent retry with exponential backoff

Replace in-memory webhook event queue with SQLite-backed delivery queue.
Failed webhook events are now retried up to 5 times with exponential
backoff (1s, 2s, 4s, 8s, 16s, capped at 300s) and jitter. Deliveries
persist across process restarts.

- WebhookDeliveryQueue: SQLite persistence, atomic claim, background worker
- Reuses existing calculateBackoff() from core/errors/recovery.ts
- Health endpoint now returns delivery stats by status
- 11 tests covering enqueue, retry, dead letter, concurrency, stats

Author: StackMemory Bot (CLI)

docs/plans/webhook-retry-exponential-backoff.md

@@ -0,0 +1,94 @@
+# Plan: Webhook Retry with Exponential Backoff
+
+## Summary
+
+Add persistent retry with exponential backoff to webhook event processing. Replace the in-memory `eventQueue` in `webhook-server.ts` with a SQLite-backed delivery queue that tracks attempts, applies exponential backoff with jitter, and respects circuit breaker state.
+
+## Existing Infrastructure to Leverage
+
+- **`src/core/errors/recovery.ts`**: `retry()`, `calculateBackoff()`, `CircuitBreaker` — all production-ready
+- **`src/integrations/linear/webhook-server.ts`**: Current in-memory queue (`eventQueue[]`, `processQueue()`)
+- **`src/core/database/sqlite-adapter.ts`**: SQLite persistence layer
+- **Error codes**: `LINEAR_WEBHOOK_FAILED`, `LINEAR_API_ERROR` already exist
+
+## Files to Change
+
+| File | Action | Purpose |
+|---|---|---|
+| `src/integrations/linear/webhook-retry.ts` | CREATE | Delivery queue + retry worker |
+| `src/integrations/linear/webhook-server.ts` | MODIFY | Replace in-memory queue with persistent queue |
+| `src/integrations/linear/__tests__/webhook-retry.test.ts` | CREATE | Tests for retry logic |
+
+## Data Model
+
+New table: `webhook_deliveries` (added inline in webhook-retry.ts, not in global migrations — this is integration-scoped)
+
+```sql
+CREATE TABLE IF NOT EXISTS webhook_deliveries (
+  id TEXT PRIMARY KEY,
+  event_type TEXT NOT NULL,
+  payload TEXT NOT NULL,
+  status TEXT NOT NULL DEFAULT 'pending',  -- pending | processing | completed | failed | dead
+  attempts INTEGER NOT NULL DEFAULT 0,
+  max_attempts INTEGER NOT NULL DEFAULT 5,
+  next_retry_at INTEGER,                   -- unix ms
+  last_error TEXT,
+  created_at INTEGER NOT NULL,
+  updated_at INTEGER NOT NULL
+);
+CREATE INDEX IF NOT EXISTS idx_webhook_deliveries_status_retry
+  ON webhook_deliveries(status, next_retry_at);
+```
+
+## Implementation Steps
+
+### Step 1: Create `webhook-retry.ts`
+
+- `WebhookDeliveryQueue` class
+  - `constructor(dbPath: string, options?: RetryConfig)` — opens/creates SQLite DB, ensures table
+  - `enqueue(eventType: string, payload: object): string` — inserts delivery, returns ID
+  - `processNext(): Promise<boolean>` — picks oldest `pending` or retriable `failed` delivery where `next_retry_at <= now`, marks `processing`, calls handler, updates status
+  - `startWorker(intervalMs?: number): void` — setInterval loop calling `processNext()`
+  - `stopWorker(): void` — clearInterval
+  - `getStats(): { pending, processing, completed, failed, dead }` — counts by status
+- Uses `calculateBackoff()` from `recovery.ts` for next_retry_at computation
+- Marks delivery `dead` after max_attempts exceeded
+- Config: `{ maxAttempts: 5, initialDelay: 1000, maxDelay: 300_000, backoffFactor: 2 }`
+
+### Step 2: Modify `webhook-server.ts`
+
+- Replace `eventQueue: LinearWebhookPayload[]` with `WebhookDeliveryQueue` instance
+- In webhook endpoint handler: call `queue.enqueue()` instead of `eventQueue.push()`
+- Start worker in `start()`, stop in `stop()`
+- Remove `processQueue()` method and `isProcessing` flag
+
+### Step 3: Write tests
+
+- Unit tests for `WebhookDeliveryQueue`:
+  - enqueue creates a delivery record
+  - processNext picks the oldest pending delivery
+  - failed delivery gets exponential backoff schedule
+  - delivery marked dead after max_attempts
+  - concurrent processNext doesn't double-process (status = processing guard)
+  - getStats returns correct counts
+
+## Acceptance Criteria
+
+- [ ] Failed webhook events are retried up to 5 times with exponential backoff
+- [ ] Backoff schedule: 1s, 2s, 4s, 8s, 16s (capped at 300s)
+- [ ] Delivery state persisted in SQLite — survives process restart
+- [ ] Dead deliveries (exceeded max attempts) are logged but not retried
+- [ ] Existing webhook signature verification unchanged
+- [ ] Tests pass with 80%+ coverage on new code
+
+## Risks
+
+- **LOW**: SQLite write contention if webhook volume is high — mitigated by WAL mode (already used)
+- **LOW**: Worker interval drift — acceptable for webhook retry cadence (not real-time)
+
+## Non-Goals
+
+- Redis/BullMQ queue (overkill for single-process webhook handler)
+- Webhook delivery UI/dashboard
+- Dead letter queue notification
+- Outbound webhook sending (this is for processing *received* webhooks)

src/integrations/linear/__tests__/webhook-retry.test.ts

@@ -0,0 +1,220 @@
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { mkdtempSync, rmSync } from 'fs';
+import { tmpdir } from 'os';
+import { join } from 'path';
+import { WebhookDeliveryQueue } from '../webhook-retry.js';
+
+describe('WebhookDeliveryQueue', () => {
+  let queue: WebhookDeliveryQueue;
+  let tempDir: string;
+
+  beforeEach(() => {
+    tempDir = mkdtempSync(join(tmpdir(), 'webhook-retry-test-'));
+    queue = new WebhookDeliveryQueue(join(tempDir, 'webhooks.db'), {
+      maxAttempts: 3,
+      initialDelay: 100,
+      maxDelay: 1000,
+      backoffFactor: 2,
+    });
+  });
+
+  afterEach(() => {
+    queue.close();
+    rmSync(tempDir, { recursive: true, force: true });
+  });
+
+  it('enqueue creates a delivery record', () => {
+    const id = queue.enqueue('Issue', { action: 'create', id: '123' });
+
+    expect(id).toBeDefined();
+    expect(typeof id).toBe('string');
+
+    const delivery = queue.getDelivery(id);
+    expect(delivery).toBeDefined();
+    expect(delivery!.event_type).toBe('Issue');
+    expect(delivery!.status).toBe('pending');
+    expect(delivery!.attempts).toBe(0);
+    expect(JSON.parse(delivery!.payload)).toEqual({
+      action: 'create',
+      id: '123',
+    });
+  });
+
+  it('processNext picks the oldest pending delivery', async () => {
+    const processed: string[] = [];
+    queue.setHandler(async (eventType) => {
+      processed.push(eventType);
+    });
+
+    queue.enqueue('Issue', { action: 'create' });
+    queue.enqueue('Comment', { action: 'update' });
+
+    await queue.processNext();
+    expect(processed).toEqual(['Issue']);
+
+    await queue.processNext();
+    expect(processed).toEqual(['Issue', 'Comment']);
+  });
+
+  it('returns false when no deliveries are available', async () => {
+    queue.setHandler(async () => {});
+    const result = await queue.processNext();
+    expect(result).toBe(false);
+  });
+
+  it('returns false when no handler is set', async () => {
+    queue.enqueue('Issue', { action: 'create' });
+    const result = await queue.processNext();
+    expect(result).toBe(false);
+  });
+
+  it('marks delivery completed on success', async () => {
+    queue.setHandler(async () => {});
+
+    const id = queue.enqueue('Issue', { action: 'create' });
+    await queue.processNext();
+
+    const delivery = queue.getDelivery(id);
+    expect(delivery!.status).toBe('completed');
+    expect(delivery!.attempts).toBe(1);
+    expect(delivery!.last_error).toBeNull();
+  });
+
+  it('failed delivery gets exponential backoff schedule', async () => {
+    let callCount = 0;
+    queue.setHandler(async () => {
+      callCount++;
+      throw new Error('Service unavailable');
+    });
+
+    const id = queue.enqueue('Issue', { action: 'create' });
+
+    // First attempt fails
+    await queue.processNext();
+
+    const delivery = queue.getDelivery(id);
+    expect(delivery!.status).toBe('failed');
+    expect(delivery!.attempts).toBe(1);
+    expect(delivery!.last_error).toBe('Service unavailable');
+    expect(delivery!.next_retry_at).toBeGreaterThan(Date.now() - 1000);
+
+    // Should not pick up the delivery again immediately (backoff not elapsed)
+    const result = await queue.processNext();
+    expect(result).toBe(false);
+    expect(callCount).toBe(1);
+  });
+
+  it('delivery marked dead after max_attempts', async () => {
+    queue.setHandler(async () => {
+      throw new Error('Permanent failure');
+    });
+
+    const id = queue.enqueue('Issue', { action: 'create' });
+
+    // Force process through all attempts by manipulating next_retry_at
+    for (let i = 0; i < 3; i++) {
+      // Set next_retry_at to past so it's immediately eligible
+      const db = (queue as any).db;
+      db.prepare(
+        `UPDATE webhook_deliveries SET next_retry_at = 0 WHERE id = ?`
+      ).run(id);
+      await queue.processNext();
+    }
+
+    const delivery = queue.getDelivery(id);
+    expect(delivery!.status).toBe('dead');
+    expect(delivery!.attempts).toBe(3);
+    expect(delivery!.last_error).toBe('Permanent failure');
+  });
+
+  it('concurrent processNext does not double-process', async () => {
+    let concurrent = 0;
+    let maxConcurrent = 0;
+
+    queue.setHandler(async () => {
+      concurrent++;
+      maxConcurrent = Math.max(maxConcurrent, concurrent);
+      await new Promise((r) => setTimeout(r, 50));
+      concurrent--;
+    });
+
+    queue.enqueue('Issue', { action: 'a' });
+
+    // Start two processNext calls simultaneously
+    const [r1, r2] = await Promise.all([
+      queue.processNext(),
+      queue.processNext(),
+    ]);
+
+    // One should process, the other should find nothing
+    expect([r1, r2].filter(Boolean).length).toBe(1);
+    expect(maxConcurrent).toBe(1);
+  });
+
+  it('getStats returns correct counts', async () => {
+    queue.setHandler(async (_type, payload) => {
+      const p = payload as { shouldFail?: boolean };
+      if (p.shouldFail) throw new Error('fail');
+    });
+
+    queue.enqueue('Issue', {});
+    queue.enqueue('Comment', { shouldFail: true });
+    queue.enqueue('Project', {});
+
+    // Process first two
+    await queue.processNext(); // Issue -> completed
+    await queue.processNext(); // Comment -> failed
+
+    const stats = queue.getStats();
+    expect(stats.completed).toBe(1);
+    expect(stats.failed).toBe(1);
+    expect(stats.pending).toBe(1);
+    expect(stats.dead).toBe(0);
+  });
+
+  it('retries succeed on later attempt', async () => {
+    let callCount = 0;
+    queue.setHandler(async () => {
+      callCount++;
+      if (callCount < 3) throw new Error('Transient error');
+    });
+
+    const id = queue.enqueue('Issue', { action: 'create' });
+
+    // First attempt fails
+    await queue.processNext();
+    expect(queue.getDelivery(id)!.status).toBe('failed');
+
+    // Force retry eligible
+    const db = (queue as any).db;
+    db.prepare(
+      `UPDATE webhook_deliveries SET next_retry_at = 0 WHERE id = ?`
+    ).run(id);
+
+    // Second attempt fails
+    await queue.processNext();
+    expect(queue.getDelivery(id)!.status).toBe('failed');
+
+    // Force retry eligible again
+    db.prepare(
+      `UPDATE webhook_deliveries SET next_retry_at = 0 WHERE id = ?`
+    ).run(id);
+
+    // Third attempt succeeds
+    await queue.processNext();
+    expect(queue.getDelivery(id)!.status).toBe('completed');
+    expect(queue.getDelivery(id)!.attempts).toBe(3);
+  });
+
+  it('worker starts and stops cleanly', () => {
+    queue.setHandler(async () => {});
+    queue.startWorker();
+
+    // Starting again is a no-op
+    queue.startWorker();
+
+    queue.stopWorker();
+    // Stopping again is a no-op
+    queue.stopWorker();
+  });
+});