Skip to content

feat: add re-audit on pricing change with email notifications. - #10

Open
jitendra-ky wants to merge 20 commits into
mainfrom
round-2-reaudit
Open

feat: add re-audit on pricing change with email notifications.#10
jitendra-ky wants to merge 20 commits into
mainfrom
round-2-reaudit

Conversation

@jitendra-ky

@jitendra-ky jitendra-ky commented May 21, 2026

Copy link
Copy Markdown
Owner

Important

Note for reviewers: The re-audit pipeline is triggered by pushing to main, which requires write access to this repo. If you don't have it, here's an alternative: submit an audit on the live site and complete the OTP verification so you're registered as a lead — then let me know. I'll change a pricing rule and trigger the re-audit script on my end. You'll receive the notification email directly in your inbox and can click through to the diff view to verify the full end-to-end flow yourself. I mentioned in the ROUND2_DEVLOG.md why I use this method.

Although I tried to add an alternative testing approach, I couldn’t complete it due to time constraints.

What this PR does

Adds a full "re-audit on pricing change" system on top of the Round 1 audit tool. Every lead who submits their AI stack is now tracked in the database across engine versions. When the audit rule engine is updated (pricing changes, new plans, revised logic), the system automatically detects every lead whose result would change, re-runs their audit, persists the new result, and sends them a single consolidated notification email with a one-click link to view an interactive diff of old vs. new recommendations.

Why

AI tool pricing is not static — Cursor restructured its plans in 2024, Claude introduced Max tiers in 2025, Copilot added Pro+. A one-time audit that goes stale is worse than no audit at all: it gives users false confidence. The core assumption is that a user who cares enough to audit their stack once will want to know when their savings opportunity has changed. This feature closes that loop automatically and turns a one-shot utility into a living recommendation engine.

How it works

Trigger: Updating src/features/audit/engine/version.ts (bumping AUDIT_ENGINE_VERSION) on main triggers a GitHub Actions workflow (.github/workflows/reaudit.yml).

Pipeline (scripts/run-reaudit.ts):

  1. Queries lead_audits for every lead whose latest row has an engine version older than AUDIT_ENGINE_VERSION
  2. Re-runs AuditService.executeAudit(tools_json) in-memory with the new rule engine
  3. Compares total_monthly_savings_usd and audit_tag — skips if unchanged
  4. If changed: inserts a new audits row + new lead_audits row (with previous_audit_id pointing to the old audit) + marks the old row is_stale=true
  5. Inserts a reaudit_notifications row (unique on lead_id + engine_version — no duplicates)
  6. Sends one email per affected lead via Resend

Diff view (/audit/[oldAuditId]?rerun=true):
The email link hits the existing /audit/[id] page with ?rerun=true. The server uses previous_audit_id to look up the new audit, then renders AuditDiffView — a side-by-side table with color-coded rows (amber=changed, green=new, red=removed, muted=unchanged), a savings delta headline, and collapsed unchanged rows.

Data flow diagram:

flowchart TD
    A["version.ts bump → push to main"]
    A --> B["reaudit.yml · GitHub Actions"]
    B --> C

    subgraph C["run-reaudit.ts"]
        direction TB
        C1["getLatestLeadAuditPerLead()"]
        C1 --> C2["Re-run AuditService → compare results"]
        C2 --> C3["createAudit() + createLeadAudit()"]
        C3 --> C4["createReauditNotification() — deduped"]
        C4 --> C5["sendReauditNotification() via Resend"]
    end

    C --> D["User clicks email link\n/audit/[oldId]?rerun=true"]
    D --> E["getNewAuditByPreviousAuditId(oldId)"]
    E --> F["AuditDiffView\nSide-by-side diff · delta headline"]
Loading

New DB tables:

  • lead_audits — one-to-many between leads and audits, tracks engine_version, is_stale, previous_audit_id
  • reaudit_notifications — dedup guard, one row per lead per engine version
  • email_verifications — OTP verification during lead capture (also Round 2)

What I cut

  • Pricing snapshot column on audits table. The spec asks for a pricing_snapshot field. I deliberately chose not to add it because pricing in this system is not external data — it's baked into the audit rule code (src/features/audit/rules/). The "pricing snapshot" is effectively the engine version + tools_json together. Adding a JSON snapshot of the pricing constants would duplicate information already recoverable from the git history of the rule files at that version. Documented clearly so reviewers can disagree.

  • One-click unsubscribe from email. The bonus feature. I prioritized the diff view (harder, higher value, more novel) over the unsubscribe link (simpler but lower user impact in a 36h window). Next thing I'd add.

  • Admin dashboard (total audits, emails sent, click-through %). The reaudit_notifications table has all the data needed to build this. Not built — no time.

  • "What changed in the AI tooling market this week" public page. Would be a compelling growth surface but requires accumulated pricing-change data over time. The infrastructure (version bumps + detection) is there; the display layer is not.

  • HTTP endpoint for triggering re-audits. The spec example shows POST /api/detect-changes. I chose GitHub Actions + a standalone script instead, because it avoids exposing a privileged endpoint on the production server and removes the need for a secret token just to call our own server. The trade-off is that a reviewer can't curl it — see testing instructions below.

How to test it manually

Option A — Full end-to-end via engine version bump (recommended):

  1. Submit an audit on the live site with any AI stack
  2. Complete the OTP email verification to register as a lead
  3. In the repo, open src/features/audit/rules/ and change any pricing constant (e.g. bump a monthly price), then bump AUDIT_ENGINE_VERSION in src/features/audit/engine/version.ts from 1.0.0 to 1.0.1 and also change something in auditEngine for example -> (src\features\audit\engine\AuditRuleEngine.ts -> line 69)
  4. Commit and push to main
  5. GitHub Actions triggers reaudit.yml → runs scripts/run-reaudit.ts with DATABASE_URL, RESEND_API_KEY, EMAIL_PROVIDER=resend, NEXT_PUBLIC_BASE_URL from repo secrets
  6. Check the email inbox used during lead capture — you'll receive a re-audit notification email listing what changed and containing a re-run link
  7. Click the link → https://credex.rocks/audit/[oldAuditId]?rerun=true → diff view loads

This runs the full pipeline directly. Useful for testing without a push.

## What's tested

- `src/app/api/audit/route.test.ts` — audit creation API, including DB persistence
- `src/features/leads/services/__tests__/` — OTP service state machine (send cooldown, attempt lockout, expiry), email verification flow
- `src/features/audit/__tests__/` — audit engine rule evaluation, savings calculations

**If I had more time, I'd test:**
- `run-reaudit.ts` pipeline with a mock DB (verify stale detection, dedup, email send)
- `getLatestLeadAuditPerLead()` query with multiple engine versions in test fixtures
- `AuditDiffView` rendering with snapshot tests for each row status (changed/new/removed/unchanged)

- New Drizzle schema: email_verifications table with otp_code,
  expires_at (10 min), last_sent_at (cooldown tracking), attempts
  counter, and verified_at timestamp
- Two indexes: email + expires_at for fast lookup; email + verified_at
  for pending-only queries
- Added query helpers: createEmailVerification, getLatestVerificationByEmail,
  deleteVerificationsForEmail, incrementVerificationAttempts,
  markVerificationComplete
- Migration: 0005_next_lady_deathstrike.sql
OtpService:
- sendOtp(email): 5-min per-email cooldown, crypto.randomInt 6-digit
  code, 10-min expiry, deletes prior pending rows before creating new
- verifyOtp(email, code): enforces expiry, max-3-attempts lockout,
  increments attempt counter on wrong guess, marks verified_at on success

EmailService (Strategy pattern):
- EMAIL_PROVIDER=mock (default): logs OTP to server console, no real send
- EMAIL_PROVIDER=resend: sends via Resend API (requires RESEND_API_KEY)
- Swap providers by changing one env var — no code changes needed

Errors: OtpCooldownError, OtpExpiredError, OtpInvalidError,
        OtpLockedError, OtpNotFoundError
Validators: sendOtpRequestSchema, verifyOtpRequestSchema (Zod)
Types: LeadResponse updated with role field
send-otp:
- Validates email via Zod, delegates to OtpService
- Returns { success: true, cooldown_seconds: 300 } on success
- 429 OTP_COOLDOWN with retry_after_seconds when within 5-min window

verify-otp:
- Validates full request body (email, otp_code required; company_name,
  role, audit_id optional)
- Verifies OTP then atomically captures lead via LeadService
- 201 for new leads, 200 for returning; all OTP error codes mapped
  to correct HTTP status (400 invalid/expired, 404 not-found, 429 locked)
New LeadCaptureModal (src/features/leads/components/):
- Step 1 (Details): email (required), company, job title (both optional)
- Step 2 (OTP): 6 individual digit boxes with paste support, 5-min
  resend countdown, inline error messages for every error code
  (OTP_INVALID shows attempts_remaining, OTP_EXPIRED/LOCKED/NOT_FOUND
  each have distinct copy)
- Step 3 (Success): is_new-aware copy, auto-dismisses after 3s

Integration:
- Both page.tsx and audit/[id]/client.tsx updated to use new modal
- localStorage flag (credex_lead_captured) persists across sessions
- Consistent 4s delay before modal appears on both pages

Cleanup:
- Deleted old src/features/audit/components/LeadCaptureModal.tsx
  (simple email+company form hitting /api/leads directly, no OTP)
New test suites (32 tests across 4 suites, all passing):

OtpService.test.ts (11 tests):
- sendOtp: cooldown enforcement, cleanup ordering, 6-digit code shape
- verifyOtp: correct code, not-found, expired, locked after 3 attempts,
  wrong code increments attempts, reports attempts_remaining correctly

EmailService.test.ts (7 tests):
- Mock provider logs to console, does not throw
- Resend provider throws when RESEND_API_KEY is absent
- Strategy selection via EMAIL_PROVIDER env var, defaults to mock

Route tests already added in API commit (14 tests):
- send-otp: success, invalid email, cooldown, missing email, 500
- verify-otp: 201 new lead, 200 returning, all OTP error codes, 500

Fix: mocked SummaryGenerationService in audit/route.test.ts to prevent
real AI call causing 5s timeout (1 test was consistently failing in CI)
Single source of truth for the audit rule engine version.
Bumping this string and pushing to main is the only action
needed to trigger a full re-audit of all stored lead audits.
createLeadAudit        - inserts a lead_audits row (onConflictDoNothing)
getLatestLeadAuditPerLead - finds each lead's latest audit row and returns
                            those on an older engine version (stale detection)
markLeadAuditStale     - sets is_stale=true on a lead_audits row
createReauditNotification - inserts a reaudit_notifications row (deduped)
updateNotificationStatus  - marks notification as sent or failed after email
LeadService: after upsertLead, creates a lead_audits row with
the current AUDIT_ENGINE_VERSION so the re-audit script can track
which engine version produced each lead's audit.

EmailService: adds sendReauditNotification() to IEmailProvider
interface and both Mock + Resend providers. Email includes savings
delta, new audit tag, and a one-click re-run link.
scripts/run-reaudit.ts
  Standalone tsx script that runs the full re-audit pipeline:
  1. Finds all leads whose latest lead_audits row is on an older version
  2. Re-runs AuditService.executeAudit() with new engine rules
  3. If total_monthly_savings_usd or audit_tag changed:
     - Creates a new auditsTable row
     - Creates a new lead_audits row (previous_audit_id for diff view)
     - Marks old lead_audits row is_stale=true
     - Creates a deduped reaudit_notifications row
  4. Sends one email per affected lead, updates notification status

.github/workflows/reaudit.yml
  Triggers on push to main only when version.ts changes.
  Runs the script directly with DATABASE_URL + RESEND_API_KEY secrets.
  No HTTP endpoint needed - all work done inside the runner.

package.json: add tsx devDependency (required to run .ts script in CI)
@vercel

vercel Bot commented May 21, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
credex Ready Ready Preview, Comment May 21, 2026 4:22pm

Implements Feature 4 of Round 2 — the side-by-side diff UI shown when
a user clicks the re-run link in their re-audit notification email.

## What changed

- src/lib/db/queries.ts
  Added getNewAuditByPreviousAuditId(oldAuditId) — reverse lookup via
  lead_audits.previous_audit_id to find the new audit written by the
  re-audit script.

- src/features/audit/utils/diffUtils.ts (new)
  Pure, DOM-free diff utility. computeAuditDiff() matches findings
  by tool_name (case-insensitive), classifies each as changed/new/
  removed/unchanged, and computes savings deltas. 100% test coverage.

- src/features/audit/components/AuditDiffView.tsx (new)
  Premium diff UI component:
  * Header banner: engine version v1.0.0 -> v1.1.0 + annual savings delta
  * Summary cards: before/after monthly savings, annual savings, audit tag
  * Per-tool diff table: amber=changed, green=new, red=removed, muted=unchanged
  * Unchanged rows collapsed by default (click chevron to expand)
  * Updated AI summary section
  * CTA bar: View Full New Report + Share Updated Results

- src/app/audit/[id]/page.tsx
  Updated Server Component to accept searchParams. When ?rerun=true:
  * Calls getNewAuditByPreviousAuditId(params.id)
  * If found: renders AuditDiffView with both old and new results
  * If not found (re-audit still running): shows processing banner + original audit
  Standard view path (?rerun absent) is completely unchanged.

- src/features/audit/__tests__/AuditDiffView.test.ts (new)
  16 unit tests covering: all four DiffRowStatus states, case-insensitive
  matching, mixed multi-tool scenarios, delta sign correctness, tag change
  detection, empty findings, formatUSD and formatDelta helpers.

## Verification
- npm run type-check: 0 errors
- npm run lint: 0 errors (14 pre-existing console warnings in scripts)
- npm run test:coverage: 146/146 tests pass, 18 suites, diffUtils 100% coverage
@jitendra-ky jitendra-ky changed the title WIP: will be ready before 10:00PM feat: add re-audit on pricing change with email notifications. May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant