Add per-domain sending ramp-up (new-domain daily limits)#370
Open
jiashuoz wants to merge 4 commits into
Open
Conversation
Add automatic, transparent per-domain sending warmup — the reputation- building ramp hosted email services apply to new senders. A domain that first becomes sending-verified begins warmup: its daily outbound volume is capped on a linear ramp (default 50/day climbing to 2000/day over 30 days) so a cold domain builds ISP reputation instead of blasting full volume on day one. Scoped per domain, since mailbox providers track reputation per sending domain. - migrations/050: warmup_status + warmup_started_at on domains. - internal/warmup: pure Schedule.DailyCap ramp math; Enforcer.Check (fail-open on read errors); ThrottleError; EngagementProvider seam + no-op for the follow-up warmup-network phase. - identity.Store: GetWarmupState; SetSendingStatus arms warmup exactly once on the first 'verified' transition and never re-arms. - usage.Store: CountDomainSendsToday (per-domain UTC-day outbound count). - httpapi: enforce warmup on send/reply/forward/test; over-cap returns 429 warmup_throttled with cap/sent/retry-after. No /v1 spec surface change (reuses the error envelope) — spec + SDK stay byte-identical. - config: `warmup:` block (disabled by default); wired in cmd/e2a. Tests: schedule curve, enforcer, 429 mapping, config parse, and DB-backed tests for the verified-stamp lifecycle and the daily counter. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Auv4sfF5kPY3v4vybLoT8L
Adversarial review of the warmup ramp found the enforcement model leaky in ways that defeated the feature. This reworks it: Enforcement moves to outbound.Sender.Send — the one chokepoint every producer of real upstream mail shares (direct sends, HITL approve API, magic-link approve, TTL auto-approve worker). Previously only the HTTP send handlers checked, so bulk-approving held drafts blasted a cold domain uncounted. Loopback self-sends and relay-domain test sends no longer reach the gate, so they are neither throttled nor counted. The numerator becomes domain_send_counters (migration 050): one row per (domain, UTC day), claimed by an atomic increment-if-below-cap at send time (usage.ReserveDomainSend). This closes three holes in the old count(*)-over-messages check: concurrent sends could jointly overshoot the cap (the check-to-insert window spanned the SES round-trip); deleting an agent cascade-deleted its message rows and rewound the counter (self-serve cap bypass); and held drafts consumed allowance on their hold day while their release day sent unmetered. Ramp-state fixes: - Enforcer honors DailyCap's done flag: a domain past its ramp is never throttled or counted again (was: permanently capped at target_daily, paying the counter query forever). - Ramp days index by UTC calendar day, matching the counter reset and retry_after (was: 24h-from-anchor, letting a mid-day-verified domain send 2x its cap inside one window). - Arming honors warmup.enabled (identity.SetWarmupArming): first verify always stamps the anchor but only arms while enabled, and migration 050 backfills anchors for pre-existing verified domains — enabling warmup later can never retroactively throttle an established sender. - config.Load seeds the schedule from warmup.DefaultSchedule, so a partial warmup: block keeps per-field defaults (was: target_daily alone produced a 1-send/day-one schedule via zero-value clamps). - Fail-open paths now log (owned by the enforcer) instead of silently disabling the ramp on persistent read errors. Plumbing: OutboundError gains Details so the 429 warmup_throttled pacing payload survives to the v1 envelope on send/reply/forward AND approve; the TTL worker defers (not auto-rejects) a throttled release; the magic-link page explains the pending state; dead EngagementProvider seam removed. No /v1 spec surface change. Tests: enforcer reserve semantics, UTC-midnight cap step, concurrent reservation (40 claims, exactly cap granted), disabled-arming anchor stamp, sender gate ordering (validation before reserve), 429 detail passthrough, partial-config defaults. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"Warmup" collides with warmup-as-a-service (synthetic engagement mail, seed inboxes) and misleads customers into expecting e2a to actively warm their domain; "quota" would read as a purchasable plan limit. What this feature actually is — an automatic daily sending limit that rises for new domains — the industry names a ramp (SES: "ramp-up"; enforced caps at SES/Mandrill/Twilio are quotas/limits, "warm-up" is the practice). Rename every surface while nothing is shipped, since the error code and config keys freeze on first release: - error code: warmup_throttled → sending_ramp_limited - config: warmup: → sending_ramp: (same fields) - columns: warmup_status/warmup_started_at → sending_ramp_status/sending_ramp_started_at (migration 050 renamed; never deployed, so a straight rename) - package: internal/warmup → internal/sendramp; seams follow (SetSendingRampGate, SetSendingRampArming, GetSendingRampState, SendingRampLimitError, SendingRampConfig) - user-facing text now says "daily sending limit is ramping up"; docs keep "domain warm-up" as a searchable alias Behavior unchanged; full test suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ring, sweep deferral Four fixes from the multi-agent review of this branch, plus the test coverage it flagged as missing: - Key domain_send_counters on the REGISTRABLE domain (eTLD+1 via x/net/publicsuffix, falling back to the exact domain when underivable). Sibling subdomains now draw from one shared daily pool, closing the bypass where verifying m1..mN.spam.example multiplied day-one volume by N. PSL-listed apexes (github.io et al) keep independent senders separate; ramp state/anchor stays per exact domain. - Meter billable outbound usage only AFTER a successful delivery in DeliverOutbound (recordOutboundUsage, both loopback and relay branches). A ramp-throttled 429 — a documented retry-daily outcome — no longer burns plan quota with nothing delivered; matches the three HITL release paths, which already recorded on success only. - Defer a ramp-throttled TTL auto-approve to the counter reset: the worker now bumps approval_expires_at by RetryAfter (new identity.DeferApprovalExpiry, guarded to pending_review). Retrying sooner cannot succeed, and without the bump the earliest-expired throttled rows head-of-line block every sweep — past batchSize rows they starve every other tenant's expirations until UTC midnight. - Honor the cap contract in ReserveDomainSend for cap <= 0: the upsert's guard lives on the DO UPDATE arm only, so the first reservation of a (domain, day) previously slipped past a non-positive cap. Unreachable via the enforcer (schedule floors the cap at 1) but the exported contract now holds for any caller. New tests drive the three ThrottleError detection seams with a REAL throttling sender (not a fabricated error): direct send maps to 429 with no usage metered and no message row; HITL approve maps to 429 with the row still pending_review; the TTL worker defers (never auto-rejects) and rotates the row out of the next sweep. Plus counterKey table tests, subdomain aggregation on the enforcer, and cap<=0 at the store. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements per-domain sending ramp-up (a.k.a. domain warm-up): an automatic, transparent daily sending limit that rises for newly sending-verified domains over a configurable window, so a cold domain builds ISP reputation instead of blasting full volume on day one — the same protection SES/Postmark/SendGrid apply to new senders.
The ramp is scoped per domain (mailbox providers track reputation per sending domain, not per agent or account). It begins automatically when a domain first becomes sending-verified while the ramp is enabled and never re-arms on a later re-verify, so a domain that has already built reputation is never dropped back to day-one limits.
Naming: deliberately not "warmup" in customer-facing surfaces (it suggests warmup-as-a-service — synthetic mail we don't send) and not "quota" (it suggests a purchasable plan limit). "Ramp" says temporary + automatic + rising, matching SES's own "ramp-up" language.
Relates to email-warmup-support.
Design (reworked after adversarial review)
Enforced at the wire, not the handler. The gate lives in
outbound.Sender.Send— the single chokepoint shared by direct sends, HITL approve (API + magic link), and the TTL auto-approve worker. Held-for-review drafts consume allowance when released, not when queued; loopback self-sends and relay-domain test sends are neither limited nor counted.Atomic counter.
domain_send_counters(one row per registrable domain (eTLD+1) + UTC day, migration 050) is claimed with a single increment-if-below-cap statement (usage.ReserveDomainSend), so concurrent bursts serialize on the row and can never jointly overshoot the cap. The counter is independent ofmessages/agent_identities, so agent churn (delete + recreate) cannot rewind it. Aggregating at the registrable domain (viax/net/publicsuffix, falling back to the exact domain when underivable) means sibling subdomains ramp on their own anchors but draw from one shared daily pool — verifyingm1..mN.spam.examplecannot multiply day-one volume by N, and PSL-listed shared apexes (github.ioet al) still keep independent senders separate.Ramp math. Linear ramp (default 50 → 2000/day over 30 days), day-indexed by UTC calendar day — the same boundary the counter resets on and
retry_after_secondspoints at. A domain past its ramp window is unlimited and uncounted forever (doneshort-circuit); "completed" is derived, never stored.Safe enable-later. First verify always stamps the ramp anchor, but the ramp arms only while
sending_ramp.enabledis true; migration 050 backfills anchors for domains verified before the feature shipped. Flipping the flag on later can therefore never retroactively limit an established sender.Limit surface. An over-cap send returns 429
sending_ramp_limitedwithdaily_cap/sent_today/retry_after_secondsdetails (carried throughOutboundError.Detailson send/reply/forward and approve). The TTL auto-approve worker defers a limited release to the counter reset (it bumpsapproval_expires_atbyretry_afterviaidentity.DeferApprovalExpiry) — it never auto-rejects on a pacing condition, and deferred rows rotate to the back of the sweep instead of head-of-line blocking other tenants' expirations. The magic-link approve page explains the message stays pending.Metering. Billable outbound usage is recorded only after a successful delivery (matching the HITL release paths) — a throttled 429, which clients are told to retry against daily, never burns plan quota.
Client surface checklist
sending_ramp_limitedwith pacing details when a domain hits its daily capsending_ramp_status/sending_ramp_started_attodomains(metadata-only), createsdomain_send_counters, and backfills anchors for pre-existing verified domains/v1endpoints; the ramp is enforced transparently on existing send paths (spec byte-identical,TestSpecGoldenNoDriftgreen)Intentionally skipped: the ramp is a pure server-side reputation optimization on existing send paths; it reuses the standard error envelope (429 +
error.code=sending_ramp_limited+ details).Operational risk
Fail-open, logged: the ramp is a reputation optimization, not a correctness gate. If the state read or slot reservation fails (transient DB blip, missing migration), the send proceeds at full volume and the enforcer logs the error — a persistently broken ramp is visible in logs, never silent.
No-op when disabled or unconfigured: disabled by default (
sending_ramp.enabled: false). A self-host that never configuressending_ramp:is unaffected. Config defaults are seeded per-field, so a partial block (e.g. onlytarget_daily) keeps the remaining defaults instead of degenerating.Idempotent arm-once: the ramp anchor is stamped exactly once, on the first
sending_status='verified'transition; re-verifies and status flaps never re-arm. Domains verified while the ramp was disabled (or before it shipped) can never be limited retroactively.Shared relay domain unaffected: the shared relay domain stays
sending_ramp_status='inactive'and is never limited.Reserved-slot conservatism: a slot claimed for a send that subsequently fails downstream stays consumed — an error burst can only slow a ramping domain, never push it over its ramp.
Known follow-ups (not in this PR): no ramp visibility on the domain API yet (429 is the only signal); the ramp day advances by calendar even on idle days; migrating an established domain onto e2a gets day-one limits unless ops pauses the ramp per-domain (
sending_ramp_status='paused', manual SQL).Test plan
usage.ReserveDomainSendDB tests — per-domain/per-day counting, refusal at cap, 40-goroutine concurrent burst grants exactly capidentity.SetSendingStatusDB tests — arms on first verify when enabled; stamps anchor without arming when disabled; never re-arms or moves the anchor on re-verify/flapsending_ramp_limitedenvelope with pacing details via theOutboundError.Detailspassthrough; plain errors unaffectedsending_ramp:block keeps per-field defaults; omitted block stays disabledAsThrottleErrorseams driven by a REAL throttling sender: direct send → 429 with no usage metered and no message row; HITL approve → 429 with the row stillpending_review; TTL worker → defers (never auto-rejects), bumpsapproval_expires_at≈retry_after, deferred row out of the next sweepcounterKeyeTLD+1 table (incl. PSL apexes + fallback), subdomain aggregation throttles against the shared pool,cap<=0refused on the INSERT arm with no phantom rowAll tests pass locally with
make test.https://claude.ai/code/session_01Auv4sfF5kPY3v4vybLoT8L