Skip to content

fix(identity): close the check-then-act race on max_domains - #901

Merged
jiashuoz merged 1 commit into
tokencanopy:mainfrom
AmirF194:fix/822-max-domains-race-condition
Aug 16, 2026
Merged

fix(identity): close the check-then-act race on max_domains#901
jiashuoz merged 1 commit into
tokencanopy:mainfrom
AmirF194:fix/822-max-domains-race-condition

Conversation

@AmirF194

Copy link
Copy Markdown
Contributor

Summary

CheckDomainCreate (the limits pre-check) and ClaimOrCreateDomain (the insert) ran as
two independent transactions with nothing serializing them: concurrent POST /v1/domains
requests could all read the same pre-insert count and all pass, so max_domains could be
exceeded by an arbitrary factor. Reported with a live reproduction against real Postgres
(12 concurrent requests all succeeded against a cap of 1).

ClaimOrCreateDomainWithLimit now takes a per-user advisory lock and re-checks the count
inside the same transaction as the INSERT, so the check and the write can no longer
interleave across requests. EnforceDomainCreate stays as a fast, richly-detailed
pre-check (plan code, upgrade URL); the transactional check inside ClaimDomain is the
race-proof source of truth for the cap itself.

Operational risk

No schema change, no behavior change on the non-concurrent path. A request that
previously slipped past the cap under a race now gets 402 limit_exceeded, the same
response the endpoint already returns for a non-racy over-cap request.

Test plan

  • New e2e test (internal/e2e, -tags integration): 8 concurrent POST /v1/domains
    against max_domains=1: fails on main (8/8 succeed, cap blown 8x), passes on this
    branch (1 created, 7 402 limit_exceeded), both against a real Postgres 16 container
  • go test ./internal/identity/... and ./internal/httpapi/... green, including two
    new unit tests for the ClaimDomain/GetLimits error-mapping paths
  • go test -p 4 -covermode=atomic ./internal/... (the Go coverage gate job's own
    command) green
  • Not verified: contention beyond 8 concurrent requests, or across more than one
    connection pool / process

Fixes #822

…opy#822)

CheckDomainCreate counted existing domains, then ClaimDomain inserted
the new row, as two independent transactions with nothing serializing
them: concurrent creates could all read the same pre-insert count and
all pass, blowing the cap by an arbitrary factor (reported with a
12-concurrent-request repro against a cap of 1, all 12 succeeding).

ClaimOrCreateDomainWithLimit now takes a per-user advisory lock and
re-checks the count inside the same transaction as the INSERT, so the
check and the insert can no longer interleave across requests.
EnforceDomainCreate stays as a fast pre-check (plan code, upgrade
URL); the transactional check inside ClaimDomain is now the race-proof
source of truth for the cap itself.

Fixes tokencanopy#822

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@AmirF194
AmirF194 requested a review from jiashuoz as a code owner August 16, 2026 07:30

@jiashuoz jiashuoz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid fix — the race is real, the advisory-lock approach is sound, and the test coverage (including a fails-on-main e2e repro) is exactly what this needed. Approving.

Verified while reviewing:

  • Lock design: hashtextextended(userID, 1) is a distinct keyspace from the seed-0 domain locks and the sender-identity: session lock; ordering (domain locks → user lock) is consistent and the user lock is taken in exactly one place, so no deadlock cycle is possible. Under READ COMMITTED the post-lock count(*) sees every insert committed before lock acquisition — properly serialized across connections, pools, and processes on the same DB.
  • The in-tx count is byte-identical to usage.CountDomainsByUser, so the backstop and CheckDomainCreate can't disagree about what consumes a slot.
  • Cap is only charged on genuine inserts; same-owner re-claims return before the count check, preserving idempotent re-register at cap.
  • GetLimits failure → 500 before ClaimDomain is called (fail-safe, and pinned by the new unit test), and DomainLimitExceededError maps to the same 402 limit_exceeded shape clients already get from limitEnvelope. No API-shape change, so no contract-scenario/spec obligations.

Two non-blocking suggestions:

  1. Consider fetching GetLimits unconditionally in handleRegisterDomain (while keeping the EnforceDomainCreate pre-check skipped on the alreadyOwned path). Today alreadyOwned passes maxDomains=0, so a DELETE racing a re-claim of the same domain would fall through to the insert branch with the cap check disabled. Contrived, but the store only charges on genuine inserts anyway, so passing the real limit costs nothing.
  2. The maxDomains <= 0 = "unlimited" sentinel inverts the pre-check's semantics (where MaxDomains=0 blocks everything). Unreachable today since a zero-cap plan 402s in the pre-check first, but worth a one-line comment at the lock site so a future zero-cap plan doesn't silently disable the backstop.

Nit: the 500 message for a GetLimits failure is identical to the EnforceDomainCreate failure message ("limits check failed") — distinguishing them would ease log triage.

@jiashuoz
jiashuoz merged commit 21a1ea4 into tokencanopy:main Aug 16, 2026
29 checks passed
@AmirF194

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review, Josh, and for catching the lock-ordering and idempotent-reclaim details. Good calls on both suggestions, I will follow up with a small PR for the unconditional GetLimits fetch and a comment at the lock site on the maxDomains <= 0 sentinel.

@AmirF194
AmirF194 deleted the fix/822-max-domains-race-condition branch August 16, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

limits: max_domains is not enforced under concurrency (check-then-act, and the endpoint has no rate limiter)

2 participants