diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 56590671..83010966 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -131,6 +131,8 @@ jobs: if (pkg.publishConfig?.registry !== "https://registry.npmjs.org") throw new Error("npm registry must be the public registry"); if (pkg.repository?.url !== "git+https://github.com/hasna/emails.git") throw new Error("repository provenance must be hasna/emails"); ' + - name: Require immutable gates on future release and deployment workflows + run: bun run deployment:policy - name: Verify generated SDK is committed run: | bun run scripts/generate-selfhost-sdk.ts @@ -141,6 +143,8 @@ jobs: run: | bun run no-cloud:source bun run no-cloud:pack + - name: Verify self-hosting release and rollback contract + run: PATH=/usr/bin:/bin ./deploy/aws/tests/static_contract.sh - name: Diff hygiene run: git diff --check @@ -167,7 +171,7 @@ jobs: with: bun-version: 1.3.14 - run: bun install --frozen-lockfile - - name: Verify self-hosted Postgres migrations, tenancy, RLS, message IDs, and send semantics + - name: Verify self-hosted Postgres migrations, tenancy, RLS, message IDs, send semantics, and store conformance env: PGHOST: 127.0.0.1 PGPORT: "5432" @@ -190,3 +194,15 @@ jobs: src/server/self-hosted/send-honesty-and-reconciliation.integration.test.ts \ src/server/self-hosted/webhooks.integration.test.ts bun test src/server/self-hosted/attachment-inventory.integration.test.ts + # store-conformance gets its OWN invocation, like attachment-inventory, and it + # is not a stylistic choice: every suite above drops schema `public` in its + # `beforeAll`, so folding this one in would make its correctness depend on + # `bun test` honouring the argument order — a property no assertion here checks. + # On its own it migrates whatever it finds and takes a fresh tenant per run, so + # nothing can wipe the schema out from under it. + # + # EMAILS_REQUIRE_POSTGRES_TESTS makes the suite assert its own reachability: the + # whole file is skipped when the connection string is absent, so without this + # flag a renamed variable or a broken service container would silently delete + # the real-service conformance evidence and leave the job green. + EMAILS_REQUIRE_POSTGRES_TESTS=1 bun test src/server/self-hosted/store-conformance.integration.test.ts diff --git a/AGENTS.md b/AGENTS.md index ea789e73..c27c2631 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,6 +2,19 @@ This file guides AI coding agents working with `@hasna/emails` - an email management CLI, MCP server, and library supporting Resend, AWS SES, and Cloudflare-routed inbound mail. +## Naming (read before "fixing" any Mailery reference) + +This product is **open-emails**: repo `hasna/emails`, package `@hasna/emails`, +bins `emails*`, env prefix `EMAILS_*`. **Mailery is a separate, unrelated +product and is not a name for this one** (owner ruling 2026-07-27). + +The string `mailery` still appears in this tree on purpose. It is either a +guard that enforces the ruling, a frozen compatibility constant (migration IDs, +the `mailery` API-key alias slug, legacy event source, `legacy-inbound@local.mailery`), +or a banned env selector that must stay named to be rejected. **Do not +bulk-rename it.** Read [docs/NAMING.md](docs/NAMING.md) first — it lists what +breaks for each one. + ## What This Package Does `@hasna/emails` manages the full email lifecycle locally: @@ -201,7 +214,7 @@ bun run dev:serve # run HTTP server in dev mode ``` src/ ├── cli/ -│ ├── index.tsx # thin orchestrator (~65 lines) +│ ├── index.tsx # thin, lazy-loading command orchestrator │ ├── utils.ts # shared helpers │ ├── tui/ # OpenTUI Emails UI dashboard │ └── commands/ # modular command files @@ -229,18 +242,18 @@ src/ ├── providers/ # provider adapters │ ├── resend.ts, ses.ts, sandbox.ts │ └── interface.ts # ProviderAdapter interface -├── mcp/ # MCP server, modular tools, and resources -├── server/serve.ts # HTTP server + REST API +├── mcp/ # MCP server, modular tools, contracts, and resources +├── server/ # local dashboard API plus self-hosted /v1 service └── index.ts # library exports ``` ## Adding New Features The codebase follows these patterns: -- **New DB table**: Add migration in `db/database.ts`, new CRUD file in `db/`, add `ensureTable`/`ensureIndex` in `ensureSchema` +- **New DB table**: Add the SQLite migration/ensure-schema work in `db/database.ts`; if self-hosted, also add an immutable migration under `server/self-hosted/migrations.ts` and store/RLS coverage - **New CLI command**: Add to appropriate `cli/commands/*.ts` file -- **New MCP tool**: Add `server.tool(...)` in `mcp/index.ts` before the Start section -- **New REST endpoint**: Add route in `server/serve.ts` +- **New MCP tool**: Add `server.tool(...)` in `mcp/tools/*.ts` and wire a new registrar from `mcp/server.ts` when needed +- **New REST endpoint**: Add local dashboard routes under `server/routes/`; add self-hosted `/v1` routes and OpenAPI under `server/self-hosted/` - **New library export**: Add to `src/index.ts` -Test: `EMAILS_DB_PATH=:memory: bun test` — must stay at 0 failures. +Test: `bun run test` — the hermetic runner owns DB isolation and must stay at 0 failures. diff --git a/CHANGELOG.md b/CHANGELOG.md index da910555..a5fe105e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ All notable changes to `@hasna/emails` are documented here. ## [Unreleased] +- **fix(ui): `emails ui` could not start in any real terminal — the packaged runtime loaded a foreign OpenTUI native library.** 1.3.4 exited immediately with `Failed to initialize OpenTUI render library: Symbol "createEventSink" not found in .../@opentui/core-linux-arm64/libopentui.so`. `@opentui/core` loads its prebuilt renderer with a bare `import("@opentui/core-")` from inside its own module, so the version-matched prebuilt in `@opentui/core/node_modules/` is what answers. `scripts/build-tui-runtime.ts` inlined core into `dist/cli/ui-runtime-bundle.js` while listing the eight platform packages as **external**, which moved that import to `dist/cli/` — it then resolved against the installed package's *parents*, never saw core's own prebuilt, and bound to whatever copy the install had hoisted (here `0.1.105`, an ABI predating the symbol core calls). The install was version-correct throughout; only the loaded `.so` was wrong. `@opentui/core` is now external — it is already a declared runtime dependency, and keeping it in `node_modules` keeps the JS and the library it `dlopen`s in one dependency tree. `web-tree-sitter` and `bun-ffi-structs` were dropped from the same list for the same reason: both are core's dependencies, neither is declared by `@hasna/emails`, so externalising them pointed at unowned copies too. Declaring the eight platform packages as our own `optionalDependencies` was rejected — it copies upstream's platform matrix into this manifest and rots on every core bump, while leaving the resolution anchor wrong. `patchBundledNativeAssetPath()` is gone with the bundling it patched around, and the runtime bundle drops from 3.4 MB to 2.1 MB. +- **test(ui): the build contract asserted the broken configuration, so the suite stayed green through a UI that could not start.** It required `scripts/build-tui-runtime.ts` to contain `"@opentui/core-linux-arm64"` and `...nativePackages` — the exact lines that caused the crash — because every assertion was a text match on the build script rather than a check of the artifact it produces. New `src/cli/tui/ui-runtime-contract.test.ts` rebuilds the bundle (never trusting a stale one), parses its imports with `Bun.Transpiler.scanImports` rather than a regex over 3 MB of bundled output, and fails if any bare import is not a declared runtime dependency of this package — the general form of the defect, not just the OpenTUI instance. It carries a positive control proving the check reports an undeclared external and passes a declared one, and a behavioural guard that a non-interactive `emails ui` exits non-zero, so a refusal can never be read as a UI that ran. +- feat(cli): every inbox and sync command now accepts `-j, --json`, emits one structured result document, and reports machine-readable failures without changing the existing human output. + +- **feat(auth): IdP-token credential class — the first committed step of the ADR-0001 identity federation.** The self-hosted server now accepts EdDSA access tokens minted by the `@hasna/tenants` IdP, verified statelessly against the JWKS URL configured via `EMAILS_IDP_JWKS_URL` (unset ⇒ the class is refused with a typed `idp_not_configured`; a JWKS outage is a typed 503, never an allow). A verified token's `sub` resolves through the new additive `idp_principal_tenants` resolution table (migration 0021 — outside RLS like `api_key_tenants`, with an IdP-tenant pin and an emails-side `revoked_at` kill switch), scopes are normalized onto the existing `emails:read`/`emails:write`/`emails:*` vocabulary, and `/v1/me` gains a third modeled `principal_type: "idp"` branch. Clients can present the token via `EMAILS_IDP_TOKEN` (session > IdP token > operator key precedence) and `emails auth whoami` reports ` (idp agent )`. Secret-free `[idp-auth]`/`[idp-jwks]` audit lines carry sub/jti/kid/reason — never the token. Existing `hasna_`/`emss_` dispatch is byte-equivalent: the IdP branch is structural JWS detection AFTER both prefix classes. + - scope AWS module cross-account SES credentials to `EMAILS_SES_*` only; generic `AWS_*` credentials are no longer injected, so unrelated SDK clients retain the task-role default chain. - **fix(status): the refusal registry is checked against the CLI, not against itself — `emails status` was still proposing a command that throws.** `src/lib/status-commands.ts` documented its source of truth as `grep -n 'serverOnly(' src/cli/commands/*.remote.ts`. That glob is wrong: `serverOnly()` is also defined and called in the SHARED modules `src/cli/commands/domain.ts` and `src/cli/commands/address.ts`, which `src/cli/index.tsx` loads in BOTH modes and whose helper throws unconditionally. Fifteen commands were missing from `NEVER_AVAILABLE_COMMANDS`, so `status-facts.remote.ts domainFixCommands` returned `emails domain status --json` for any failed/errored domain, `agent-context.ts buildNextActions` promoted `fix_commands[0]` into `next_actions`, and `isCommandAvailableInMode` waved it through — the exact "remedy that refuses" defect the registry exists to remove, reintroduced one command over. Local mode was hit the same way through `domain-readiness.ts` fix_commands (`emails domain check|dns|verify|setup-cloudflare`, all four unconditional refusals). Also fixed: `cli_equivalents.provision_address` and the `create_receive_address` workflow proposed `emails address provision` (refuses everywhere) — now `emails address add` plus an explicit `emails address set-owner` step, because `address add` takes only `--provider`/`--name` and the workflow was registering an owner it never attached; only one of the three workflow lists was mode-filtered, now all three are; and the `Usable domains:` footer and the MCP domains resource `cli_equivalent` both advertised `emails domain status`, now `emails domain list`. diff --git a/README.md b/README.md index 087cc279..a4e9c797 100644 --- a/README.md +++ b/README.md @@ -4,11 +4,15 @@ Open-source email infrastructure for local SQLite workflows and operator-owned s [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE) +> **This product is `open-emails`. Mailery is not a name for it.** (Owner +> ruling, 2026-07-27 — see [docs/NAMING.md](docs/NAMING.md).) +> > This package publishes as **`@hasna/emails`** and ships the `emails`, > `emails-mcp`, and `emails-serve` bins. `@hasna/mailery` on npm is the > abandoned 0.6.x line, and the `mailery*` bins belong to the separate cloud CLI > — neither is this package. The env prefix is `EMAILS_*`. This stays a -> **cloud-free** OSS package: a hosted Mailery cloud is a separate product. +> **cloud-free** OSS package: Mailery is a separate, unrelated product, not a +> hosted version of this one. ## Install @@ -23,6 +27,10 @@ bun install -g @hasna/emails Emails has exactly two modes: `local` and `self_hosted`. Local mode keeps SQLite, files, and credentials on the current machine. Self-hosted mode connects to an Emails service deployed in user-owned infrastructure. Provider integrations always use user-supplied credentials; the package has no hosted account or control-plane service. +Local provider credentials are envelope-encrypted with a root key kept outside +SQLite. Rotation, locked-keyring recovery, and backup rebind procedures are in +[Provider credential storage](docs/PROVIDER_SECRETS.md). + ## Quick Start ```bash @@ -31,19 +39,17 @@ Emails has exactly two modes: `local` and `self_hosted`. Local mode keeps SQLite AWS_PROFILE=emails-operator emails provider add --name production-ses --type ses --region us-east-1 emails provider add --name production-resend --type resend --api-key ... -# Set up a domain (buy + DNS + SES in one command) -emails domain setup example.com --provider --email you@example.com ... - -# Or connect a domain you already own without buying it -emails domains connect example.com --provider --dry-run -emails domains connect example.com --provider --dns-provider route53 --no-register-provider - -# Or configure DNS for an existing domain via Cloudflare -emails domain setup-cloudflare example.com --provider +# Register a domain you already own and have verified with the provider +emails domain adopt example.com --provider -# Check public DNS before changing inbound routing +# See the DNS records the domain must publish, then confirm what is live +emails domain dns example.com --provider emails domain check example.com +# Buy a domain first, if you do not own one yet +emails domain available example.com +emails domain buy example.com --email you@example.com ... + # SES send-only setup preserves existing MX, such as Google Workspace emails domain adopt example.com --provider --no-inbound @@ -79,10 +85,17 @@ The source of truth follows the mode; it is not a per-domain choice. A domain created or connected through this client is owned by the app's `/v1` database, so `source_of_truth` is reported as `postgres` and is not an input. -| Mode | Who owns the mail source of truth | Domain setup path | -| --- | --- | --- | -| `local` | The local SQLite/files install | `emails domains add`; DNS checks are advisory unless using a real send/receive provider. | -| `self_hosted` | Your PostgreSQL/S3/SES or equivalent infrastructure | `emails domains connect`, then publish the returned DNS tasks and enable inbound/outbound when evidence is ready. | +| Mode | Who owns the mail source of truth | +| --- | --- | +| `local` | The local SQLite/files install | +| `self_hosted` | Your PostgreSQL/S3/SES or equivalent infrastructure | + +The domain setup path is the same either way, because none of it is served over +the wire: `emails domain add` (or `emails domain adopt` for a domain the +provider has already verified), then `emails domain dns ` for the +records to publish, then `emails domain check ` to confirm what is live. +`emails aws setup-inbound` creates the S3 bucket and SES receipt rules when the +domain should also receive. Authentication records are required only for the capability you enable: @@ -95,7 +108,9 @@ Authentication records are required only for the capability you enable: before moving from `p=none` to stricter policies. Self-hosted clients must set `EMAILS_MODE=self_hosted`, -`EMAILS_SELF_HOSTED_URL`, and `EMAILS_SELF_HOSTED_API_KEY`. The service uses +`EMAILS_SELF_HOSTED_URL`, and one bearer credential: +`EMAILS_SESSION_TOKEN`, `EMAILS_IDP_TOKEN`, or +`EMAILS_SELF_HOSTED_API_KEY` (in that precedence order). The service uses `EMAILS_DATABASE_URL`, `EMAILS_API_SIGNING_KEY`, `EMAILS_AUTH_ALLOWED_EMAIL_DOMAINS`, and `EMAILS_AUTH_FROM`; Postgres is authoritative and there is no hybrid SQLite synchronization mode. @@ -113,16 +128,12 @@ revoke the old key after the rollback window closes. ## Emails UI (`emails ui`) -A full-screen OpenTUI mail client with a responsive dashboard shell. Wide -terminals use a two-column admin layout with persistent navigation, mailbox -metrics, operations health, folders, actions, and a focused workspace. Inbox on -wide terminals uses a split message list + preview reader. Narrow terminals collapse to -a compact single-column view with the same Inbox, Compose, Domains, and -Settings dialog. Inbox starts at all addresses and can be filtered to one email -address when needed. Mailbox source status is exposed through CLI/API/MCP -surfaces without treating provider credentials as inboxes. Live read-state, -local refresh, background auto-pull, and an `auto`/`light`/`dark` color theme -keep the mailbox current and readable across terminals. +A full-screen Solid/OpenTUI mail client with a persistent mailbox sidebar and a +workspace for message lists, the reader, and domain status. Inbox can be scoped +to all addresses or one address and filtered by ingestion source, folder, +label, and search. Grouping, digests, attachment/link/raw views, live read +state, local refresh, background auto-pull, and `auto`/`light`/`dark` themes are +available in both local and self-hosted clients. ```bash emails ui @@ -130,20 +141,24 @@ emails ui --mailbox unread ``` The app uses visible buttons and the Shortcuts command palette for actions. -Mailbox filtering is handled by the mailbox dialog, which lists all mailboxes -and configured/observed recipient addresses. Sidebar labels filter mailbox -content, and mail categories show Primary, Social, Promotions, Updates, -and Forums separately from custom labels. Reader shows -attachments with size/type. Composer writes **markdown** rendered to HTML on -send. Settings opens as a simple menu dialog for sync, defaults, and display -controls. Folders: Inbox · Unread · Starred · Sent · Archived · Spam · Trash. +The address and source dialogs select mailbox scope; sidebar labels filter +mailbox content, and mail categories show Primary, Social, Promotions, Updates, +and Forums separately from custom labels. Reader dialogs expose attachments, +links, and raw details. Composer writes **markdown** rendered to HTML on send. +Settings controls sync, defaults, and display. Folders: Inbox · Unread · +Starred · Sent · Archived · Spam · Trash. ## Command Structure +The table below covers every primary root namespace. Standalone compatibility +aliases and the full subcommand matrix are in [docs/CLI.md](docs/CLI.md); the +runtime `emails --help` output remains the option-level source of +truth. + ``` emails ui # Mailbox UI - inbox, compose, domains, settings emails provider # provider credentials/capabilities (ses, resend, sandbox) -emails domain # add/verify/buy/setup/dns/check domains +emails domain / domains # domain records, purchase, DNS checks, warming emails domain warm # domain warming schedules: warm, warm-status, warm-list, # warm-pause, warm-resume, warm-complete, warm-delete emails address # manage sender addresses (add, suspend, activate, quota) @@ -155,9 +170,10 @@ emails owner # ownership: register human/agent owners emails alias # per-domain aliases + catch-all routing emails forwarding # app-level forwarding for locally received/synced mail emails sendkey # scoped send keys (restrict an agent to its own addresses) +emails send-intent # inspect/reconcile uncertain self-hosted send outcomes emails send # send an email emails reply / forward # reply (in-thread) or forward a sent/inbound email -emails email # sent email: list, search, show, replies, conversation +emails email # sent email: list, search, show, replies, thread emails inbox # mailbox folders, sources, sync, read/star/archive/label, watch emails template # email templates emails contact # contacts (suppression list) @@ -165,14 +181,19 @@ emails group # recipient groups emails sequence # drip sequences emails schedule # scheduled emails: list, cancel, run emails db # self-hosted PostgreSQL migration and status commands +emails self-hosted key # operator API-key create/list/rotate/revoke +emails auth # self-hosted signup/login/logout/tenant sessions +emails keys # tenant-scoped API keys for the active organization +emails whoami # current self-hosted principal and organization emails aws # AWS setup: SES receipt rules, S3 inbound bucket -emails config # configuration (key=value) emails stats # delivery statistics (--inbox for received mail) emails analytics # email analytics emails doctor # system diagnostics emails doctor delivery # diagnose missing inbound mail for one address -emails serve # local HTTP server + dashboard + /api management routes +emails provision # registered but intentionally NOT IMPLEMENTED +emails serve # local dashboard or self-hosted /v1 service, by mode emails mcp # install MCP server +emails remove # remove MCP configuration from supported agent clients ``` ### Compact Output and Gradual Disclosure @@ -185,7 +206,7 @@ more: ```bash emails address list # compact table emails address list --verbose # expanded owner/admin/quota rows -emails domain status --verbose # includes per-domain issue and fix lines +emails domains status # per-domain records and DNS state emails provider list --limit 50 # explicit larger page emails contact list --suppressed # compact filtered contact list emails template show # detail path for template bodies @@ -194,8 +215,6 @@ emails forwarding list --source ops@example.com emails agent context # compact agent context summary emails agent context --verbose # full redacted context snapshot emails agent context --json # full machine-readable context -emails config list --verbose # full redacted config values -emails config keys --verbose # include examples for every key emails email show # detail path for one sent email emails inbox read # detail path for one inbound email emails inbox attachments --limit 100 --direction inbound --json @@ -261,7 +280,8 @@ emails forwarding run --provider # future mail only emails forwarding run --provider --backfill # intentionally include older synced mail # Address lifecycle -emails address provision ops@example.com --provider --owner Atlas +emails address add ops@example.com --provider +emails address set-owner ops@example.com --owner Atlas emails address suggest --domain example.com emails address suspend # block sending from this address emails address activate @@ -308,9 +328,11 @@ SES send-only provisioning does not require changing root MX and is the safest path when an existing mailbox provider already receives mail. Publishing SES inbound MX is only for domains that should receive through -SES/S3. Commands that can add SES inbound MX refuse to proceed when public MX -already belongs to another provider. `--force-mx-switch` is available for -intentional migrations after confirming mailbox ownership can move. +SES/S3. `emails domain adopt` refuses to wire SES inbound when public MX already +belongs to another provider; `--force-mx-switch` overrides it for intentional +migrations after confirming mailbox ownership can move. `emails aws +setup-inbound` writes no DNS at all — it prints the MX record for you to +publish. ## MCP Server @@ -349,10 +371,12 @@ emails-mcp # stdio transport (default) ## REST API -`emails serve` exposes the local dashboard and management API: +`emails serve` selects the server by deployment mode: -- **Dashboard / management API** under `/api/*` for providers, domains, - addresses, messages, stats, sources, and mailbox views. +- In local mode it exposes the static dashboard and its unauthenticated, + loopback-oriented management API under `/api/*` on `127.0.0.1:3900`. +- In `self_hosted` mode it exposes the authenticated PostgreSQL-backed `/v1` + service on `0.0.0.0:8080`; `/openapi.json` is the formal wire contract. - Scoped send keys remain part of the local send authorization model; there is no separate hosted-agent API surface in this OSS server. @@ -398,6 +422,18 @@ runInTransaction(db, () => { closeDatabase(); ``` +The provider factory and the DNS helpers are exported too. `providerDnsPublishing` +answers whether a provider type publishes DNS records at all, so an empty record +list renders as "nothing to publish here" rather than "nothing found" — a +`sandbox` provider captures mail locally and has no DKIM/SPF/DMARC of its own: + +```ts +import { getAdapter, providerDnsPublishing, formatDnsTable } from "@hasna/emails"; + +const records = await getAdapter(provider).getDnsRecords("example.com"); +console.log(formatDnsTable(records, providerDnsPublishing(provider))); +``` + ## Inbound Email (AWS SES -> S3) ```bash @@ -435,12 +471,20 @@ allowlists before it confirms or syncs a notification. ## Self-Hosted Runtime (PostgreSQL/S3/SES) -The server uses operator-owned Postgres and provider accounts. A client must configure `EMAILS_MODE=self_hosted`, `EMAILS_SELF_HOSTED_URL`, and `EMAILS_SELF_HOSTED_API_KEY`. The service requires `EMAILS_DATABASE_URL`, `EMAILS_API_SIGNING_KEY`, `EMAILS_SEND_PROVIDER=ses|resend`, `EMAILS_AUTH_ALLOWED_EMAIL_DOMAINS`, and `EMAILS_AUTH_FROM`. SES uses the deployment IAM role; Resend uses `RESEND_API_KEY`. +The server uses operator-owned Postgres and provider accounts. A client must +configure `EMAILS_MODE=self_hosted`, `EMAILS_SELF_HOSTED_URL`, and one of +`EMAILS_SESSION_TOKEN`, `EMAILS_IDP_TOKEN`, or +`EMAILS_SELF_HOSTED_API_KEY`. The service requires `EMAILS_DATABASE_URL`, +`EMAILS_API_SIGNING_KEY`, `EMAILS_SEND_PROVIDER=ses|resend`, +`EMAILS_AUTH_ALLOWED_EMAIL_DOMAINS`, and `EMAILS_AUTH_FROM`. SES uses the +deployment IAM role; Resend uses `RESEND_API_KEY`. See +[docs/AUTHENTICATION.md](docs/AUTHENTICATION.md) for signup, sessions, +tenant-scoped keys, and optional IdP verification. `EMAILS_AUTH_ALLOWED_EMAIL_DOMAINS` is the allowlist of email domains that may sign up, log in, or be invited (comma- or space-separated globs, `*` matching one DNS label — e.g. `example.com` or `example.*`), and `EMAILS_AUTH_FROM` is the sender identity for confirmation/reset/invite mail. **Neither has a default and the service refuses to boot without them**: this package ships no domain and no sender of its own, so a default would either lock your auth surface to someone else's organisation or open signup to everyone. See [docs/SELF_HOSTED_RUNTIME.md](docs/SELF_HOSTED_RUNTIME.md). -Self-hosted client commands fail closed when the mode, URL, or API key is -missing or invalid. With `--json`, the CLI emits one structured error object on +Self-hosted client commands fail closed when the mode, URL, or selected bearer +credential is missing or invalid. With `--json`, the CLI emits one structured error object on stderr, exits nonzero, and leaves stdout empty. It does not open, create, or fall back to the local SQLite database. diff --git a/bun.lock b/bun.lock index 685e8ed4..33dd6b22 100644 --- a/bun.lock +++ b/bun.lock @@ -13,7 +13,7 @@ "@aws-sdk/client-sqs": "^3.1072.0", "@aws-sdk/client-sts": "^3.1072.0", "@aws-sdk/credential-provider-ini": "^3.972.55", - "@hasna/contracts": "0.4.2", + "@hasna/contracts": "0.8.4", "@hasna/domains": "0.0.35", "@hasna/events": "^0.1.8", "@hasna/mcp-harness": "0.1.0", @@ -159,7 +159,7 @@ "@babel/types": ["@babel/types@7.29.7", "", { "dependencies": { "@babel/helper-string-parser": "^7.29.7", "@babel/helper-validator-identifier": "^7.29.7" } }, "sha512-4zBIxpPzowiZpusoFkyGVwakdRJUyuH5PxQ/PrqghfdFWWasvnCdPfQXHrenDai+gyLARulZjZowCOj6fjT4pA=="], - "@hasna/contracts": ["@hasna/contracts@0.4.2", "", { "dependencies": { "commander": "^13.1.0", "zod": "^3.25.76" }, "bin": { "contracts": "dist/cli/index.js", "contracts-cli": "dist/cli/index.js" } }, "sha512-xD7ZWQXR+AjYAJVpaMsQ+8V0SABiJSKVeI0B1WbPByYeGz6daDtgyZ6c3q0c2S6uzIEtUxlGHPundr675636ew=="], + "@hasna/contracts": ["@hasna/contracts@0.8.4", "", { "dependencies": { "commander": "^13.1.0", "zod": "^3.25.76" }, "bin": { "contracts": "dist/cli/index.js", "contracts-cli": "dist/cli/contracts-cli.js" } }, "sha512-a/flLzdu8cbCUrScNQT3VBMl2HgwbjUYWrmwQ45x19X6F8FPiOf2gbb43MT0vlKTLUPEonh5jSoEPdjp44nAGw=="], "@hasna/domains": ["@hasna/domains@0.0.35", "", { "dependencies": { "@aws-sdk/client-route-53": "^3.1067.0", "@aws-sdk/client-route-53-domains": "^3.1067.0", "@aws-sdk/credential-provider-ini": "3.972.53", "@hasna/contracts": "^0.5.2", "@modelcontextprotocol/sdk": "^1.29.0", "chalk": "^5.4.1", "commander": "^13.1.0", "ink": "^5.2.0", "ink-text-input": "^6.0.0", "pg": "^8.21.0", "react": "^18.3.1", "zod": "^3.24.2" }, "optionalDependencies": { "@hasna/events": "^0.1.7" }, "peerDependencies": { "@hasna/contacts": "^0.6.19" }, "optionalPeers": ["@hasna/contacts"], "bin": { "domains": "dist/cli/index.js", "domains-mcp": "dist/mcp/index.js", "domains-serve": "dist/server/index.js" } }, "sha512-926OuguAuUsTQ1YOXmbU2fhCFwDBfwIGqK0oVJQjffMJXV+rYC/4Aowj2HMgWjWiF/S+FW4oj/e3B89bE4YxhA=="], diff --git a/deploy/aws/tests/attachment_repair_runtime_report.ts b/deploy/aws/tests/attachment_repair_runtime_report.ts index 0212d12b..0f5d7172 100644 --- a/deploy/aws/tests/attachment_repair_runtime_report.ts +++ b/deploy/aws/tests/attachment_repair_runtime_report.ts @@ -91,7 +91,6 @@ export async function generateAttachmentRepairRuntimeReport( }; const deps: AttachmentRepairMaintenanceDeps = { env: { - EMAILS_MODE: "self_hosted", EMAILS_DATABASE_URL: "postgresql://redacted", EMAILS_INGEST_S3_BUCKET: "canonical-inbound", ECS_CONTAINER_METADATA_URI_V4: "http://169.254.170.2/v4/static-contract", diff --git a/deploy/aws/tests/static_contract.sh b/deploy/aws/tests/static_contract.sh index beff7651..739ee147 100755 --- a/deploy/aws/tests/static_contract.sh +++ b/deploy/aws/tests/static_contract.sh @@ -608,13 +608,6 @@ release_132_section="$( capture { print } ' "$changelog" )" -unreleased_section="$( - awk ' - /^## \[Unreleased\]$/ { capture = 1 } - capture && /^## / && $0 != "## [Unreleased]" { exit } - capture { print } - ' "$changelog" -)" expected_release_132_section='## 1.3.2 (2026-07-26) - fail closed on malformed JSON, wrong response envelopes, and missing required @@ -625,8 +618,6 @@ expected_release_132_section='## 1.3.2 (2026-07-26) asynchronous inbox data source, and generated `@hasna/emails/selfhost` client; validation errors identify the endpoint and invalid field without including credentials or response-body contents.' -expected_unreleased_sha256='40e9d4fc08e67cd4f7d38b053c5c9031dd3e8e403d68bc7e40f83a87bc00ba20' -actual_unreleased_sha256="$(printf '%s' "$unreleased_section" | sha256sum | awk '{ print $1 }')" unreleased_line="$(grep -Fn '## [Unreleased]' "$changelog" | cut -d: -f1)" release_132_line="$(grep -Fn '## 1.3.2 (2026-07-26)' "$changelog" | cut -d: -f1)" release_131_line="$(grep -Fn '## 1.3.1 (2026-07-26)' "$changelog" | cut -d: -f1)" @@ -637,9 +628,8 @@ if [ "$(grep -Fxc '## [Unreleased]' "$changelog" || true)" != "1" ] \ || [ -z "$release_131_line" ] \ || [ "$unreleased_line" -ge "$release_132_line" ] \ || [ "$release_132_line" -ge "$release_131_line" ] \ - || [ "$actual_unreleased_sha256" != "$expected_unreleased_sha256" ] \ || [ "$release_132_section" != "$expected_release_132_section" ]; then - echo "1.3.2 changelog must contain exactly its two release bullets below the full Unreleased section" >&2 + echo "1.3.2 changelog must preserve its exact frozen release section and ordering" >&2 exit 1 fi diff --git a/docs/AUTHENTICATION.md b/docs/AUTHENTICATION.md new file mode 100644 index 00000000..8c0d4f20 --- /dev/null +++ b/docs/AUTHENTICATION.md @@ -0,0 +1,86 @@ +# Self-hosted authentication + +Authentication applies to the operator-owned self-hosted `/v1` service. The +local SQLite dashboard has a separate loopback-oriented trust boundary and does +not use these accounts. + +## Client configuration + +A self-hosted client sets the service URL and one bearer credential: + +```bash +export EMAILS_SELF_HOSTED_URL="https://emails.example.com" +export EMAILS_SELF_HOSTED_API_KEY="..." # operator or tenant API key +``` + +Setting the service URL is what selects the hosted client store; there is no +separate deployment-mode variable. + +The accepted credential variables, in precedence order, are: + +1. `EMAILS_SESSION_TOKEN` — an opaque user session created by `emails auth login`; +2. `EMAILS_IDP_TOKEN` — an access token from the configured identity provider; +3. `EMAILS_SELF_HOSTED_API_KEY` — an HMAC application or tenant API key. + +`EMAILS_CLIENT_ENV_SECRET` may point to a `secrets` vault entry containing +`EMAILS_SELF_HOSTED_URL` and any one of those credentials. +`emails auth login`, `logout`, and `switch-tenant` update that entry when the +pointer is configured. Without it, a login token exists only in the current CLI +process and is not durable across later invocations. + +## User and organization commands + +```bash +emails auth signup --email owner@example.com --tenant-name Example +emails auth verify-email +emails auth verify-email --resend --email owner@example.com +emails auth login --email owner@example.com +emails auth whoami +emails auth switch-tenant another-org +emails auth logout +``` + +Passwords are prompted without echo when omitted. In non-interactive use they +must be provided as options, so prefer a protected execution environment and do +not place them in shell history or source files. + +The service permits signup, login, and invitations only for domains matching +`EMAILS_AUTH_ALLOWED_EMAIL_DOMAINS`. Signup requires email verification using +mail sent from `EMAILS_AUTH_FROM`; both service variables are required and have +no built-in defaults. + +`emails auth bootstrap` is the one-time primary-owner path. It requires the +operator API key selected by the server's paired bootstrap email/KID settings; +matching the email alone is not authorization. + +## Two API-key scopes + +These similarly named commands manage different key classes: + +- `emails self-hosted key create/list/rotate/revoke` runs on the operator host + against Postgres and manages application keys used to establish the service. +- `emails keys create/list/revoke` calls `/v1/keys` as an owner/admin session and + manages tenant-scoped keys for the active organization. A new plaintext token + is shown once; lists never return tokens or hashes. + +Scoped send keys under `emails sendkey` are a third, narrower authorization +mechanism: they bind an owner/agent to allowed From addresses rather than +authenticating a general `/v1` client. + +## Optional IdP tokens + +Migration `0021_idp_principal_tenants` adds the resolution mapping for IdP +principals. When the service sets `EMAILS_IDP_JWKS_URL`, it can verify Ed25519 +access tokens with audience `emails`, map the token subject to an Emails tenant, +and enforce `emails:read`, `emails:write`, or `emails:*` scopes. The client puts +that token in `EMAILS_IDP_TOKEN`. + +This is currently the verifier slice only: no `emails auth idp map/list/revoke` +commands are shipped. An unmapped or locally revoked principal fails closed, +and an unset JWKS URL refuses the IdP credential class. IdP-side revocation +stops new tokens, while an already issued token remains valid until expiry +unless its Emails mapping is revoked. + +See [ADR-0001](adr/0001-adopt-tenants-idp-for-identity.md) and +[ADR-0002](adr/0002-agent-identity-signup-and-scopes.md) for the accepted target +architecture and the unimplemented later phases. diff --git a/docs/CLI.md b/docs/CLI.md new file mode 100644 index 00000000..70ac7a28 --- /dev/null +++ b/docs/CLI.md @@ -0,0 +1,102 @@ +# CLI reference + +This page describes the command tree shipped by `@hasna/emails` 1.3.3. It was +checked against the live `--help` output in both `local` and `self_hosted` +modes. Use `emails --help` for every option and argument; Commander +help is the option-level source of truth. + +## Global options + +`emails` accepts `--json`, `--quiet`, `--verbose`, `--version`, and `--help`. +With `--json`, successful structured output is written to stdout and structured +errors are written to stderr. + +## Root command tree + +| Root command | Subcommands or purpose | +| --- | --- | +| `provider` | `add`, `list`, `remove`, `update`, `status`, `check`, `sync` | +| `domain` | `add`, `connect`, `adopt`, `list`, `dns`, `verify`, `status`, `usable`, `move-provider`, `remove`, `check`, `setup-cloudflare`, warming commands, `available`, `buy`, `purchase-status`, `list-registered`, `setup` | +| `domains` | `list`, `status`, `add`, `connect`, `dns`, `verify`, `check`, `enable-inbound`, `enable-outbound`, `disable-outbound` | +| `address` | `add`, `list`, `owner`, ownership changes/history, `suggest`, `provision`, `verify`, `set-verified`, `remove`, `suspend`, `activate`, `quota` | +| `send` | Send one message; supports templates, attachments, scheduling, tracking, and idempotency options where the selected store supports them. | +| `email` | `list`, `search`, `show`, `replies`, `thread`, `send` | +| `webhook` | `listen` for provider event webhooks. | +| `template` | `add`, `list`, `show`, `remove` | +| `contact` / `contacts` | `list`, `suppress`, `unsuppress` | +| `group` | `create`, `list`, `show`, `members`, `add`, `remove-member`, `delete` | +| `sequence` | `create`, `list`, `show`, `pause`, `archive`, enrollment commands, and `step add/list/remove` | +| `schedule` / `scheduled` | `list`, `cancel`, `run` | +| `inbox` | Code waiting, list/search/read, mailbox/source status, state changes, attachments, deletion, S3 sync, realtime setup/watch, SMTP listen, and local open. | +| `owner` | `register`, `list`, `addresses` | +| `alias` | `add`, `catch-all`, `global`, `list`, `remove`, `resolve` | +| `sendkey` | `create`, `list`, `revoke`, `check` | +| `send-intent` | `uncertain`, `reconcile` | +| `forwarding` | `add`, `list`, `enable`, `disable`, `remove`, `run`, `explain` | +| `aws` | `setup-inbound`, `status` | +| `agent` | `context` | +| `daemon` | `status`, `restart` | +| `logs` | `tail` | +| `db` | `migrate`, `status` for the self-hosted Postgres schema. | +| `self-hosted` | `key create/list/rotate/revoke` for operator application keys. | +| `auth` | `signup`, `login`, `logout`, `whoami`, `switch-tenant`, `verify-email`, `bootstrap` | +| `keys` | `list`, `create`, `revoke` tenant-scoped API keys. | +| `ui` | Start the full-screen OpenTUI client. | +| `serve` | Start the local dashboard or self-hosted service selected by mode. | +| `mcp` | Print or install MCP configuration for Claude Code, Codex, or Gemini. | +| `remove` / `uninstall` | Remove MCP configuration from supported agent clients. | +| `status` | Redacted health and next actions. | +| `stats`, `analytics`, `monitor` | Delivery statistics and monitoring. | +| `doctor` | Diagnostics; `doctor delivery
` diagnoses missing inbound mail. | +| `provision` | Registered compatibility namespace; intentionally not implemented (see below). | + +Standalone aliases are also shipped for common actions: `addresses`, `log`, +`search`, `show`, `replies`, `conversation`, `test`, `export`, `pull`, +`preview`, `scheduler`, `batch`, `completion`, `verify-email`, `code`, `links`, +`forward`, `reply`, and `whoami`. + +## Commands that intentionally refuse + +Registration in help does not imply implementation. The following compatibility +and design-target commands fail with an actionable "not implemented in this +build" error in every deployment mode: + +- every `emails provision *` subcommand; +- `emails domain connect`, `verify`, `status`, `setup-cloudflare`, and `setup`; +- `emails domains connect`, `verify`, `enable-inbound`, `enable-outbound`, and + `disable-outbound`; +- `emails address provision`. + +Use `emails domains status` for stored domain state, `emails domain check` for +live public DNS, `emails domain adopt` for an already verified provider domain, +and `emails aws setup-inbound` for SES/S3 inbound wiring. + +## Mode differences + +The root command names are the same in both modes, but storage and capability +checks may refuse an operation that the selected store cannot perform. +`emails inbox attachments` (cursor-based attachment inventory) is present only +for the self-hosted client; `emails inbox attachment ` exists in both +modes. `emails serve` defaults to the local dashboard at `127.0.0.1:3900` in +local mode and the self-hosted `/v1` service at `0.0.0.0:8080` in +`self_hosted` mode. + +## Other shipped bins + +`emails-mcp` uses stdio by default. `--http` opts into Streamable HTTP, +`-p/--port` selects the port, and HTTP refuses to start without +`EMAILS_MCP_HTTP_TOKEN`. `--stdio`, `--version`, and `--help` are also +available. + +`emails-serve` starts the same mode-selected HTTP service and also ships these +operator commands: + +- `ingest-worker` +- `ingest-s3-backfill` +- `attachment-repair-canary` +- `attachment-repair-ledger` +- `inbound-provenance-audit` +- `inbound-provenance-fence` + +Run `emails-serve --help` before an operator workflow; these commands have +strict environment, provenance, and argument requirements. diff --git a/docs/DEPLOYMENT_CUTOVER.md b/docs/DEPLOYMENT_CUTOVER.md index a37b19ef..1d6b0111 100644 --- a/docs/DEPLOYMENT_CUTOVER.md +++ b/docs/DEPLOYMENT_CUTOVER.md @@ -3,13 +3,20 @@ This repository intentionally has no automatic deployment workflow. Merging or tagging the repository cannot publish a package, push an image, or update AWS. +Any future release or deployment workflow is repository-policy gated by +`scripts/deployment-workflow-policy.mjs`. Before its first publishing or +deployment mutation it must run the immutable candidate gate documented in +`docs/IMMUTABLE_DEPLOYMENT_GATE.md`; a separate mutating job must have a normal +`needs: immutable-deployment-gate` dependency. CI rejects a missing, bypassable, +or incomplete gate before that workflow can merge. + The fast-uri quarantine is resolved: the eligible security update cleared the full seven-day managed quarantine window on 2026-07-26 10:42:54.497 Europe/Bucharest and is pinned in the current package manifest and lockfile. Future third-party dependency changes remain subject to the managed quarantine process. Before a future `workflow_dispatch` deployment is introduced, an operator must -provide a Mailery-owned infrastructure manifest and least-privilege role in the +provide an operator-owned infrastructure manifest and least-privilege role in the target user's AWS account. The workflow must use `APP=emails`, require an explicit environment approval, and must not contain a Hasna account ID, bucket, cluster, database URL, secret path, or default endpoint. diff --git a/docs/DOMAIN_READINESS.md b/docs/DOMAIN_READINESS.md index efdf0889..30cdfad8 100644 --- a/docs/DOMAIN_READINESS.md +++ b/docs/DOMAIN_READINESS.md @@ -12,17 +12,29 @@ not supported as provider backends. A sending domain is ready only after ownership, DKIM and SPF evidence is valid. Inbound readiness additionally requires an active provider route and durable -source such as SES to S3/SQS. DNS mutations require an explicit plan or dry run; -Emails never purchases a domain or changes MX records implicitly. +source such as SES to S3/SQS. -Useful checks: +No shipped command publishes DNS. `emails domain dns` prints the records to +publish and `emails aws setup-inbound` prints the MX record it needs, both for +you to apply at your DNS provider; nothing purchases a domain or changes MX +implicitly. (`emails domain buy` purchases explicitly, and `emails domain adopt` +refuses to wire SES inbound when public root MX belongs to another provider +unless `--force-mx-switch` is passed.) + +Useful checks — these run in every configuration, because they resolve public +DNS and need no server: ```bash -emails domain check example.com -emails provision domain example.com --provider --dry-run -emails domain verify example.com +emails domain dns example.com --provider # records the domain must publish +emails domain check example.com # what is actually published, plus root-MX owner ``` +`emails domain verify`, `emails domain status`, `emails domains connect`, +`emails domains enable-inbound|enable-outbound|disable-outbound` and +`emails provision *` are NOT implemented in this build. Running any of them +prints what is missing and which command to use instead. + Self-hosted API clients must explicitly configure `EMAILS_MODE=self_hosted`, -`EMAILS_SELF_HOSTED_URL`, and `EMAILS_SELF_HOSTED_API_KEY`. No endpoint, account, -database, bucket or secret path is supplied by the package. +`EMAILS_SELF_HOSTED_URL`, and one of `EMAILS_SESSION_TOKEN`, `EMAILS_IDP_TOKEN`, +or `EMAILS_SELF_HOSTED_API_KEY`. No endpoint, account, database, bucket or secret +path is supplied by the package. diff --git a/docs/EMAILS-UI-CLI-ROADMAP.md b/docs/EMAILS-UI-CLI-ROADMAP.md index 3a85f708..de6a27fd 100644 --- a/docs/EMAILS-UI-CLI-ROADMAP.md +++ b/docs/EMAILS-UI-CLI-ROADMAP.md @@ -1,19 +1,27 @@ -# Emails UI + CLI Roadmap +# Archived Emails UI + CLI roadmap Created: 2026-06-18 -## Baseline +Status: completed historical work log for the 0.6.47–0.6.49 releases. Paths, +package versions, test counts, and smoke commands below describe those releases +and are not a current CLI or build contract. In particular, the current UI is +under `src/cli/tui-solid/`, the React-era `src/cli/tui/App.tsx` is gone, and the +old `emails sandbox` namespace is gone. See [opentui-ui-spike.md](opentui-ui-spike.md) +and [CLI.md](CLI.md) for the current implementation. + +## Original baseline - `EMAILS_DB_PATH=:memory: bun test` passes: 1559 tests, 0 failures. -- Main terminal UI is `src/cli/tui-solid/App.tsx`, loaded by `emails ui`. -- Legacy `src/cli/tui/App.tsx` re-exports the Solid/OpenTUI app. +- Main terminal UI was being moved to `src/cli/tui-solid/App.tsx`, loaded by + `emails ui`; the compatibility component named later in this log no longer + exists. - Dashboard frontend is one static file: `dashboard/index.html`. - Dashboard API routes are split under `src/server/routes/`. - Existing inbound data already stores attachment metadata and paths. - Existing TUI reader already renders markdown/html into readable terminal text. - Existing stored summaries from previous versions are readable from `email_agent_runs` / `email_triage` and preferred in `getMessageBody`. -## Findings +## Original findings - Branding still says `Emails`/`Open Emails Dashboard` in the web dashboard, while TUI uses `Emails` in a secondary line under the selected inbox. - TUI has `Search`, but not a broader compact filter dialog for address/read/star/label/sort. diff --git a/docs/FEATURE-CONVENTIONS.md b/docs/FEATURE-CONVENTIONS.md index a5dc837f..667775f5 100644 --- a/docs/FEATURE-CONVENTIONS.md +++ b/docs/FEATURE-CONVENTIONS.md @@ -1,25 +1,33 @@ # Feature conventions for agents Use this checklist when adding behavior to `@hasna/emails`. The project is a -CLI, MCP server, REST dashboard API, and public library over the same local -SQLite store, so new behavior should land in the right layer and get regression -coverage there. +CLI, MCP server, local dashboard API, self-hosted `/v1` service, and public +library. Storage behavior may need both the local SQLite store and the HTTP/ +Postgres path, so new behavior should land at the shared store seam where +possible and get regression coverage at each exposed layer. ## DB-backed feature -1. Add schema in `src/db/database.ts`. -2. Add idempotent `ensureSchema` coverage for the same table, columns, and - indexes. -3. Add CRUD helpers in `src/db/.ts`. -4. Add focused tests in `src/db/.test.ts`. -5. Use `EMAILS_DB_PATH=:memory:` or a temp DB path in tests. +1. Add local schema in `src/db/database.ts` and idempotent `ensureSchema` + coverage for the same table, columns, and indexes. +2. If the feature is self-hosted, add an immutable migration in + `src/server/self-hosted/migrations.ts` and tenant/RLS coverage where the row + is tenant-scoped. +3. Put shared operations on the relevant `src/store/` repository and implement + them for SQLite and HTTP, or document a typed capability refusal. +4. Add focused DB/store tests and extend the shared conformance suite when the + operation belongs to the store contract. +5. Use `EMAILS_DB_PATH=:memory:` or a temp DB path for local tests; real + Postgres suites are gated by `EMAILS_TEST_POSTGRES_URL`. Regression example: ownership lives in `src/db/owners.ts`, `src/lib/address-ownership.ts`, and `src/db/owners.test.ts`. ## CLI command -1. Register in the nearest `src/cli/commands/*.ts` module. +1. Register in the nearest `src/cli/commands/*.ts` module. If behavior differs + by selected store, keep the shared command shape stable and route the + implementation through the existing local/remote facade. 2. Return structured data through the shared `output(data, formatted)` callback whenever practical. 3. If a command still logs directly, `--json` must stay parseable through the @@ -33,7 +41,8 @@ Regression examples: `src/cli/cli-contract.test.ts`, ## MCP tool -1. Register in `src/mcp/tools/*.ts`. +1. Register in `src/mcp/tools/*.ts` and wire any new registrar from + `src/mcp/server.ts`. 2. Return JSON text, not human-only prose, for agent-facing results. 3. Let the MCP contract wrapper add `cli_equivalent` and structured errors. 4. Add HTTP transport tests for high-use tools, not only direct helper tests. @@ -43,7 +52,9 @@ Regression examples: `src/mcp/http.test.ts` and ## REST endpoint -1. Add routes in `src/server/routes/*.ts`, keeping `serve.ts` thin. +1. Add local dashboard routes in `src/server/routes/*.ts`, keeping `serve.ts` + thin. Add self-hosted `/v1` behavior to the service/store and its OpenAPI + contract in `src/server/self-hosted/`. 2. Redact provider credentials before returning provider-shaped objects. 3. Prefer route-dispatcher tests for fast API parity coverage. @@ -62,7 +73,7 @@ Before publishing a release: ```bash bun run build -EMAILS_DB_PATH=:memory: bun test +bun run test npm pack --dry-run ``` diff --git a/docs/IMMUTABLE_DEPLOYMENT_GATE.md b/docs/IMMUTABLE_DEPLOYMENT_GATE.md new file mode 100644 index 00000000..9f283845 --- /dev/null +++ b/docs/IMMUTABLE_DEPLOYMENT_GATE.md @@ -0,0 +1,143 @@ +# Immutable deployment gate + +This gate is mandatory for any future package release or deployment workflow. +The repository still contains no automatic publishing or deployment workflow; +adding one does not weaken this prerequisite. `bun run deployment:policy` scans +every workflow and fails CI unless each mutating job either runs the gate before +the mutation or normally depends on a job named `immutable-deployment-gate`. +`continue-on-error`, `always()`, late gates, and suppressed gate exits are +rejected. + +Run the gate from an exact source checkout after the candidate package has been +published and the candidate image has been pushed by a separately approved +process: + +```bash +bun scripts/immutable-deployment-gate.mjs run +``` + +The runner requires these public bindings: + +- `DEPLOYMENT_GATE_CONFIG` — reviewed JSON config described below. +- `DEPLOYMENT_GATE_EVIDENCE` — a new output path. An existing path is refused; + the output is created mode 0600 without overwrite. +- `DEPLOYMENT_GATE_EXECUTION_ID` — unique workflow run/attempt identity, such as + `${{ github.run_id }}:${{ github.run_attempt }}`. Evidence cannot be replayed + under another execution identity. +- `DEPLOYMENT_GATE_CANDIDATE_SHA`, `DEPLOYMENT_GATE_PACKAGE_VERSION`, + `DEPLOYMENT_GATE_PACKAGE_INTEGRITY`, and + `DEPLOYMENT_GATE_IMAGE_REFERENCE`. +- The equivalent four `DEPLOYMENT_GATE_ROLLBACK_*` values for the reviewed + schema-compatible rollback release. + +It also requires masked workflow secrets: + +- `EMAILS_GATE_API_KEY` — read key for the dedicated synthetic fixture tenant. +- `EMAILS_GATE_OTHER_TENANT_API_KEY` — a valid key for a different tenant. +- `EMAILS_GATE_DATABASE_URL` — source database used for the migration/restore + drill. +- `EMAILS_GATE_RESTORE_ADMIN_URL` — maintenance connection allowed to create and + drop only the named isolated restore database. +- `EMAILS_GATE_RESTORE_URL` — connection to that isolated restore database. + +The config has this exact shape. Values below are placeholders, not defaults: + +```json +{ + "schema_version": 1, + "candidate": { + "package_name": "@hasna/emails", + "package_version": "", + "package_integrity": "sha512-", + "source_sha": "<40-lowercase-hex>", + "image_reference": "@sha256:<64-lowercase-hex>" + }, + "rollback": { + "package_name": "@hasna/emails", + "package_version": "", + "package_integrity": "sha512-", + "source_sha": "<40-lowercase-hex>", + "image_reference": "@sha256:<64-lowercase-hex>" + }, + "target": { + "base_url": "https://" + }, + "database": { + "restore_database": "emails_deployment_gate_" + }, + "latency": { + "max_probe_ms": 5000 + }, + "evidence": { + "max_age_seconds": 900 + }, + "fixture": { + "classification": "synthetic_designated_test", + "production_data": false, + "message_body_disclosure": false, + "recipient": "deployment-gate@example.test", + "search_token": "", + "ordered_message_ids": ["", ""], + "content": { + "message_id": "", + "field": "text_body", + "sha256": "" + }, + "attachments": [ + { + "message_id": "", + "index": 0, + "filename": "gate-one.txt", + "sha256": "" + }, + { + "message_id": "", + "index": 0, + "filename": "gate-two.txt", + "sha256": "" + } + ] + } +} +``` + +The runner rejects extra config and evidence fields. It verifies registry +metadata for the exact package version, including exact npm `gitHead` and +integrity, then installs both candidate and rollback packages into fresh private +prefixes. It pulls both immutable images and checks their registry digest and +OCI revision/version labels. + +The clean-installed candidate CLI must then pass self-hosted status, list, +search, exact read and body hash, offset pagination, cursor-based attachment +inventory, authenticated attachment download and byte hash, unauthenticated +denial, cross-tenant HTTP and CLI denial, and the configured per-probe latency +budget. The process receives an empty HOME plus a poison `EMAILS_DB_PATH`; status +must say `self_hosted`, expose no local data directory, and leave the poison +directory untouched. This makes a silent ambient SQLite fallback a hard failure. + +For database compatibility, the gate takes a private custom-format backup, +hashes it, restores it into the dedicated prefixed database, compares aggregate +data and exact migration-ledger fingerprints, applies candidate migrations, and +requires no pending migrations. The clean-installed rollback CLI must accept +that migrated ledger with no pending migration. Finally the original backup is +restored again and both fingerprints must match the source. The temporary dump, +downloaded attachment, clean-install homes, and restore database are removed at +the end. Production backup retention remains a separate cutover requirement. + +Evidence contains only immutable package/image bindings, pass/fail check names, +latencies, aggregate fixture counts, and hashes. It never contains API keys, +database URLs, tenant identifiers, message identifiers, subjects, recipients, +message bodies, or attachment bytes. The gate accepts only synthetic designated +messages addressed under the reserved `.test` domain; it cannot be pointed at +ordinary production mail to manufacture body-disclosure evidence. + +Immediately before the mutating step, evidence can be rechecked with the same +reviewed environment: + +```bash +bun scripts/immutable-deployment-gate.mjs verify +``` + +Verification rejects stale evidence, another run identity, candidate or rollback +SHA/package/integrity drift, image-digest drift, any missing/skipped check, +latency over budget, disclosure flags, or backup/restore fingerprint drift. diff --git a/docs/NAMING.md b/docs/NAMING.md new file mode 100644 index 00000000..ed37234c --- /dev/null +++ b/docs/NAMING.md @@ -0,0 +1,143 @@ +# Naming: this product is open-emails, not Mailery + +Status: binding. Owner ruling by Andrei Hasna, 2026-07-27. + +## The ruling + +The owner's verbatim wording is recorded in the Knowledge entry +`k_ms2x7nlw_ek0nrh` (tag `convention`), not quoted here: it uses the commercial +vocabulary that this package's own boundary guard bans from every tracked file +(`scripts/no-cloud-scan-lib.mjs`, "hosted implementation vocabulary"). Quoting +it verbatim in this file would turn CI red. That is the guard working, not a +problem to route around. + +In substance: Mailery is retired as a brand name for this product, which is +renamed to open-emails; Mailery becomes a different, unrelated commercial +product. + +Stated as rules: + +1. The OSS email core is **open-emails**. It is the repository + `hasna/emails`, the npm package `@hasna/emails`, and the `emails`, + `emails-mcp`, and `emails-serve` bins. +2. **`@hasna/mailery` is a legacy package name.** It is the abandoned 0.6.x + line (0.6.20-0.6.116, last published 2026-07-08). It is not this package + and must never be revived by publishing this tree under that name. +3. **Mailery is reserved for a separate, unrelated future commercial product** + and **must not be used to refer to the email product** — not in docs, not in + code, not in commit messages, not in issue titles, not in conversation. + +`open-emails` is the product/brand name. It does not change the repository or +package name: per the org-wide OSS naming convention the GitHub repo is +`hasna/` and the npm package is `@hasna/` with no `open-` prefix, +while `open-` is the brand and the local workspace folder name. So +`open-emails` -> `hasna/emails` -> `@hasna/emails` is already correct and +consistent, and nothing about this ruling requires renaming either. + +## What this ruling does NOT ask you to do + +The rename it describes has already happened — but only just, and not +smoothly. An earlier version of this file claimed it "landed on 2026-07-11 in +PR #21" and had been settled since. That was wrong on both counts. The +published package name changed six times: + +| Commit | Date | Name | +| --- | --- | --- | +| `f303797` | 2026-03-12 | `@hasna/emails` | +| `9a28e6a` | 2026-06-17 | `@hasna/mailery` | +| `8eac4ed` | 2026-07-09 | `@hasna/emails` (PR #21) | +| `a14a792` | 2026-07-13 | an `@hasnaxyz`-scoped variant (itself banned as a typo-squat) | +| `39e292f` | 2026-07-14 | `@hasna/emails` | +| `8b5c7d0` | 2026-07-21 | `@hasna/mailery` | +| `80faf15` | 2026-07-25 | `@hasna/emails` | + +`80faf15` is the operative rename, and it is two days older than the ruling. +So the owner ruling is not restating a long-settled fact — it is closing a +question that genuinely kept reopening. Treat any instruction to rename this +package *to* `@hasna/mailery` as superseded by the ruling, whatever its date. + +The word "mailery" still appears in this tree, and **almost all of those +occurrences must stay**. They fall into three groups, none of which is +branding: + +### 1. Guards that enforce this ruling (the large majority) + +`src/package-identity.test.ts` pins the package name, repository URL, and bin +list, and asserts CI checks them. `src/no-cloud-boundary.test.ts` plus +`scripts/no-cloud-scan-lib.mjs` scan every tracked file for banned cloud and +Mailery patterns. `src/lib/mode.test.ts` proves the banned env selectors are +rejected. + +These files contain the string "mailery" **because their job is to keep it +out**. Deleting the word from them deletes the enforcement. A "clean up all +mailery references" pass over this repo would silently disarm the mechanism +that makes the ruling true. + +### 2. Compatibility with production state that already exists + +Renaming any of these breaks deployed systems, so they are frozen: + +| Reference | Where | What breaks if renamed | +| --- | --- | --- | +| Migration IDs `0001_mailery_selfhosted_core` .. `0005_mailery_selfhosted_resources` | `src/server/self-hosted/migrations.ts` | Migration IDs are immutable ledger entries. Renaming them makes deployed databases re-run applied migrations. | +| API key alias slug `"mailery"` | `hasna.contract.json` (`apiKeyAppAliases`), `src/server/self-hosted/{keys,api-key-verifier,serve}.ts` | API keys already minted under the `mailery` slug stop verifying — this revokes live credentials. | +| `legacy-inbound@local.mailery` | `src/db/database.ts` | A retired synthetic inbound identity present in deployed SQLite databases. Renaming orphans real rows. | +| `LEGACY_MAILERY_EVENT_SOURCE` / `mailery.v1` | `src/lib/emails-events.ts` | Wire compatibility for events emitted by older versions. | +| `mailery_mode` config key migration | `src/lib/config.ts` | The forward-migration path for existing on-disk config. | + +### 3. Rejected input, deliberately named + +`MAILERY_*` and `HASNA_MAILERY_*` environment variables are **banned +selectors**, not aliases — `src/lib/mode.ts` and +`src/server/self-hosted/env.ts` reject them loudly rather than honoring them. +The canonical prefix is `EMAILS_*`. The names must remain listed in order to be +refused with a useful error. + +## Mailery is a different product + +`hasnatools/platform-mailery` (private, `mailery.co`) and `hasnatools/mailery` +(public, issues-only) belong to that separate product, and they keep the name. +That separation is enforced from both sides: at `platform-mailery` GitHub main +`7ff7ca4` its only `@hasna/*` dependencies are `@hasna/domains` and +`@hasna/feedback`, `docs/COMMERCIAL_CONTRACT.md` names `@hasna/emails` a +non-dependency, and `src/contracts/commercial-contract.test.ts` asserts it is +absent; this repo in turn stays cloud-free with the `mailery*` bin names left +free for that product's CLI. + +**Check that against GitHub, not a local checkout.** The on-disk clone at +`~/workspace/hasnatools/platform/platform-mailery` is dozens of commits stale, +predates the contract, and still carries `"@hasna/mailery": "0.6.93"`. An +adversarial reviewer read it and concluded — wrongly — that the two products +are coupled. + +The practical consequence for this repo: **`@hasna/emails` has no hosted +counterpart.** Its deployment modes are operator-owned. Do not describe +`mailery.co` as the hosted version of this package, and do not add a client for +it here. + +This supersedes, for this product only, the `mailery.co` example in the +deployment doctrine (knowledge `k_mryqb555_2osk2w`), which cites it as the +`platform-` wrapper of an OSS product. The doctrine's two-mode model +stands; its choice of example does not, because the ruling is newer. + +## If you are an agent about to "fix" a naming inconsistency here + +Read this file first, then check whether the string you are about to change is +a guard, a compatibility constant, or a banned-input name. If it is any of the +three, leave it alone. + +Outside those three groups there are exactly three remaining prose uses, all +reviewed and all deliberately kept, because each describes a historical fact +rather than naming the product: + +- `README.md` and `docs/SELF_HOSTED_RUNTIME.md` — "the active Mailery-era key", + meaning the key minted under the `mailery` alias slug. Renaming the phrase + would obscure which key an operator has to rotate. +- `docs/DEPLOYMENT_CUTOVER.md` — "released Mailery migration ids/checksums", + which is literally what those frozen IDs are. + +A fourth ("a Mailery-owned infrastructure manifest") was genuine residue and is +fixed; infrastructure here is operator-owned, never Mailery's. + +Renaming a published package, a repository, or a deployed resource requires +explicit owner approval for that specific step. diff --git a/docs/PLAN-MODE-REMOVAL.md b/docs/PLAN-MODE-REMOVAL.md new file mode 100644 index 00000000..93d4248d --- /dev/null +++ b/docs/PLAN-MODE-REMOVAL.md @@ -0,0 +1,362 @@ +# PLAN — Deleting the deployment-mode axis (open-emails) + +> Status: IN PROGRESS — phases 1–3 have landed and phase 4 is actively collapsing +> repository families (verified 2026-07-29). The live ratchet currently reports +> 14 two-arm families; use its test output rather than the baseline counts below. +> Owner: agents. Gates: `src/mode-axis-ratchet.test.ts`, `src/store-seam.test.ts`. +> Every number in this file was measured on `6646cc8` (`origin/main`) with Bun 1.3.14. Re-measure +> before trusting one; the commands are in §10. +> +> **A NOTE ON SPELLING, and it is load-bearing.** This file is inside the ratchet's scanned corpus, +> and six of its eleven counters match plain text in *any* tracked file — docs included. Writing the +> deployment-mode variable, the mode predicate, the mode reader/resolver/parser or the resource-gate +> call *with its parentheses* would raise a ceiling and turn CI red for the document describing the +> deletion. So this plan names those identifiers by role and by `file:line` instead of by spelling, +> exactly as `src/mode-axis-ratchet.test.ts` does with its own metric keys. Do not "fix" it. + +## 1. Goal + +Delete the deployment-mode axis outright. Not deprecate it, not hide it behind a default, not keep a +compatibility shim: **remove the variable, the two-arm modules, the dispatch layer and every branch +that reads them**, and leave no dead code behind. + +The axis is a forbidden "switch" under the adopted OSS boundary policy, whose model is **one +deployment story — "you run it"**. Where the data physically lives is a deployment detail of *your* +installation, not a product variant with two personalities, two code paths and two truths. + +## 2. The binding constraint — exactly TWO client stores, forever + +| Store | What it is | Status | +|---|---|---| +| `SqliteEmailStore` | The local, default, on-disk SQLite database. | to build (§6, phase 1) | +| `HttpEmailStore` | A client of an Emails API. | to build (§6, phase 2) | + +**There is no third, and Postgres is not a candidate.** Postgres is the *server's internal storage* +and is reached only through the API — **never a client transport**. An earlier design proposed a +client-side Postgres store as its central idea. That design is **rejected**. If a caller means +"local", it means SQLite; if it means "somewhere else", it means the API. + +This is stated here, in `src/store/email-store.ts:8-12`, and enforced structurally by the seam guard, +because a third client store *is* the deployment-mode axis growing back under a new name. Anyone who +finds themselves adding one has mis-read this plan. + +## 3. Why halfway is not an option + +The one-line version, and the strongest argument in the program: + +> The deployment-mode variable set to `self_hosted` means **opposite things in two of the three +> shipped binaries.** In the `emails` CLI it means *"become an HTTP client"*. In `emails-serve` it +> means *"become a Postgres server"*. (`emails-mcp` reads it too, and inherits the CLI's meaning.) + +Two definitions of the same predicate exist today and prove it: + +| Definition | File | Question it answers | +|---|---|---| +| client | `src/db/self-hosted-store.ts:170` | "am I an HTTP client?" | +| server | `src/server/self-hosted/env.ts:52` | "am I a Postgres server?" | + +One variable, two contradictory semantics. No amount of documentation fixes that; renaming it moves +the contradiction; defaulting it hides the contradiction. Only deletion resolves it. + +## 4. Baseline state on `6646cc8` — what existed and what it measured + +### 4.1 The ratchet (`src/mode-axis-ratchet.test.ts`) + +Eleven metrics, each pinned as a **ceiling that may only decrease** (`<=`, never `===`). Lowering one +is the point of the program and needs no argument; raising one must be argued in review. The table +below is the historical measurement on `6646cc8`, when every metric sat exactly +at its ceiling and no phase had landed. It is not the current tree's count: + +| Metric | Ceiling = today | What it counts | +|---|---|---| +| `twoArmFamilies` | 43 | facades with two or more implementation arms — **structural, no identifier** | +| `remoteArmModules` | 43 | `src/**/*.remote.*` HTTP-arm modules | +| `routedFacadeDefinitions` | 30 | definitions of the dispatch helper — in 30 of the 43 facades; the other 13 dispatch without it | +| `routedCallExpressions` | 293 | dispatch call sites — exports whose implementation is picked at runtime | +| `selfHostedResourceBranches` | 47 | mode-gated resource-gate branches inside `*.local.*` | +| `selfHostedResourceReferences` | 203 | the same gate anywhere in the tree (superset of the above) | +| `isSelfHostedModeReferences` | 70 | the mode predicate — **both** definitions from §3 | +| `getEmailsModeReferences` | 78 | the process-wide mode read | +| `resolveEmailsModeReferences` | 74 | mode resolution, both variants | +| `normalizeEmailsModeReferences` | 16 | the parser that admits the two mode values | +| `emailsModeEnvReferences` | 242 | the variable itself, tree-wide: TypeScript, docs, Terraform, compose | + +Corpus on `6646cc8`: **638 tracked files, 637 scanned, 7,631,658 characters** (adding this file makes +it 639/638 and moves no metric — the spelling discipline above is what buys that); floors are 500 files +and 5,000,000 characters, and they live inside the scan function so no single test can be run over a +corpus small enough to satisfy `<=` trivially. **228 test files hold 169 of the 242 variable +references** — so a mass test deletion is the obvious fake reduction, and the floors exist to make it +fail. (The prose *inside* the ratchet still quotes 624/623/~7.4M and 226/159 from when it was +written. Those are comments, not assertions; the ceilings are what is enforced.) + +**Ten of the eleven counters are keyed on names.** A mechanical rename of the dispatch helper, the +variable and the arm-file suffix drives all ten to zero with all 43 families, 293 dispatch sites and +242 references still standing. `twoArmFamilies` is the one computed from file structure, which is why +it is listed first and why the deletion PR must be **read**, not merely measured. + +### 4.2 The seam (`src/store/`) — types only at the baseline + +At the baseline it was declared but not implemented. The current tree contains +both SQLite and HTTP stores plus the shared conformance suite described in +phases 1–3. Historical baseline figures follow: + +| Fact | Value on `6646cc8` | +|---|---| +| Repositories on `EmailStore` | 25 (23 mapped 1:1 onto `src/db/*` families + `sendIntents`, `attachmentRepair`) | +| Declared operations | **70**, and every one returns `Promise>` | +| Capability-gated operations | 27, across 7 capability keys | +| Ungated operations | 43 — still returning `Outcome`, see §5 | +| Conformance cases shipped | **0** (`CONFORMANCE_CASES` is frozen empty; all 7 capabilities are uncovered) | +| Implementations | **none** | +| Consumers | **none** — `src/storage.ts` re-exports the types, `src/store-seam.test.ts` guards them | + +What the guard already forbids, so it does not have to be re-litigated per PR: no classes, no +inheritance, no `implements`-able parent, no default bodies, no member-selection/subtraction from an +existing type, no runtime merging of defaults, no scope identifier in any signature, and no +synchronous operation. `src/store/**` imports nothing from outside itself. + +### 4.3 The two-arm families — 43 of them + +| Area | Count | Families | +|---|---|---| +| `src/db/` | 21 | address-lifecycle, addresses, aliases, contacts, domains, email-content, email-digests, emails, events, forwarding, groups, inbound, owners, providers, provisioning, sandbox, scheduled, send-keys, sequences, templates, warming | +| `src/lib/` | 13 | analytics, batch, delivery-doctor, doctor, email-digest, forwarding, s3-sync, send, stats, status-facts, sync, verification-code, webhook | +| `src/cli/commands/` | 6 | daemon, email-log, inbox, misc, serve, sync | +| `src/cli/tui/` | 1 | data | +| `src/mcp/` | 2 | resources, tools/email-ops | + +Two `src/db` families (`threads`, `webhook-receipts`) are already single-armed and have repositories +on the seam waiting for them. + +The HTTP arm is synchronous **only** because `src/db/self-hosted-store.ts:259` shells out to `curl` +through `spawnSync` to fake a blocking network call. That bridge dies with the axis, which is why +every seam operation is a `Promise` even where SQLite could answer synchronously. + +## 5. Two hard rules + +**5.1 Every operation returns the outcome type — not only the capability-gated ones.** + +An earlier draft of the seam returned plain values from operations "every store can always perform" +and reserved `Outcome` for the gated ones. That left **42 of 65 operations with no way to say +"I cannot"** (`src/store/repositories.ts:46-56`), so for the list operations, the unread count and the +message counts, **an empty array or a zero was the only expressible answer**. That is precisely the +lie this refactor exists to remove, so the split rule is gone: the refusal channel is universal, a +caller cannot reach `.value` without checking `ok`, and `tsc` — not code review — enforces it. Today +that means 70 of 70 operations carry it, 27 of them additionally capability-gated. Do not +reintroduce the split under any argument about ergonomics. + +**5.2 The mode-gated branches inside the local-arm modules die WITH the axis, never before.** + +The 47 `selfHostedResourceBranches` are the local arm asking whether it is really the local arm. They +**fail loud** today: when the client is configured for the API and the endpoint does not exist yet, +the call throws rather than degrading. Delete them early — before the axis and the dispatch layer are +gone — and the same calls fall through to local SQLite and **silently serve local rows under a +self-hosted configuration.** That is the split-brain bug the module was written to close +(`src/db/self-hosted-resource.local.ts:1-15`). Deleting them is part of phase 9 and of no earlier +phase. + +## 6. The phases + +Dependency order. Each phase is one or more PRs; **every PR is individually green and individually +reviewable**, and none is large enough that a reviewer will wave it through. Every phase from 4 +onward lowers at least one ratchet metric, and the lowered numbers are the diff a reviewer reads +first. + +| # | Lands | Gate | +|---|---|---| +| 1 | `SqliteEmailStore` + shared conformance suite — **landed** | uniform-coverage assertion passes; every capability covered | +| 2 | `HttpEmailStore` against the same suite — **landed** | both stores execute every case | +| 3 | Configuration-driven store resolution — **landed** | both-configured is a boot error, proven by test | +| 4 | `src/db` families migrated, lowest fan-in first | ratchet drops each PR | +| 5 | CLI command families + TUI data layer | strictly after their db families | +| 6 | MCP surfaces collapsed, mode guards deleted | one registration path, no mode read in `src/mcp/**` | +| 7 | One mailbox view over the store | both `MailDataSource` backends retired | +| 8 | One send service | exactly-once ledger present in every configuration | +| 9 | **The axis deleted** | **ratchet reads zero on all eleven metrics** | +| 10 | Mode exports dropped from the published surface | major version | + +**Phase 1 — `SqliteEmailStore` behind the seam.** The first implementation, plus the **shared +conformance suite** the second one will be held to. The suite's runner must assert that *every* +implementation executed *every* case (`assertUniformCaseCoverage`); a case list that shrinks +mid-run, a store counted twice, or an empty run must all throw rather than certify. The gate for this +PR is that `capabilityCoverageGaps()` returns empty — an unexercised capability is exactly where a +refusal quietly degenerates into "returns nothing". + +**Phase 2 — `HttpEmailStore`.** Same suite, no new cases written to suit it. Where the API cannot do +something, the store declares the capability false and returns the typed refusal; the harness already +fails a store that answers an unavailable capability with `{ ok: true, value: [] }`, and fails one +that refuses a capability it declared available. Both directions matter. + +**Phase 3 — configuration-driven resolution.** Selection follows from **which of the database path or +the API URL is configured** — not from a mode word. **Both configured is a hard boot error, never a +precedence rule.** A precedence rule is the axis with extra steps: it answers "which one wins?" +instead of "which one did you mean?", and the wrong answer is silent. One configured → that store. +Neither → the default local SQLite path. Both → refuse to start, naming both settings. + +`src/store-resolution.ts` is that resolution. Two further configurations it refuses rather than +resolves, both of which would otherwise pick a store silently: an API URL with no credential (that +store answers 401 to everything, which is indistinguishable from one that legitimately declines +everything), and a set vault pointer whose payload is not in the environment (the pointer names an +API, so falling through to local SQLite serves local rows under an API configuration). It reads +storage settings only, and a source-text assertion — not a convention — keeps it from reading the +axis this program is deleting. + +Phase 3 also closed the gap phase 2 documented: `/v1` had **no route that recorded a message without +sending it**, so `createMessage`/`upsertMessage` — ungated on the seam, hence with no capability to +declare false and no legal refusal — could not be served for outbound input at all. +`POST /v1/messages/record` serves both, in either direction, and refuses the four send-ledger +columns so a row recorded there can never carry a fence the send path did not produce; the +inbound-only 409 on `POST /v1/messages` is unchanged. The conformance suite now runs against the +**real service** over HTTP with Postgres behind it in the `selfhost-postgres` job (40 passed / +8 refused / 0 failed, with neutering controls that must turn it red), which is how the live run +found and fixed the Postgres upsert writing its whole column set on a replay — resetting read, star +and label state on the one operation whose contract is that a replay changes nothing it was not +told about. + +**Phase 4 — migrate the two-arm `src/db` families, one at a time, lowest consumer fan-in first.** +Fan-in measured on `6646cc8` by resolving relative import specifiers and counting the **non-test** +modules that import the family from outside its own arms: + +| Tier | Families (prod fan-in) | +|---|---| +| first | address-lifecycle 2, email-content 2, events 2, scheduled 2 | +| then | aliases 3, forwarding 3, groups 3, inbound 3, sandbox 3, send-keys 3, sequences 3, warming 3 | +| then | email-digests 4, emails 4, owners 4, provisioning 4, templates 4, contacts 5 | +| last | addresses 7, domains 8, providers 12 | + +Each family's PR deletes its `.remote.*` arm, its facade's dispatch helper and its dispatch call +sites, and lowers `twoArmFamilies`, `remoteArmModules`, `routedFacadeDefinitions` and +`routedCallExpressions` together. The `*.local.*` mode-gated branches inside it stay (§5.2). + +**Phase 5 — CLI command families and the TUI data layer.** Strictly after the db families they +import; `src/cli/tui/data` has the largest fan-in of the 43 two-arm families (16 production +importers; next is `src/db/providers` at 12) and goes last within the phase. The branches on the data +source's `mode` label — 5 at `src/cli/commands/send.ts:136` and +`src/cli/commands/inbox.local.ts:178,496,1006,1011`, plus 3 pass-throughs at +`src/cli/commands/inbox.remote.ts:427,679` and the TUI state, as recorded in +`src/store/descriptor.ts:6-9` — are deployment-mode decisions wearing a data-source costume and go +with them. **That recorded count of 8 is the sites at the top of the call chain, not all of them**: +the two pass-throughs feed two further branches on the same value at +`src/cli/commands/inbox.remote.ts:1095,1107`, and `src/lib/mail-data-source.ts:487` compares it for +memoisation. Grep the label, do not work from the list. Do not replace it with another +narrow union; `StoreDescriptor.kind` is deliberately `string` so no `switch` over it can be +exhaustive (`src/store/descriptor.ts`). + +**Phase 6 — collapse the MCP surfaces and delete the mode guards.** `src/mcp/resources.ts`, +`src/mcp/tools/email-ops.ts`, `src/mcp/tools/sequences.ts` and `src/mcp/tools/infrastructure.ts` each +choose a registration or refuse a tool based on the mode read. One registration path, one tool set; +a tool that a given store genuinely cannot serve refuses with the capability refusal rather than +being unregistered, so the answer to "why is this tool missing?" stops depending on an environment +variable. + +**Phase 7 — one mailbox view over the store.** `src/lib/mail-data-source.ts` (510 lines) and +`src/lib/self-hosted-mail-data-source.ts` (1,660 lines) are two implementations of the same inbox. +Replace both with one view built on `MessagesRepository` / `InboundRepository` / `ThreadsRepository`, +and retire the backends. This is the phase that removes the largest block of duplicated behaviour and +the parity tests that exist only to compare the two. + +**Phase 8 — one send service.** The idempotency-fenced send ledger is the strongest arm's and the +local arm has never had it. `SendIntentsRepository` declares 12 operations, all capability-gated: 10 +on `sendIntentLedger` and 2 (`markSendBlocked`, `evaluateOutboundPolicy`) on `outboundPolicy`. The +interface's own header comment claims all 12 are gated on the ledger; it is wrong, and the split +matters — a store can hold the fence without being able to evaluate policy, and vice versa. +One send service carries the **exactly-once ledger in every configuration**. A store that cannot hold +the fence transactionally must refuse the send, not approximate it: a non-atomic reserve hands the +same intent to two senders and mails the message twice, which is strictly worse than refusing. + +**Phase 9 — delete the axis.** `src/lib/mode.ts`, the dispatch layer, the `curl`-based bridge in +`src/db/self-hosted-store.ts`, the mode-gated branches in the `*.local.*` modules, and every remaining +mode-gated branch. **Gated on the ratchet reading zero on all eleven metrics**, at which point +`src/mode-axis-ratchet.test.ts` and `scripts/mode-axis-ratchet-lib.mjs` are deleted along with the +axis they measure. Zero is *necessary and not sufficient* — see §4.1 — so this PR gets a real read, +not a numbers check. + +**Phase 10 — drop the mode exports from the published surface.** `src/storage.ts` currently exports +**fourteen** mode symbols from `@hasna/emails/storage`: six functions, four constants and four types +(`src/storage.ts:2-21`). Count the constants — they are what the tree-wide metric sees, and omitting +them understates the phase. Removing them is a breaking +change to a public package: **major version**, with the removal listed explicitly in the changelog. +No re-export shim, no deprecated alias — a shim is a compatibility layer for an axis that no longer +exists, which is the same mistake in a smaller box. + +## 7. Traps already paid for + +Each of these cost real time in this repository. They are recorded so the next person does not buy +them again. + +1. **A guard that scans a tarball built without running its build step certifies an empty artifact.** + The historical vacuous run of the pack scan packed 6 entries, scanned 5 files holding zero product + code, and printed a clean bill of health (recorded at `scripts/no-cloud-artifact-scan.mjs:11-17`). + It now carries two independent floors (100 files **and** 1,000,000 bytes), because a file count + alone is cleared by 100 empty stubs and a byte total alone by one big file. A real artifact scans + **756 text files and 12,646,868 bytes** after `bun run build` on `6646cc8` — the script's own + comment still says 696 files and 8.0 MB, so its "the byte floor is about an eighth of today's + payload" rationale is now nearer a twelfth. The floor is still doing its job; the prose is stale. + **Any new guard needs a floor, and the floor belongs inside the function every assertion goes + through — not in one test that can be skipped.** +2. **Ban patterns without positive controls can be neutered while CI stays green.** A pattern that + stops matching silently passes everything; a pattern widened until it matches anything blocks + unrelated work. Every ban pattern and every ratchet metric therefore carries fixtures that MUST + match and fixtures that MUST NOT, checked against the pattern itself rather than against repo + content — because repo counts are *supposed* to reach zero, and a "this metric found something" + check would have to be deleted exactly when it matters most. **This includes path selectors**: a + one-character typo in an arm-file selector once drove a 47-unit metric to zero with every other + assertion green (`src/mode-axis-ratchet.test.ts:42-47`; the `pathHits`/`pathMisses` fixtures and + the assertion at `:259-262` are the fix). Traps 1 and 2 survive only as prose in the guards + themselves — there is no commit left to read them from, which is the reason to keep writing them + down. +3. **A ratchet keyed on symbol names is defeated by a rename.** Ten of the eleven metrics are; that is + why `twoArmFamilies` is computed from file structure and why the metric list is asserted to contain + exactly one identifier-independent counter. **Any future ratchet needs at least one structural + metric.** +4. **A fidelity check must compare structurally, not through a normaliser that flattens the very + difference it is hunting.** The ratchet originally normalised the one compatibility bridge out of + its own corpus — subtracting an occurrence of the exact text it counts — and now measures the raw + corpus (`c96011b`). The no-cloud guard's remaining bridge is pinned by unique structural anchors + plus a `sha256` digest of the complete byte range, so insertion, reordering, duplication or + movement to another path all fail closed instead of being normalised away. + +## 8. Non-goals + +- **Do NOT remove org/tenancy scoping.** The amended boundary policy places the org model in the OSS + layer; only commercial metered operation sits in the paid hosted product. Tenancy vocabulary is + legitimate here and the no-cloud guard deliberately does not flag it. Note that scope is *structural* + on the seam — no signature takes a scope identifier, because the tenant is fixed at construction — + and that property is asserted, not conventional. +- **Do NOT delete existing mail.** No phase migrates, rewrites or drops stored messages. A store swap + changes where reads go, not what exists. +- **Do NOT take a hard dependency on the identity package.** It plugs in behind the seam later. Taking + the dependency now couples the deletion to an unrelated integration and gives the axis somewhere to + hide. +- Do not rename anything as a substitute for deleting it (§4.1). + +## 9. Definition of done + +1. All eleven ratchet metrics read zero, and the ratchet and its scan library are deleted. +2. No `*.remote.*` module, no dispatch helper, no dispatch call site, no mode variable, predicate, + reader, resolver or parser remains in the tree — including docs, Terraform and compose. +3. Exactly two implementations of `EmailStore` exist, both passing the same conformance suite with + uniform case coverage and no capability gaps. +4. The published surface carries no mode symbol, and the major version records the removal. +5. No compatibility shim, no deprecated alias, no dead branch. + +## 10. Verification + +Run all four on every PR in this program; they are the same four this plan's own PR was checked with, +green on `6646cc8`: + +``` +bunx tsc --noEmit # clean +bun run test # 2528 pass, 138 skip, 0 fail (2666 tests, 228 files) +bun run no-cloud:source # 26 pass, 0 fail +bun run build && bun run no-cloud:pack # build first, or trap #1 applies to you +``` + +The ratchet runs inside `bun run test`; to read the numbers on their own: + +``` +bun test src/mode-axis-ratchet.test.ts src/store-seam.test.ts +``` + +A failing ratchet prints the metric, its ceiling, the new count and the highest-count files. A +*lowered* count never fails — that is the whole design. diff --git a/docs/PLAN-PROVISIONING.md b/docs/PLAN-PROVISIONING.md index ae5e13c4..6d142105 100644 --- a/docs/PLAN-PROVISIONING.md +++ b/docs/PLAN-PROVISIONING.md @@ -1,8 +1,10 @@ # PLAN — Automated Domain → Email Address Provisioning (open-emails) -> Status: NOT IMPLEMENTED (2026-07-25). Owner: agents. Companion plan: `open-domains/docs/PLAN-PROVISIONING.md`. -> The orchestrator/daemon/round-trip modules this plan describes were built but never wired to any -> shipped entrypoint, and were removed as dead code. `emails provision *` fails loud in every mode. +> Status: DESIGN TARGET, NOT IMPLEMENTED (verified 2026-07-29). Owner: agents. +> Companion plan: `open-domains/docs/PLAN-PROVISIONING.md`. +> No shipped orchestrator, daemon, round-trip runner, or `/v1` provisioning route exists. +> Some schema/state-machine helpers remain, but no entrypoint drives them. +> `emails provision *` and the `provision_*` MCP tools fail loud in every mode. > The BrandSight/GCD DNS client below was likewise removed (enterprise-contract-only, never reachable). > This plan turns open-emails into a system that **gives users and agents real email addresses on > domains we own**, fully automatically: buy/verify the domain, wire DNS through Cloudflare, set up @@ -31,14 +33,15 @@ in Cloudflare regardless of where the domain was bought. | Cloudflare DNS auto-publish (DKIM/SPF/DMARC/MX) | `src/lib/cloudflare-dns.ts` `setupEmailDns()` | Direct Cloudflare REST client, no connector dependency. | | Resend send + domain | `src/providers/resend.ts` | Send-only + domain create/verify. | | S3/SES inbound sync | `src/lib/s3-sync.ts`, `inbox sync-s3` | Active stored-mail path. | -| Cloudflare routing inbound | `src/lib/cloudflare-routing.ts` | Active forwarding/routing setup; stored body requires SES/S3 or Worker/webhook persistence. | -| Partial orchestration | `src/mcp/tools/infrastructure.ts` | `setup_domain_for_email`, `setup_cloudflare_dns`, `setup_ses_inbound`. | +| Cloudflare routing inbound | Not implemented | Cloudflare may own public MX, but this package has no provider-native routing-rule client. App-level `emails forwarding` runs only after mail enters Emails. | +| One-shot local infrastructure helpers | `src/mcp/tools/infrastructure.ts` | `setup_domain_for_email`, `setup_cloudflare_dns`, and `setup_ses_inbound` work in local mode with the executing machine's cloud credentials; they are not a resumable provisioner. | | Address / domain / provider DB | `src/db/{addresses,domains,providers}.ts` | Schema exists; needs extension (§5). | | Cross-repo link | imports `@hasna/domains` | r53 buy/zone functions already imported. | -**Critical gap:** `setup_domain_for_email` currently creates a **Route53 hosted zone** and writes DNS -there. New rule: **DNS is ALWAYS Cloudflare.** This flow must be refactored to delegate NS to -Cloudflare and publish records via `cloudflare-dns.ts` (§6, T-E2). +`setup_domain_for_email` already creates/reuses a Cloudflare zone, delegates a +Route 53 Domains registration to its nameservers when possible, and publishes +records through `cloudflare-dns.ts`. The remaining critical gap is durable, +resumable orchestration and a server-side authorization boundary for that work. ## 3. Provider capability matrix (2026 — from research) @@ -61,6 +64,10 @@ Cloudflare and publish records via `cloudflare-dns.ts` (§6, T-E2). ## 4. Architecture — the provisioning state machine +Everything from this section onward is proposed behavior unless explicitly +listed as existing in section 2. See `docs/PROVISIONING.md` for current operator +commands. + A domain and each address move through an explicit, resumable state machine (persisted in DB so the daemon can resume after crash/restart): @@ -99,7 +106,8 @@ Every transition is idempotent and re-entrant. Each state records `attempts`, `l - New table **`provisioning_events`**: append-only audit (`entity_type`, `entity_id`, `from_state`, `to_state`, `detail_json`, `created_at`) — powers `emails provision status` and the dashboard. -Add a SQLite migration in `src/db/database.ts` and the matching `pg-migrations.ts` for the remote storage path. +Add a SQLite migration in `src/db/database.ts` and a matching self-hosted +Postgres migration in `src/server/self-hosted/migrations.ts`. ## 6. New / changed code modules diff --git a/docs/PROVIDER_SECRETS.md b/docs/PROVIDER_SECRETS.md new file mode 100644 index 00000000..32af4189 --- /dev/null +++ b/docs/PROVIDER_SECRETS.md @@ -0,0 +1,68 @@ +# Provider credential storage + +Local provider credentials are envelope-encrypted. Ordinary `providers` rows +retain their historical credential columns only so older databases can be +migrated; current rows keep every one of those columns `NULL`. Ciphertext, +nonces, authentication tags and wrapped per-provider data keys live in +`provider_secrets`. The AES-256-GCM root keys never enter SQLite. + +By default the root-key keyring is +`~/.hasna/secrets/open-emails-provider-credentials.keyring.json` (mode `0600`, +inside a mode `0700` directory). Operators may instead bind a separately +protected keyring with `EMAILS_PROVIDER_SECRETS_KEY_FILE`, or inject one +base64/hex 32-byte recovery key with `EMAILS_PROVIDER_SECRETS_KEY`. Inline keys +cannot be rotated by the CLI and should be supplied by an OS keyring or secret +manager, not committed to an environment file. + +On first open, legacy plaintext values are encrypted and cleared in one SQLite +savepoint. Secure deletion, compaction and WAL truncation remove the superseded +bytes before a post-migration database backup is safe. SQLite triggers reject +later attempts to put credentials back into ordinary provider rows. + +The local execution boundary unwraps a credential only when constructing a +provider adapter. Provider list/get DTOs, REST/MCP responses, exports and generic +resources expose metadata only. Error messages identify a provider or root-key +ID, never a credential, ciphertext plaintext, or command argument. + +## Rotation and recovery + +Use: + +```bash +emails provider secrets status +emails provider secrets rotate-root +emails provider secrets rewrap +emails provider secrets revoke-root --yes +``` + +Rotation stages the new root in the external keyring, atomically rewraps every +data key, and only then makes the new root active. The old root remains for +in-flight work and crash recovery until explicit revocation. Revocation fails +while any envelope still references the key. + +A database backup is intentionally insufficient on its own. Back up the +keyring separately under the protection used for other root credentials. To +restore on another machine, restore the database, bind the original keyring +with `EMAILS_PROVIDER_SECRETS_KEY_FILE`, verify `provider secrets status`, then +optionally rotate and revoke the transported root. Opening a database with +encrypted provider rows and no matching root key fails closed; it does not +generate a replacement or attempt a send. + +Self-hosted operation does not store provider credentials in tenant provider +rows. The service sender uses operator-injected secret-manager values or its +least-privilege workload role. Provider resources remain tenant-scoped metadata, +and clients never retrieve service credentials. + +## Threat model and review checklist + +Protected: SQLite files, WAL/journal files, ordinary database backups, public +DTOs, diagnostics, logs and exports. Compromise of both the database and the +external root-key keyring is outside this boundary; rotate upstream provider +credentials and the root key after such a compromise. A process already +authorized to send necessarily sees a credential briefly in memory. + +Independent review should verify the migration savepoint and raw-byte purge, +AES-GCM AAD binding (`provider id`, `revision`, and purpose), root-key separation, +locked-keyring failure paths, DTO projections, rotation crash points, restore +rebind instructions, and tenant isolation in the self-hosted store. This +document records the review surface; it is not itself an independent sign-off. diff --git a/docs/PROVISIONING.md b/docs/PROVISIONING.md index b502a681..242ff1d4 100644 --- a/docs/PROVISIONING.md +++ b/docs/PROVISIONING.md @@ -1,98 +1,93 @@ -# Email-address provisioning (open-emails) +# Email-address provisioning -> **STATUS: NOT IMPLEMENTED in the shipped package.** Every `emails provision *` -> command and every `provision_*` MCP tool fails loud. There is no local -> provisioning orchestrator (it was unreachable from all shipped entrypoints and -> was deleted) and the self-hosted server exposes no provisioning route and runs -> no reconciler. The rest of this document describes the intended design, not -> current behaviour. **Provisioning today is manual — see "What actually works" -> below.** +> **Status: the stateful provisioning workflow is not implemented.** Every +> `emails provision *` command and the MCP tools `provision_domain`, +> `provision_address`, and `provision_status` returns an actionable error. The +> package ships no provisioning reconciler or `/v1` provisioning route. -Give users and agents **real email addresses on domains we own**, fully -automatically: buy the domain, wire DNS through Cloudflare, set up SES sending + -receiving, create addresses, and verify by sending mail back and forth. +The registered commands preserve compatibility and make the missing capability +explicit; registration in `--help` is not a claim that they work. The internal +provisioning records and state-machine helpers are not connected to a shipped +orchestrator. -## What actually works today -``` -emails domain adopt ours.com --provider # register an already-verified domain + wire SES inbound (S3) + catch-all -emails aws setup-inbound # create the S3 bucket + SES receipt rules -emails address add andrew@ours.com --provider -emails domain list --json # what is registered -emails inbox sync-s3 # pull inbound mail from the S3 bucket -``` +## Supported operator workflow -## One command (design target — NOT implemented) -``` -emails provision domain ours.com --provider --add-mx # SES identity + publish DNS in Cloudflare -emails provision address andrew@ours.com --provider --receive ses-s3 -emails address provision andrew@ours.com --provider --receive ses-s3 # address-first alias -emails provision status -``` -For buying + delegating first, use `@hasna/domains` (`domains domain buy --wait --dns cloudflare`) or the `setup_domain_for_email` MCP tool (which now buys, creates the Cloudflare zone, delegates NS, registers with SES, and publishes DNS **in Cloudflare**). - -Ownership is separate from address creation. Use `emails address owner ` -to inspect owner/admin state, `emails address set-owner --owner ` -for initial assignment, and the explicit `transfer-owner`, `unassign-owner`, and -`owner-history` commands when ownership changes need an audit trail. - -## The pipeline -1. **Buy** (Route53, `@hasna/domains`) — the only reliable self-serve API. -2. **DNS → always Cloudflare** — create the zone, delegate registrar NS to it. -3. **Send** — SES domain identity (any `*@domain` can send); Resend secondary. -4. **Receive** — one of three ingestion-source strategies (none are IMAP mailboxes): - - `ses-s3` (default): SES receipt rule → S3 → `emails inbox sync-s3` → SQLite. This is an ingestion source feeding the local mailbox. - - `cf-routing`: Cloudflare Email Routing forward/Worker (no stored body unless a Worker persists it). - - `resend-webhook`: Resend `email.received` webhook (no stored mailbox body unless persisted). -5. **Validate** — `emails test roundtrip` sends tokened mail back and forth and confirms receipt. - -## There is no IMAP/POP mailbox anywhere -No provider (SES, Cloudflare, Resend) exposes an IMAP/POP inbox. Providers are -credentials/capabilities; sources are ingestion streams. For the `ses-s3` -strategy, SES drops raw MIME into S3 and `emails inbox sync-s3` parses it into -the local mailbox store. Query the synced store with `emails inbox mailboxes`, -`emails inbox sources`, or mailbox list/search commands instead of expecting -direct provider mailbox access. - -## State machine + daemon -Domains and addresses move through an explicit, resumable lifecycle -(`src/lib/provision/state-machine.ts`); the reconciler daemon -(`src/daemon/provisioner.ts`) advances any entity whose `next_check_at` is due, -crash-safe because all state lives in the DB. - -Useful health checks: +For a domain already registered and verified in SES, use the manual path: +```bash +emails provider add --name production-ses --type ses --region us-east-1 +emails domain adopt example.com --provider --no-inbound +emails domain dns example.com --provider +emails domain check example.com --provider +emails address add hello@example.com --provider ``` -emails status -emails daemon status -emails inbox sync-status -emails doctor delivery andrew@ours.com -emails logs tail --component daemon + +`domain dns` prints expected records and `domain check` reads public DNS. Neither +publishes DNS. `domain verify`, despite appearing in help for compatibility, is +not implemented; use `domain check` for the live result. + +When SES should also receive the domain, omit `--no-inbound` from `domain adopt` +or run the explicit inbound setup: + +```bash +emails aws setup-inbound \ + --domain example.com \ + --bucket operator-owned-inbound-bucket \ + --provider + +# Publish the MX value printed by setup-inbound, then ingest mail: +emails inbox sync-s3 --source +emails inbox sources +emails inbox mailboxes ``` -## Credentials (`emails doctor`) -- **AWS** (SES send/inbound, Route53 buy): `AWS_PROFILE` or keys, region us-east-1. -- **Cloudflare** (DNS + Email Routing): `CLOUDFLARE_API_TOKEN` *or* - `CLOUDFLARE_API_KEY`+`CLOUDFLARE_EMAIL` + `CLOUDFLARE_ACCOUNT_ID`. -- **Resend** (optional): `RESEND_API_KEY`. -- **SES sandbox**: new accounts send only to verified identities (200/day, 1/sec); - request production access with the `ses-sandbox` helper (PutAccountDetails). - -## Proven live -Verified end-to-end: 3 funny `.com` domains bought, DNS in Cloudflare, SES DKIM -verified, 3 addresses/domain, **144/144 emails** sent via the `emails` CLI and -received (SES→S3→SQLite). See `docs/PLAN-PROVISIONING.md` for the architecture. - -## AWS account architecture (self-hosted operator example) -| Concern | AWS account | Notes | -|---|---|---| -| **SES** (send + inbound) | Operator mail account | Production access. All domain identities, MAIL FROM, and receipt rules live here; configure S3 with `inbound_s3_bucket` / `inbound_s3_buckets`. | -| **Domain purchase** (Route53 Domains) | Operator registrar account | Run `domains domain buy` with the operator's AWS profile or ambient credentials. | -| **DNS** | Operator Cloudflare account | Always Cloudflare — DKIM/SPF/DMARC/MAIL-FROM/inbound-MX + Email Routing. | -| **Send (secondary)** | Resend | Provider integrated; sends proven. Free plan caps Resend-verified domains at 1. | - -`emails config set inbound_s3_bucket ` makes `emails inbox sync-s3` default to that inbound bucket (no `--bucket` needed). Inbound buckets must block public access and use server-side encryption; keep the versioning policy explicit because raw MIME objects are the mailbox for the `ses-s3` strategy. `emails doctor` reports SES sandbox/production + provisioning creds. - -### Integration status (priority: SES, Resend, Cloudflare) -- **SES**: verified + send/receive tested with operator-owned domains. -- **Resend**: ✅ send tested end-to-end (Resend send → our domain → SES inbound). -- **Cloudflare**: ✅ DNS + Email Routing client. +Both `domain adopt` and `aws setup-inbound` mutate the AWS account selected by +the executing machine's profile/credentials. In a self-hosted client they are +still operator-side infrastructure commands; the `/v1` server does not perform +that work. Run them only from an operator-controlled machine against the +intended account. + +`domain adopt` refuses SES inbound setup if another provider owns public root +MX. Use `--force-mx-switch` only for an intentional inbound migration. For a +send-only SES identity behind Google Workspace or another mailbox provider, +keep `--no-inbound` and preserve the existing MX. + +The inbound bucket may also come from `EMAILS_INBOUND_S3_BUCKET` or the +`inbound_s3_bucket` config value written by `domain adopt`. There is no +`emails config` command in this build. + +## Local MCP infrastructure helpers + +Local mode also exposes direct, operator-credentialed infrastructure tools: + +- `setup_domain_for_email` can buy through Route 53 Domains, create a + Cloudflare zone, delegate nameservers, register the mail provider domain, and + publish email DNS in Cloudflare; +- `setup_cloudflare_dns` publishes DKIM/SPF/DMARC and optional MX records; +- `setup_ses_inbound` creates the S3 bucket and SES receipt rules. + +These are one-shot infrastructure helpers, not the missing resumable +`provision_*` workflow. They are refused in `self_hosted` mode because otherwise +they would mutate infrastructure using the client machine's ambient cloud +credentials while recording state in the operator's shared service. + +## Mailbox model + +SES, Resend, and Cloudflare routing are capabilities or ingestion paths, not +IMAP/POP mailboxes. SES can archive raw MIME to S3; `emails inbox sync-s3` parses +it into the selected Emails store. Cloudflare Email Routing forwards mail and +does not create a stored mailbox here. Resend inbound is persisted only when a +configured webhook delivers it to this application. + +App-level forwarding under `emails forwarding` runs only after this package has +received or synced the source message. If Google Workspace, Microsoft 365, or +another provider owns root MX and mail never enters Emails, configure forwarding +at that provider. + +## Unimplemented target + +The intended resumable domain/address state machine, retry/status commands, +daemon, and round-trip acceptance runner remain design work. Their historical +design is preserved in [PLAN-PROVISIONING.md](PLAN-PROVISIONING.md); it is not an +operator runbook and its example future commands must not be used as current +instructions. diff --git a/docs/SELF_HOSTED_RUNTIME.md b/docs/SELF_HOSTED_RUNTIME.md index 0ca11ab3..61375239 100644 --- a/docs/SELF_HOSTED_RUNTIME.md +++ b/docs/SELF_HOSTED_RUNTIME.md @@ -8,10 +8,29 @@ Client configuration: ```bash export EMAILS_MODE=self_hosted export EMAILS_SELF_HOSTED_URL="https://emails.example.com" -export EMAILS_SELF_HOSTED_API_KEY="..." +export EMAILS_SELF_HOSTED_API_KEY="..." # or EMAILS_SESSION_TOKEN / EMAILS_IDP_TOKEN emails inbox list ``` +The client chooses `EMAILS_SESSION_TOKEN`, then `EMAILS_IDP_TOKEN`, then +`EMAILS_SELF_HOSTED_API_KEY` when more than one is present. A client-env vault +entry referenced by `EMAILS_CLIENT_ENV_SECRET` may carry the URL and any one of +those credentials. See [AUTHENTICATION.md](AUTHENTICATION.md) for the account, +tenant-key, and optional IdP flows. + +For a repeatable read-only client check, run the published smoke from the exact +checked-out release on every client station: + +```bash +./scripts/self-hosted-client-smoke.sh +``` + +It refuses local database selectors, then runs `emails --version`, remote +status, provider-list, and a one-row inbox read. It performs no send or mailbox +mutation and emits only an aggregate pass record; command responses stay in a +private temporary directory. This is the smoke referenced by +[STATION_LOCAL_RETIREMENT.md](STATION_LOCAL_RETIREMENT.md). + Service configuration: ```bash @@ -21,6 +40,7 @@ export EMAILS_API_SIGNING_KEY="..." # 32+ characters export EMAILS_SEND_PROVIDER=ses # or resend export EMAILS_AUTH_ALLOWED_EMAIL_DOMAINS="example.com" # required; your own domains export EMAILS_AUTH_FROM="no-reply@example.com" # required; a verified sender identity +export EMAILS_IDP_JWKS_URL="https://id.example.com/v1/.well-known/jwks.json" # optional IdP verifier export EMAILS_AWS_REGION=us-east-1 # SES identity — pick ONE: # (a) nothing: sign with the deployment IAM role of the account the service runs in @@ -49,6 +69,10 @@ emails self-hosted key create emails-serve ``` +The current migration ledger ends at `0021_idp_principal_tenants`. A release +image used after that migration must recognize 0021; an older image fails +readiness on the unknown applied ledger row and is not a rollback target. + ## Auth: signup domain allowlist and sender identity Two auth variables are **required** and have **no defaults** — the service refuses diff --git a/docs/STATION_LOCAL_RETIREMENT.md b/docs/STATION_LOCAL_RETIREMENT.md new file mode 100644 index 00000000..cd5e92f0 --- /dev/null +++ b/docs/STATION_LOCAL_RETIREMENT.md @@ -0,0 +1,166 @@ +# Retiring station-local Emails state + +This runbook is the coordinated gate for retiring local SQLite state from two +client stations after migration to an operator-owned Emails service. It is not +evidence that a station has already been retired. Store all evidence outside the +repository in an access-controlled case directory; database, attachment, cache, +and backup manifests can reveal private paths and mailbox metadata. + +The operation is fail-closed. Do not stop a writer until both stations have +passed the same published smoke and both pre-stop backups have been independently +restore-verified. Do not quarantine either station until both stations have also +passed the stopped-writer recheck. Never delete or shorten the retention of a +backup during this procedure. + +## Evidence contract + +Create one sealed evidence bundle per station. The two station identifiers must +be distinct, and both bundles must name the same service tenant, release commit, +and SHA-256 of `scripts/self-hosted-client-smoke.sh`. Each bundle records: + +- the station identifier, executor, timestamps, release commit, and service + tenant identifier (never a token or database credential); +- the aggregate JSON emitted by `./scripts/self-hosted-client-smoke.sh`, with a + zero exit status, plus the smoke-script SHA-256; +- a canonical source manifest with an explicit row for the SQLite database and + each `-wal`, `-shm`, and `-journal` sidecar, recording `MISSING` when a sidecar + does not exist, and a recursive file/size/SHA-256 manifest for every local + attachment or cache root; +- deterministic local row counts and the reviewed, semantically comparable + remote counts used for parity. Record the exact query/CLI command and schema + version beside every count; local and Postgres table names are not themselves + proof of semantic parity; +- a backup manifest, its checksum, restore location, restore-test result, + SQLite `PRAGMA integrity_check` result, and restored row counts; +- the independent backup verifier's identity and timestamp. The verifier must + not be that station's retirement executor; +- the quarantine path, rollback owner, retention owner, UTC retain-until time, + and the ticket or policy that authorizes any later retention decision. + +The state manifest is authoritative, not the default path. The usual database +is `~/.hasna/emails/emails.db` and usual downloaded attachments are below +`~/.hasna/emails/attachments`, but `HASNA_EMAILS_DB_PATH`, `EMAILS_DB_PATH`, and +`inbound_emails.attachment_paths` can name other locations. Discover those +locations before the gate. Reject symlinks, non-regular database artifacts, +relative paths, duplicate destinations, and a quarantine nested under any active +state path. Keep client configuration and credentials out of the state move. + +## 1. Two-station pre-stop barrier + +On each station, with both database-path variables **unset**, set the canonical +self-hosted client environment and run the exact script from the reviewed +release: + +```bash +test "${HASNA_EMAILS_DB_PATH+x}" != x +test "${EMAILS_DB_PATH+x}" != x +test -n "${EMAILS_CLIENT_ENV_SECRET:-}" || test -n "${EMAILS_SELF_HOSTED_URL:-}" +./scripts/self-hosted-client-smoke.sh >"$PRIVATE_EVIDENCE_DIR/pre-stop-smoke.json" +``` + +Independently restore and inspect each pre-stop backup. Compare its checksum, +integrity result, database counts, attachment/cache file count, byte count, and +content manifest with the bundle. A successful copy command is not a restore +test. The coordinator signs a two-station barrier only after both smoke records +and both independent backup reviews pass. A failure or missing bundle stops the +operation on both stations. + +## 2. Stop and fence every local writer + +Inventory and stop the station's actual processes: daemon or launch service, +SMTP listener, webhook server, MCP process, scheduler, sync job, TUI auto-pull, +cron job, and any long-running CLI. The inventory is site-specific; a process +name alone is not proof. Record service-manager state and prove no process has +the database, WAL, SHM, journal, attachment roots, or cache roots open. + +Do this on both stations before either state move. Do not run a local-mode Emails +command after the fence: resolving the default local store can create +`~/.hasna/emails/emails.db` and invalidate the evidence. + +## 3. Recheck stopped state and final backups + +While every writer remains stopped: + +1. Rebuild the canonical source manifest twice and require identical path, + type, size, and SHA-256 records. Explicitly recheck the database, `-wal`, + `-shm`, and `-journal` states even when they are absent. +2. Copy the exact database and existing sidecars to a private validation + directory. Run integrity and deterministic count queries against that copy, + not the fenced source, then prove a third source manifest is unchanged. +3. Recompute the attachment/cache file count, byte count, and per-file hashes. + Compare them to the stopped-state manifest and the parity evidence. +4. Produce a final fenced backup and have the independent verifier restore it, + rerun integrity/count checks, and compare its attachment/cache manifest. + +Seal the final source manifest, counts, final-backup manifest, verifier record, +and an exact source-to-quarantine move plan. If a source changes, a count differs, +integrity is not `ok`, a backup cannot be restored, or either station lacks its +final verifier record, do not move state on either station. + +## 4. Move exact state to recoverable quarantine + +The reviewed quarantine must be private, outside every active Emails path, on +the same filesystem as each source so each move is a rename rather than an +unverified copy-and-remove. It must not exist at a path the runtime scans or +creates. Create it with mode `0700`; its files retain mode `0600` or stricter. + +Execute only the sealed source-to-destination plan. Move the database, every +sidecar whose manifest state is present, and each exact attachment/cache root. +Never glob and never move the whole `~/.hasna/emails` directory: it can contain +the client configuration needed for remote operation. Refuse an existing +destination. Record every successful rename, then generate a quarantine +manifest with paths normalized back to their original source names and require +it to equal the final source manifest byte-for-byte. + +Write these non-secret controls beside the quarantine manifest: + +- `original-paths.tsv` (exact rollback mapping); +- `retention-owner` and `retain-until`; +- `rollback-owner` and the approval ticket; +- checksums of both station evidence bundles and the final backup manifests. + +The independently verified backups remain in their protected backup locations. +Do not move, replace, prune, or delete them as part of quarantine. + +## 5. Remote proof and non-recreation proof + +On both stations, keep local writers disabled and local database variables +unset. Run the same published smoke again, followed by any station-specific +read-only workflows in the parity record. Preserve their aggregate results. + +Before and after those commands, require that every original database/sidecar +and attachment/cache source in the move plan is absent. Recheck after the normal +client service/MCP startup interval as well; a one-shot check does not catch a +scheduled writer. Require the quarantine manifest and hashes to remain +unchanged. Any recreated database, WAL, SHM, journal, attachment directory, or +cache path is a failed retirement: stop the recreating process and return to the +stopped-state gate. + +The operation is complete only when both post-move smoke records pass, both +stations still have no active local state, both quarantine manifests match, and +the coordinator signs the joint result. + +## Rollback + +Rollback requires the named rollback owner and a new approval; one station must +not silently diverge from the other. Stop all Emails clients and writers, verify +the quarantine and protected-backup checksums, and require every original target +path to be absent. Restore a copy of each quarantined item to the exact path in +`original-paths.tsv`, preserving ownership and modes, then compare the restored +manifest, integrity result, and counts with the sealed final evidence before +enabling any local writer. If quarantine validation fails, restore from the +independently restore-tested final backup instead. A rollback consumes neither +backup and does not authorize backup deletion. + +Record which store is authoritative before restarting. Never configure an +Emails API and a local database path in the same process: the client deliberately +refuses that ambiguous two-store configuration. + +## Retention handoff + +The retention owner acknowledges custody of both protected backup sets, +quarantines, manifests, and rollback instructions. Retention extends through the +latest rollback, legal, incident-response, or operator-policy deadline. Expiry +only creates a review; it is not an automatic deletion instruction. Any later +destruction is a separate approved operation outside this goal. Cutover success +is never authority to delete a backup. diff --git a/docs/adr/0001-adopt-tenants-idp-for-identity.md b/docs/adr/0001-adopt-tenants-idp-for-identity.md new file mode 100644 index 00000000..d7022542 --- /dev/null +++ b/docs/adr/0001-adopt-tenants-idp-for-identity.md @@ -0,0 +1,245 @@ +# ADR-0001 — Adopt `@hasna/tenants` as the identity authority for `@hasna/emails` + +- Status: **Accepted** (owner ruling, 2026-07-28: *"we need to fix all this and we + need to adhere to hasna/tenants"*) +- Date: 2026-07-28 +- Deciders: owner; drafted by the tenants-federation working session +- Companion: [ADR-0002 — Agent identity, signup, and scopes](0002-agent-identity-signup-and-scopes.md) +- Related design: [`docs/design/multi-tenancy-auth.md`](../design/multi-tenancy-auth.md) + (the private multi-tenancy build this ADR federates and then supersedes in part) + +## Context + +### What emails has today (verified in this tree) + +`@hasna/emails` grew a complete, **private** identity system as part of the +multi-tenancy build (design doc §3–§6, migrations `0012`/`0013` in +`src/server/self-hosted/migrations.ts`): + +- Its own `tenants`, `users`, `memberships`, `sessions`, `invitations`, + `api_key_tenants`, `send_key_tenants` tables (migrations.ts ~:1489+). These are + the **resolution layer**: read before a tenant is known, deliberately outside + the generic resource surface and outside RLS. +- Its own signup/login/verify/bootstrap-owner surface (`/v1/auth/*`, + `src/server/self-hosted/auth/service.ts`) and CLI (`src/cli/commands/auth.ts`). +- Two credential classes, dispatched by prefix in `resolveRequestContext` + (auth/service.ts:119): `hasna_…` HMAC API keys verified via + `@hasna/contracts/auth` and mapped to a tenant through `api_key_tenants`, and + `emss_…` opaque user sessions. The server derives the tenant from the + credential; the client never sends one. +- Tenant isolation in three layers (design §6): a tenant-scoped store, forced + Postgres RLS keyed on `app.current_tenant` (with the boot guard in + `rls-guard.ts`), and `NOT NULL tenant_id` on every data table. + +Production (the deployed self-hosted service, server 1.3.0) runs this with real tenants, +~325 addresses and ~170k messages. **Agents currently authenticate by +materializing the owner's client-env bundle from the vault** — every agent is +the owner. That works and is the problem: no per-agent identity, no per-agent +revocation, no per-agent audit line. + +### What the org IdP provides + +`@hasna/tenants` 0.2.0 is the org tenant-auth IdP: tenants, users, +memberships, **service principals**, sessions, an OTP login front door, and +**asymmetric (EdDSA/Ed25519) access tokens** with a published JWKS. It is +explicitly distinct from `@hasna/identities` (the agent registry). + +The token wire contract (open-tenants `src/idp/tokens.ts`) is: + +- Compact JWS, header `{ alg: "EdDSA", kid, typ: "at+jwt" }`. +- Claims `{ iss, aud, sub, tid, pt, scope, iat, exp, jti }`: + `iss` is the **fixed issuer string `identities`** (an org-wide wire + contract, not a code dependency on the agent registry); `aud` is the app slug + the token is for; `sub` is the principal id (user or service principal); + `tid` is the IdP tenant UUID; `pt` is `user | service`; `scope` uses the + `:` / `:*` / `*` grammar shared with + `@hasna/contracts/auth`. +- TTL is server-bounded at **≤ 24 h** — a caller can shorten a token's life, + never extend it. +- Verification is **stateless**: any app holding the published JWKS + (`/v1/.well-known/jwks.json`) verifies signature, issuer, audience and expiry + offline. The IdP never holds another app's signing secret, so an IdP-side + compromise cannot forge an app's HMAC keys, and an app-side compromise cannot + mint IdP tokens. + +### Verified gaps (2026-07-28) — named, not worked around + +1. **No deployed IdP instance.** The expected IdP host's `/.well-known/jwks.json` + returns HTTP 404 `{"error":"no matching host route"}`; no vault entries for a + tenants API URL exist; the installed `tenants` CLI (0.2.0) requires + `HASNA_TENANTS_API_URL` and has nothing to point at. + → prerequisite task filed against `open-tenants` (deploy an org instance). +2. **No service-principal signup or token path.** The `service_principals` + table and `store.createServicePrincipal` exist, but no HTTP route or CLI + verb creates one, and `POST /v1/auth/token` mints `pt: "user"` tokens from + sessions only. → prerequisite task filed against `open-tenants` + (service-principal enrollment + `pt: "service"` issuance; see ADR-0002). +3. **Revocation is invisible to stateless verifiers.** The jti denylist is + enforced only by the tenants service's own `/v1` surface. An app verifying + via JWKS alone keeps accepting a revoked token until it expires (≤ 24 h). + → prerequisite task filed against `open-tenants` (introspection/revocation + feed), and this ADR designs around the bound honestly (below). + +## Decision + +**`@hasna/tenants` is the identity authority for emails.** The target state is: +every WHO-question — human users, agents/service principals, login, OTP, +session issuance, token issuance, credential revocation — is answered by the +org IdP. Emails keeps only the **data plane**: tenant-scoped mail rows, RLS, +per-tenant roles/scopes, and an explicit mapping from IdP principals to +emails tenants. + +Concretely: + +1. **Federation of identity, local ownership of mail data.** Emails keeps its + own `tenants` rows and `tenant_id`/RLS machinery as the *data-scoping* + boundary (WHAT-MAIL). It accepts **access tokens minted by the IdP**, verified + statelessly against the published JWKS, and maps the token's `sub` to an + emails tenant through an explicit, additive mapping table + (`idp_principal_tenants`, mirroring `api_key_tenants`). The IdP owns WHO; + emails owns WHAT-MAIL and *which IdP principal may act in which mail + tenant*. +2. **The private auth surface becomes a legacy shim with a stated sunset** + (migration plan below). New signups — human AND agent — go through the IdP + from day one once an instance is deployed. Nothing existing breaks at any + step; the `hasna_` API-key path stays a supported credential class until + every issued key has been migrated and revoked. +3. **Fail-closed adoption.** Until an operator configures the JWKS source + (`EMAILS_IDP_JWKS_URL`), the IdP credential class is refused with a + typed error. Verification pins the wire contract (issuer `identities`, + EdDSA, `at+jwt`, audience `emails` — plus the `mailery` alias for parity + with the API-key verifier in `api-key-verifier.ts`). + +### Naming: the credential class is called `idp` inside emails + +Everything emails-side is named after the issuer's ROLE — `idp` — never after +the org-infrastructure noun the tenants package uses in its own prose: +`EMAILS_IDP_JWKS_URL`, `EMAILS_IDP_TOKEN`, `idp_principal_tenants`, +`principal_type: "idp"`, `[idp-auth]`/`[idp-jwks]` audit tags, `idp_*` typed +reasons. Two reasons. First, precision: from this product's point of view the +counterparty IS an identity provider; which org runs it is irrelevant to the +verification code. Second, this repo's no-cloud boundary +(`scripts/no-cloud-scan-lib.mjs`, "hosted implementation vocabulary") bans the +hosted-infrastructure noun across the whole corpus with no path allowance — +that guard protects the product's operator-owned identity and this programme +deliberately does not weaken it. No wire value is affected: the issuer string, +claims shape, and scope grammar carry no such vocabulary. + +### Why not full delegation of the tenancy tables + +The rejected alternative is deleting emails' `tenants`/`users`/`memberships` +tables and pointing every `tenant_id` FK at the IdP's database (or resolving +tenancy per-request from the IdP). Rejected because: + +- **Availability coupling.** Mail ingest, RLS policy evaluation and the + ingest worker resolve tenants on every request/message. A synchronous IdP + dependency puts IdP availability in the mail hot path; an IdP outage + would stop mail. Token verification via cached JWKS has no such coupling. +- **Referential integrity across services.** 27 data tables FK + `tenant_id → tenants(id)` locally and RLS policies compare against a local + GUC. Cross-database FKs don't exist; dropping the FKs to point at a remote + system trades a real integrity guarantee for a convention. +- **The product must work standalone.** `@hasna/emails` is an OSS self-hosted + product. A single operator on their own box must be able to run it without + standing up a second identity service. Federation is opt-in configuration; + delegation would be a hard dependency. +- **Migration risk.** The prod dataset (325 addresses, 170k messages) is keyed + to local tenant UUIDs; rekeying data tenancy to IdP-issued UUIDs is a + destructive migration with nothing to buy — the mapping table gives the same + end-state semantics additively. +- **The IdP is 0.2.0 and not deployed.** A system of record cannot be a + service that does not yet run anywhere. Federation lets emails ship the + verifying side now and light it up when the prerequisite lands. + +Direction of truth, stated once: **the IdP owns identity and the issuance / +revocation of identity credentials. Emails owns the authorization mapping +(IdP principal → emails tenant + role/scopes) and all mail data.** The +mapping is explicit — never inferred from slug equality or email domain — so +every cross-domain grant is a deliberate, auditable row. + +## Migration plan — committed, phased, nothing breaks at any step + +Every phase is additive or flag-gated; every removal is gated on measured zero +traffic, not on calendar time. Existing credentials keep working within each +phase; a credential class is only retired after its inventory reaches zero. + +**Phase 1 — IdP-token verification slice** *(this repo, first committed step — +not a proof of concept)*. The server verifies EdDSA IdP tokens against a +configured JWKS URL (fail-closed when unconfigured, typed refusal), maps `sub` +through the new additive `idp_principal_tenants` table, and `emails auth +whoami` works with an IdP token. Existing `hasna_`/`emss_` behaviour is +byte-equivalent (proved by the existing suite). Ships dark until the IdP +deploys. + +**Phase 2 — Mapping management + agent onboarding.** `emails auth idp +map|list|revoke` (admin/owner session or operator key), IdP auth audit lines, +and the end-to-end agent flow of ADR-0002 once the IdP's service-principal +prerequisite lands. The owner-bundle agent pattern is deprecated the day this +lands: new agents get service principals, never the owner's credentials. + +**Phase 3 — Human sign-in federates.** `emails auth login` gains the IdP path +(OTP via the IdP front door; the IdP mints an `aud: emails` token; emails +verifies it and mints its own short-lived `emss_` session from it, so all +existing session-based server code and the dashboard keep working — sessions +become *derivative* of an IdP authentication event, not an independent root of +trust). Additive column `users.idp_sub` links local users to IdP principals +on first federated login. Password login remains available but is marked +deprecated; new signups are directed to the IdP. The private password/signup +path is put behind an operator flag whose default still allows it (no +breakage). + +**Phase 4 — Credential migration: API keys and send keys.** +1. *Inventory*: enumerate active `hasna_` keys per tenant + (`api_keys ⨝ api_key_tenants`) and send keys (`send_key_tenants`), each with + `last_used_at`. +2. *Re-issue*: for each key, create an IdP service principal (or link the + owning human), add the `idp_principal_tenants` row, and switch the + consumer to IdP tokens. +3. *Cutover per credential*: watch audit lines until the legacy kid goes + quiet; then revoke that key (`emails keys revoke`). Per-key cutover — never + big-bang. +4. *Retire issuance first*: once inventory trends to zero, key **minting** is + disabled (typed 410 pointing at the IdP flow) while **verification** of + the remaining tail continues. The class is removed only at zero inventory, + in a major version. + +**Phase 5 — Sunset the private auth shim.** Remove private signup/OTP/password +login once: (a) every active user has `idp_sub` linked, (b) audit shows zero +password logins and zero legacy-key verifications over an agreed window, and +(c) owner sign-off. Local `users`/`memberships` rows survive as the +authorization/role layer (renamed conceptually to "principal directory"); +`password_hash` and the verification/reset token tables are dropped. Emails' +`tenants` table survives indefinitely — it is the data-plane boundary, not an +identity artifact. + +Phases 2–5 are tracked as tasks under the "IdP federation + agent +signup" umbrella (todos project `1631772c`) with explicit dependencies, +including the three `open-tenants` prerequisite tasks named above. + +## Revocation, stated honestly + +Until the `open-tenants` introspection prerequisite lands, IdP-side revocation +of a principal stops **new** tokens immediately but leaves already-minted +tokens valid for up to their ≤ 24 h TTL against stateless verifiers. Emails +therefore keeps a local kill switch: `idp_principal_tenants.revoked_at` +fails that principal closed on the next request regardless of token validity. +"Revocation via the IdP kills access everywhere" is exact for issuance, +bounded by 24 h for outstanding tokens, and immediate when paired with the +emails-side mapping revocation — ADR-0002 specifies the operator flow. + +## Consequences + +- Agents and humans get one org-wide identity that works across apps; emails + stops being an identity island. Per-agent revocation and audit become real. +- Two new operational requirements: a deployed IdP (prerequisite task) and + JWKS reachability from the emails server (cached, with fail-closed refusal + and a typed unavailability error — never fail-open). +- The emails codebase carries three credential classes during the migration + window. Cost accepted; the dispatch point is a single function + (`resolveRequestContext`) and each class is independently testable. +- The wire contract (issuer string, claims shape, scope grammar) becomes + load-bearing across repos. It must graduate into `@hasna/contracts` so + tenants and every verifying app import one definition — ADR-0002 §Contracts + describes the shape; implementation there is deliberately out of scope here. +- Until contracts owns it, both repos pin the contract values in tests; a + drift breaks a test, not production. diff --git a/docs/adr/0002-agent-identity-signup-and-scopes.md b/docs/adr/0002-agent-identity-signup-and-scopes.md new file mode 100644 index 00000000..7c3bda45 --- /dev/null +++ b/docs/adr/0002-agent-identity-signup-and-scopes.md @@ -0,0 +1,191 @@ +# ADR-0002 — Agent identity: signup flow, scope model, audit, and contracts + +- Status: **Accepted** (owner ruling, 2026-07-28 — see ADR-0001) +- Date: 2026-07-28 +- Deciders: owner; drafted by the tenants-federation working session +- Depends on: [ADR-0001 — Adopt `@hasna/tenants` as the identity authority](0001-adopt-tenants-idp-for-identity.md) + +## Context + +Agents today authenticate to emails by materializing the **owner's** client-env +bundle from the vault: every agent is indistinguishable from the owner, cannot +be individually revoked, and leaves no per-agent audit trail. The owner ruling +makes per-agent identity mandatory: an agent gets its **own revocable +identity**, issued by the org IdP, never a copy of anyone's credential +bundle. + +The IdP building blocks exist (`service_principals` table, `pt: "service"` +token claims) but the issuance surface does not — see the prerequisites named +in ADR-0001. This ADR fixes the end-to-end flow, the CLI ownership, the scope +model, and the audit events so both repos build toward one design. + +## Decision 1 — The signup flow, end to end + +``` + (1) create (2) mint token + operator ──► tenants CLI/API ──► service_principal ──► IdP token (pt=service, + │ in @hasna/tenants aud=emails, scope=[…], + │ sub=, ≤24h) + │ (3) map — one deliberate grant │ + └──► emails: idp_principal_tenants row │ + (sub → emails tenant [+ revoked_at]) ▼ + (4) agent calls emails /v1 with the + token; server verifies via JWKS, + maps sub → tenant, enforces scopes + (5) revoke: IdP disable (kills issuance, + ≤24h residual) and/or emails map + revoke (immediate, mail only) +``` + +1. **Create the agent's identity in the IdP** (owning tenant, display name, + granted scopes). This is the `open-tenants` prerequisite: principal + creation plus an **enrollment credential** the agent holds — the agent + exchanges it daily for short-lived IdP tokens; it never holds a password + or a long-lived app key. Proposed verbs (to land in `open-tenants`): + + ``` + tenants principals create --tenant --name \ + --scope emails:read --scope emails:write + tenants principals token --app emails [--ttl ] + tenants principals disable + tenants principals list --tenant + ``` + +2. **Mint an IdP token** for `--app emails`. Claims per the ADR-0001 wire + contract; `pt: "service"`, `sub` = principal id, TTL ≤ 24 h. + +3. **Grant mail access in emails** — an explicit mapping row, created by an + emails admin/owner (or the operator key): + + ``` + emails auth idp map [--tenant ] [--note ] + emails auth idp list + emails auth idp revoke + ``` + + The mapping records the IdP tenant (`tid`) observed at grant time; a token + whose `tid` no longer matches is refused (a principal moved between IdP + tenants does not silently keep old mail access). + +4. **The agent uses the token** as its bearer credential (client env: + `EMAILS_IDP_TOKEN`); `emails auth whoami` shows the IdP principal, its + emails tenant and effective scopes. The server derives everything from the + token + mapping; the client never sends a tenant. + +5. **Revocation** — two independent kill switches, both auditable: + - IdP: `tenants principals disable` stops all new tokens for every app at + once (residual ≤ 24 h for outstanding tokens until the introspection + prerequisite lands — stated honestly in ADR-0001). + - Emails: `emails auth idp revoke ` sets `revoked_at` on the mapping + and fails that principal closed on the next request — immediate, mail + only. + +### CLI ownership: tenants-CLI-first (rejected: `emails auth agent-signup`) + +Principal **creation lives in the `tenants` CLI**; the `emails` CLI only +accepts the result (verify + map). An `emails auth agent-signup` that creates +IdP principals was rejected because: + +- It would require IdP admin credentials to flow through the emails CLI — a + confused-deputy magnet and precisely the credential-forwarding pattern this + programme removes. +- The pattern must generalize to N apps × 1 IdP. Per-app signup verbs mean N + reimplementations of principal creation and N privileged credential paths; + one IdP-owned verb means one. +- The two-step ceremony (IdP creates WHO; emails grants WHAT-MAIL) is the + security property, not friction: each side's admin makes an explicit, + auditable decision in their own domain. +- Cost accepted: onboarding touches two CLIs. Mitigated by `emails auth idp + map` printing the exact `tenants` command when the referenced principal does + not exist yet, and by one onboarding skill composing both. + +## Decision 2 — Scope model + +IdP tokens carry emails scopes in the shared `:` grammar +(`@hasna/contracts/auth` scope helpers; wildcard on the grant side only): + +| Grant | Meaning | Existing gate it satisfies | +| --- | --- | --- | +| `emails:read` | read-mailbox: list/read messages, domains, addresses, contacts | every read gate | +| `emails:write` | send-as + mutations: send, drafts, contacts, templates | every write gate | +| `emails:*` | admin: everything an API key can do, incl. tenant-operator maintenance (`isTenantOperator`) | wildcard | + +This deliberately reuses the exact scope vocabulary the API-key class already +enforces, so one scope check (`hasAllScopes`) serves all credential classes +and no route grows a special IdP branch. Two constraints named for later, +not smuggled in now: + +- **Finer send-only scope** (`emails:send` distinct from `emails:write`): + requires splitting today's write gate; tracked as scope-registry work in + contracts (below), adopted by emails and the IdP's grant UI together. +- **Send-as address restriction** (an agent may send only from + `agent-x@domain`): an authorization property of the *mapping*, not the + token — a designed extension column on `idp_principal_tenants` + (`allowed_from_addresses`), enforced where send authority is already + checked. Out of the first slices. + +Role gates are unaffected: IdP principals are never `principalType:"user"`, +so human/session-only surfaces (member management, invitations, password +flows) remain unreachable with an IdP token regardless of scopes — same +containment the API-key class has today. + +## Decision 3 — Audit events + +IdP auth reuses the structured, secret-free audit discipline of the API-key +path (`[api-auth]` lines in serve.ts). New events, all carrying `sub`, `jti`, +outcome and typed reason — never the token: + +| Event | When | Minimum fields | +| --- | --- | --- | +| `idp.auth.allow` / `idp.auth.deny` | every IdP-token authentication decision | outcome, sub, tid, jti, kid, reason, method, path, status, at | +| `idp.map.create` | mapping row created | sub, emails tenant, granted by (user id / kid), note | +| `idp.map.revoke` | mapping revoked | sub, emails tenant, revoked by, reason | +| `idp.jwks.refresh` / `idp.jwks.error` | JWKS cache refresh / fetch failure | url host, kid set, error class | + +Phase 1 emits these as structured log lines (parity with `[api-auth]`); a +durable `audit_log` table is a later, separate decision shared with the +API-key path. The IdP audits issuance on its side (`jti` is the join key +between the two audit domains). + +## Decision 4 — What belongs in `@hasna/contracts` (describe only — no +implementation in this programme) + +A new `@hasna/contracts/auth` idp module, so tenants, emails, and every next +app share one definition instead of pinning copies: + +- **Claims schema**: `IdpAccessTokenClaims` = `{ iss, aud, sub, tid, pt, + scope, iat, exp, jti }` with `pt: "user" | "service"`. +- **Wire constants**: the fixed issuer string (`identities` today — renaming it + is a coordinated org-wide change and stays out of scope), the algorithm + (`EdDSA`), the token type (`at+jwt`), the ≤ 24 h TTL ceiling. +- **JWKS shapes**: the Ed25519 public JWK and JWKS document types, plus the + well-known path convention. +- **Verify surface**: options (jwks, expected audience, leeway) and a **closed + set of typed failure reasons** (`malformed | unsupported_alg | missing_kid | + unknown_kid | bad_signature | issuer_mismatch | audience_mismatch | expired | + not_yet_valid`) so every app refuses identically and audit lines are + comparable org-wide. +- **Structural detector** (is this bearer token an IdP JWS?) so credential + dispatch is uniform across apps. +- **Scope registry pattern**: per-app scope-name constants (emails contributes + `emails:read` / `emails:write` / `emails:*`) validated by the existing scope + grammar helpers, giving the IdP's grant surface a source of truth for what + each app accepts. +- **Documented mapping-table pattern**: each verifying app owns a + `idp_principal_tenants`-shaped table (`sub → app tenant`, `tid` pin, + `revoked_at`) — a convention description, not shipped code. + +Until that module exists, open-tenants (`idp/tokens.ts`) and emails (the +Phase-1 verifier) each pin the wire values in their own tests; the contracts +migration then replaces both implementations with one import and deletes the +duplicated pins. + +## Consequences + +- An agent is onboarded in two explicit, auditable steps and revoked in one; + the owner-bundle pattern is retired as Phase 2 of ADR-0001 lands. +- The `open-tenants` prerequisites (deployed instance; principal + create/token/disable; introspection) gate the end-to-end flow and are filed + as tasks there — emails ships its verifying half first, fail-closed. +- One scope vocabulary spans credential classes; scope evolution (send-only, + send-as restriction) has a named home instead of ad-hoc per-app drift. diff --git a/docs/design/multi-tenancy-auth.md b/docs/design/multi-tenancy-auth.md index 524dd0ef..accfab3c 100644 --- a/docs/design/multi-tenancy-auth.md +++ b/docs/design/multi-tenancy-auth.md @@ -65,7 +65,7 @@ sending `Authorization: Bearer `, both fed by one config resolver: - Async mail/inbox: `SelfHostedMailDataSource.request()` — `src/lib/self-hosted-mail-data-source.ts:322` (fetch). - Config: `resolveSelfHostedConfig()` — `src/db/self-hosted-store.ts:70` → `{baseUrl, apiKey}`. - Env (via `src/lib/client-env.ts`): vault pointer `EMAILS_CLIENT_ENV_SECRET` → - `EMAILS_MODE`, `EMAILS_SELF_HOSTED_URL`, `EMAILS_SELF_HOSTED_API_KEY`. + the deployment-mode selector, `EMAILS_SELF_HOSTED_URL`, `EMAILS_SELF_HOSTED_API_KEY`. The code runs on a **single DB DSN** (`EMAILS_DATABASE_URL`, `env.ts`) used by both `migrate.ts` and `serve.ts`. A least-privilege app role is only *aspirational* (a comment @@ -881,7 +881,7 @@ no hybrid synchronization mode. Passing an explicit Bun `Database` handle to the public library always selects that caller-owned SQLite database, even when the process is otherwise configured as a self-hosted client. -The self-hosted schema is additive through migrations 0012–0016: +The self-hosted schema is additive through migrations 0012–0021: - 0012 adds tenant/identity/session/membership/API-key binding and tenant-scopes existing resources with a default-tenant backfill; @@ -891,7 +891,13 @@ The self-hosted schema is additive through migrations 0012–0016: - 0015 adds multiple verified email identities per user, the singleton primary super-admin flag, and an audit-safe bootstrap ledger; - 0016 makes inbound-domain ownership atomic and quarantines/removes stale, - pending, or unverified legacy routes before new writers run. + pending, or unverified legacy routes before new writers run; +- 0017 records immutable inbound source/object provenance; +- 0018 adds durable recovery metadata for uncertain provider send outcomes; +- 0019 adds inbox performance rollups; +- 0020 adds the tenant-scoped attachment-repair ledger; +- 0021 adds the IdP-principal-to-Emails-tenant resolution table described by + ADR-0001. It remains outside RLS because it is read before a tenant is known. User sessions, tenant API keys, invitations, memberships, password reset, email verification, tenant switching, and multiple login email identities are wired @@ -908,10 +914,12 @@ email/KID as paired nullable API-only settings and never hardcodes a person or accepts the token in Terraform. Production ordering is a hard gate: drain old API/worker/ingest writers; deploy -new-code-compatible migration tooling; run migrations through 0016; then start -only the new API and worker tasks. After 0016, never roll back to an old unscoped -writer. A rollback may restore the previous new-code-compatible image, but not a -pre-tenancy writer. +new-code-compatible migration tooling; run migrations through 0021; then start +only API and worker tasks whose migration set recognizes every applied row. +After 0016, never roll back to an old unscoped writer; after 0021, a 0020-only +image is also invalid because the migration ledger fails closed on an unknown +applied row. Recovery is a compatible roll-forward, not a pre-tenancy or +pre-0021 image rollback. Required release evidence is: full local suite, real PostgreSQL migration + multi-tenancy + RLS + message-ID suites, generated SDK sync, public package diff --git a/docs/macos-app.md b/docs/macos-app.md index 62ff64c2..8f0005b7 100644 --- a/docs/macos-app.md +++ b/docs/macos-app.md @@ -1,140 +1,25 @@ -# Emails — macOS desktop app +# macOS desktop app -Emails ships a native macOS desktop app alongside the `@hasna/emails` CLI. It is a UI -copycat of [open-notes](https://github.com/hasna/notes)' "Hasna Notes" app, retargeted to -email: a thin AppKit **WKWebView shell** hosting an offline web UI, bridging real mail -from the local Emails SQLite store. +Status: **not shipped and not buildable from the current repository**. -It lives in the SAME repo as the CLI: +The package currently ships the `emails`, `emails-mcp`, and `emails-serve` +bins, the OpenTUI client (`emails ui`), and the browser dashboard served by +`emails serve`. It does not contain a Swift package or native desktop-app +source: `Package.swift`, `Sources/`, and the former native app's `web/` assets +are absent. -``` -Package.swift swift-tools 6.0, platforms [.macOS("26.0")] -Sources/EmailsCore/ read the SQLite store + build the mutation CLI argv -Sources/EmailsApp/ the WKWebView host (AppKit) — injects __BOOT__, bridges `mail` -Sources/EmailsSmoke/ CLI smoke harness (TDD; XCTest is unavailable under CLT) -web/ the 3-pane mail UI (index.html / styles.css / app.js) -scripts/build_emails_app.sh build + assemble dist/Emails.app (run on a Mac) -scripts/run_on_apple_mac.sh rsync to a Mac, run the smoke test, then build there -``` - -## Architecture — the read/write split (key decision) - -Emails deliberately splits reads from writes: - -- **Reads go straight to SQLite (read-only).** `EmailsCore.MailStore` opens - `~/.hasna/emails/emails.db` with `SQLITE_OPEN_READONLY` and reads `inbound_emails` - (received + synced-sent), plus the outbound `emails` log joined with `email_content`. - This is the fast path that powers the boot payload and every refresh. The app **never - writes to the database itself.** -- **Writes ALWAYS go through the `emails` CLI.** Every mutation — send, reply, mark-read, - archive, star, label, trash/spam, refresh — is delegated to the `emails inbox …` / - `emails send` / `emails refresh` commands. The CLI owns provider auth, - inbound refresh, threading headers (In-Reply-To/References), and write-path invariants; - re-implementing those in Swift would drift from the source of truth and risk corrupting - the shared DB. `EmailsCore.EmailsCLI` builds the exact argv (pure + unit-tested) and - shells out. - -This mirrors open-notes' shell/bridge structure (WKWebView + `__BOOT__` + a message -handler) while swapping the Markdown-file store for the email SQLite store and the -direct-disk writes for CLI-delegated mutations. - -### The bridge contract - -| Notes (reference) | Emails | -|-------------------|---------| -| message handler `notes` | message handler **`mail`** | -| `window.HasnaNotes` | **`window.HasnaMail`** | -| `__BOOT__ = {notes, machines, thisMachine}` | **`__BOOT__ = {threads, folders, thisAddress}`** | - -At launch the shell injects `window.__BOOT__` as a document-start user script, so `app.js` -renders from disk on first paint (no sample fallback in the app). The web posts -`{action, …}` messages to `window.webkit.messageHandlers.mail`; the shell runs the -matching CLI command on a background thread and pushes fresh data back via -`window.HasnaMail.hydrate(...)` plus an `actionResult(...)` ack. - -`app.js` is **dual-mode**: in a plain browser (screenshots/dev) there is no bridge, so it -falls back to `sampleBoot()` and applies mutations optimistically in-memory only. - -#### Actions the web can post on `mail` - -`markRead {id, unread?}` · `archive {id, undo?}` · `star {id, undo?}` · -`label {id, label, remove?}` · `trash {id, confirmed}` · `spam {id, confirmed}` · -`reply {id, body, html?}` · `send {to[], cc[], subject, body, from, html?}` · `refresh {}` · -`shareAttachment {path, requestId}` (→ `attachments upload`). - -## Persistence - -App-level UI state belongs under `~/.hasna/apps/emails/` (e.g. theme is currently in -`localStorage`). The mail data itself is **not** owned by the app — it lives in the shared -`~/.hasna/emails/emails.db` and is managed by the `emails` CLI. - -## HTML email rendering - -Message HTML is isolated in a **sandboxed `