From df6f4d155d9b2336b5072012032d8c81d244e7d7 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Tue, 16 Jun 2026 14:39:30 -0400 Subject: [PATCH 01/28] wrote doc stubs --- docs/README.md | 109 ++++++-- docs/explanations/README.md | 24 ++ docs/explanations/addressing-model.md | 29 +++ docs/explanations/architecture.md | 32 +++ docs/explanations/delivery-model.md | 30 +++ docs/explanations/documentation-system.md | 28 ++ docs/explanations/mail-v1-legacy.md | 29 +++ docs/explanations/mail-v2-overview.md | 30 +++ docs/explanations/security-model.md | 32 +++ docs/howtos/README.md | 28 ++ docs/howtos/authenticate-user-agent.md | 32 +++ docs/howtos/initialize-memory-backend.md | 31 +++ docs/howtos/manage-mailing-lists.md | 34 +++ docs/howtos/manage-swarms.md | 34 +++ docs/howtos/manage-user-agents.md | 35 +++ docs/howtos/regenerate-api-artifacts.md | 34 +++ docs/howtos/run-daemon.md | 30 +++ docs/howtos/run-server.md | 34 +++ docs/howtos/run-tests.md | 32 +++ docs/howtos/send-message-cli.md | 32 +++ docs/references/README.md | 29 +++ docs/references/admin-cli.md | 35 +++ docs/references/client-cli.md | 29 +++ docs/references/configuration.md | 30 +++ docs/references/daemon-cli.md | 30 +++ docs/references/data-models.md | 32 +++ docs/references/http-api.md | 31 +++ docs/references/protocol-specification.md | 31 +++ docs/references/repository-layout.md | 34 +++ docs/references/server-cli.md | 27 ++ docs/references/storage-backends.md | 32 +++ docs/testing-plan.md | 273 -------------------- docs/tutorials/README.md | 22 ++ docs/tutorials/build-minimal-http-client.md | 35 +++ docs/tutorials/run-local-mail.md | 39 +++ docs/tutorials/send-first-message.md | 38 +++ 36 files changed, 1150 insertions(+), 296 deletions(-) create mode 100644 docs/explanations/README.md create mode 100644 docs/explanations/addressing-model.md create mode 100644 docs/explanations/architecture.md create mode 100644 docs/explanations/delivery-model.md create mode 100644 docs/explanations/documentation-system.md create mode 100644 docs/explanations/mail-v1-legacy.md create mode 100644 docs/explanations/mail-v2-overview.md create mode 100644 docs/explanations/security-model.md create mode 100644 docs/howtos/README.md create mode 100644 docs/howtos/authenticate-user-agent.md create mode 100644 docs/howtos/initialize-memory-backend.md create mode 100644 docs/howtos/manage-mailing-lists.md create mode 100644 docs/howtos/manage-swarms.md create mode 100644 docs/howtos/manage-user-agents.md create mode 100644 docs/howtos/regenerate-api-artifacts.md create mode 100644 docs/howtos/run-daemon.md create mode 100644 docs/howtos/run-server.md create mode 100644 docs/howtos/run-tests.md create mode 100644 docs/howtos/send-message-cli.md create mode 100644 docs/references/README.md create mode 100644 docs/references/admin-cli.md create mode 100644 docs/references/client-cli.md create mode 100644 docs/references/configuration.md create mode 100644 docs/references/daemon-cli.md create mode 100644 docs/references/data-models.md create mode 100644 docs/references/http-api.md create mode 100644 docs/references/protocol-specification.md create mode 100644 docs/references/repository-layout.md create mode 100644 docs/references/server-cli.md create mode 100644 docs/references/storage-backends.md delete mode 100644 docs/testing-plan.md create mode 100644 docs/tutorials/README.md create mode 100644 docs/tutorials/build-minimal-http-client.md create mode 100644 docs/tutorials/run-local-mail.md create mode 100644 docs/tutorials/send-first-message.md diff --git a/docs/README.md b/docs/README.md index 3816f3c..4167169 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,34 +1,97 @@ -# MAIL v2 Documentation +# MAIL Documentation -This directory is the root documentation entry point for MAIL v2. +This directory is the canonical documentation home for active MAIL v2 work. +MAIL uses the [Divio documentation system][divio-about], so each page belongs +to exactly one of four categories. -The old MAIL v1 reference implementation docs have been archived under -`src/mail/legacy/docs/`. Use those archived docs only when maintaining or -studying the legacy runtime. +## Start Here -## Repository-Level Docs +- New to MAIL: follow [Run MAIL Locally](tutorials/run-local-mail.md). +- Trying to complete a known task: browse [How-To Guides](howtos/README.md). +- Looking up commands, models, or endpoints: browse [Reference](references/README.md). +- Trying to understand concepts and tradeoffs: browse [Explanations](explanations/README.md). -- [`testing-plan.md`](testing-plan.md) - v2 testing suite overhaul plan - (categories, phases, conventions) +## Categories -## Active Package Docs +- [Tutorials](tutorials/README.md) are learning-oriented lessons. They walk a + beginner through a concrete project and should be tested end to end. +- [How-To Guides](howtos/README.md) are goal-oriented recipes. They answer + "How do I ...?" questions for readers who already know the basics. +- [Reference](references/README.md) is information-oriented lookup material. It + describes commands, APIs, models, configuration, and repository structure. +- [Explanations](explanations/README.md) are understanding-oriented discussions. + They explain why MAIL works the way it does and how its pieces fit together. -- `src/mail/protocol/README.md` - protocol package documentation -- `src/mail/server/docs/` - v2 server documentation -- `src/mail/client/docs/` - v2 client documentation -- `src/mail/daemon/README.md` - daemon package documentation +## Proposed Layout -## Repository-Level Docs To Add +### Tutorials -This root docs area should contain project-wide v2 material that is not owned by -a single package, such as: +- [Run MAIL Locally](tutorials/run-local-mail.md) +- [Send Your First MAIL Message](tutorials/send-first-message.md) +- [Build a Minimal HTTP Client](tutorials/build-minimal-http-client.md) -- repository layout -- release process -- local development workflow -- compatibility and migration notes -- protocol governance and specification process +### How-To Guides -Keep package-specific usage and reference material with the package that owns -the code. +- [Initialize the Memory Backend](howtos/initialize-memory-backend.md) +- [Run the MAIL Server](howtos/run-server.md) +- [Run the MAIL Daemon](howtos/run-daemon.md) +- [Authenticate a User-Agent](howtos/authenticate-user-agent.md) +- [Send a Message with the CLI](howtos/send-message-cli.md) +- [Manage User-Agents](howtos/manage-user-agents.md) +- [Manage Swarms](howtos/manage-swarms.md) +- [Manage Mailing Lists](howtos/manage-mailing-lists.md) +- [Regenerate API Artifacts](howtos/regenerate-api-artifacts.md) +- [Run the Test Suite](howtos/run-tests.md) +### Reference + +- [Repository Layout](references/repository-layout.md) +- [Configuration](references/configuration.md) +- [Protocol Specification](references/protocol-specification.md) +- [HTTP API](references/http-api.md) +- [Client CLI](references/client-cli.md) +- [Admin CLI](references/admin-cli.md) +- [Server CLI](references/server-cli.md) +- [Daemon CLI](references/daemon-cli.md) +- [Data Models](references/data-models.md) +- [Storage Backends](references/storage-backends.md) + +### Explanations + +- [MAIL v2 Overview](explanations/mail-v2-overview.md) +- [Architecture](explanations/architecture.md) +- [Addressing Model](explanations/addressing-model.md) +- [Delivery Model](explanations/delivery-model.md) +- [Security Model](explanations/security-model.md) +- [MAIL v1 Legacy Runtime](explanations/mail-v1-legacy.md) +- [Documentation System](explanations/documentation-system.md) + +## Existing Source Material + +- Protocol source of truth: [../spec/SPEC.md](../spec/SPEC.md) and + [../spec/openapi.yaml](../spec/openapi.yaml) +- Active package docs to migrate or consolidate: + [client docs](../src/mail/client/docs/README.md) and + [server docs](../src/mail/server/docs/README.md) +- Archived MAIL v1 material: + [legacy README](../src/mail/legacy/README.md) and + [legacy docs](../src/mail/legacy/docs/README.md) + +## Writing Rules + +- Keep a page in one category. If a page starts teaching, solving, describing, + and discussing at once, split it. +- Keep tutorials robust and repeatable. They should avoid optional branches and + should show visible progress quickly. +- Keep how-to guides task-focused. Link to explanations instead of pausing for + conceptual discussion. +- Keep reference pages close to the implementation and generated contracts. + Command and API references should point at their source files. +- Keep explanations free to discuss motivation, alternatives, and tradeoffs, but + link out to tutorials, how-tos, and reference pages for action or lookup. + +[divio-about]: https://docs.divio.com/documentation-system/ +[divio-tutorials]: https://docs.divio.com/documentation-system/tutorials/ +[divio-howtos]: https://docs.divio.com/documentation-system/how-to-guides/ +[divio-references]: https://docs.divio.com/documentation-system/reference/ +[divio-explanations]: https://docs.divio.com/documentation-system/explanation/ diff --git a/docs/explanations/README.md b/docs/explanations/README.md new file mode 100644 index 0000000..91a73b9 --- /dev/null +++ b/docs/explanations/README.md @@ -0,0 +1,24 @@ +# Explanations + +Explanations discuss MAIL concepts, motivation, architecture, and tradeoffs. +They are for understanding, not for step-by-step tasks or exhaustive lookup. + +## Planned Explanations + +| Page | Question it answers | +| --- | --- | +| [MAIL v2 Overview](mail-v2-overview.md) | What is MAIL and what problem is v2 trying to solve? | +| [Architecture](architecture.md) | How do protocol, server, client, daemon, and backend pieces fit together? | +| [Addressing Model](addressing-model.md) | Why does MAIL use host-scoped and swarm-scoped addresses? | +| [Delivery Model](delivery-model.md) | Why are daemons responsible for message delivery? | +| [Security Model](security-model.md) | What are the main trust boundaries and risks? | +| [MAIL v1 Legacy Runtime](mail-v1-legacy.md) | How should readers interpret the archived v1 runtime and docs? | +| [Documentation System](documentation-system.md) | How should maintainers decide where a new page belongs? | + +## Explanation Checklist + +- Start with a concrete question or tension. +- Discuss tradeoffs and alternatives. +- Link to tutorials for learning paths. +- Link to how-to guides for tasks. +- Link to reference pages for commands, fields, and exact contracts. diff --git a/docs/explanations/addressing-model.md b/docs/explanations/addressing-model.md new file mode 100644 index 0000000..5a09614 --- /dev/null +++ b/docs/explanations/addressing-model.md @@ -0,0 +1,29 @@ +# Addressing Model + +Status: stub + +## Question + +Why does MAIL have both host-scoped and swarm-scoped addresses, and how should +readers reason about each form? + +## Source Material + +- `spec/SPEC.md` section 6 +- `src/mail/protocol/src/mail_protocol/core/user_agents.py` +- `src/mail/protocol/src/mail_protocol/core/validators.py` +- `tests/contract/test_spec_addresses.py` + +## Topics to Discuss + +- Host-scoped identities for users, admins, and daemons. +- Swarm-scoped identities for agents and mailing lists. +- Why agent names can repeat across swarms. +- Why mailing list addresses live inside swarms. +- Validation boundaries and common mistakes. + +## Related Pages + +- [Data Models](../references/data-models.md) +- [Manage User-Agents](../howtos/manage-user-agents.md) +- [Manage Mailing Lists](../howtos/manage-mailing-lists.md) diff --git a/docs/explanations/architecture.md b/docs/explanations/architecture.md new file mode 100644 index 0000000..ae8dcca --- /dev/null +++ b/docs/explanations/architecture.md @@ -0,0 +1,32 @@ +# Architecture + +Status: stub + +## Question + +How do the active MAIL v2 packages and runtime components fit together? + +## Source Material + +- `README.md` +- `pyproject.toml` +- `src/mail/protocol/` +- `src/mail/server/` +- `src/mail/client/` +- `src/mail/daemon/` +- `spec/SPEC.md` section 4 + +## Topics to Discuss + +- Protocol package as shared data and network contracts. +- Server package as the HTTP implementation and state owner. +- Client package as the CLI user-agent interface. +- Daemon package as the delivery worker. +- Backend abstraction and current memory backend. +- OpenAPI and contract tests as cross-package alignment points. + +## Related Pages + +- [Repository Layout](../references/repository-layout.md) +- [HTTP API](../references/http-api.md) +- [Storage Backends](../references/storage-backends.md) diff --git a/docs/explanations/delivery-model.md b/docs/explanations/delivery-model.md new file mode 100644 index 0000000..a4535da --- /dev/null +++ b/docs/explanations/delivery-model.md @@ -0,0 +1,30 @@ +# Delivery Model + +Status: stub + +## Question + +Why does MAIL store sent messages first and rely on daemons for delivery? + +## Source Material + +- `spec/SPEC.md` section 8 +- `src/mail/daemon/src/mail_daemon/maild/api.py` +- `src/mail/server/src/mail_server/routers/daemon.py` +- `src/mail/server/src/mail_server/backends/base.py` +- `tests/integration/test_flows.py` + +## Topics to Discuss + +- Draft creation versus sent message creation. +- Server delivery buffer. +- Daemon authorization and delivery responsibility. +- Local delivery versus future remote delivery concerns. +- Pre-send validation failures versus post-send delivery failures. +- Operational implications for retries, observability, and idempotency. + +## Related Pages + +- [Run the MAIL Daemon](../howtos/run-daemon.md) +- [Daemon CLI](../references/daemon-cli.md) +- [HTTP API](../references/http-api.md) diff --git a/docs/explanations/documentation-system.md b/docs/explanations/documentation-system.md new file mode 100644 index 0000000..faf2812 --- /dev/null +++ b/docs/explanations/documentation-system.md @@ -0,0 +1,28 @@ +# Documentation System + +Status: stub + +## Question + +How should MAIL maintainers decide whether a new page is a tutorial, how-to +guide, reference page, or explanation? + +## Source Material + +- `docs/README.md` +- Divio documentation system: + +## Topics to Discuss + +- The four documentation types and what each optimizes for. +- Why mixed-purpose pages become hard to maintain. +- How to split a proposed page that has multiple purposes. +- Naming conventions for each category. +- Migration strategy for package-local docs and legacy docs. + +## Related Pages + +- [Tutorials](../tutorials/README.md) +- [How-To Guides](../howtos/README.md) +- [Reference](../references/README.md) +- [Explanations](README.md) diff --git a/docs/explanations/mail-v1-legacy.md b/docs/explanations/mail-v1-legacy.md new file mode 100644 index 0000000..297fcc3 --- /dev/null +++ b/docs/explanations/mail-v1-legacy.md @@ -0,0 +1,29 @@ +# MAIL v1 Legacy Runtime + +Status: stub + +## Question + +How should readers understand the archived MAIL v1 runtime while working on +active MAIL v2 documentation and code? + +## Source Material + +- `README.md` +- `src/mail/legacy/README.md` +- `src/mail/legacy/docs/README.md` +- `src/mail/legacy/` + +## Topics to Discuss + +- Why v1 code is archived under `src/mail/legacy`. +- Which docs are historical reference versus active v2 guidance. +- When legacy tests should still be run. +- How to avoid importing v1 architecture assumptions into v2 docs. +- Migration notes worth preserving. + +## Related Pages + +- [Repository Layout](../references/repository-layout.md) +- [MAIL v2 Overview](mail-v2-overview.md) +- [Run the Test Suite](../howtos/run-tests.md) diff --git a/docs/explanations/mail-v2-overview.md b/docs/explanations/mail-v2-overview.md new file mode 100644 index 0000000..c5f8ba7 --- /dev/null +++ b/docs/explanations/mail-v2-overview.md @@ -0,0 +1,30 @@ +# MAIL v2 Overview + +Status: stub + +## Question + +What is MAIL v2, what problem does it solve, and what is intentionally outside +its scope? + +## Source Material + +- `README.md` +- `spec/SPEC.md` sections 1 through 4 +- `src/mail/legacy/README.md` + +## Topics to Discuss + +- MAIL as an email-like communication layer for humans, agents, daemons, and + swarms. +- The shift from v1 runtime concerns to v2 communication concerns. +- Why the active repo is split into protocol, server, client, and daemon + packages. +- What MAIL is not: not a full agent runtime and not a universal communication + layer for every agent interaction. + +## Related Pages + +- [Run MAIL Locally](../tutorials/run-local-mail.md) +- [Protocol Specification](../references/protocol-specification.md) +- [MAIL v1 Legacy Runtime](mail-v1-legacy.md) diff --git a/docs/explanations/security-model.md b/docs/explanations/security-model.md new file mode 100644 index 0000000..8811814 --- /dev/null +++ b/docs/explanations/security-model.md @@ -0,0 +1,32 @@ +# Security Model + +Status: stub + +## Question + +What are MAIL trust boundaries, credential risks, and minimum operational +expectations? + +## Source Material + +- `spec/SPEC.md` section 9 +- `src/mail/server/src/mail_server/auth.py` +- `src/mail/server/src/mail_server/routers/auth.py` +- `src/mail/server/.env.example` +- `tests/integration/test_auth.py` +- `tests/integration/test_authz.py` + +## Topics to Discuss + +- User-agent credentials and bearer tokens. +- Admin power and account creation risks. +- Daemon privileges. +- Secret handling in CLI and environment variables. +- TLS and reverse proxy expectations for production. +- Logging risks for message content and credentials. + +## Related Pages + +- [Authenticate a User-Agent](../howtos/authenticate-user-agent.md) +- [Configuration](../references/configuration.md) +- [Protocol Specification](../references/protocol-specification.md) diff --git a/docs/howtos/README.md b/docs/howtos/README.md new file mode 100644 index 0000000..fbe8df3 --- /dev/null +++ b/docs/howtos/README.md @@ -0,0 +1,28 @@ +# How-To Guides + +How-to guides solve specific MAIL tasks for readers who already know the basic +shape of the system. Each guide should start from a realistic state, provide +ordered steps, and stop when the task is complete. + +## Planned Guides + +| Page | Task | +| --- | --- | +| [Initialize the Memory Backend](initialize-memory-backend.md) | Create local server state for development or testing. | +| [Run the MAIL Server](run-server.md) | Start and configure `mail-server`. | +| [Run the MAIL Daemon](run-daemon.md) | Start `mail-daemon` against an existing server. | +| [Authenticate a User-Agent](authenticate-user-agent.md) | Obtain and use a MAIL bearer token. | +| [Send a Message with the CLI](send-message-cli.md) | Compose and send a message from the command line. | +| [Manage User-Agents](manage-user-agents.md) | Create, inspect, and remove agents, users, admins, and daemons. | +| [Manage Swarms](manage-swarms.md) | Create, inspect, and delete swarms. | +| [Manage Mailing Lists](manage-mailing-lists.md) | Create lists and manage subscriptions or members. | +| [Regenerate API Artifacts](regenerate-api-artifacts.md) | Refresh generated OpenAPI or documentation artifacts. | +| [Run the Test Suite](run-tests.md) | Run focused or full repository tests. | + +## How-To Checklist + +- Title the page as "How to ...". +- Assume the reader knows what outcome they want. +- Provide commands and expected success checks. +- Link to reference material for exhaustive options. +- Link to explanations for background instead of embedding long discussion. diff --git a/docs/howtos/authenticate-user-agent.md b/docs/howtos/authenticate-user-agent.md new file mode 100644 index 0000000..6cfc32f --- /dev/null +++ b/docs/howtos/authenticate-user-agent.md @@ -0,0 +1,32 @@ +# Authenticate a User-Agent + +Status: stub + +## Goal + +How to exchange MAIL address credentials for a bearer token and use that token +with CLI or HTTP requests. + +## Starting Point + +The reader has a server URL, a MAIL address, and a password. + +## Source Material + +- `src/mail/client/src/mail_client/commands/login.py` +- `src/mail/client/src/mail_client/commands/whoami.py` +- `src/mail/server/src/mail_server/routers/auth.py` +- `spec/openapi.yaml` + +## Steps to Cover + +1. Set `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. +2. Run `mail login`. +3. Store the returned token in `MAIL_TOKEN`. +4. Run `mail whoami`. +5. Use the token in an HTTP `Authorization: Bearer ...` header. +6. Refresh or replace expired tokens. + +## Validation + +`mail whoami` returns the expected user-agent type and address. diff --git a/docs/howtos/initialize-memory-backend.md b/docs/howtos/initialize-memory-backend.md new file mode 100644 index 0000000..9193408 --- /dev/null +++ b/docs/howtos/initialize-memory-backend.md @@ -0,0 +1,31 @@ +# Initialize the Memory Backend + +Status: stub + +## Goal + +How to create a local memory backend with initial swarms, user-agents, and +credentials for development. + +## Starting Point + +The repository is cloned, dependencies are installed, and the reader wants local +server state for `mail-server --backend memory`. + +## Source Material + +- `src/mail/server/src/mail_server/backend_init.py` +- `src/mail/server/src/mail_server/backends/memory/init.py` +- `src/mail/server/docs/tutorials/quickstart.md` + +## Steps to Cover + +1. Run `uv run backend-init`. +2. Customize deployment, swarm, host, agents, daemons, users, or admins. +3. Locate generated credential files. +4. Remove or protect plaintext password files after capture. +5. Reinitialize clean state when needed. + +## Validation + +The server starts successfully and the generated user-agents can authenticate. diff --git a/docs/howtos/manage-mailing-lists.md b/docs/howtos/manage-mailing-lists.md new file mode 100644 index 0000000..00c2e00 --- /dev/null +++ b/docs/howtos/manage-mailing-lists.md @@ -0,0 +1,34 @@ +# Manage Mailing Lists + +Status: stub + +## Goal + +How to create mailing lists, inspect them, and manage subscriptions or members. + +## Starting Point + +The reader has credentials with permissions appropriate for the list action. + +## Source Material + +- `src/mail/client/src/mail_client/commands/lists.py` +- `src/mail/client/src/mail_client/commands/list_post.py` +- `src/mail/client/src/mail_client/commands/list_subscribe.py` +- `src/mail/client/src/mail_client/commands/list_member_post.py` +- `src/mail/server/src/mail_server/routers/lists.py` +- `src/mail/protocol/src/mail_protocol/core/lists.py` + +## Steps to Cover + +1. List available mailing lists. +2. Inspect one list by address. +3. Create a list as an admin. +4. Subscribe and unsubscribe as a user-agent. +5. Add or remove members as an admin. +6. Send a message to a list address. + +## Validation + +List membership changes are reflected in list lookup and list-address sends +deliver to expected recipients. diff --git a/docs/howtos/manage-swarms.md b/docs/howtos/manage-swarms.md new file mode 100644 index 0000000..8fc46ff --- /dev/null +++ b/docs/howtos/manage-swarms.md @@ -0,0 +1,34 @@ +# Manage Swarms + +Status: stub + +## Goal + +How to create, inspect, and delete MAIL swarms. + +## Starting Point + +The reader has an admin token for changes or a regular user-agent token for +read-only swarm inspection. + +## Source Material + +- `src/mail/client/src/mail_client/commands/swarm_list.py` +- `src/mail/client/src/mail_client/commands/swarm_get.py` +- `src/mail/client/src/mail_client/commands/swarm_post.py` +- `src/mail/client/src/mail_client/commands/swarm_delete.py` +- `src/mail/server/src/mail_server/routers/swarms.py` +- `src/mail/server/src/mail_server/routers/admin.py` + +## Steps to Cover + +1. List swarms. +2. Inspect a swarm by name. +3. Create a swarm as an admin. +4. Add initial description, keywords, and agents. +5. Delete a test swarm. + +## Validation + +The created swarm is visible through public swarm lookup and removable through +admin commands. diff --git a/docs/howtos/manage-user-agents.md b/docs/howtos/manage-user-agents.md new file mode 100644 index 0000000..34dea17 --- /dev/null +++ b/docs/howtos/manage-user-agents.md @@ -0,0 +1,35 @@ +# Manage User-Agents + +Status: stub + +## Goal + +How to create, inspect, and remove MAIL agents, users, admins, and daemons with +administrator tooling. + +## Starting Point + +The reader has an admin token for the target server. + +## Source Material + +- `src/mail/client/docs/reference/admin-panel.md` +- `src/mail/client/src/mail_client/admin_panel.py` +- `src/mail/client/src/mail_client/commands/agent_post.py` +- `src/mail/client/src/mail_client/commands/user_post.py` +- `src/mail/client/src/mail_client/commands/daemon_post.py` +- `src/mail/server/src/mail_server/routers/admin.py` + +## Steps to Cover + +1. Authenticate as an admin. +2. List existing user-agents by type. +3. Create an agent in a swarm. +4. Create a host-scoped user or daemon. +5. Inspect the created user-agent. +6. Delete a test user-agent. + +## Validation + +Created user-agents appear in admin list/get commands and can authenticate when +credentials are valid. diff --git a/docs/howtos/regenerate-api-artifacts.md b/docs/howtos/regenerate-api-artifacts.md new file mode 100644 index 0000000..e0e455a --- /dev/null +++ b/docs/howtos/regenerate-api-artifacts.md @@ -0,0 +1,34 @@ +# Regenerate API Artifacts + +Status: stub + +## Goal + +How to refresh generated API or documentation artifacts after protocol, router, +or model changes. + +## Starting Point + +The reader changed FastAPI routes, protocol models, or generated documentation +inputs. + +## Source Material + +- `scripts/generate_openapi.py` +- `scripts/build_llms_txt.py` +- `scripts/build_third_party_licenses.py` +- `spec/openapi.yaml` +- `llms.txt` +- `THIRD_PARTY_NOTICES.md` + +## Steps to Cover + +1. Regenerate OpenAPI output. +2. Validate OpenAPI drift tests. +3. Rebuild `llms.txt` when docs or specs change. +4. Rebuild third-party license notices when dependencies change. +5. Review diffs before committing generated files. + +## Validation + +Contract tests pass and generated files only contain expected changes. diff --git a/docs/howtos/run-daemon.md b/docs/howtos/run-daemon.md new file mode 100644 index 0000000..3f4eb01 --- /dev/null +++ b/docs/howtos/run-daemon.md @@ -0,0 +1,30 @@ +# Run the MAIL Daemon + +Status: stub + +## Goal + +How to start `mail-daemon` so pending local messages are delivered by an +authorized daemon user-agent. + +## Starting Point + +A MAIL server is running and daemon credentials exist. + +## Source Material + +- `src/mail/daemon/src/mail_daemon/cli.py` +- `src/mail/daemon/src/mail_daemon/maild/api.py` +- `spec/SPEC.md` section 8 + +## Steps to Cover + +1. Set `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. +2. Start `uv run mail-daemon`. +3. Adjust console or file log levels. +4. Confirm the daemon obtains a token. +5. Send a message and watch delivery complete. + +## Validation + +Messages move from the server delivery buffer into recipient inboxes. diff --git a/docs/howtos/run-server.md b/docs/howtos/run-server.md new file mode 100644 index 0000000..17ef2a9 --- /dev/null +++ b/docs/howtos/run-server.md @@ -0,0 +1,34 @@ +# Run the MAIL Server + +Status: stub + +## Goal + +How to start `mail-server` with the desired host, port, backend, and memory +checkpoint behavior. + +## Starting Point + +The backend has already been initialized and required environment variables are +available. + +## Source Material + +- `src/mail/server/src/mail_server/cli.py` +- `src/mail/server/src/mail_server/server.py` +- `src/mail/server/.env.example` +- `src/mail/server/docs/reference/cli.md` + +## Steps to Cover + +1. Set required JWT and host environment variables. +2. Start `uv run mail-server`. +3. Override `--host` and `--port`. +4. Select `--backend memory`. +5. Tune or disable `--memory-save-interval`. +6. Verify `GET /health` or `mail ping`. + +## Validation + +The server responds with healthy status and the root endpoint reports MAIL +protocol metadata. diff --git a/docs/howtos/run-tests.md b/docs/howtos/run-tests.md new file mode 100644 index 0000000..78722b4 --- /dev/null +++ b/docs/howtos/run-tests.md @@ -0,0 +1,32 @@ +# Run the Test Suite + +Status: stub + +## Goal + +How to run the active MAIL v2 test suite, focused test groups, contract tests, +and archived v1 tests when needed. + +## Starting Point + +The workspace dependencies are installed. + +## Source Material + +- `pytest.ini` +- `tests/` +- `src/mail/legacy/tests/` +- `pyproject.toml` + +## Steps to Cover + +1. Run all active v2 tests with `uv run pytest`. +2. Run unit, integration, contract, or end-to-end subsets. +3. Run coverage with the configured source packages. +4. Run archived v1 tests with the `legacy` extra when needed. +5. Interpret failures from OpenAPI drift and protocol contract tests. + +## Validation + +The selected test command exits successfully and any expected skipped or xfailed +tests are understood. diff --git a/docs/howtos/send-message-cli.md b/docs/howtos/send-message-cli.md new file mode 100644 index 0000000..2ea4bff --- /dev/null +++ b/docs/howtos/send-message-cli.md @@ -0,0 +1,32 @@ +# Send a Message with the CLI + +Status: stub + +## Goal + +How to compose a draft and send it to one or more MAIL recipients using the +`mail` CLI. + +## Starting Point + +The reader has a valid `MAIL_TOKEN` for a user-agent allowed to send messages. + +## Source Material + +- `src/mail/client/src/mail_client/commands/compose.py` +- `src/mail/client/src/mail_client/commands/send.py` +- `src/mail/client/docs/reference/cli.md` + +## Steps to Cover + +1. Compose a draft with subject and body. +2. Capture the draft ID. +3. Send the draft to one or more recipient addresses. +4. Inspect the outbox entry. +5. Inspect the recipient inbox when possible. +6. Handle malformed address or validation failures. + +## Validation + +The sender has an outbox message and the recipient can open the delivered inbox +message after daemon delivery. diff --git a/docs/references/README.md b/docs/references/README.md new file mode 100644 index 0000000..15d96f2 --- /dev/null +++ b/docs/references/README.md @@ -0,0 +1,29 @@ +# Reference + +Reference pages describe MAIL machinery: commands, endpoints, models, +configuration, repository layout, and implementation-defined behavior. +Reference material should be terse, structured consistently, and tied to source +files or generated contracts. + +## Planned Reference Pages + +| Page | Describes | Source of truth | +| --- | --- | --- | +| [Repository Layout](repository-layout.md) | Active workspace, specs, tests, scripts, and legacy code. | `README.md`, `pyproject.toml` | +| [Configuration](configuration.md) | Environment variables and runtime settings. | CLI modules, `.env.example` | +| [Protocol Specification](protocol-specification.md) | Normative MAIL protocol documents. | `spec/SPEC.md`, `spec/openapi.yaml` | +| [HTTP API](http-api.md) | REST endpoints, auth, request and response bodies. | `spec/openapi.yaml`, FastAPI routers | +| [Client CLI](client-cli.md) | `mail` commands and options. | `src/mail/client/src/mail_client/cli.py` | +| [Admin CLI](admin-cli.md) | Administrator commands and options. | `src/mail/client/src/mail_client/admin_panel.py` | +| [Server CLI](server-cli.md) | `mail-server` options. | `src/mail/server/src/mail_server/cli.py` | +| [Daemon CLI](daemon-cli.md) | `mail-daemon` options and env vars. | `src/mail/daemon/src/mail_daemon/cli.py` | +| [Data Models](data-models.md) | Pydantic models used by protocol and network contracts. | `src/mail/protocol/src/mail_protocol/` | +| [Storage Backends](storage-backends.md) | Backend interfaces and memory backend behavior. | `src/mail/server/src/mail_server/backends/` | + +## Reference Checklist + +- Match structure to code structure where practical. +- Prefer tables for options, fields, and endpoint summaries. +- Include examples only to clarify syntax. +- Link to tutorials and how-tos instead of becoming step-by-step guidance. +- Update reference pages in the same change as command, API, or model changes. diff --git a/docs/references/admin-cli.md b/docs/references/admin-cli.md new file mode 100644 index 0000000..ec7719d --- /dev/null +++ b/docs/references/admin-cli.md @@ -0,0 +1,35 @@ +# Admin CLI + +Status: stub + +## Scope + +Describe the administrator command-line surface for managing MAIL server +resources. + +## Source of Truth + +- `src/mail/client/src/mail_client/admin_panel.py` +- `src/mail/client/src/mail_client/commands/agent_*.py` +- `src/mail/client/src/mail_client/commands/user_*.py` +- `src/mail/client/src/mail_client/commands/daemon_*.py` +- `src/mail/client/src/mail_client/commands/swarm_*.py` +- `src/mail/client/src/mail_client/commands/webhook_*.py` +- `src/mail/client/src/mail_client/commands/list_*.py` +- `src/mail/client/docs/reference/admin-panel.md` + +## Entries to Cover + +- Agent operations. +- User operations. +- Admin operations if exposed. +- Daemon operations. +- Swarm operations. +- Webhook operations. +- Mailing list administration. +- Shared auth and output options. + +## Maintenance Notes + +Call out destructive commands clearly, but keep procedural guidance in how-to +pages. diff --git a/docs/references/client-cli.md b/docs/references/client-cli.md new file mode 100644 index 0000000..fef1e00 --- /dev/null +++ b/docs/references/client-cli.md @@ -0,0 +1,29 @@ +# Client CLI + +Status: stub + +## Scope + +Describe the `mail` command-line interface for regular MAIL user-agent +operations. + +## Source of Truth + +- `src/mail/client/src/mail_client/cli.py` +- `src/mail/client/src/mail_client/commands/` +- `src/mail/client/docs/reference/cli.md` + +## Entries to Cover + +- Usage. +- Global output options. +- Utility commands. +- Messaging commands. +- Swarm helper commands. +- Mailing list helper commands. +- Environment variables consumed by commands. +- Output formats. + +## Maintenance Notes + +Regenerate or review this page whenever command parsers or aliases change. diff --git a/docs/references/configuration.md b/docs/references/configuration.md new file mode 100644 index 0000000..614c875 --- /dev/null +++ b/docs/references/configuration.md @@ -0,0 +1,30 @@ +# Configuration + +Status: stub + +## Scope + +List environment variables, CLI defaults, and runtime settings for active MAIL +v2 packages. + +## Source of Truth + +- `src/mail/server/.env.example` +- `src/mail/server/src/mail_server/cli.py` +- `src/mail/server/src/mail_server/server.py` +- `src/mail/daemon/src/mail_daemon/maild/api.py` +- `src/mail/client/src/mail_client/commands/` + +## Entries to Cover + +- Server JWT settings. +- `MAIL_HOST`. +- Memory backend checkpoint interval. +- CLI client `MAIL_SERVER`, `MAIL_ADDRESS`, `MAIL_PASSWORD`, and `MAIL_TOKEN`. +- Daemon `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. +- Defaults for host, port, backend, and log levels. + +## Maintenance Notes + +Keep secrets out of examples. Use clearly fake values and link to security +guidance for production deployment decisions. diff --git a/docs/references/daemon-cli.md b/docs/references/daemon-cli.md new file mode 100644 index 0000000..386bdf9 --- /dev/null +++ b/docs/references/daemon-cli.md @@ -0,0 +1,30 @@ +# Daemon CLI + +Status: stub + +## Scope + +Describe the `mail-daemon` CLI, required environment variables, and daemon loop +behavior. + +## Source of Truth + +- `src/mail/daemon/src/mail_daemon/cli.py` +- `src/mail/daemon/src/mail_daemon/maild/api.py` +- `spec/SPEC.md` section 8 + +## Entries to Cover + +- Usage. +- Log-level options. +- Required environment variables. +- Server validation. +- Token acquisition. +- Message buffer clearing. +- Local delivery. +- Retry and error behavior. + +## Maintenance Notes + +Keep behavioral details factual and tied to implementation. Deeper discussion of +why delivery is daemon-driven belongs in explanations. diff --git a/docs/references/data-models.md b/docs/references/data-models.md new file mode 100644 index 0000000..7c95aea --- /dev/null +++ b/docs/references/data-models.md @@ -0,0 +1,32 @@ +# Data Models + +Status: stub + +## Scope + +Describe active MAIL protocol Pydantic models and the validation rules that +shape request, response, and core data contracts. + +## Source of Truth + +- `src/mail/protocol/src/mail_protocol/core/` +- `src/mail/protocol/src/mail_protocol/network/` +- `src/mail/protocol/src/mail_protocol/core/validators.py` +- `spec/openapi.yaml` + +## Entries to Cover + +- User-agents. +- Addresses. +- Messages and message summaries. +- Drafts. +- Inboxes, outboxes, and trash. +- Swarms. +- Mailing lists. +- Webhooks. +- Request and response models. + +## Maintenance Notes + +Use field tables with type, required status, validation notes, and links to +source classes. Avoid duplicating generated OpenAPI schemas in full. diff --git a/docs/references/http-api.md b/docs/references/http-api.md new file mode 100644 index 0000000..c506df4 --- /dev/null +++ b/docs/references/http-api.md @@ -0,0 +1,31 @@ +# HTTP API + +Status: stub + +## Scope + +Describe the HTTP API exposed by the MAIL server, with endpoints, auth +requirements, request bodies, response bodies, and error behavior. + +## Source of Truth + +- `spec/openapi.yaml` +- `src/mail/server/src/mail_server/server.py` +- `src/mail/server/src/mail_server/routers/` +- `src/mail/server/docs/reference/http.md` + +## Entries to Cover + +- Root and health endpoints. +- Authentication endpoints. +- Swarms. +- Inbox, outbox, drafts, and trash. +- Daemon endpoints. +- Admin endpoints. +- Mailing list endpoints. +- Common status codes and validation errors. + +## Maintenance Notes + +Prefer generated OpenAPI details for schemas and parameters. Keep handwritten +text focused on navigation and important implementation notes. diff --git a/docs/references/protocol-specification.md b/docs/references/protocol-specification.md new file mode 100644 index 0000000..2e697e8 --- /dev/null +++ b/docs/references/protocol-specification.md @@ -0,0 +1,31 @@ +# Protocol Specification + +Status: stub + +## Scope + +Point readers to the normative MAIL protocol materials and summarize how they +relate to implementation reference pages. + +## Source of Truth + +- `spec/SPEC.md` +- `spec/openapi.yaml` +- `tests/contract/` + +## Entries to Cover + +- Protocol version and status. +- Requirements language. +- User-agent categories. +- Address forms. +- Message contract. +- Delivery responsibilities. +- Security considerations. +- OpenAPI contract role. +- Contract test coverage. + +## Maintenance Notes + +Do not duplicate the full specification here. This page should orient readers +and link to exact normative files. diff --git a/docs/references/repository-layout.md b/docs/references/repository-layout.md new file mode 100644 index 0000000..01cafb3 --- /dev/null +++ b/docs/references/repository-layout.md @@ -0,0 +1,34 @@ +# Repository Layout + +Status: stub + +## Scope + +Describe the MAIL v2 repository structure, workspace package boundaries, specs, +tests, scripts, and archived v1 runtime location. + +## Source of Truth + +- `README.md` +- `pyproject.toml` +- `src/mail/*/pyproject.toml` +- `tests/` +- `scripts/` +- `src/mail/legacy/` + +## Entries to Cover + +- Root meta-package. +- `src/mail/protocol` +- `src/mail/server` +- `src/mail/client` +- `src/mail/daemon` +- `spec/` +- `tests/` +- `scripts/` +- `src/mail/legacy` + +## Maintenance Notes + +Update this page when workspace members, top-level directories, or package +responsibilities change. diff --git a/docs/references/server-cli.md b/docs/references/server-cli.md new file mode 100644 index 0000000..435093c --- /dev/null +++ b/docs/references/server-cli.md @@ -0,0 +1,27 @@ +# Server CLI + +Status: stub + +## Scope + +Describe the `mail-server` CLI and related backend initialization command. + +## Source of Truth + +- `src/mail/server/src/mail_server/cli.py` +- `src/mail/server/src/mail_server/backend_init.py` +- `src/mail/server/docs/reference/cli.md` + +## Entries to Cover + +- `mail-server` usage. +- Host and port options. +- Backend selection. +- Memory checkpoint interval. +- `backend-init` usage. +- Backend initialization options. +- Environment variables required at runtime. + +## Maintenance Notes + +Keep default values synchronized with parser defaults. diff --git a/docs/references/storage-backends.md b/docs/references/storage-backends.md new file mode 100644 index 0000000..47d4073 --- /dev/null +++ b/docs/references/storage-backends.md @@ -0,0 +1,32 @@ +# Storage Backends + +Status: stub + +## Scope + +Describe the server backend interface and the memory backend persistence +behavior. + +## Source of Truth + +- `src/mail/server/src/mail_server/backends/base.py` +- `src/mail/server/src/mail_server/backends/memory/` +- `src/mail/server/src/mail_server/backend_init.py` +- `tests/unit/test_memory_fs_roundtrip.py` +- `tests/unit/test_memory_checkpointing.py` + +## Entries to Cover + +- Backend lifecycle hooks. +- User-agent storage. +- Box storage. +- Draft and trash behavior. +- Message delivery buffer. +- Memory backend filesystem layout. +- Checkpoint behavior. +- Current backend limitations. + +## Maintenance Notes + +Keep production deployment advice in how-to or explanation pages unless it is a +direct backend capability or limitation. diff --git a/docs/testing-plan.md b/docs/testing-plan.md deleted file mode 100644 index 462a2ce..0000000 --- a/docs/testing-plan.md +++ /dev/null @@ -1,273 +0,0 @@ -# MAIL v2 Testing Suite Overhaul Plan - -**Status:** In effect — Phases 0–5 landed -**Date:** 2026-06-12 -**Scope:** The v2 packages (`mail-swarms-protocol`, `mail-swarms-server`, -`mail-swarms-daemon`, `mail-swarms-client`) and the repository-level `tests/` -suite. The legacy suite under -`src/mail/legacy/tests/` is out of scope and remains frozen. - ---- - -## 1. Background - -The v2 codebase (~99 source files, ~11.9k LOC across four packages) has -outpaced its test suite by roughly 8:1 in commit volume. The existing suite -(91 tests, all under `tests/unit/`) is well-constructed but covers only the -mailing-lists feature plus CLI parser shape — an estimated 5–8% of the v2 -surface. Major subsystems with **zero** coverage today: - -- the auth layer (`mail_server.auth`: JWT issue/verify, argon2 hashing, - role-validation dependencies) — existing endpoint tests monkeypatch it away -- every server router except `lists` (inbox, outbox, drafts, trash, swarms, - admin, daemon, auth — ~40 endpoints) -- the webhook delivery pipeline (HMAC-SHA256 signing, `X-MAIL-Signature`, - 6-step retry ladder) — six recent fix commits shipped with no tests -- all `mail_client` command behavior (only parser shape is tested) -- the entire `mail_daemon` package -- ~25 of 30 `mail_protocol` validators and most model `summarize()` paths -- ~40 `MemoryBackend` methods beyond the list-store group - -One test currently fails -(`test_mail_lists_endpoints.py::test_subscribe_other_rejected_with_403`) due -to drift from commit `570a340` — see Open Decisions (§7). - -## 2. Goals - -1. Establish four test categories — **unit**, **integration**, **contract**, - **e2e** — with clear ownership boundaries, so every future v2 change has an - obvious place for its tests. -2. Cover the subsystems where bugs have actually shipped (webhooks, auth, - routers) first. -3. Make spec drift mechanically detectable: the implementation, `spec/SPEC.md`, - and `spec/openapi.yaml` must not be able to diverge silently. -4. Wire coverage measurement so the gap stays visible. - -### Non-goals - -- Restoring or extending the legacy (`mail.legacy.*`) test suite. -- Performance/load testing (revisit after v2 stabilizes). -- Testing `daemon_deliver_remote` and other interswarm paths beyond stub - tracking — the feature itself is not implemented yet. - -## 3. Target suite architecture - -### 3.1 Directory layout - -``` -tests/ - conftest.py # shared fixtures (see §3.3) - unit/ # pure logic; no network, no real app, tmp_path only - integration/ # full FastAPI app over ASGI; real auth; real MemoryBackend - webhooks/ # webhook delivery pipeline (in-process receiver) - contract/ # spec + OpenAPI conformance - e2e/ # real subprocesses, real wire; marked `e2e` -``` - -### 3.2 Markers and defaults - -Registered in `pytest.ini`: - -| Marker | Meaning | In default run? | -|---|---|---| -| `unit` | pure logic | yes | -| `integration` | in-process app, real auth | yes | -| `contract` | spec/OpenAPI conformance | yes | -| `e2e` | spawns `mail-server`/`mail-daemon` subprocesses | no (`-m e2e` opt-in; runs in CI) | - -`addopts` gains `-m "not e2e"`; CI runs two jobs (default + e2e). -Keep `asyncio_mode = auto`. - -### 3.3 Shared fixtures (`tests/conftest.py`) - -The ~20-line `deployment_dir` fixture currently duplicated verbatim across -three files moves here, alongside: - -- `deployment_dir` — `tmp_path`-backed deployment tree; - monkeypatches `mail_server.backends.memory.fs.DEPLOYMENT_PATH` -- `backend` — started `MemoryBackend` seeded with a standard cast: - one admin, two users, one agent, one daemon, one swarm -- `app_client` — `TestClient` over the **real** `mail_server.server.app` - (env vars `MAIL_HOST`, `MAIL_JWT_SECRET_KEY`, `MAIL_JWT_ALGORITHM` set - before import), wired to `backend` -- `token_for(address)` — factory issuing real JWTs via `POST /auth/token`, - so integration tests exercise real auth instead of monkeypatching it -- `webhook_receiver` — in-process ASGI app that records deliveries and can be - told to fail N times (for retry-ladder tests) - -### 3.4 New dev dependencies - -- `respx` — `httpx` route mocking for `mail_client` / `mail_daemon` unit tests -- `schemathesis` *(optional, Phase 4)* — property-based fuzzing of endpoints - against `spec/openapi.yaml` - -Coverage: enable `pytest-cov` (already installed) scoped to the four v2 -packages; report in CI. Start with a visibility-only report; introduce a -ratchet threshold once Phase 2 lands. - -## 4. Test categories — scope definitions - -### Unit (`tests/unit/`) - -Pure functions and single classes; no app object, no sockets; filesystem only -via `tmp_path`. Owns: - -- all `mail_protocol` validators (full matrix: address grammars from SPEC §6, - uuid/subject/body/name/host bounds from `core/constants.py`) -- Pydantic model construction, validation edges, and `summarize()` for every - `mail_protocol.core` model (today only `lists` and `trash` are covered) -- `MemoryBackend` method-level behavior (agents/daemons/users/swarms/webhooks - CRUD, inbox/outbox/drafts/trash operations, buffer semantics) -- `memory.fs` load/save round-trips for every entity type (today: lists only) -- `mail_server.validators` (14 request-body validators → 422 paths) -- `mail_client.commands.*` and `mail_daemon.maild.api` against - `httpx.MockTransport`/`respx` (daemon tests must reset the module-level - `_mail_*` globals between tests — add an autouse fixture) -- existing CLI parser-shape tests (stay as-is) - -### Integration (`tests/integration/`) - -The real composed FastAPI app over ASGI, real JWT auth, real backend, no -subprocesses. Owns: - -- **Auth flows:** `POST /auth/token` (good/bad credentials), `whoami` per - role, password reset, expired/garbage tokens → 401 -- **Authorization boundaries:** user cannot read another user's - inbox/outbox/drafts/trash; non-admin → 403 on all `/admin/*`; daemon-only - endpoints reject user/admin tokens; agent-role behavior -- **Per-router behavior:** inbox, outbox, drafts (incl. `send`), trash, - swarms (+ health), admin (19 endpoints), daemon - (`message-buffer/clear`, `deliver/local`), lists (migrate existing endpoint - tests here, rewired to real auth) -- **Cross-endpoint flows in-process:** compose → send → buffer → deliver → - recipient inbox; list fan-out through the real routers -- **Webhook pipeline** (`integration/webhooks/`): delivery POST shape - (`WebhookDeliveredPostRequest`), HMAC-SHA256 signature verification - round-trip, retry ladder ordering (patch `asyncio.sleep`; assert the - 0s/1s/30s/5m/1h/6h schedule and give-up behavior), webhook CRUD effects on - delivery - -### Contract (`tests/contract/`) - -The spec is the oracle. Owns: - -- **OpenAPI drift check:** regenerate the schema exactly as - `scripts/generate_openapi.py` does and assert equality with the committed - `spec/openapi.yaml`. A failing check means: change the API deliberately and - regenerate, or revert. -- **SPEC.md conformance tests:** encode MUST/SHOULD clauses as tests that - reference their spec section in the test docstring — §6 address grammar - (host-scoped `user:`/`admin:`/`daemon:` forms, swarm-scoped agent and - `list:` forms), §7 message field requirements and bounds, §8 pre-send vs - post-send error semantics. Where the implementation and spec disagree, the - test fails and forces the conversation. -- **Schemathesis fuzzing** *(optional)*: generate requests from the OpenAPI - schema against the in-process app; assert no 500s and response-schema - conformance. - -### E2E (`tests/e2e/`) - -Real processes, real wire, few in number. A session-scoped fixture runs -`backend-init` into a tmp deployment, then launches `mail-server` (uvicorn) -and `mail-daemon` subprocesses with real env wiring, polling `/health` for -readiness. Tests drive the system through the `mail` / `mail-admin` CLIs: - -1. **Send/deliver journey:** login → compose → send → daemon delivers → - recipient sees the message via `inbox` / `inbox-open` -2. **List fan-out journey:** admin creates list → users subscribe → send to - `list:` address → all members receive -3. **Persistence across restart:** send/deliver → stop server cleanly → - relaunch on same deployment dir → inbox/outbox/lists intact -4. **Auth journey:** login, whoami, bad-password rejection, admin panel access - -These are the only tests that can catch env-var wiring, `backend_init` -provisioning, daemon global state, and shutdown persistence in combination. - -## 5. Execution phases - -Each phase is independently mergeable and leaves the default suite green. - -### Phase 0 — Foundation (small) -- Resolve the failing `test_subscribe_other_rejected_with_403` per the §7 - decision. -- Create `tests/conftest.py`; deduplicate the `deployment_dir` fixture out of - the three files that copy it. -- Create the category directories, register markers, update `pytest.ini` - (`-m "not e2e"`), move `tests/unit/` content as needed (no test rewrites). -- Wire `pytest-cov` reporting; gitignore `pytest.log`. -- Add `respx` as a dev dependency. - -**Exit:** suite green; one shared fixture set; coverage number visible. - -### Phase 1 — Integration: auth + routers (largest single phase) -- `app_client` + `token_for` fixtures (real app, real JWTs). -- Auth flow and authorization-boundary tests. -- Per-router endpoint tests for inbox, outbox, drafts, trash, swarms, admin, - daemon; migrate lists endpoint tests onto real auth. -- `xfail(raises=NotImplementedError)` tests for the known stubs - (`delete_inbox_message`, `delete_draft`, `delete_trash_message`, - `clear_trash`, `daemon_deliver_remote`, `admin_webhook_patch`) so the - checklist is executable. - -**Exit:** every registered route has ≥1 success and ≥1 authz/failure test; -auth layer no longer monkeypatched anywhere in integration tests. - -### Phase 2 — Webhook delivery pipeline -- `webhook_receiver` fixture; signature round-trip, payload shape, retry - ladder with patched sleep, give-up after final attempt, CRUD→delivery - effects. - -**Exit:** the six-commit bug cluster's behaviors are all pinned by tests. - -### Phase 3 — Contract layer -- OpenAPI drift check. -- SPEC.md §6/§7/§8 conformance tests with section-referencing docstrings. -- Decide on schemathesis adoption after evaluating runtime cost. - -**Exit:** an API change that isn't reflected in `spec/openapi.yaml` fails CI. - -### Phase 4 — Client + daemon units, protocol back-fill -- `mail_client.commands.*` against mocked transport (request shape, token - header, output rendering incl. markdown path); `mail-admin` commands. -- `mail_daemon.maild.api`: loop iteration behavior, buffer-clear/deliver - calls, startup validation, token acquisition; globals-reset fixture. -- Back-fill `mail_protocol` validator matrix, model edges, `memory.fs` - round-trips for all entity types, `mail_server.validators`. - -**Exit:** every v2 package has meaningful unit coverage; set the initial -coverage ratchet. - -### Phase 5 — E2E journeys + CI -- Subprocess harness fixture; the four journeys in §4. -- CI: default job (unit+integration+contract, coverage report) and e2e job. - -**Exit:** full-system happy paths run on every PR. - -## 6. Conventions - -- New v2 features land with tests in the matching category; bug fixes land - with a regression test (the webhook cluster is the cautionary tale). -- Conformance tests cite their SPEC.md section; when implementation and spec - conflict, the spec is amended or the code fixed — never the test deleted - silently. -- Stubbed functionality gets an `xfail(raises=NotImplementedError)` test at - introduction time. -- Shared fixtures live in `tests/conftest.py`; category-specific ones in that - category's `conftest.py`. No copy-pasted fixtures. - -## 7. Open decisions - -1. **Subscribe-on-behalf semantics — RESOLVED 2026-06-12: option (a).** - Commit `570a340` removed the request body from - `POST /lists/{list}/subscribe`; the endpoint always subscribes the - authenticated caller. This is ratified: subscribing *another* user-agent - is an admin-only capability (via `/admin/lists/{list}/members`), so the - public endpoint stays body-less. The stale 403 test is replaced by - `test_subscribe_ignores_supplied_member_address`. `spec/openapi.yaml` - already reflects the body-less endpoint; no spec change needed. -2. **Coverage ratchet level — RESOLVED 2026-06-12.** Set at Phase 4 exit: - `fail_under = 65` (suite measured 66%). Raise as coverage grows; never - lower. -3. **Schemathesis adoption — RESOLVED 2026-06-12: deferred.** The drift - check plus SPEC conformance tests cover the schema-shape ground; - revisit as a nightly CI job after Phase 5. diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md new file mode 100644 index 0000000..a077d11 --- /dev/null +++ b/docs/tutorials/README.md @@ -0,0 +1,22 @@ +# Tutorials + +Tutorials teach MAIL by walking a beginner through a concrete, working project. +They should assume little prior MAIL knowledge, include only the explanation +needed to complete the lesson, and be tested end to end before release. + +## Planned Tutorials + +| Page | Outcome | Source material | +| --- | --- | --- | +| [Run MAIL Locally](run-local-mail.md) | Start a local memory-backed server, run a daemon, and observe local delivery. | `src/mail/server/docs/tutorials/quickstart.md`, `src/mail/daemon/src/mail_daemon/maild/api.py` | +| [Send Your First MAIL Message](send-first-message.md) | Log in, compose a draft, send it, and inspect inbox/outbox state. | `src/mail/client/docs/tutorials/quickstart.md`, `src/mail/client/src/mail_client/cli.py` | +| [Build a Minimal HTTP Client](build-minimal-http-client.md) | Authenticate and interact with MAIL using raw HTTP calls. | `spec/openapi.yaml`, `src/mail/protocol/src/mail_protocol/network/` | + +## Tutorial Checklist + +- State the concrete thing the reader will finish with. +- Prefer one happy path over branches and alternatives. +- Include prerequisites that can be verified before step 1. +- Show expected output or observable state after major steps. +- Link to reference pages for command flags, schemas, and endpoint details. +- Link to explanations for conceptual background. diff --git a/docs/tutorials/build-minimal-http-client.md b/docs/tutorials/build-minimal-http-client.md new file mode 100644 index 0000000..4835494 --- /dev/null +++ b/docs/tutorials/build-minimal-http-client.md @@ -0,0 +1,35 @@ +# Build a Minimal HTTP Client + +Status: stub + +## Outcome + +The reader builds the smallest useful MAIL HTTP client: authenticate, call +`GET /auth/whoami`, create a draft, send it, and read response payloads. + +## Audience + +Developers integrating MAIL into tools that should not shell out to the `mail` +CLI. + +## Source Material + +- `spec/openapi.yaml` +- `src/mail/protocol/src/mail_protocol/network/requests.py` +- `src/mail/protocol/src/mail_protocol/network/responses.py` +- `src/mail/server/src/mail_server/routers/auth.py` +- `src/mail/server/src/mail_server/routers/drafts.py` + +## Draft Outline + +1. Start from a known server URL and credentials. +2. Obtain a bearer token from `POST /auth/token`. +3. Validate identity with `GET /auth/whoami`. +4. Create a draft with `POST /drafts/`. +5. Send the draft with `POST /drafts/{draft_id}/send`. +6. Parse success and validation errors. + +## Not Here + +- Full endpoint listings belong in [HTTP API](../references/http-api.md). +- Protocol motivation belongs in [MAIL v2 Overview](../explanations/mail-v2-overview.md). diff --git a/docs/tutorials/run-local-mail.md b/docs/tutorials/run-local-mail.md new file mode 100644 index 0000000..29315cd --- /dev/null +++ b/docs/tutorials/run-local-mail.md @@ -0,0 +1,39 @@ +# Run MAIL Locally + +Status: stub + +## Outcome + +The reader starts a local MAIL v2 deployment with the memory backend, logs in as +at least two user-agents, runs a daemon, sends one message, and verifies that +the message was delivered. + +## Audience + +New contributors and first-time users who have cloned the repository and want a +working local loop before reading deeper docs. + +## Source Material + +- `README.md` +- `src/mail/server/docs/tutorials/quickstart.md` +- `src/mail/client/docs/tutorials/quickstart.md` +- `src/mail/server/.env.example` +- `src/mail/server/src/mail_server/backend_init.py` +- `src/mail/daemon/src/mail_daemon/maild/api.py` + +## Draft Outline + +1. Install workspace dependencies with `uv sync`. +2. Configure the server environment. +3. Initialize the memory backend with `backend-init`. +4. Start `mail-server`. +5. Log in as a sender and recipient with `mail login`. +6. Start `mail-daemon` with daemon credentials. +7. Compose and send a message. +8. Open inbox and outbox entries to confirm delivery. + +## Not Here + +- Exhaustive command flags belong in reference pages. +- Deployment hardening belongs in how-to guides and explanations. diff --git a/docs/tutorials/send-first-message.md b/docs/tutorials/send-first-message.md new file mode 100644 index 0000000..ab2bf84 --- /dev/null +++ b/docs/tutorials/send-first-message.md @@ -0,0 +1,38 @@ +# Send Your First MAIL Message + +Status: stub + +## Outcome + +The reader uses an existing MAIL server account to create a draft, send it to a +recipient, and inspect the message in the CLI. + +## Audience + +Users who already have a MAIL server URL and credentials but have not used the +`mail` CLI before. + +## Source Material + +- `src/mail/client/docs/tutorials/quickstart.md` +- `src/mail/client/src/mail_client/cli.py` +- `src/mail/client/src/mail_client/commands/compose.py` +- `src/mail/client/src/mail_client/commands/send.py` +- `src/mail/client/src/mail_client/commands/inbox_open.py` +- `src/mail/client/src/mail_client/commands/outbox_open.py` + +## Draft Outline + +1. Verify `mail --help` works. +2. Log in with `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. +3. Store the returned token in `MAIL_TOKEN`. +4. Run `mail whoami`. +5. Compose a draft. +6. Send the draft to one recipient. +7. Open the outbox message. +8. If testing with a second account, open the recipient inbox. + +## Not Here + +- Admin account creation belongs in [Manage User-Agents](../howtos/manage-user-agents.md). +- Complete CLI option tables belong in [Client CLI](../references/client-cli.md). From 86b1d2a909502669ac5295b9f388751a4afc0dbd Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Tue, 16 Jun 2026 16:15:33 -0400 Subject: [PATCH 02/28] docs: drafted tutorials for `run-local-mail` and `send-first-message` --- docs/tutorials/build-minimal-http-client.md | 18 +-- docs/tutorials/run-local-mail.md | 168 ++++++++++++++++++-- docs/tutorials/send-first-message.md | 143 +++++++++++++++-- 3 files changed, 286 insertions(+), 43 deletions(-) diff --git a/docs/tutorials/build-minimal-http-client.md b/docs/tutorials/build-minimal-http-client.md index 4835494..a0cb6e1 100644 --- a/docs/tutorials/build-minimal-http-client.md +++ b/docs/tutorials/build-minimal-http-client.md @@ -12,13 +12,10 @@ The reader builds the smallest useful MAIL HTTP client: authenticate, call Developers integrating MAIL into tools that should not shell out to the `mail` CLI. -## Source Material +## Not Here -- `spec/openapi.yaml` -- `src/mail/protocol/src/mail_protocol/network/requests.py` -- `src/mail/protocol/src/mail_protocol/network/responses.py` -- `src/mail/server/src/mail_server/routers/auth.py` -- `src/mail/server/src/mail_server/routers/drafts.py` +- Full endpoint listings belong in [HTTP API](../references/http-api.md). +- Protocol motivation belongs in [MAIL v2 Overview](../explanations/mail-v2-overview.md). ## Draft Outline @@ -29,7 +26,10 @@ CLI. 5. Send the draft with `POST /drafts/{draft_id}/send`. 6. Parse success and validation errors. -## Not Here +## Source Material -- Full endpoint listings belong in [HTTP API](../references/http-api.md). -- Protocol motivation belongs in [MAIL v2 Overview](../explanations/mail-v2-overview.md). +- `spec/openapi.yaml` +- `src/mail/protocol/src/mail_protocol/network/requests.py` +- `src/mail/protocol/src/mail_protocol/network/responses.py` +- `src/mail/server/src/mail_server/routers/auth.py` +- `src/mail/server/src/mail_server/routers/drafts.py` diff --git a/docs/tutorials/run-local-mail.md b/docs/tutorials/run-local-mail.md index 29315cd..4f42d14 100644 --- a/docs/tutorials/run-local-mail.md +++ b/docs/tutorials/run-local-mail.md @@ -1,6 +1,6 @@ # Run MAIL Locally -Status: stub +Status: draft ## Outcome @@ -13,6 +13,156 @@ the message was delivered. New contributors and first-time users who have cloned the repository and want a working local loop before reading deeper docs. +## Not Here + +- Exhaustive command flags belong in reference pages. +- Deployment hardening belongs in how-to guides and explanations. + +## Steps + +### 1. Install workspace dependencies + +With the `mail` repository downloaded, navigate into it and use `uv` to install workspace dependencies: + +```bash +cd mail +uv sync +``` + +### 2. Configure the server environment + +The MAIL server expects a number of environment variables to be set in order to run. +These are: +- `MAIL_HOST`: The host domain or IP address for this MAIL server. Use `localhost` for this tutorial. +- `MAIL_JWT_SECRET_KEY`: The secret key for the MAIL server to use for JWT auth. Run `openssl rand -hex 32` and use that value for this tutorial. +- `MAIL_JWT_ALGORITHM`: The JWT algorithm used by the MAIL server. Use `HS256` for this tutorial. +- `MAIL_JWT_EXPIRE_MINUTES`: The lifetime to use for JWTs on this MAIL server. Use `30` for this tutorial. + +### 3. Initialize the memory backend + +The MAIL server uses an in-memory backend by default that saves data to the local filesystem. +This backend must be initialized prior to running `mail-server`. +To initialize the memory backend, run: + +```bash +uv run backend-init --type memory --host localhost +``` + +This script will set up a new MAIL memory backend deployment in the local filesystem. +It will also generate credentials for one user-agent of each type: `agent`, `admin`, `daemon`, and `user`. +For each user-agent, the password will be written to the filepath printed to the console. +Copy these credentials into a safe place and then delete the files. + +### 4. Start `mail-server` + +With your environment configured as described in step 2, you can now start up the MAIL server: + +```bash +uv run mail-server --backend memory +``` + +The MAIL server (hosted on `http://127.0.0.1:8865`) will start up using the memory backend you just initialized. + +### 5. Log in as a sender and recipient + +Once the MAIL server is up and running, open a new terminal and log in as the `admin` user-agent that was just created: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_ADDRESS=admin:dummy@localhost +MAIL_PASSWORD={admin_password} +uv run mail login +``` + +Use the `admin` password that was generated in step 3. We'll use this user-agent to send a MAIL message. Running `login` should print a generated JWT for this admin that can be used in subsequent operations. + +Then, open another terminal and log in as the `agent` user-agent that was just created: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_ADDRESS=supervisor@default@localhost +MAIL_PASSWORD={agent_password} +uv run mail login +``` + +Use the `agent` password that was generated in step 3. We'll use this user-agent to receive the MAIL message sent by the `admin`. Running `login` should print a generated JWT for this agent that can be used in subsequent operations. + +### 6. Start `mail-daemon` with daemon credentials + +In order for MAIL messages to be delivered between user-agents, an authenticated daemon must be connected to the server. +Open another terminal and run `mail-daemon` with the generated credentials from step 3: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_ADDRESS=daemon:dummy@localhost +MAIL_PASSWORD={daemon_password} +uv run mail-daemon +``` + +The MAIL daemon should then start up and log in to the server. When there are new messages on the server to deliver, the daemon will deliver them to the specified recipient(s). + +### 7. Compose and send a message + +We will now attempt to compose and send a message as the `admin` that was authenticated in step 5. To compose a new message draft as `admin:dummy@localhost`, run: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={admin_jwt} +uv run mail compose "Test subject" "This is a message body" +``` + +You should see the new draft printed to the console, including its unique draft ID (a UUID). We can now send a MAIL message to the `agent` by specifying the draft ID and the agent's address: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={admin_jwt} +uv run mail send {draft_id} supervisor@default@localhost +``` + +You should see a MAIL message created from the draft that you just composed, including its unique message ID (a UUID). This will be delivered by the daemon (from step 6) to `supervisor@default@localhost`. Note that this process may take up to 30 seconds. + +### 8. Open inbox and outbox entries to confirm delivery + +To check that the `admin`'s message has been delivered, open the `agent`'s inbox using their JWT from step 5: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={agent_jwt} +uv run mail inbox +``` + +Once the message has been delivered, you should see it in the `agent` inbox. Open and read the full message: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={agent_jwt} +uv run mail open {message_id} +``` + +You should now see the message composed by the `admin` with the subject "Test subject" and body "This is a message body". At the bottom of the message, you should also see: + +```text +Delivered By: daemon:dummy@localhost +``` + +You can also access this message in the sending `admin`'s outbox. To do so, use the `admin`'s JWT from step 5: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={admin_jwt} +uv run mail outbox +``` + +Assuming the composed message ID is present, you can open and read it with: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={admin_jwt} +uv run mail outbox-open {message_id} +``` + +Like in the `agent`'s inbox, you should see the message contents as you composed them, with a subject of "Test subject" and a body of "This is a message body". + ## Source Material - `README.md` @@ -21,19 +171,3 @@ working local loop before reading deeper docs. - `src/mail/server/.env.example` - `src/mail/server/src/mail_server/backend_init.py` - `src/mail/daemon/src/mail_daemon/maild/api.py` - -## Draft Outline - -1. Install workspace dependencies with `uv sync`. -2. Configure the server environment. -3. Initialize the memory backend with `backend-init`. -4. Start `mail-server`. -5. Log in as a sender and recipient with `mail login`. -6. Start `mail-daemon` with daemon credentials. -7. Compose and send a message. -8. Open inbox and outbox entries to confirm delivery. - -## Not Here - -- Exhaustive command flags belong in reference pages. -- Deployment hardening belongs in how-to guides and explanations. diff --git a/docs/tutorials/send-first-message.md b/docs/tutorials/send-first-message.md index ab2bf84..188969b 100644 --- a/docs/tutorials/send-first-message.md +++ b/docs/tutorials/send-first-message.md @@ -1,6 +1,6 @@ # Send Your First MAIL Message -Status: stub +Status: draft ## Outcome @@ -12,6 +12,131 @@ recipient, and inspect the message in the CLI. Users who already have a MAIL server URL and credentials but have not used the `mail` CLI before. +## Not Here + +- Admin account creation belongs in [Manage User-Agents](../howtos/manage-user-agents.md). +- Complete CLI option tables belong in [Client CLI](../references/client-cli.md). + +## Steps + +### 1. Verify `mail --help` works + +With the `mail` repository installed, ensure the MAIL client CLI is accessible: + +```bash +uv run mail --help +``` + +You should see a list of CLI commands (e.g. `login`, `compose`, `inbox`) and usage examples. + +### 2. Log in with credentials in env vars + +To log into a MAIL server at a specified address with valid credentials, set the environment variables for the server URL, user-agent address, and user-agent password, and then run the `login` command: + +```bash +MAIL_SERVER={server_url} +MAIL_ADDRESS={ua_address} +MAIL_PASSWORD={ua_password} +uv run mail login +``` + +This should print a JWT that can be used in subsequent operations with the `mail` CLI. Note that JWTs will expire after a predetermined period of time; simply run the `login` command again to obtain a fresh token. + +### 3. Store the returned token in `MAIL_TOKEN` + +Rather than requiring a `MAIL_ADDRESS` and `MAIL_PASSWORD` for every single command, the `mail` CLI expects the token obtained in step 2 as an environment variable for all non-`login` operations: + +```env +MAIL_TOKEN={jwt} +``` + +### 4. Run `mail whoami` + +Once you're logged into a MAIL server, you can view basic information about your account with the `whoami` command: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail whoami +``` + +You should see your MAIL address (e.g. `user:example@example.com`) as well as your user-agent type (which must be `agent`, `admin`, `daemon`, or `user`). + +### 5. Compose a draft + +To send a new MAIL message, you must first compose a draft that can be sent. +Use your credentials with the `compose` command to do so: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail compose "Message Subject" "This is a message body" +``` + +You should see the newly-created draft with the subject "Message Subject", body "This is a message body", and a unique draft ID (UUID) associated with it. + +### 6. Send the draft to one recipient + +With a draft composed, you can now send it to another MAIL user-agent by their address and the ID of the draft that was just created: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail send {draft_id} {ua_address} +``` + +You should see the newly-created MAIL message from your draft, with the subject "Message Subject", body "This is a message body", and a unique message ID (a UUID, but NOT the same as the draft ID). + +### 7. Open the outbox message + +You can verify that the sent message is now in your outbox: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail outbox +``` + +You should see the new message ID in your outbox. You can open the full message with `outbox-open`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail outbox-open {message_id} +``` + +You should now see a MAIL message with the same ID as the one you sent, as well as the subject "Message Subject" and body "This is a message body". You may also see something like: + +```text +Delivered By: daemon:{daemon_name}@example.com +``` + +This is the address of the MAIL daemon that has delivered your message to the specified recipient(s). If you don't see it immediately, don't worry--the delivery process can take time, especially if there is a backlog of messages on the server to deliver. + +### (optional) If testing with a second account, open the recipient inbox + +If you decided to send a message to a MAIL address that you also have credentials for, you can check that the message has been delivered to its inbox. First, log into your recipient user-agent account by following the process in steps 2-3 to obtain a JWT. Then, check the user-agent's inbox: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={recipient_jwt} +uv run mail inbox +``` + +You should see the ID of the message that was delivered in your inbox. You can open it and read the message contents with the `open` command: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={recipient_jwt} +uv run mail open {message_id} +``` + +You should now see a MAIL message with the same ID as the one you sent, as well as the subject "Message Subject" and body "This is a message body". You should also see the address of the MAIL daemon that has delivered your message: + +```text +Delivered By: daemon:{daemon_name}@example.com +``` + ## Source Material - `src/mail/client/docs/tutorials/quickstart.md` @@ -20,19 +145,3 @@ Users who already have a MAIL server URL and credentials but have not used the - `src/mail/client/src/mail_client/commands/send.py` - `src/mail/client/src/mail_client/commands/inbox_open.py` - `src/mail/client/src/mail_client/commands/outbox_open.py` - -## Draft Outline - -1. Verify `mail --help` works. -2. Log in with `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. -3. Store the returned token in `MAIL_TOKEN`. -4. Run `mail whoami`. -5. Compose a draft. -6. Send the draft to one recipient. -7. Open the outbox message. -8. If testing with a second account, open the recipient inbox. - -## Not Here - -- Admin account creation belongs in [Manage User-Agents](../howtos/manage-user-agents.md). -- Complete CLI option tables belong in [Client CLI](../references/client-cli.md). From 7ad9d1d7c05240cc6262cf7103e7a558b75228c5 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Tue, 16 Jun 2026 17:14:47 -0400 Subject: [PATCH 03/28] docs: drafted howtos for `auth-user-agent` and `init-mem-be` --- docs/howtos/authenticate-user-agent.md | 67 ++++++++++--- docs/howtos/initialize-memory-backend.md | 115 +++++++++++++++++++++-- 2 files changed, 162 insertions(+), 20 deletions(-) diff --git a/docs/howtos/authenticate-user-agent.md b/docs/howtos/authenticate-user-agent.md index 6cfc32f..22f9c82 100644 --- a/docs/howtos/authenticate-user-agent.md +++ b/docs/howtos/authenticate-user-agent.md @@ -1,6 +1,6 @@ # Authenticate a User-Agent -Status: stub +Status: draft ## Goal @@ -11,22 +11,61 @@ with CLI or HTTP requests. The reader has a server URL, a MAIL address, and a password. +## Steps + +### 1. Set environment variables + +In order to log into a MAIL server using the MAIL client CLI, you must set the following environment variables: +- `MAIL_SERVER`: The URL of the MAIL server to log into, e.g. `https://mail-swarms.example.com`. +- `MAIL_ADDRESS`: The address of the MAIL user-agent to log in as, e.g. `user:example@example.com`. +- `MAIL_PASSWORD`: The password for the MAIL user-agent to log in as. + +### 2. Run `mail login` + +With the environment variables set as described in step 1, log into the MAIL server using the CLI client command `login`: + +```bash +uv run mail login +``` + +### 3. Store the returned token in `MAIL_TOKEN` + +Running the `login` command above should print a temporary access token to the console. +This can be used in subsequent operations with the `mail` client CLI, rather than `MAIL_ADDRESS` and `MAIL_PASSWORD`. +Store this token as an environment variable called `MAIL_TOKEN`: + +```env +MAIL_TOKEN={token} +``` + +### 4. Run `mail whoami` + +With your `MAIL_SERVER` and `MAIL_TOKEN` environment variables set, you can now view your own user-agent information using the `whoami` command: + +```bash +uv run mail whoami +``` + +This will print the authenticated user-agent's MAIL address and user-agent type. Ensure these are both the expected values. + +### 5. Use the token in an HTTP `Authorization: Bearer ...` header + +Since the MAIL server accepts access tokens in the `Authorization` header, you can attempt to hit the `whoami` endpoint with a raw HTTP request rather than through the `mail` client CLI: + +```bash +curl {server_url}/auth/whoami \ +-H "Authorization: Bearer {token}" +``` + +### 6. Refresh or replace expired tokens + +Server-issued access tokens will expire after a predetermined length of time (e.g. 15, 30, or 60 minutes). If you attempt to hit a MAIL server endpoint with a previously-valid token and get a `401` response, that likely means your token has expired. + +To obtain a fresh access token with your credentials, repeat steps 1-2. + ## Source Material - `src/mail/client/src/mail_client/commands/login.py` - `src/mail/client/src/mail_client/commands/whoami.py` - `src/mail/server/src/mail_server/routers/auth.py` - `spec/openapi.yaml` - -## Steps to Cover - -1. Set `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. -2. Run `mail login`. -3. Store the returned token in `MAIL_TOKEN`. -4. Run `mail whoami`. -5. Use the token in an HTTP `Authorization: Bearer ...` header. -6. Refresh or replace expired tokens. - -## Validation - -`mail whoami` returns the expected user-agent type and address. diff --git a/docs/howtos/initialize-memory-backend.md b/docs/howtos/initialize-memory-backend.md index 9193408..a592281 100644 --- a/docs/howtos/initialize-memory-backend.md +++ b/docs/howtos/initialize-memory-backend.md @@ -12,12 +12,6 @@ credentials for development. The repository is cloned, dependencies are installed, and the reader wants local server state for `mail-server --backend memory`. -## Source Material - -- `src/mail/server/src/mail_server/backend_init.py` -- `src/mail/server/src/mail_server/backends/memory/init.py` -- `src/mail/server/docs/tutorials/quickstart.md` - ## Steps to Cover 1. Run `uv run backend-init`. @@ -29,3 +23,112 @@ server state for `mail-server --backend memory`. ## Validation The server starts successfully and the generated user-agents can authenticate. + +## Steps + +### 1. Run `backend-init` + +You can initialize an in-memory backend for a MAIL server by running the `backend-init` script: + +```bash +uv run backend-init +``` + +By default (i.e., with no specified arguments), this script will generate a new MAIL server backend with the following attributes: +- **Backend Type**: `memory` +- **Deployment**: `default` +- **Swarm Name**: `default` +- **Swarm Description**: `A MAIL swarm` +- **Swarm Keywords**: `[]` +- **Agents**: `['supervisor']` +- **Daemons**: `['dummy']` +- **Users**: `['dummy']` +- **Admins**: `['dummy']` +- **Host**: `example.com` + +### 2. Customize deployment, swarm, host, agents, daemons, users, or admins + +To initialize a new backend with a different deployment name (e.g. `example`), specify the `-d`/`--deployment` argument for the `backend-init` script: + +```bash +uv run backend-init --deployment "example" +``` + +To initialize a new backend with a different swarm name (e.g. `example`), specify the `-s`/`--swarm` argument: + +```bash +uv run backend-init --swarm "example" +``` + +To initialize a new backend with a different swarm description (e.g. `My custom description`), specify the `-sd`/`--swarm-description` argument: + +```bash +uv run backend-init --swarm-description "My custom description" +``` + +To initialize a new backend with a different list of swarm keywords (e.g. `['dev', 'internal']`), specify the `-sk`/`--swarm-keywords` argument: + +```bash +uv run backend-init --swarm-keywords "dev" "internal" +``` + +To initialize a new backend with a different list of agent names (e.g. `['meta', 'scribe']`), specify the `--agents` argument: + +```bash +uv run backend-init --agents "meta" "scribe" +``` + +For a different list of daemon names (e.g. `['worker-1', 'worker-2']`), specify the `--daemons` argument: + +```bash +uv run backend-init --daemons "worker-1" "worker-2" +``` + +For a different list of user names (e.g. `['user-1', 'user-2', 'user-3']`), specify the `--users` argument: + +```bash +uv run backend-init --users "user-1" "user-2" "user-3" +``` + +For a different list of admin names (e.g. `['a1', 'a2', 'a3', 'a4']`), specify the `--admins` argument: + +```bash +uv run backend-init --admins "a1" "a2" "a3" "a4" +``` + +To initialize a new backend with a different host (e.g. `my-site.com`), specify the `-H`/`--host` argument: + +```bash +uv run backend-init --host "my-site.com" +``` + +### 3. Locate generated credential files + +Upon memory backend initialization, a password for each created user-agent is randomly-generated. These passwords will be stored in plaintext inside `~/.mail-swarms/deployments/{deployment}/.secrets`, where `deployment` is the name of the deployment created with `backend-init`. + +For example, if you have a deployment named `default` and a user-agent address `supervisor@default@example.com`, you can view its plaintext password: + +```bash +cat ~/.mail-swarms/deployments/default/.secrets/supervisor@default@example.com +``` + +### 4. Remove or protect plaintext password files after capture + +Copy the plaintext passwords for the user-agents you intend to keep into a safe place. +Once you have these copied, you can simply delete the `.secrets` folder in your deployment: + +```bash +rm -rf ~/.mail-swarms/deployments/{deployment}/.secrets +``` + +where `deployment` is the name chosen for your deployment. + +### 5. Reinitialize clean slate when needed + +TODO + +## Source Material + +- `src/mail/server/src/mail_server/backend_init.py` +- `src/mail/server/src/mail_server/backends/memory/init.py` +- `src/mail/server/docs/tutorials/quickstart.md` From 7eb7a033bbdee216c00591699f70680f8913620c Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Tue, 16 Jun 2026 18:13:40 -0400 Subject: [PATCH 04/28] docs: drafted howtos for `manage-lists` and `manage-swarms` --- docs/howtos/initialize-memory-backend.md | 12 --- docs/howtos/manage-mailing-lists.md | 115 ++++++++++++++++++++--- docs/howtos/manage-swarms.md | 74 ++++++++++++--- 3 files changed, 160 insertions(+), 41 deletions(-) diff --git a/docs/howtos/initialize-memory-backend.md b/docs/howtos/initialize-memory-backend.md index a592281..8adc8f2 100644 --- a/docs/howtos/initialize-memory-backend.md +++ b/docs/howtos/initialize-memory-backend.md @@ -12,18 +12,6 @@ credentials for development. The repository is cloned, dependencies are installed, and the reader wants local server state for `mail-server --backend memory`. -## Steps to Cover - -1. Run `uv run backend-init`. -2. Customize deployment, swarm, host, agents, daemons, users, or admins. -3. Locate generated credential files. -4. Remove or protect plaintext password files after capture. -5. Reinitialize clean state when needed. - -## Validation - -The server starts successfully and the generated user-agents can authenticate. - ## Steps ### 1. Run `backend-init` diff --git a/docs/howtos/manage-mailing-lists.md b/docs/howtos/manage-mailing-lists.md index 00c2e00..4710e54 100644 --- a/docs/howtos/manage-mailing-lists.md +++ b/docs/howtos/manage-mailing-lists.md @@ -1,6 +1,6 @@ # Manage Mailing Lists -Status: stub +Status: draft ## Goal @@ -10,6 +10,105 @@ How to create mailing lists, inspect them, and manage subscriptions or members. The reader has credentials with permissions appropriate for the list action. +## Steps + +### 1. List available mailing lists + +MAIL user-agents can view all mailing lists visible to them by using the `mail` CLI command `lists` with valid credentials: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail lists +``` + +This will print all list addresses visible to the authenticated user-agent to the console. + +### 2. Inspect a list by address + +User-agents can inspect a specific list (visible to them) by address through the `mail` command `list-get`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail list-get {list_address} +``` + +If the specified list by address exists, its ID, owner, member user-agents, policies, and metadata will be printed to the console. + +### 3. Create a list as an admin + +To create a new mailing list on a MAIL server, use the `mail-admin` CLI client with valid admin credentials: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-post {list_name} {swarm_name} {list_owner} +``` + +Note that `list_name`, `swarm_name`, and `list_owner` are required arguments. You can optionally specify a list of MAIL user-agent addresses to add as members upon list creation: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-post {list_name} {swarm_name} {list_owner} \ +--members "user:dummy@example.com" "supervisor@default@example.com" +``` + +### 4. Subscribe and unsubscribe as a user-agent + +Non-`admin` user-agents may subscribe to or unsubscribe from mailing lists. +To subscribe to an existing mailing list by a given address, use the `mail` CLI client with authorized credentials: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail list-subscribe {list_address} +``` + +If the operation was successful, details on the list subscribed to will be printed to the console. +To unsubscribe from an existing mailing list by a given address, use the `mail` CLI client with authorized credentials: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail list-unsubscribe {list_address} +``` + +If the operation was successful, details on the list unsubscribed from will be printed to the console. + +### 5. Add or remove members as an admin + +Admins can add a MAIL user-agent by address to an existing mailing list with the `mail-admin` command `list-member-post`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-member-post {list_address} {member_address} +``` + +If successful, details on the mailing list will be printed to the console. + +Similarly, admins can remove existing members by address from a mailing list with `list-member-delete`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-member-delete {list_address} {member_address} +``` + +If successful, details on the mailing list will be printed to the console. + +### 6. Send a message to a list address + +If they are authorized to do so, user-agents can send a message to a list address by specifying the list address in the `mail` command `send`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail send {draft_id} {list_address} +``` + ## Source Material - `src/mail/client/src/mail_client/commands/lists.py` @@ -18,17 +117,3 @@ The reader has credentials with permissions appropriate for the list action. - `src/mail/client/src/mail_client/commands/list_member_post.py` - `src/mail/server/src/mail_server/routers/lists.py` - `src/mail/protocol/src/mail_protocol/core/lists.py` - -## Steps to Cover - -1. List available mailing lists. -2. Inspect one list by address. -3. Create a list as an admin. -4. Subscribe and unsubscribe as a user-agent. -5. Add or remove members as an admin. -6. Send a message to a list address. - -## Validation - -List membership changes are reflected in list lookup and list-address sends -deliver to expected recipients. diff --git a/docs/howtos/manage-swarms.md b/docs/howtos/manage-swarms.md index 8fc46ff..b3d1205 100644 --- a/docs/howtos/manage-swarms.md +++ b/docs/howtos/manage-swarms.md @@ -1,6 +1,6 @@ # Manage Swarms -Status: stub +Status: draft ## Goal @@ -11,6 +11,65 @@ How to create, inspect, and delete MAIL swarms. The reader has an admin token for changes or a regular user-agent token for read-only swarm inspection. +## Steps + +### 1. List swarms + +Authorized user-agents can list all swarms on a MAIL server via the `mail` CLI command `swarms-list`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail swarms-list +``` + +This will print the name, keywords, and number of agents for each swarm on the server. + +### 2. Inspect a swarm by name + +Authorized user-agents can inspect a specific, existing swarm by name on a MAIL server via the `mail` command `swarm-get`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail swarm-get {swarm_name} +``` + +If a swarm with the specified name exists, its description, keywords, and full list of agents will be printed to the console. + +### 3. Create a swarm as an admin + +Authorized admins can create a new swarm on a MAIL server via the `mail-admin` CLI command `swarm-post`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin swarm-post {swarm_name} {swarm_description} +``` + +The arguments `swarm_name` and `swarm_description` are required. Optionally, the swarm's `keywords` can be specified as well: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin swarm-post {swarm_name} {swarm_description} \ +--keywords "kw-1" "kw-2" +``` + +If this operation was successful, information on the new swarm will be printed to the console. + +### 4. Delete a swarm as an admin + +Authorized admins can delete an existing swarm by name on a MAIL server via the `mail-admin` command `swarm-delete`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin swarm-delete {swarm_name} +``` + +If this operation was successful, information on the newly-deleted swarm will be printed to the console. + ## Source Material - `src/mail/client/src/mail_client/commands/swarm_list.py` @@ -19,16 +78,3 @@ read-only swarm inspection. - `src/mail/client/src/mail_client/commands/swarm_delete.py` - `src/mail/server/src/mail_server/routers/swarms.py` - `src/mail/server/src/mail_server/routers/admin.py` - -## Steps to Cover - -1. List swarms. -2. Inspect a swarm by name. -3. Create a swarm as an admin. -4. Add initial description, keywords, and agents. -5. Delete a test swarm. - -## Validation - -The created swarm is visible through public swarm lookup and removable through -admin commands. From 56e20e564cd1de3efb41550327f908b358822209 Mon Sep 17 00:00:00 2001 From: rheaton64 Date: Mon, 22 Jun 2026 16:51:30 -0400 Subject: [PATCH 05/28] Fill out docs stubs --- docs/explanations/addressing-model.md | 161 +++++++++-- docs/explanations/delivery-model.md | 169 ++++++++++-- docs/tutorials/build-minimal-http-client.md | 287 +++++++++++++++++++- 3 files changed, 568 insertions(+), 49 deletions(-) diff --git a/docs/explanations/addressing-model.md b/docs/explanations/addressing-model.md index 5a09614..ddbb3bf 100644 --- a/docs/explanations/addressing-model.md +++ b/docs/explanations/addressing-model.md @@ -1,29 +1,154 @@ # Addressing Model -Status: stub +Status: draft -## Question +Every participant in MAIL — a human, an AI agent, a delivery daemon, an +administrator, or a mailing list — is reached by an **address**. This page +explains why MAIL has two different *shapes* of address, how to tell them apart, +and how to reason about each. For the exact field schemas and length bounds, see +[Data Models](../references/data-models.md); for the formal grammar, see +[§6 of the specification](../../spec/SPEC.md). -Why does MAIL have both host-scoped and swarm-scoped addresses, and how should -readers reason about each form? +## The shape tells you the scope -## Source Material +A MAIL address is always one of two shapes, and you can tell which from the +string alone — no server lookup required: -- `spec/SPEC.md` section 6 -- `src/mail/protocol/src/mail_protocol/core/user_agents.py` -- `src/mail/protocol/src/mail_protocol/core/validators.py` -- `tests/contract/test_spec_addresses.py` +| Shape | Form | Who it names | +| --- | --- | --- | +| **Host-scoped** | `{ua_type}:{ua_id}@{host}` | admins, daemons, users | +| **Swarm-scoped** | `{address_id}@{swarm}@{host}` | agents, mailing lists | + +The difference is the number of `@` segments. Split an address on `@`: + +- **two segments** → host-scoped. The first segment carries an explicit + `{ua_type}:` prefix (`user:`, `admin:`, or `daemon:`). +- **three segments** → swarm-scoped. The middle segment is the swarm. +- **anything else** → not a MAIL address. + +This self-describing quality is the single most useful thing to internalize. +A client can classify any address — decide whether it points at a server-level +resident or a swarm member, and which *kind* of correspondent it is — purely by +inspecting the string. The reference validator (`validate_mail_address`) does +exactly this, and so can your own code: the shape is the type tag. + +## Host-scoped addresses + +Host-scoped addresses are defined at the level of the MAIL **server**, not at the +level of any swarm inside it. They take the form `{ua_type}:{ua_id}@{host}`, +where `ua_type` is one of `admin`, `daemon`, or `user`: + +```text +admin:root@example.com an administrator +daemon:worker-1@example.com a delivery daemon +user:alice@example.com a human user +``` + +These are the server's permanent residents. A **user** is a human; an **admin** +is a human (or operator) with server-level privileges; a **daemon** is the +autonomous worker that actually carries messages between inboxes. None of them +belongs to a particular swarm — they exist at the host, so their identifiers are +unique across the *entire* server. There is only ever one `user:alice@example.com`. + +## Swarm-scoped addresses + +A **swarm** is an abstract collection of agent addresses and mailing lists — a +way to scope a discrete multi-agent deployment inside a server. Addresses that +live inside a swarm take the form `{address_id}@{swarm}@{host}`, and `address_id` +comes in exactly two flavors: + +```text +sage@chorus@example.com an agent (bare name, no prefix) +list:welfare-discourse@chorus@host a mailing list (the list: prefix) +``` + +An **agent** is named by a bare identifier; a **mailing list** is named with a +`list:` prefix. That prefix is the *only* prefixed swarm-scoped form — there is +no `agent:` prefix, because a bare name already means "agent." (`agent:sage@…` +is therefore invalid; it should be `sage@…`.) + +## Why agent names repeat across swarms + +Here is the design choice that the two-scope split exists to enable: **an agent's +name is unique only within its swarm.** The following two addresses name two +*different* agents, and both are valid simultaneously: + +```text +supervisor@swarm-1@example.com +supervisor@swarm-2@example.com +``` -## Topics to Discuss +This matters because swarms are meant to be independent deployments. Each one +should be free to name its agents naturally — every swarm can have a +`supervisor`, a `planner`, a `researcher` — without coordinating a globally +unique name with every other swarm on the server. The swarm segment is what +disambiguates them. Mailing list names work the same way: `list:all@swarm-1` and +`list:all@swarm-2` coexist. -- Host-scoped identities for users, admins, and daemons. -- Swarm-scoped identities for agents and mailing lists. -- Why agent names can repeat across swarms. -- Why mailing list addresses live inside swarms. -- Validation boundaries and common mistakes. +Contrast this with host-scoped identifiers (users, admins, daemons), which carry +no swarm segment and so *must* be unique across the whole server. The scope a +name lives in determines how widely it has to be unique. That is the heart of the +model: **host-scoped names are server-unique; swarm-scoped names are only +swarm-unique.** + +## Why mailing lists live inside a swarm + +A mailing list is a **fan-out target**, not a user-agent. It owns no inbox; when +a message is addressed to `list:{list_id}@{swarm}@{host}`, the server expands the +list and delivers a copy to each member. Lists are swarm-scoped for the same +reason agents are: they belong to a particular deployment, and naming them inside +a swarm lets list names repeat across swarms without collision. The `list:` +prefix is what distinguishes a list from an agent that happens to share the +swarm-scoped shape. + +## How an address is validated + +Identifiers in MAIL — every `ua_id`, `agent`, `swarm`, and `list_id` — must be +**slugs**: lowercase alphanumerics separated by single hyphens, matching + +```text +^[a-z0-9]+(?:-[a-z0-9]+)*$ +``` + +So `welfare-discourse` is valid; `Welfare_Discourse`, `-leading`, `trailing-`, +and `double--hyphen` are not. Each identifier must be at least one character and +SHOULD be at most 32 (the reference implementation enforces 32 as a hard cap). +The `host` segment must be a valid domain name — or, in the reference +implementation, an IP address. + +Putting it together, validation is just the classification from the top of this +page plus a slug check on each part: + +1. Split on `@`. Two segments → expect `{ua_type}:{ua_id}`, with `ua_type` in + `{admin, daemon, user}`. Three segments → the first is either a bare agent + slug or `list:{list_id}`; the middle is the swarm. +2. Slug-check each identifier; validate the host. +3. Reject anything that does not fit either shape. + +## Common mistakes + +These all fail validation — and each is worth recognizing, because the error +("invalid MAIL address structure") is the same for several distinct causes: + +| Address | Why it is rejected | +| --- | --- | +| `alice@example.com` | A two-segment address needs a `ua_type:` prefix. Use `user:alice@example.com`. | +| `agent:sage@chorus@example.com` | Agents are bare; only lists take a prefix. Use `sage@chorus@example.com`. | +| `bot:alice@example.com` | `ua_type` must be `admin`, `daemon`, or `user`. | +| `user:Alice@example.com` | Identifiers are lowercase slugs — no capitals, no spaces. | +| `sage@Chorus@example.com` | The swarm segment must be a slug too. | +| `a@b@c@d`, `alice`, `""` | Neither the two-segment nor three-segment shape. | ## Related Pages -- [Data Models](../references/data-models.md) -- [Manage User-Agents](../howtos/manage-user-agents.md) -- [Manage Mailing Lists](../howtos/manage-mailing-lists.md) +- [Data Models](../references/data-models.md) — the user-agent and address schemas, with exact length bounds. +- [Manage User-Agents](../howtos/manage-user-agents.md) — creating admins, agents, daemons, and users. +- [Manage Mailing Lists](../howtos/manage-mailing-lists.md) — creating and addressing lists. +- [Build a Minimal HTTP Client](../tutorials/build-minimal-http-client.md) — uses these addresses in practice. + +## Source Material + +- `spec/SPEC.md` §5 (User-Agents) and §6 (Addresses) +- `src/mail/protocol/src/mail_protocol/core/user_agents.py` +- `src/mail/protocol/src/mail_protocol/core/validators.py` +- `tests/contract/test_spec_addresses.py` diff --git a/docs/explanations/delivery-model.md b/docs/explanations/delivery-model.md index a4535da..b2fbc67 100644 --- a/docs/explanations/delivery-model.md +++ b/docs/explanations/delivery-model.md @@ -1,30 +1,161 @@ # Delivery Model -Status: stub +Status: draft -## Question +In MAIL, **sending a message is not the same as delivering it.** When a +user-agent sends, the message is written to the server and placed in the sender's +outbox — but it is not yet in anyone's inbox. A separate, authorized worker (a +*daemon*) carries it the rest of the way. This page explains why MAIL splits +those two acts, how the hand-off works, and what it means operationally. For the +hands-on version of the first half, see +[Build a Minimal HTTP Client](../tutorials/build-minimal-http-client.md); for the +formal contract, see [§8 of the specification](../../spec/SPEC.md). -Why does MAIL store sent messages first and rely on daemons for delivery? +## Sending is not delivering -## Source Material +The central idea is a deliberate separation: -- `spec/SPEC.md` section 8 -- `src/mail/daemon/src/mail_daemon/maild/api.py` -- `src/mail/server/src/mail_server/routers/daemon.py` -- `src/mail/server/src/mail_server/backends/base.py` -- `tests/integration/test_flows.py` +1. **Send** — the sender writes a message to the server. It lands in the sender's + outbox, marked as not-yet-delivered, and its id is queued for delivery. +2. **Deliver** — a daemon later picks the message up and the server files a copy + into each recipient's inbox. + +So a message has two observable states — *sent* and *delivered* — and MAIL makes +the gap between them explicit rather than hiding it. Everything below follows +from taking that separation seriously. + +## Two steps to a message: draft, then send + +Creating a message is itself two steps, which is worth understanding before +delivery enters the picture: + +- **Create a draft** (`POST /drafts`) with only a `subject` and a `body`. A draft + has no recipients. +- **Send the draft** (`POST /drafts/{draft_id}/send`) with the `recipients`. This + is the moment the `MAILMessage` is assembled — a fresh `message_id`, the + sender, the recipients, the subject and body, a `sent_at` timestamp — stored in + the sender's outbox with no delivery stamp yet, and enqueued for delivery. + +Recipients are bound at *send* time, not at draft time, so a draft is a reusable +subject-and-body that can be sent more than once or to different addresses. (The +draft remains in your drafts box after sending.) For why recipients are addressed +the way they are, see [Addressing Model](addressing-model.md). + +## The delivery buffer + +The server keeps a **delivery buffer**: the list of `message_id`s awaiting +delivery. Sending a draft enqueues its id there. The buffer holds *ids*, not +copies — the message itself lives in storage and in the sender's outbox; the +buffer is just the server's "still needs carrying" worklist. + +## The daemon carries the mail + +A **daemon** is a distinct user-agent whose entire job is delivery. The spec +constrains it tightly: a daemon MUST NOT alter the messages it carries, and +SHOULD NOT compose messages of its own. Delivery is a narrow, privileged, +auditable role — not something every agent does for itself. + +A daemon authenticates exactly like any other user-agent (password grant → bearer +token), and the server verifies that the caller really is a daemon before +honoring delivery calls. It then runs a simple loop (the reference daemon pauses +~30 seconds between iterations): + +```text +sender server daemon + │ POST /drafts/{id}/send │ + │ ───────────────────▶ store in outbox │ + │ enqueue id in buffer │ + │ │ + │ ◀───── POST /daemon/message-buffer/clear + │ return pending ids, │ + │ empty the buffer ──────────▶ │ + │ │ + │ ◀───── POST /daemon/deliver/local {ids} + │ for each id: │ + │ file copy into each inbox │ + │ stamp outbox delivered_at, │ + │ delivered_by = daemon │ + │ return delivered summaries ─▶ │ +``` + +In other words: clearing the buffer hands the daemon the pending ids *and* empties +the buffer; the delivery call is where the server actually files copies into +recipients' inboxes and stamps the sender's outbox entry with the delivery time +and the delivering daemon's address. -## Topics to Discuss +## Sent versus delivered, made observable -- Draft creation versus sent message creation. -- Server delivery buffer. -- Daemon authorization and delivery responsibility. -- Local delivery versus future remote delivery concerns. -- Pre-send validation failures versus post-send delivery failures. -- Operational implications for retries, observability, and idempotency. +Because delivery is a separate step, MAIL exposes where a message is in its +journey. Each outbox entry carries two nullable fields: + +- `delivered_at` — `null` while the message is still waiting; a timestamp once a + daemon has carried it. +- `delivered_by` — the address of the daemon that delivered it. + +A `null` `delivered_at` means *sent, awaiting delivery*; a populated one means +*delivered*. The recipient's inbox entry likewise records which daemon delivered +it. This is the surface a client uses to show a message's status honestly — +"Sent" versus "Delivered by `daemon:…`" — rather than pretending the two are the +same. (Delivery is also where the server can fire webhooks so recipients are +notified of new mail rather than having to poll; see [HTTP API](../references/http-api.md).) + +## Local versus remote delivery + +The implemented delivery path is **local**: `POST /daemon/deliver/local` carries +messages between user-agents on the *same* server. A second endpoint, +`POST /daemon/deliver/remote`, is reserved for future delivery of messages that +arrive from other MAIL servers; it is not yet implemented. For now, treat +delivery as a within-host operation. + +## Pre-send versus post-send errors + +MAIL draws a sharp line between failures that happen *before* a message is +accepted and failures that happen *after*: + +- **Pre-send errors (§8.1)** are synchronous and caught at create or send time. A + malformed subject or body means the message is never created; a malformed + recipient address means it is never delivered. The sender is told immediately, + as a `4xx` response (a validation failure returns `422` with a `detail` + explaining what was wrong). Nothing is queued. +- **Post-send errors (§8.2)** happen after a valid message is in the system. If a + daemon cannot deliver it, the message MUST be preserved and the daemon SHOULD + log the error. These failures are asynchronous and recoverable — the message is + not lost. + +The boundary is the useful thing to remember: before send, an error is the +sender's to fix and the message may not exist at all; after send, the message is +durable and getting it delivered is the daemon's responsibility. + +## Operational implications + +- **Decoupling.** A sender never blocks on a recipient, or even on a daemon being + online. Sending is just a write. If no daemon is connected, messages simply wait + in the buffer (and sit undelivered in the outbox) until one runs. +- **Latency.** The reference daemon polls on an interval (~30 seconds by + default), so delivery is prompt but not instantaneous — expect up to a poll + interval of lag, especially under a backlog. +- **Observability.** Track delivery through the outbox (`delivered_at` / + `delivered_by`) and the daemon's logs. The daemon also warns when the number of + ids it cleared does not match the number the server reports as delivered. +- **Durability and retries.** Messages are stored server-side and preserved on + delivery failure (§8.2). Note the shape of the loop, though: clearing the buffer + empties it, so once a daemon has claimed a batch, delivering it is that daemon's + responsibility. Run a reliable daemon and watch its logs rather than assuming + failed items are automatically re-queued. +- **The unit of delivery is the `message_id`.** The daemon hands the server a + batch of ids to deliver; idempotency and retry policy live at that granularity. ## Related Pages -- [Run the MAIL Daemon](../howtos/run-daemon.md) -- [Daemon CLI](../references/daemon-cli.md) -- [HTTP API](../references/http-api.md) +- [Build a Minimal HTTP Client](../tutorials/build-minimal-http-client.md) — performs the draft → send half over raw HTTP. +- [Run the MAIL Daemon](../howtos/run-daemon.md) — running the daemon that does the carrying. +- [Addressing Model](addressing-model.md) — how recipients are named. +- [Daemon CLI](../references/daemon-cli.md) and [HTTP API](../references/http-api.md) — the daemon commands and `/daemon` endpoints. + +## Source Material + +- `spec/SPEC.md` §7 (Messages) and §8 (Delivery) +- `src/mail/server/src/mail_server/routers/daemon.py` +- `src/mail/daemon/src/mail_daemon/maild/api.py` +- `src/mail/server/src/mail_server/backends/base.py` (and `backends/memory/api.py` for the reference behavior) +- `tests/integration/test_flows.py` diff --git a/docs/tutorials/build-minimal-http-client.md b/docs/tutorials/build-minimal-http-client.md index a0cb6e1..bae6ed9 100644 --- a/docs/tutorials/build-minimal-http-client.md +++ b/docs/tutorials/build-minimal-http-client.md @@ -1,30 +1,293 @@ # Build a Minimal HTTP Client -Status: stub +Status: draft ## Outcome -The reader builds the smallest useful MAIL HTTP client: authenticate, call -`GET /auth/whoami`, create a draft, send it, and read response payloads. +You will talk to a MAIL server using nothing but HTTP — no `mail` CLI — and walk +away with a small shell script that authenticates, confirms your identity, +creates a draft, sends it, and reads the server's responses. The same handful of +calls translate directly into any language's HTTP library. ## Audience -Developers integrating MAIL into tools that should not shell out to the `mail` -CLI. +Developers integrating MAIL into their own tools and services who want to speak +to the server directly over HTTP rather than shelling out to the `mail` CLI. +Comfort with `curl` and JSON is assumed. ## Not Here - Full endpoint listings belong in [HTTP API](../references/http-api.md). - Protocol motivation belongs in [MAIL v2 Overview](../explanations/mail-v2-overview.md). +- Why sending is a two-step (draft, then send) is covered in [Delivery Model](../explanations/delivery-model.md). -## Draft Outline +## Prerequisites -1. Start from a known server URL and credentials. -2. Obtain a bearer token from `POST /auth/token`. -3. Validate identity with `GET /auth/whoami`. -4. Create a draft with `POST /drafts/`. -5. Send the draft with `POST /drafts/{draft_id}/send`. -6. Parse success and validation errors. +- A running MAIL server plus a user-agent address and password. If you do not + have one, complete [Run MAIL Locally](run-local-mail.md) first — this tutorial + reuses its `admin:dummy@localhost` account and `http://127.0.0.1:8865` server. +- `curl` to make requests, and `jq` to read and extract JSON. (`jq` is only for + convenience; every call works without it.) + +Set these in your shell once — every step below uses them: + +```bash +export MAIL_SERVER="http://127.0.0.1:8865" +export MAIL_ADDRESS="admin:dummy@localhost" +export MAIL_PASSWORD="" +``` + +## Steps + +### 1. Confirm the server is reachable + +Before authenticating, verify the server is up. The health endpoint needs no token: + +```bash +curl -s "$MAIL_SERVER/health" +``` + +```json +{"status":"ok"} +``` + +(`GET /` returns the protocol name, version, and uptime if you want to confirm +the version you are targeting.) + +### 2. Obtain a bearer token + +MAIL uses the OAuth2 "password" flow, so credentials are sent as **form fields**, +not as a JSON body: + +```bash +curl -s -X POST "$MAIL_SERVER/auth/token" \ + -d "grant_type=password" \ + --data-urlencode "username=$MAIL_ADDRESS" \ + --data-urlencode "password=$MAIL_PASSWORD" +``` + +The response carries a JSON Web Token: + +```json +{"access_token":"eyJhbGciOiJIUzI1NiIs...","token_type":"bearer","metadata":{}} +``` + +Capture the token so the authenticated calls can reuse it: + +```bash +export MAIL_TOKEN=$(curl -s -X POST "$MAIL_SERVER/auth/token" \ + -d "grant_type=password" \ + --data-urlencode "username=$MAIL_ADDRESS" \ + --data-urlencode "password=$MAIL_PASSWORD" | jq -r .access_token) +``` + +Tokens expire after the server's configured lifetime (`MAIL_JWT_EXPIRE_MINUTES`). +When a call starts returning `401`, request a fresh token the same way. + +### 3. Confirm your identity + +Every authenticated request carries the token in an `Authorization: Bearer` +header. Use `whoami` to check that the token works and to see who the server +thinks you are: + +```bash +curl -s "$MAIL_SERVER/auth/whoami" -H "Authorization: Bearer $MAIL_TOKEN" | jq +``` + +The user-agent is nested one level inside the response envelope: + +```json +{ + "user_agent": { + "user_agent": { + "ua_type": "admin", + "admin_id": "dummy", + "host": "localhost" + } + }, + "metadata": {} +} +``` + +`ua_type` is one of `agent`, `user`, `admin`, or `daemon`, and the remaining +fields depend on it — an `agent`, for instance, carries `name`, `swarm`, and +`host` instead of `admin_id`. See [Addressing Model](../explanations/addressing-model.md) +for how these compose into a full address. + +### 4. Create a draft + +A draft holds only a `subject` and a `body`; recipients are chosen later, at send +time. Send the draft as JSON: + +```bash +curl -s -X POST "$MAIL_SERVER/drafts" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"subject":"Hello over HTTP","body":"My first MAIL message, sent with curl."}' | jq +``` + +The new draft comes back wrapped in an `entry`: + +```json +{ + "entry": { + "draft": { + "draft_id": "0f8e6e2a-1c4d-4a9b-8e7f-2b6c1d0a9f3e", + "subject": "Hello over HTTP", + "body": "My first MAIL message, sent with curl.", + "created_at": "2026-06-16T23:11:00Z", + "updated_at": null + }, + "sent_at": null, + "sent_by": null + }, + "metadata": {} +} +``` + +Capture the `draft_id` for the next step: + +```bash +export DRAFT_ID=$(curl -s -X POST "$MAIL_SERVER/drafts" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"subject":"Hello over HTTP","body":"My first MAIL message, sent with curl."}' \ + | jq -r .entry.draft.draft_id) +``` + +### 5. Send the draft + +Choose one or more recipient addresses and send the draft. The recipients are +supplied now, not at draft time: + +```bash +curl -s -X POST "$MAIL_SERVER/drafts/$DRAFT_ID/send" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"recipients":["supervisor@default@localhost"]}' | jq +``` + +The response is the assembled message. It has a `message_id` distinct from the +`draft_id`, plus the recipients you just chose: + +```json +{ + "message": { + "message_id": "5a2c1b9d-7e3f-4c8a-9d21-0b4e6f8a1c2d", + "sender": "admin:dummy@localhost", + "recipients": ["supervisor@default@localhost"], + "subject": "Hello over HTTP", + "body": "My first MAIL message, sent with curl.", + "sent_at": "2026-06-16T23:11:05Z", + "metadata": {} + }, + "metadata": {} +} +``` + +The server has now stored the message; a running daemon delivers a copy to each +recipient's inbox. (MAIL stores first and delivers via daemon — see +[Delivery Model](../explanations/delivery-model.md). To watch the message land, +follow the inbox steps in [Run MAIL Locally](run-local-mail.md).) + +### 6. Read success and error payloads + +Two conventions make every response predictable: + +- **Success** responses wrap their payload and always include a `metadata` + object. A single resource uses a key like `entry` or `message`; list endpoints + (such as `GET /drafts`) use `entries`. +- **Errors** return a non-2xx status and a JSON object with a `detail` string. + Always check the status code — a parser that only reads the body can miss the + failure. + +Print the status code alongside the body with `-w`. An invalid or missing token +returns `401`: + +```bash +curl -s -w "\n%{http_code}\n" "$MAIL_SERVER/auth/whoami" \ + -H "Authorization: Bearer not-a-real-token" +``` + +```text +{"detail":"could not validate credentials"} +401 +``` + +A request that reaches the server but fails validation returns `422`, with +`detail` naming the offending field and the reason. For example, omitting the +required `body` when creating a draft: + +```bash +curl -s -w "\n%{http_code}\n" -X POST "$MAIL_SERVER/drafts" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"subject":"only a subject"}' +``` + +```text +{"detail":"request body validation failed: 1 validation error for DraftPostRequest\nbody\n Field required ..."} +422 +``` + +Sending to a malformed address, or with an empty `recipients` list, fails the +same way — the `detail` tells you which field and why. + +### 7. Put it together: a minimal client + +Every call above, assembled into one script. Save it as `mailclient.sh`: + +```bash +#!/usr/bin/env bash +set -euo pipefail + +MAIL_SERVER="${MAIL_SERVER:-http://127.0.0.1:8865}" +MAIL_ADDRESS="${MAIL_ADDRESS:-admin:dummy@localhost}" +: "${MAIL_PASSWORD:?set MAIL_PASSWORD}" +recipient="${1:?usage: mailclient.sh }" + +# 1. authenticate +token=$(curl -s -X POST "$MAIL_SERVER/auth/token" \ + -d grant_type=password \ + --data-urlencode "username=$MAIL_ADDRESS" \ + --data-urlencode "password=$MAIL_PASSWORD" | jq -r .access_token) +auth=(-H "Authorization: Bearer $token") + +# 2. confirm identity +echo "authenticated as: $(curl -s "${auth[@]}" "$MAIL_SERVER/auth/whoami" \ + | jq -r '.user_agent.user_agent.ua_type')" + +# 3. create a draft +draft_id=$(curl -s -X POST "$MAIL_SERVER/drafts" "${auth[@]}" \ + -H "Content-Type: application/json" \ + -d '{"subject":"Hello over HTTP","body":"Sent by mailclient.sh"}' \ + | jq -r .entry.draft.draft_id) +echo "draft created: $draft_id" + +# 4. send it +message_id=$(curl -s -X POST "$MAIL_SERVER/drafts/$draft_id/send" "${auth[@]}" \ + -H "Content-Type: application/json" \ + -d "{\"recipients\":[\"$recipient\"]}" \ + | jq -r .message.message_id) +echo "message sent: $message_id" +``` + +Run it with a recipient address: + +```bash +MAIL_PASSWORD="" ./mailclient.sh supervisor@default@localhost +``` + +```text +authenticated as: admin +draft created: 0f8e6e2a-1c4d-4a9b-8e7f-2b6c1d0a9f3e +message sent: 5a2c1b9d-7e3f-4c8a-9d21-0b4e6f8a1c2d +``` + +That is a complete MAIL client in four calls — authenticate, identify, draft, +send. Port the same requests into your language's HTTP library, reuse the bearer +token across calls, and you have native MAIL integration with no dependency on +the CLI. ## Source Material From 54e649b288ca66d44a6eb596e7bb96c4a4944bed Mon Sep 17 00:00:00 2001 From: minichorus-pm Date: Tue, 23 Jun 2026 16:18:18 -0400 Subject: [PATCH 06/28] docs: webhook delivery contract, manage-webhooks how-to, and build-webhook-receiver tutorial MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three new pages plus three index updates, addressing the webhook documentation gap in the v2 docs branch (the first of the two docs offers from the dev-list thread on Final Prep for v2). ## docs/explanations/webhook-delivery.md (new) The conceptual contract. What webhooks are for, the payload shape (with v2's reply_to and tags from PR #74), the HMAC-SHA256 scheme (timestamp.body, not body.timestamp — bug-shape worth flagging explicitly), the headers MAIL sends, the receiver verification checklist, the retry ladder (6 attempts: immediate, +1s, +30s, +5min, +1h, +6h — total window ~7h31m), the retry conditions (timeout / 5xx / 429 retry; 2xx / other 4xx don't), and the 'inbox is source of truth' contract that shapes how receivers should handle internal failures. Sources verified against src/mail/server/src/mail_server/backends/base.py (_handle_webhook_delivered and _webhook_delivered_post on origin main; the v2-docs branch is currently behind main on the schema changes from PR #74). Wrote against the canonical main-branch shape so the docs match the v2 release contract; the docs branch needs the rebase before merge. ## docs/howtos/manage-webhooks.md (new) Operator's guide. Secret generation, POST /admin/webhooks (with events and url), GET /admin/webhooks for listing, GET /admin/webhooks/{id} for inspection, PATCH /admin/webhooks/{id} for URL / secret rotation (event types are immutable in v2), DELETE /admin/webhooks/{id}. Includes the rotation coordination note (both sides must update at the same moment to avoid signature failures in flight). ## docs/tutorials/build-webhook-receiver.md (new) Implementer's walkthrough. A single-file FastAPI receiver with: - verify_signature using HMAC over raw bytes (calls out the two most common bugs: re-encoded JSON breaking signature; missing the timestamp.body prefix). - is_duplicate / mark_processed for event_id-based dedup with a 24-hour garbage-collection window. - is_timestamp_in_window for 5-min skew rejection. - The full endpoint composing them with the right error codes (503 if secret not configured, 408 for skew, 403 for bad signature, 200 with status=duplicate for retries). - Registration command for end-to-end test. - Diagnostic checklist for the 'nothing arrives' case. ## Index updates docs/{explanations,howtos,tutorials}/README.md each gain a row linking to the new page. ## Followups in scope of the original offer - docs/howtos/manage-mailing-lists.md exists as a stub today; drafting that one is the second piece of the offer and will land as a separate commit on the same branch. - docs/references/http-api.md is a stub overall (not just for webhooks). The webhook-specific endpoints could be sketched there in a later pass; deferred so this commit stays focused on the conceptual + tutorial layer. ## Verification The four facts that are easiest to get wrong (and that I had ground truth on from the chorus-side webhook receiver): - HMAC inputs: 'timestamp.raw_body' not 'raw_body.timestamp'. - X-MAIL-Timestamp value: Unix seconds as a STRING, used in both the HMAC and the header so receivers can recompute from the header alone. - Signature header format: 'sha256='. - Body bytes: payload.model_dump_json() (Pydantic's canonical JSON), posted as-is — re-encoded JSON has different bytes and breaks verification. All four match what _webhook_delivered_post does in src/mail/server/src/mail_server/backends/base.py:653. --- docs/explanations/README.md | 1 + docs/explanations/webhook-delivery.md | 263 ++++++++++++++++++++++ docs/howtos/README.md | 1 + docs/howtos/manage-webhooks.md | 148 +++++++++++++ docs/tutorials/README.md | 1 + docs/tutorials/build-webhook-receiver.md | 265 +++++++++++++++++++++++ 6 files changed, 679 insertions(+) create mode 100644 docs/explanations/webhook-delivery.md create mode 100644 docs/howtos/manage-webhooks.md create mode 100644 docs/tutorials/build-webhook-receiver.md diff --git a/docs/explanations/README.md b/docs/explanations/README.md index 91a73b9..03706b2 100644 --- a/docs/explanations/README.md +++ b/docs/explanations/README.md @@ -12,6 +12,7 @@ They are for understanding, not for step-by-step tasks or exhaustive lookup. | [Addressing Model](addressing-model.md) | Why does MAIL use host-scoped and swarm-scoped addresses? | | [Delivery Model](delivery-model.md) | Why are daemons responsible for message delivery? | | [Security Model](security-model.md) | What are the main trust boundaries and risks? | +| [Webhook Delivery](webhook-delivery.md) | What is the webhook contract — payload shape, HMAC signing, retry behavior, and the inbox-is-source-of-truth assumption? | | [MAIL v1 Legacy Runtime](mail-v1-legacy.md) | How should readers interpret the archived v1 runtime and docs? | | [Documentation System](documentation-system.md) | How should maintainers decide where a new page belongs? | diff --git a/docs/explanations/webhook-delivery.md b/docs/explanations/webhook-delivery.md new file mode 100644 index 0000000..fec2591 --- /dev/null +++ b/docs/explanations/webhook-delivery.md @@ -0,0 +1,263 @@ +# Webhook Delivery + +Status: draft + +## Scope + +How MAIL notifies external consumers when mail is delivered: the event +shape, the security model, the retry behavior, and the assumptions the +contract places on receivers. + +This document is for implementers building a webhook consumer (a +service that receives MAIL events and routes them somewhere else). +The matching how-to for *registering* webhooks via the admin API is +[Manage Webhooks](../howtos/manage-webhooks.md); the matching tutorial +for *building* a receiver end-to-end is [Build a Webhook +Receiver](../tutorials/build-webhook-receiver.md). + +## What webhooks are for + +The MAIL inbox is the durable surface: mail lives there, indexed by +recipient, and any authenticated user-agent can poll it via the +inbox endpoints. Webhooks are a *push* alternative: when a message +is delivered to a recipient's inbox, the MAIL server fires an HTTP +`POST` to one or more registered URLs with a structured payload, so +downstream services can react without polling. + +Webhooks do not replace the inbox. The inbox is the source of truth. +A webhook that fails to deliver is a notification missed; the message +itself is still readable by the recipient via the normal inbox API. +This shapes the security and retry contract below. + +## Event types + +The only `event` value in v2 is `mail.delivered`. A future release +may add other event types; receivers should reject events whose +`event` field they do not recognize, but should not fail registration +on the presence of an unknown event in the `events` array (the +`/admin/webhooks` validator already gates that). + +## Payload shape + +Every `mail.delivered` event is delivered as a JSON request body +shaped like: + +```json +{ + "event": "mail.delivered", + "event_id": "evt_", + "delivered_at": "2026-06-24T19:31:00.000000+00:00", + "message": { + "message_id": "msg_", + "reply_to": null, + "sender": "alice@chorus@example.com", + "recipient": "bob@chorus@example.com", + "subject": "Daily briefing", + "body": "…", + "tags": [], + "sent_at": "2026-06-24T19:30:55.123456+00:00", + "swarm": "chorus", + "metadata": {} + } +} +``` + +Field notes: + +- `event_id` is unique per delivery attempt SET but is reused across + retries (see [Retries](#retries)). Receivers MUST treat + `event_id` as the dedup key — a webhook receiver that processes + the same `event_id` more than once is a bug. +- `message_id` is prefixed with `msg_`. The bare UUID is stored on + the canonical `MAILMessage`; the prefix is added at webhook + payload construction time. Use the prefixed form when fetching the + full message via the inbox API. +- `reply_to`, when set, is the prefixed `message_id` of the original + message this is replying to. +- `tags` is a list of slug-shaped strings the sender attached. +- `metadata.list_address`, when present, indicates the delivery + originated from a list expansion. Use it to surface the originating + list to the end-recipient. + +Refer to [Data Models](../references/data-models.md) for the full +field-by-field schema of the inner `MAILMessageInWebhook`. + +## Security model + +### Why HMAC + +MAIL emits webhooks to URLs configured by an administrator. The +receiver needs to verify that an incoming request actually came from +MAIL (not from a third party who guessed or scanned the URL). The +shared mechanism is an HMAC signature over the request body, computed +with a secret known only to MAIL and the receiver. + +### What gets signed + +MAIL computes the signature as: + +``` +signature = HMAC-SHA256(secret, f"{timestamp}.{raw_body}") +``` + +Where: + +- `timestamp` is the value of the `X-MAIL-Timestamp` header (Unix + seconds since the epoch, as a string). +- `raw_body` is the *exact byte sequence* of the request body. MAIL + signs `payload.model_dump_json()` (Pydantic's canonical JSON + serialization) and posts those same bytes as the request body. + Re-encoding via `json=...` would produce different bytes (different + key order, whitespace, type coercion) and break verification. + +The `secret` is the value supplied when the webhook was registered. + +### Headers sent on every webhook POST + +| Header | Value | +| ------------------ | ------------------------------------------------- | +| `Content-Type` | `application/json` | +| `X-MAIL-Event-Id` | The `event_id` from the payload. | +| `X-MAIL-Timestamp` | Unix seconds since epoch, as a string. | +| `X-MAIL-Signature` | `sha256=` where `` is the HMAC digest. | +| `User-Agent` | `Multi-Agent-Interface-Layer-Server/2.0.0 (...)` | + +### Receiver verification + +A correct receiver does the following on every request: + +1. Read `X-MAIL-Timestamp` and reject the request (`408` or `400`) + if it is more than 5 minutes from the receiver's clock. This + bounds the replay window. +2. Read `X-MAIL-Signature` and strip the `sha256=` prefix. +3. Recompute `HMAC-SHA256(secret, f"{timestamp}.{raw_body}")` over + the raw request body bytes (NOT the parsed JSON). +4. Compare to the received digest using a constant-time comparison. + Reject (`403`) if they differ. +5. Read `X-MAIL-Event-Id` and check it against a recent-events store. + If it has been processed in the last ~24 hours, return `200` with + a no-op response (the request is a retry; the original processing + stands). +6. Process the event. Return `200` (or `202`) on success. + +Step 5 is where the dedup contract lives. MAIL retries on transient +failure (see below) and reuses the same `event_id` across retries. +A receiver that does not dedup will process the same delivery +multiple times under load or after any transient outage. + +### What if the secret isn't configured + +A receiver that has registered a webhook but does not yet have the +secret in its environment SHOULD reject incoming requests with +`503 Service Unavailable` (not `403`). `403` would suggest a real +authentication failure; `503` correctly signals "I'm not ready, +please retry." + +## Retries + +MAIL fires up to **six attempts** per event, with the following +delays between attempts: + +| Attempt | Delay before this attempt | Cumulative wall-clock | +| ------- | ------------------------- | ---------------------- | +| 1 | (immediate) | 0 | +| 2 | 1 second | ~1 s | +| 3 | 30 seconds | ~31 s | +| 4 | 5 minutes | ~5 min | +| 5 | 1 hour | ~1 h | +| 6 | 6 hours | ~7 h | + +After the sixth attempt, MAIL gives up. The total retry window is +roughly **7 hours and 31 seconds** from the first attempt. + +A retry is triggered when `_webhook_delivered_post` returns `True`, +which happens for any of: + +- `httpx.TimeoutException` on the request. +- A `5xx` status code from the receiver. +- A `429 Too Many Requests` status code. + +A retry is NOT triggered (and the event is considered delivered or +abandoned) for: + +- A `2xx` status code (success). +- A `4xx` status code other than `429` (the receiver explicitly + rejected the request; retries won't change that). + +### Implications for receivers + +- A receiver that needs to throttle MAIL's webhook firing should + return `429` rather than starve. MAIL backs off cleanly. +- A receiver that detects a permanently malformed payload should + return `4xx` (not `5xx`). MAIL will not retry, which is the + correct behavior — the next event will succeed. +- A receiver should NOT return `5xx` for "I couldn't route this + internally but I have the message stored." That makes MAIL retry + unnecessarily. Instead, return `200` — MAIL's inbox is the source + of truth; the routing failure does not need MAIL's help to + recover. + +## The "inbox is source of truth" contract + +This is the single most important assumption a receiver makes: + +> If a webhook delivery fails, the message is not lost. The +> recipient can still poll their MAIL inbox via the regular HTTP +> API. The webhook is a notification — its failure shapes UX, not +> correctness. + +In practice this means: + +- A receiver that successfully verifies the signature, accepts the + event_id as new, but then fails internally while processing the + event SHOULD STILL RETURN `200`. The event is recorded as + processed; the internal failure is the receiver's problem to + recover from (it can read the message from the MAIL inbox on its + own schedule). +- A receiver MUST NOT return `5xx` to "force MAIL to retry." MAIL's + retries are for transport failures, not for receiver-internal + bugs. The retry schedule above is short enough that downstream + systems can fail and recover quickly without webhook help. + +## Reliability and ordering + +The webhook contract guarantees: + +- **At-least-once delivery** within the 7-hour retry window. After + retries exhaust, the message is still in the recipient's inbox + and can be fetched there. +- **Per-event idempotency via `event_id`.** Receivers MUST dedup on + `event_id` to handle retries correctly. + +The contract does NOT guarantee: + +- **Ordering.** Webhooks for related events (e.g., several mails to + the same recipient in rapid succession) may arrive out of order + due to retry interleavings or concurrent firing. Receivers MUST + treat each event independently. The `sent_at` and `delivered_at` + timestamps can be used to reconstruct ordering if needed. +- **Exactly-once delivery.** Dedup by `event_id` collapses the + at-least-once delivery to at-most-once *processing* in the + receiver's domain, but MAIL itself can fire the same event_id up + to six times. +- **Synchronous delivery.** Webhook firing happens asynchronously + on the MAIL server. A successful `POST /drafts/{id}/send` (or + similar) does not block on webhook delivery. + +## See also + +- [Manage Webhooks](../howtos/manage-webhooks.md) — registering, + inspecting, and deleting webhooks via the admin API. +- [Build a Webhook Receiver](../tutorials/build-webhook-receiver.md) + — step-by-step tutorial walking through the signature + verification, dedup, and processing of a real receiver. +- [Delivery Model](delivery-model.md) — broader context on how MAIL + routes a message from sender to recipient inbox. +- [Security Model](security-model.md) — the broader auth and + authorization model the webhook contract sits within. +- [Data Models](../references/data-models.md) — formal field-by-field + schemas for `MAILWebhook`, `MAILMessageInWebhook`, and the + envelope. +- [HTTP API](../references/http-api.md) — the formal route list, + including the webhook firing target shape and the admin + registration endpoints. diff --git a/docs/howtos/README.md b/docs/howtos/README.md index fbe8df3..67587c2 100644 --- a/docs/howtos/README.md +++ b/docs/howtos/README.md @@ -16,6 +16,7 @@ ordered steps, and stop when the task is complete. | [Manage User-Agents](manage-user-agents.md) | Create, inspect, and remove agents, users, admins, and daemons. | | [Manage Swarms](manage-swarms.md) | Create, inspect, and delete swarms. | | [Manage Mailing Lists](manage-mailing-lists.md) | Create lists and manage subscriptions or members. | +| [Manage Webhooks](manage-webhooks.md) | Register, inspect, update, and delete webhook subscriptions. | | [Regenerate API Artifacts](regenerate-api-artifacts.md) | Refresh generated OpenAPI or documentation artifacts. | | [Run the Test Suite](run-tests.md) | Run focused or full repository tests. | diff --git a/docs/howtos/manage-webhooks.md b/docs/howtos/manage-webhooks.md new file mode 100644 index 0000000..573c0e5 --- /dev/null +++ b/docs/howtos/manage-webhooks.md @@ -0,0 +1,148 @@ +# Manage Webhooks + +Status: draft + +## Goal + +How to register, inspect, update, and delete webhook subscriptions on +a MAIL server using the admin API. Webhooks let downstream services +receive `mail.delivered` events without polling — see [Webhook +Delivery](../explanations/webhook-delivery.md) for the conceptual +contract. + +## Starting Point + +You have admin credentials for the MAIL server, and you know the +public URL the webhook should fire against. You have a shared secret +already agreed with the receiver, or you're prepared to generate one. + +## Steps + +### 1. Generate a secret (if you don't already have one) + +Webhook signatures use an HMAC-SHA256 with a shared secret. The +secret must be known to both MAIL and the receiver, and not exposed +elsewhere. A reasonable generator: + +```bash +python -c "import secrets; print(secrets.token_urlsafe(32))" +``` + +Save the resulting string in a secure location accessible to the +receiver process. The receiver loads it from a config file or env +var; MAIL stores it on the registered webhook record. + +### 2. Register the webhook + +`POST /admin/webhooks` with the receiver URL, the events to +subscribe to, and the secret. v2 supports one event type +(`mail.delivered`); future versions may add more. + +```bash +ADMIN_TOKEN="$(cat ~/.mail/admin.token)" +SECRET="…" # from step 1 + +curl -sS -X POST "$MAIL_SERVER/admin/webhooks" \ + -H "Authorization: Bearer $ADMIN_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$(jq -n \ + --arg url "https://my-receiver.example.com/mail/webhook" \ + --arg secret "$SECRET" \ + '{url: $url, events: ["mail.delivered"], secret: $secret}')" +``` + +A successful response returns the new webhook record (including its +generated `webhook_id`): + +```json +{ + "webhook": { + "webhook_id": "wh_abc12345-…", + "url": "https://my-receiver.example.com/mail/webhook", + "events": ["mail.delivered"], + "secret": "…" + }, + "metadata": {} +} +``` + +Save the `webhook_id` — you'll need it to inspect, update, or delete +the registration later. The secret is also stored in MAIL's backend; +the receiver only needs its own copy. + +### 3. List all registered webhooks + +`GET /admin/webhooks` returns the IDs of every webhook on the +server: + +```bash +curl -sS "$MAIL_SERVER/admin/webhooks" \ + -H "Authorization: Bearer $ADMIN_TOKEN" +``` + +```json +{ + "webhook_ids": ["wh_abc12345-…", "wh_def67890-…"], + "metadata": {} +} +``` + +To get the full record for a specific webhook, use +`GET /admin/webhooks/{webhook_id}`: + +```bash +curl -sS "$MAIL_SERVER/admin/webhooks/wh_abc12345-…" \ + -H "Authorization: Bearer $ADMIN_TOKEN" +``` + +### 4. Update an existing webhook + +`PATCH /admin/webhooks/{webhook_id}` can change the receiver URL or +rotate the secret. The webhook_id and the subscribed events are +immutable; to change events you must delete and re-register. + +```bash +curl -sS -X PATCH "$MAIL_SERVER/admin/webhooks/wh_abc12345-…" \ + -H "Authorization: Bearer $ADMIN_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"url": "https://new-receiver.example.com/mail/webhook", "secret": "new-secret"}' +``` + +When rotating a secret, coordinate with the receiver so both sides +update at the same moment; otherwise webhooks delivered between the +two updates will fail signature verification on the receiver side. + +### 5. Delete a webhook + +`DELETE /admin/webhooks/{webhook_id}` removes the registration. MAIL +will stop firing webhooks to that URL immediately. + +```bash +curl -sS -X DELETE "$MAIL_SERVER/admin/webhooks/wh_abc12345-…" \ + -H "Authorization: Bearer $ADMIN_TOKEN" +``` + +In-flight retries for events that were already being delivered when +the webhook was deleted are not interrupted; if a retry attempt +succeeds, the receiver still gets the event. After the retry +schedule exhausts (or succeeds), no further events fire. + +## Validation + +After registering a webhook, you can confirm it works end-to-end by: + +1. Sending a test message to a recipient on the server. +2. Watching the receiver's logs for an incoming `POST` with the + expected event_id, signature, and payload. +3. Confirming the receiver returns `200`. (A non-2xx response will + trigger MAIL's retry ladder; see [Webhook + Delivery](../explanations/webhook-delivery.md#retries).) + +## See also + +- [Webhook Delivery](../explanations/webhook-delivery.md) — the + contract MAIL emits and the receiver must verify. +- [Build a Webhook Receiver](../tutorials/build-webhook-receiver.md) + — implementer's tutorial for writing a receiver from scratch. +- [HTTP API](../references/http-api.md) — full route and response + reference. diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md index a077d11..3c67bf4 100644 --- a/docs/tutorials/README.md +++ b/docs/tutorials/README.md @@ -11,6 +11,7 @@ needed to complete the lesson, and be tested end to end before release. | [Run MAIL Locally](run-local-mail.md) | Start a local memory-backed server, run a daemon, and observe local delivery. | `src/mail/server/docs/tutorials/quickstart.md`, `src/mail/daemon/src/mail_daemon/maild/api.py` | | [Send Your First MAIL Message](send-first-message.md) | Log in, compose a draft, send it, and inspect inbox/outbox state. | `src/mail/client/docs/tutorials/quickstart.md`, `src/mail/client/src/mail_client/cli.py` | | [Build a Minimal HTTP Client](build-minimal-http-client.md) | Authenticate and interact with MAIL using raw HTTP calls. | `spec/openapi.yaml`, `src/mail/protocol/src/mail_protocol/network/` | +| [Build a Webhook Receiver](build-webhook-receiver.md) | Build a correct HTTP receiver for MAIL's `mail.delivered` webhook events. | `src/mail/server/src/mail_server/backends/base.py`, `docs/explanations/webhook-delivery.md` | ## Tutorial Checklist diff --git a/docs/tutorials/build-webhook-receiver.md b/docs/tutorials/build-webhook-receiver.md new file mode 100644 index 0000000..19f6fce --- /dev/null +++ b/docs/tutorials/build-webhook-receiver.md @@ -0,0 +1,265 @@ +# Build a Webhook Receiver + +Status: draft + +## Goal + +Walk through building a correct MAIL webhook receiver from scratch. +By the end you'll have a small HTTP server that verifies signatures, +dedupes retries, and processes `mail.delivered` events. + +The companion explainer is [Webhook +Delivery](../explanations/webhook-delivery.md). This tutorial assumes +you've read it; we'll reference its sections rather than restate the +contract. + +## Prerequisites + +- A running MAIL server you can register webhooks against (see + [Run a Local MAIL](run-local-mail.md)). +- Python 3.12+ with `fastapi`, `uvicorn`, and `httpx`. +- A receiver URL MAIL can reach. For local development, a tunnel + (e.g., `cloudflared`) or both processes on the same host both + work. + +## The receiver, end to end + +We'll build a single-file FastAPI app that: + +1. Accepts `POST /mail/webhook`. +2. Verifies the `X-MAIL-Timestamp`, `X-MAIL-Signature`, and the + `Content-Type`. +3. Dedupes against `X-MAIL-Event-Id`. +4. Parses the payload, then prints a one-line summary. + +### Set up + +Create a new directory and install the dependencies: + +```bash +mkdir mail-receiver && cd mail-receiver +uv init +uv add fastapi uvicorn +``` + +### The signature verification function + +The signature scheme is documented in detail in [Webhook Delivery → +Security model](../explanations/webhook-delivery.md#security-model). +The receiver's job is to recompute `HMAC-SHA256(secret, +f"{timestamp}.{raw_body}")` over the *raw bytes* it received (not +the parsed JSON), then compare in constant time. + +```python +import hashlib +import hmac + + +def verify_signature( + *, raw_body: bytes, timestamp: str, signature: str, secret: str +) -> bool: + """ + Return True iff ``signature`` is a valid HMAC-SHA256 over + ``f"{timestamp}.{raw_body}"`` keyed by ``secret``. + + Signature comes in as ``"sha256="``; strip the prefix before + comparison. + """ + if not signature.startswith("sha256="): + return False + received = signature[len("sha256=") :] + + expected = hmac.new( + key=secret.encode("utf-8"), + msg=f"{timestamp}.".encode("utf-8") + raw_body, + digestmod=hashlib.sha256, + ).hexdigest() + + return hmac.compare_digest(received, expected) +``` + +A common bug at this step is to recompute the HMAC over the *parsed +and re-serialized* JSON body, which produces different bytes than +the originally-signed body and breaks verification. Always operate +on the raw bytes you received on the wire. + +Another common bug is to forget the `f"{timestamp}."` prefix. The +signed message is `timestamp.body`, not just `body`. + +### Dedup against event_id + +MAIL retries on transient failures (see [Retries](../explanations/webhook-delivery.md#retries)) +and reuses the same `event_id` for every attempt. A correct +receiver remembers recently-processed event_ids and short-circuits +duplicates. For this tutorial, an in-memory set is enough: + +```python +from collections import deque +from datetime import datetime, timezone + +PROCESSED_EVENTS: deque[tuple[str, datetime]] = deque(maxlen=10_000) + + +def is_duplicate(event_id: str) -> bool: + """ + Return True iff ``event_id`` has already been processed. + Garbage-collects entries older than 24 hours on each call. + """ + now = datetime.now(timezone.utc) + # Drop expired entries from the left. + while PROCESSED_EVENTS and (now - PROCESSED_EVENTS[0][1]).total_seconds() > 86400: + PROCESSED_EVENTS.popleft() + return any(e[0] == event_id for e in PROCESSED_EVENTS) + + +def mark_processed(event_id: str) -> None: + PROCESSED_EVENTS.append((event_id, datetime.now(timezone.utc))) +``` + +For production use, replace this with a real durable store (SQLite, +Redis, a database table) so dedup survives restarts. The 24-hour +window matches MAIL's retry exhaustion behavior with a safety +margin. + +### Timestamp skew window + +Reject any request whose `X-MAIL-Timestamp` is more than 5 minutes +from the receiver's clock. This bounds the replay window and catches +clock-drift bugs early. + +```python +import time + +SKEW_WINDOW_SECONDS = 5 * 60 + + +def is_timestamp_in_window(timestamp: str) -> bool: + try: + sent_at = int(timestamp) + except ValueError: + return False + return abs(int(time.time()) - sent_at) <= SKEW_WINDOW_SECONDS +``` + +### The full receiver + +```python +import os + +from fastapi import FastAPI, HTTPException, Request + +SECRET = os.environ.get("MAIL_WEBHOOK_SECRET") + +app = FastAPI() + + +@app.post("/mail/webhook") +async def mail_webhook(request: Request) -> dict[str, object]: + if SECRET is None: + # Not configured yet. Tell MAIL to retry later. + raise HTTPException( + status_code=503, detail="Webhook secret not configured." + ) + + timestamp = request.headers.get("X-MAIL-Timestamp") + signature = request.headers.get("X-MAIL-Signature") + event_id = request.headers.get("X-MAIL-Event-Id") + + if not timestamp: + raise HTTPException(status_code=400, detail="Missing X-MAIL-Timestamp.") + if not signature: + raise HTTPException(status_code=403, detail="Missing X-MAIL-Signature.") + if not event_id: + raise HTTPException(status_code=400, detail="Missing X-MAIL-Event-Id.") + + if not is_timestamp_in_window(timestamp): + raise HTTPException(status_code=408, detail="Timestamp outside skew window.") + + raw_body = await request.body() + + if not verify_signature( + raw_body=raw_body, timestamp=timestamp, signature=signature, secret=SECRET + ): + raise HTTPException(status_code=403, detail="Invalid signature.") + + if is_duplicate(event_id): + return {"status": "duplicate", "event_id": event_id} + + import json + + payload = json.loads(raw_body) + message = payload["message"] + + # Process the event. For this tutorial, just print. + print( + f"[mail.delivered] {message['sender']} → {message['recipient']}: " + f"{message['subject']}" + ) + + mark_processed(event_id) + return {"status": "ok", "event_id": event_id} +``` + +Run it with: + +```bash +MAIL_WEBHOOK_SECRET="" uv run uvicorn receiver:app --port 8000 +``` + +### Register with MAIL + +In another shell, register the receiver per [Manage +Webhooks](../howtos/manage-webhooks.md): + +```bash +curl -sS -X POST "$MAIL_SERVER/admin/webhooks" \ + -H "Authorization: Bearer $ADMIN_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"url": "http://localhost:8000/mail/webhook", + "events": ["mail.delivered"], + "secret": ""}' +``` + +### Send a test message + +Send a message to any recipient on the MAIL server. The receiver +should log: + +``` +[mail.delivered] alice@chorus@example.com → bob@chorus@example.com: Hello +``` + +If nothing arrives: + +- Check the MAIL server logs for outgoing POST attempts. A `403` + back from your receiver usually means the secret strings don't + match. +- Check that `MAIL_WEBHOOK_SECRET` in the receiver's env matches + the secret you registered exactly (no extra whitespace). +- Verify the receiver is reachable from MAIL's host (e.g., curl + from the MAIL host to your receiver URL). + +## What this tutorial leaves out + +- **Durable dedup.** Replace the in-memory set with a real store + before deploying. +- **Internal routing.** This receiver just prints. In production, + you'd route the event to whatever downstream service needs it + (a chat surface, a database write, a queue, etc.). +- **The "inbox is source of truth" contract.** If your internal + routing fails after signature verification succeeds, return + `200` anyway — the message lives in the MAIL inbox and your + service can recover on its own. See [Webhook Delivery → The + "inbox is source of truth" + contract](../explanations/webhook-delivery.md#the-inbox-is-source-of-truth-contract). +- **Observability.** Log every received event, every failed + signature, every dedup hit. Webhook receivers are silent failure + modes if you don't. + +## See also + +- [Webhook Delivery](../explanations/webhook-delivery.md) — the + contract this tutorial implements. +- [Manage Webhooks](../howtos/manage-webhooks.md) — registering and + rotating webhooks via the admin API. +- [HTTP API](../references/http-api.md) — formal route reference. From 929c7b5b69d7ebb9b1d4f6168e8f94ae2fa7c50e Mon Sep 17 00:00:00 2001 From: minichorus-pm Date: Tue, 23 Jun 2026 16:21:49 -0400 Subject: [PATCH 07/28] docs: mailing-lists explanation + manage-mailing-lists polish MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second of the two docs offers from the dev-list thread on Final Prep for v2. Adds the conceptual explainer that was missing and fills the gaps in the existing how-to. ## docs/explanations/mailing-lists.md (new) The conceptual model. Coverage: - What a list is: a swarm-scoped, addressable fan-out target with no inbox. - The address shape: list:@@, with the list: prefix as the distinguishing marker against the agent / user / admin shapes. - Why lists exist: three concrete problems (broadcast in one send, stable address for changing audience, policy separated from membership). - Anatomy: name, swarm, host, owner, members, policy, metadata. Canonical address immutable; policy and members mutable. - The policy shape: three independent enumerations (visibility, join_policy, send_policy), each with v1's honored variant flagged and the deferred variants explained as forward-looking wire-format reservations. - How messages flow: local delivery picks up list:, looks up list by address, fans out per-member with metadata.list_address, nested-list members get skipped, webhook firing is per-member. - Admin vs user-agent permission split: admin owns create / patch / member add-remove / delete and gets unconditional reads; user-agents get policy-gated read / subscribe / unsubscribe / send. - Addressability examples (table of the four address shapes). - Things lists are NOT: not a queue or buffer, not a history store, not a privacy boundary in v1. Sources verified against src/mail/protocol/src/mail_protocol/core/lists.py — the policy enum and 'forward-looking' framing are paraphrased from the inline docstring. ## docs/howtos/manage-mailing-lists.md (polished) The existing how-to covered list / list-get / create / subscribe / unsubscribe / member-post / member-delete / send. Added: - An updated Starting Point that points at the new explanation and names the address-shape convention explicitly. - Step 6 (NEW): update list policy via list-patch with the v1 honored variants and a pointer to the deferred-variant discussion. - Step 7 (NEW): delete a list via list-delete with the in-flight-messages-already-expanded note. - Step 8 (was step 6): send to a list, plus a forward reference to the webhook delivery doc for how the list_address metadata surfaces on the wire. - See-also section with cross-links to mailing-lists, webhook-delivery, addressing-model, and http-api. ## docs/explanations/README.md Index row added for the new mailing-lists explainer. --- docs/explanations/README.md | 1 + docs/explanations/mailing-lists.md | 235 ++++++++++++++++++++++++++++ docs/howtos/manage-mailing-lists.md | 84 +++++++++- 3 files changed, 316 insertions(+), 4 deletions(-) create mode 100644 docs/explanations/mailing-lists.md diff --git a/docs/explanations/README.md b/docs/explanations/README.md index 91a73b9..36abf6c 100644 --- a/docs/explanations/README.md +++ b/docs/explanations/README.md @@ -12,6 +12,7 @@ They are for understanding, not for step-by-step tasks or exhaustive lookup. | [Addressing Model](addressing-model.md) | Why does MAIL use host-scoped and swarm-scoped addresses? | | [Delivery Model](delivery-model.md) | Why are daemons responsible for message delivery? | | [Security Model](security-model.md) | What are the main trust boundaries and risks? | +| [Mailing Lists](mailing-lists.md) | What is a list? How does it expand, what does its policy mean, and how do admin and user-agent permissions split? | | [MAIL v1 Legacy Runtime](mail-v1-legacy.md) | How should readers interpret the archived v1 runtime and docs? | | [Documentation System](documentation-system.md) | How should maintainers decide where a new page belongs? | diff --git a/docs/explanations/mailing-lists.md b/docs/explanations/mailing-lists.md new file mode 100644 index 0000000..cc46b8f --- /dev/null +++ b/docs/explanations/mailing-lists.md @@ -0,0 +1,235 @@ +# Mailing Lists + +Status: draft + +## Scope + +The conceptual model behind MAIL's mailing lists: what they are, how +they're addressed, how messages flow through them, how the policy +shape works, and how admin and user-agent permissions split. The +matching how-to for *operating* lists via the CLI is [Manage +Mailing Lists](../howtos/manage-mailing-lists.md); the formal route +reference is in [HTTP API](../references/http-api.md). + +## What a list is + +A MAIL list is a **swarm-scoped, addressable fan-out target**. It is +not a user-agent and it does not own an inbox. When a sender +addresses a message to a list, MAIL's local delivery path expands +the list and delivers one copy of the message to each member +(see [Local versus remote delivery in the Delivery +Model](delivery-model.md#local-versus-remote-delivery) for the +broader pipeline). + +The address shape is: + +``` +list:@@ +``` + +The `list:` prefix is what distinguishes a list address from the +other three address shapes (`agent`, `user`, `admin`); see +[Addressing Model](addressing-model.md) for the full address +taxonomy. Subscribers (members) are themselves addressable +user-agents on the same host — typically the same swarm, though +cross-swarm membership is possible. + +## Why lists exist + +Three concrete problems lists solve cleanly: + +- **One sender, many recipients with a single send.** Without + lists, a sender broadcasting to N recipients has to issue N + send requests (or send to one address and have the receiver + re-broadcast). Lists let the fan-out happen server-side in + one atomic dispatch. +- **Stable address for a changing audience.** Members can be + added or removed without the sender needing to know. A + `list:announcements@chorus@chrn.ai` address persists; the + set of recipients behind it can change daily. +- **Policy control on send / join / visibility separated from + membership.** Who can join, who can post, and who can see + the list are three different questions; the list's `policy` + object addresses each independently. + +## Anatomy of a list + +A MAIL list has the following fields (see +[`MAILList`](../references/data-models.md) for the formal +schema): + +| Field | Meaning | +| --- | --- | +| `name` | The swarm-scoped identifier (e.g., `announcements`). | +| `swarm` | The swarm this list belongs to. | +| `host` | The MAIL host the list lives on. | +| `owner` | The MAIL address of the user-agent that created or owns the list. | +| `members` | The current list of subscribed user-agent addresses. | +| `policy` | The visibility / join / send policy (see below). | +| `metadata` | Free-form key/value pairs for downstream consumers. | + +Once created, the canonical address (`name`, `swarm`, `host`) is +**immutable for the life of the list**. The `policy` and `members` +fields can change; admin-side patches in v1 are limited to policy +edits. + +## The policy shape + +The `policy` object has three fields, each an enumeration with the +v1 variant the server actually honors flagged below: + +```python +class MAILListPolicy: + visibility: "public" | "private" # v1 honors: public + join_policy: "open" | "approval" | "admin-only" # v1 honors: open + send_policy: "open" | "members-only" | "admin-only" # v1 honors: open +``` + +The wire format reserves all enumerations now so future +contributions can extend the server without changing the protocol +shape. Other variants pass protocol-layer validation but are +rejected at the endpoint layer in v1 with `501 Not Implemented`. + +What "open" means in each field: + +- `visibility: public` — the list address appears in `GET /lists` + and `GET /lists/{addr}` for any authenticated user-agent. +- `join_policy: open` — any user-agent can `POST + /lists/{addr}/subscribe` to add themselves as a member without + admin intervention. +- `send_policy: open` — any user-agent can address a message to + the list and have it expanded. + +A v1 list is therefore effectively a **public open-open** list: +anyone can see it, anyone can join, anyone can post. + +The deferred variants (`approval`, `admin-only`, `members-only`, +`private`) define the structure that v1.1+ can fill in. Designing +a list with `join_policy: admin-only` today means the policy is +recorded faithfully but the server returns `501` on any +self-subscribe attempt; readers can use that signal to know "this +list will become admin-managed when the server honors it." + +## How messages flow through a list + +When a sender addresses a message to `list:@@`, +the following happens on the receiving MAIL server: + +1. **Local delivery picks up the list address.** The recipient + prefix `list:` triggers the list-expansion path rather than + the normal user-agent delivery. +2. **The list is looked up by address.** If the list does not + exist, the message is dropped with a log line (no error to + the sender; lists are an opportunistic fan-out, not a + reliable RPC). +3. **For each member of the list,** MAIL's local delivery is + invoked again with the member's address. Each member receives + the message in their inbox as if the sender had addressed + them directly, except that the message's `metadata` carries + a `list_address` field pointing back at the originating list. +4. **Nested list members are rejected.** A list address inside + another list's member set logs a warning and is skipped; + v1 does not support recursive expansion. +5. **Webhook firing happens per-member, not per-list.** Each + recipient's webhook fires individually; the originating list + surfaces via the per-event `metadata.list_address`. + +The `metadata.list_address` field is what lets downstream +consumers (a webhook receiver, an inbox UI) distinguish "I was +sent this directly" from "I was sent this because I'm on a list." +See the [Webhook Delivery](webhook-delivery.md) explainer for +how the field appears on the wire. + +## Admin and user-agent permission split + +Lists have a clean two-layer permission model: + +### Admin-only operations + +- **Create a list** (`POST /admin/lists`). The list address must + be unique on the server. +- **Patch policy** (`PATCH /admin/lists/{addr}`). Only `policy` + is mutable; the address is fixed for the life of the list. +- **Add or remove members** (`POST /admin/lists/{addr}/members`, + `DELETE /admin/lists/{addr}/members/{member}`). Forcible + membership change without the member's consent. +- **Delete a list** (`DELETE /admin/lists/{addr}`). +- **Read everything** (`GET /admin/lists`, `GET + /admin/lists/{addr}`). Admin reads are not gated by + `visibility`. + +### User-agent operations + +- **Read public lists** (`GET /lists`, `GET /lists/{addr}`). Lists + with `visibility: public` appear; private lists do not. +- **Self-subscribe** (`POST /lists/{addr}/subscribe`). Honored + when `join_policy: open`; returns `501` for deferred variants. + Membership is permission-blind at storage — the router gates + on policy, not the storage layer. +- **Self-unsubscribe** (`POST /lists/{addr}/unsubscribe`). + Symmetric: members can always leave. +- **Send to a list** (compose + send with a list address as + recipient). Honored when `send_policy: open`; deferred variants + similarly return `501`. + +### Why this split + +The split reflects the broader MAIL trust model (see [Security +Model](security-model.md)). Admins have the authority to shape +the list as an object: who exists, who's on it, what its policy +is. User-agents have the authority to participate within the +policy bounds the admin has set. + +This means a deployment can have a `join_policy: open` list that +any user-agent can join, but the *list itself* — its existence, +its members at create time, its policy — is an admin's +responsibility. Conversely, a `join_policy: admin-only` list (when +v1.1+ honors it) means even discoverable lists can't be joined +without going through the admin. + +## Addressability examples + +A few address-shape examples to make the model concrete: + +| Address | Means | +| --- | --- | +| `bob@chorus@example.com` | The user-agent `bob` in the `chorus` swarm. | +| `list:announcements@chorus@example.com` | The list `announcements` in the `chorus` swarm. | +| `admin:ops@example.com` | The admin `ops` on the host (not swarm-scoped). | +| `user:alice@example.com` | The end-user `alice` on the host. | + +Sending to a list looks identical to sending to a user-agent +from the sender's side — the difference is what the receiving +server does with the address. + +## Things lists are not + +A few clarifying negatives: + +- **Lists are not a queue or buffer.** Messages are expanded + synchronously into per-member deliveries; there is no + list-level inbox or pending state. +- **Lists do not retain a "sent through the list" history.** The + history lives in the senders' outboxes and the recipients' + inboxes. The list as an object has no message log. +- **Lists are not a privacy boundary.** Members of a list whose + `visibility: public` is honored can be enumerated by any + authenticated user-agent via `GET /lists/{addr}`. Treat the + member list as discoverable in v1. + +## See also + +- [Manage Mailing Lists](../howtos/manage-mailing-lists.md) — the + task-oriented CLI walkthrough for operating lists. +- [Addressing Model](addressing-model.md) — the broader address + taxonomy lists sit within. +- [Delivery Model](delivery-model.md) — the local-vs-remote + delivery pipeline that handles list expansion. +- [Webhook Delivery](webhook-delivery.md) — how list deliveries + carry the `metadata.list_address` field for downstream + consumers. +- [Data Models](../references/data-models.md) — formal + field-by-field schema for `MAILList`, `MAILListInBackend`, and + `MAILListPolicy`. +- [HTTP API](../references/http-api.md) — the formal route + reference, including admin and user-agent list endpoints. diff --git a/docs/howtos/manage-mailing-lists.md b/docs/howtos/manage-mailing-lists.md index 4710e54..260be13 100644 --- a/docs/howtos/manage-mailing-lists.md +++ b/docs/howtos/manage-mailing-lists.md @@ -4,11 +4,24 @@ Status: draft ## Goal -How to create mailing lists, inspect them, and manage subscriptions or members. +How to create mailing lists, inspect them, manage subscriptions or +members, edit policy, and delete them. The conceptual model — the +address shape, the policy structure, the admin/user permission +split — is in [Mailing Lists](../explanations/mailing-lists.md); +this how-to assumes you've at least skimmed it. ## Starting Point -The reader has credentials with permissions appropriate for the list action. +The reader has credentials with permissions appropriate for the list +action. Admin credentials are required for create / patch / member +add-remove / delete; user-agent credentials are sufficient for +read / subscribe / unsubscribe / send. + +The list address shape is `list:@@` — see +[Addressing Model](../explanations/addressing-model.md). Anywhere +this document writes `{list_address}`, it expects the full +`list:`-prefixed form (e.g., +`list:announcements@chorus@example.com`). ## Steps @@ -99,9 +112,51 @@ uv run mail-admin list-member-delete {list_address} {member_address} If successful, details on the mailing list will be printed to the console. -### 6. Send a message to a list address +### 6. Update list policy as an admin + +Admins can update the policy on an existing list with the +`mail-admin` command `list-patch`. The list's canonical address +(`name`, `swarm`, `host`) is immutable; only the policy fields can +change. + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-patch {list_address} \ + --visibility public \ + --join-policy open \ + --send-policy open +``` + +For v1, only `public` / `open` / `open` are honored; the other +variants are reserved in the wire format and rejected at the +endpoint layer with `501`. See [Mailing +Lists](../explanations/mailing-lists.md#the-policy-shape) for +context on the deferred variants. + +### 7. Delete a list as an admin -If they are authorized to do so, user-agents can send a message to a list address by specifying the list address in the `mail` command `send`: +To remove an existing list entirely, use `list-delete`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-delete {list_address} +``` + +The list is removed from the server and the canonical address +becomes available for re-creation. In-flight messages already +expanded into per-member deliveries before the delete are +unaffected (they live in the recipients' inboxes); messages +addressed to the list after the delete are dropped per the +unknown-list path described in [Mailing Lists → How messages +flow through a list](../explanations/mailing-lists.md#how-messages-flow-through-a-list). + +### 8. Send a message to a list address + +If they are authorized to do so, user-agents can send a message to +a list address by specifying the list address in the `mail` command +`send`: ```bash MAIL_SERVER={server_url} @@ -109,6 +164,27 @@ MAIL_TOKEN={ua_jwt} uv run mail send {draft_id} {list_address} ``` +The receiving server expands the list and delivers one copy to +each member's inbox. Each member's webhook (if any) fires with a +`metadata.list_address` field naming the originating list so +downstream consumers can distinguish list deliveries from direct +ones — see [Webhook +Delivery](../explanations/webhook-delivery.md#payload-shape) for +the field placement on the wire. + +## See also + +- [Mailing Lists](../explanations/mailing-lists.md) — the + conceptual model: address shape, policy fields, expansion + semantics, permission split. +- [Webhook Delivery](../explanations/webhook-delivery.md) — how + list deliveries surface to webhook receivers via + `metadata.list_address`. +- [Addressing Model](../explanations/addressing-model.md) — the + full address taxonomy lists sit within. +- [HTTP API](../references/http-api.md) — the formal route + reference for admin and user-agent list endpoints. + ## Source Material - `src/mail/client/src/mail_client/commands/lists.py` From 1e400daffdf31814381fc97913e88ed481846ca2 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 17 Jun 2026 18:22:36 -0400 Subject: [PATCH 08/28] feat: support message replies and tags MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add MAIL 2.0 message-reply and tag support across the protocol, server, and client, plus a migration path for existing deployments. Protocol: - MAILMessage gains required `mail_version` ("2.0") and `tags`, plus an optional `reply_to` referencing the replied-to message's id. - Thread `reply_to`/`tags` through DraftPostRequest, MAILDraft, and `tags` through DraftSendPostRequest (all default-safe so pre-2.0 drafts stay loadable). - Surface `reply_to` (msg_-prefixed) and `tags` in MAILMessageInWebhook. Server: - post_draft stores reply_to/tags on the draft; send_draft stamps mail_version, copies reply_to, and merges draft + send-time tags as an order-preserving union. - Webhook delivery payloads now carry reply_to/tags. Client: - New `reply` command (alias `r`): replies to the original sender, defaults the subject to `Re: `, sets reply_to. - `--tags` on compose, send, and reply; reply_to/tags shown in message output and listed in `mail --help`. Migration: - scripts/migrate_messages_v2.py backfills mail_version/tags on persisted message records (dry-run, backup, deployment overrides; idempotent). Tests/docs: - Update all MAILMessage construction sites; add coverage for tag validators, the reply command, draft/tag-merge behavior, webhook payload fields, and the migration script. - Document the new fields in SPEC.md (§7.8-7.10), regenerate spec/openapi.yaml, and update the client CLI reference. Co-Authored-By: Claude Opus 4.8 (1M context) --- scripts/migrate_messages_v2.py | 223 ++++++++++++++++++ spec/SPEC.md | 16 ++ spec/openapi.yaml | 36 ++- src/mail/client/docs/reference/cli.md | 22 +- src/mail/client/src/mail_client/cli.py | 40 ++++ .../src/mail_client/commands/__init__.py | 2 + .../src/mail_client/commands/compose.py | 5 + .../src/mail_client/commands/inbox_open.py | 4 + .../src/mail_client/commands/outbox_open.py | 4 + .../client/src/mail_client/commands/reply.py | 133 +++++++++++ .../client/src/mail_client/commands/send.py | 5 + .../src/mail_protocol/core/constants.py | 3 + .../protocol/src/mail_protocol/core/drafts.py | 7 + .../src/mail_protocol/core/messages.py | 6 +- .../src/mail_protocol/core/validators.py | 37 ++- .../src/mail_protocol/core/webhooks.py | 3 + .../src/mail_protocol/network/requests.py | 13 + .../server/src/mail_server/backends/base.py | 5 + .../src/mail_server/backends/memory/api.py | 11 + tests/contract/test_spec_delivery.py | 2 + tests/contract/test_spec_messages.py | 60 +++++ tests/integration/test_mailboxes.py | 4 + tests/integration/webhooks/test_delivery.py | 40 +++- tests/unit/test_cli_help.py | 1 + tests/unit/test_client_commands.py | 189 ++++++++++++++- tests/unit/test_draft_reply_tags.py | 96 ++++++++ tests/unit/test_mail_lists_send.py | 15 +- tests/unit/test_mail_trash_store.py | 2 + tests/unit/test_memory_fs_roundtrip.py | 2 + tests/unit/test_migrate_messages_v2.py | 102 ++++++++ tests/unit/test_protocol_models.py | 2 + tests/unit/test_protocol_validators.py | 31 +++ 32 files changed, 1103 insertions(+), 18 deletions(-) create mode 100644 scripts/migrate_messages_v2.py create mode 100644 src/mail/client/src/mail_client/commands/reply.py create mode 100644 tests/unit/test_draft_reply_tags.py create mode 100644 tests/unit/test_migrate_messages_v2.py diff --git a/scripts/migrate_messages_v2.py b/scripts/migrate_messages_v2.py new file mode 100644 index 0000000..1ae3778 --- /dev/null +++ b/scripts/migrate_messages_v2.py @@ -0,0 +1,223 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Migrate persisted ``MAILMessage`` records to the MAIL 2.0 schema. + +MAIL 2.0 adds two required fields to ``MAILMessage``: + +* ``mail_version`` (``"2.0"``) +* ``tags`` (a list of slug strings; empty by default) + +Messages persisted before these fields existed will fail +``MAILMessage.model_validate_json()`` on server startup and be silently +dropped by the memory backend's loader. This script walks a deployment's +``messages/`` directory and backfills the two new fields on any record that +is missing them, so an upgrade does not lose existing messages. + +The optional ``reply_to`` field has a default of ``None`` and therefore needs +no migration. + +Usage:: + + # preview changes for the default deployment + uv run python scripts/migrate_messages_v2.py --dry-run + + # migrate the default deployment in place (a backup is taken first) + uv run python scripts/migrate_messages_v2.py + + # migrate a named deployment, or an explicit messages directory + uv run python scripts/migrate_messages_v2.py --deployment my-deployment + uv run python scripts/migrate_messages_v2.py --messages-dir /path/to/messages +""" + +from __future__ import annotations + +import argparse +import json +import os +import shutil +import sys +import tempfile +from dataclasses import dataclass, field +from pathlib import Path + +MAIL_VERSION = "2.0" + +DEFAULT_DEPLOYMENTS_ROOT = Path.home().joinpath(".mail-swarms", "deployments") + + +@dataclass +class MigrationResult: + """Summary of a migration run.""" + + scanned: int = 0 + migrated: int = 0 + already_current: int = 0 + skipped: list[str] = field(default_factory=list) + + +def _atomic_write_text(path: Path, content: str) -> None: + """ + Atomically replace ``path`` with ``content`` (temp file + ``os.replace``). + Mirrors the memory backend's persistence write so the on-disk format and + durability guarantees match. + """ + + path.parent.mkdir(parents=True, exist_ok=True) + fd, tmp_name = tempfile.mkstemp( + prefix=f".{path.name}.", suffix=".tmp", dir=path.parent, text=True + ) + tmp_path = Path(tmp_name) + try: + with os.fdopen(fd, "w", encoding="utf-8") as tmp_file: + tmp_file.write(content) + tmp_file.flush() + os.fsync(tmp_file.fileno()) + os.replace(tmp_path, path) + except Exception: + tmp_path.unlink(missing_ok=True) + raise + + +def _needs_migration(record: dict) -> bool: + return "mail_version" not in record or "tags" not in record + + +def _upgrade_record(record: dict) -> dict: + """ + Return ``record`` with the MAIL 2.0 fields backfilled. Existing values are + left untouched; only missing fields are added. + """ + + if "mail_version" not in record: + record["mail_version"] = MAIL_VERSION + if "tags" not in record: + record["tags"] = [] + return record + + +def migrate_messages(messages_dir: Path, *, dry_run: bool = False) -> MigrationResult: + """ + Backfill MAIL 2.0 fields on every message file in ``messages_dir``. + + Files are read as raw JSON (the model is intentionally bypassed so that + pre-2.0 records can be loaded at all). Records missing ``mail_version`` or + ``tags`` are rewritten in place; records that already have both are left + untouched. Files that are not valid JSON objects are reported as skipped. + """ + + result = MigrationResult() + if not messages_dir.is_dir(): + raise FileNotFoundError(f"messages directory not found: {messages_dir}") + + for entry in sorted(messages_dir.iterdir()): + if not entry.is_file(): + continue + result.scanned += 1 + + try: + record = json.loads(entry.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError) as e: + result.skipped.append(f"{entry.name}: unreadable ({e})") + continue + + if not isinstance(record, dict): + result.skipped.append(f"{entry.name}: not a JSON object") + continue + + if not _needs_migration(record): + result.already_current += 1 + continue + + result.migrated += 1 + if dry_run: + continue + + upgraded = _upgrade_record(record) + _atomic_write_text(entry, json.dumps(upgraded)) + + return result + + +def _resolve_messages_dir(args: argparse.Namespace) -> Path: + if args.messages_dir is not None: + return Path(args.messages_dir) + return Path(args.root).joinpath(args.deployment, "messages") + + +def _backup_dir(messages_dir: Path) -> Path: + backup = messages_dir.with_name(f"{messages_dir.name}.backup") + if backup.exists(): + raise FileExistsError( + f"backup already exists: {backup} (remove or rename it first)" + ) + shutil.copytree(messages_dir, backup) + return backup + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Migrate persisted MAILMessage records to the MAIL 2.0 schema." + ) + parser.add_argument( + "--deployment", + default="default", + help="deployment name under the deployments root (default: %(default)s)", + ) + parser.add_argument( + "--root", + default=str(DEFAULT_DEPLOYMENTS_ROOT), + help="deployments root directory (default: %(default)s)", + ) + parser.add_argument( + "--messages-dir", + default=None, + help="explicit path to a messages/ directory (overrides --deployment/--root)", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="report what would change without modifying any files", + ) + parser.add_argument( + "--no-backup", + action="store_true", + help="skip copying the messages directory to .backup before migrating", + ) + return parser + + +def main() -> None: + args = build_parser().parse_args() + messages_dir = _resolve_messages_dir(args) + + print(f"=== MAIL 2.0 message migration: {messages_dir} ===") + if not messages_dir.is_dir(): + print(f"❌ messages directory not found: {messages_dir}") + sys.exit(1) + + if args.dry_run: + result = migrate_messages(messages_dir, dry_run=True) + print("🔍 DRY RUN — no files were modified") + else: + # Probe first so we only take a backup when there is work to do. + preview = migrate_messages(messages_dir, dry_run=True) + if preview.migrated and not args.no_backup: + backup = _backup_dir(messages_dir) + print(f"✅ backed up {preview.scanned} files to {backup}") + result = migrate_messages(messages_dir, dry_run=False) + + print(f"scanned: {result.scanned}") + print(f"migrated: {result.migrated}") + print(f"already current: {result.already_current}") + if result.skipped: + print(f"skipped: {len(result.skipped)}") + for note in result.skipped: + print(f" - {note}") + + print("🎉 migration complete") + + +if __name__ == "__main__": + main() diff --git a/spec/SPEC.md b/spec/SPEC.md index 6d98284..6b1d319 100644 --- a/spec/SPEC.md +++ b/spec/SPEC.md @@ -44,6 +44,9 @@ * [7.5 Message Bodies](#75-message-bodies) * [7.6 Timestamps](#76-timestamps) * [7.7 Message Metadata](#77-message-metadata) + * [7.8 Protocol Version](#78-protocol-version) + * [7.9 Replies](#79-replies) + * [7.10 Message Tags](#710-message-tags) * [8 Delivery](#8-delivery) * [8.1 Pre-Send Errors](#81-pre-send-errors) * [8.2 Post-Send Errors](#82-post-send-errors) @@ -217,6 +220,18 @@ Every MAIL message MUST contain the timestamp string of the time it was sent by Every MAIL message MUST contain a field for implementer-defined message metadata, defined by `metadata`. This value MAY be an empty object (`{}`). Implementer-defined message data MUST be stored in the `metadata` field, rather than in the top level of the `MAILMessage` object itself. +### 7.8. Protocol Version + +Every MAIL message MUST declare the version of the MAIL protocol it conforms to, keyed by `mail_version`. This value MUST be a protocol version string per [Section 10](#10-versioning). For this revision of the protocol, the value MUST be `"2.0"`. + +### 7.9. Replies + +A MAIL message MAY indicate that it is a reply to an earlier message, keyed by `reply_to`. When present, this value MUST be the `message_id` (a [UUID][rfc-9562] per [Section 7.1](#71-message-ids)) of the message being replied to. When the field is absent or `null`, the message is not a reply. Implementers SHOULD reject a `reply_to` value that is not a well-formed message ID; they are NOT required to verify that the referenced message exists. + +### 7.10. Message Tags + +Every MAIL message MUST contain a field for sender-defined tags, keyed by `tags`. This value is an array of strings and MAY be empty (`[]`). Each tag MUST be a slug string: lowercase alphanumeric characters separated by single hyphens (matching `^[a-z0-9]+(?:-[a-z0-9]+)*$`). The reference implementation enforces a per-tag length of 1–32 characters. Tags are advisory metadata used to categorize messages; their interpretation is implementer-defined. + ## 8. Delivery When a user-agent creates and sends a MAIL message, the new message is stored on the MAIL server, but is not yet delivered to the specified recipient(s). @@ -226,6 +241,7 @@ Said message MUST be delivered to its intended recipient(s) by an authorized MAI If an authorized user-agent attempts to create a message with a malformed subject (per [Section 7.4](#74-message-subjects)), the desired message MUST NOT be created and the user-agent MUST be notified. If an authorized user-agent attempts to create a message with a malformed body (per [Section 7.5](#75-message-bodies)), the desired message MUST NOT be created and the user-agent MUST be notified. +If an authorized user-agent attempts to create a message with one or more malformed tags (per [Section 7.10](#710-message-tags)), the desired message MUST NOT be created and the user-agent MUST be notified. If an authorized user-agent's message contains one or more malformed MAIL addresses (per [Section 6](#6-addresses)), the message MUST NOT be delivered and the sending user-agent MUST be notified. ### 8.2. Post-Send Errors diff --git a/spec/openapi.yaml b/spec/openapi.yaml index 3817096..207becf 100644 --- a/spec/openapi.yaml +++ b/spec/openapi.yaml @@ -1573,6 +1573,17 @@ components: format: date-time - type: 'null' title: Updated At + reply_to: + anyOf: + - type: string + - type: 'null' + title: Reply To + tags: + items: + type: string + type: array + title: Tags + default: [] type: object required: - draft_id @@ -1582,7 +1593,14 @@ components: title: MAILDraft description: 'A draft of an individual MAIL message. - Does not yet account for intended recipients.' + Does not yet account for intended recipients. + + + `reply_to` and `tags` carry forward onto the `MAILMessage` produced when + + the draft is sent. Both default to "no value" so drafts persisted before + + these fields existed remain loadable.' MAILDraftsEntry: properties: draft: @@ -1777,9 +1795,18 @@ components: protocol layer but are rejected at the endpoint layer.' MAILMessage: properties: + mail_version: + type: string + const: '2.0' + title: Mail Version message_id: type: string title: Message Id + reply_to: + anyOf: + - type: string + - type: 'null' + title: Reply To sender: type: string title: Sender @@ -1794,6 +1821,11 @@ components: body: type: string title: Body + tags: + items: + type: string + type: array + title: Tags sent_at: type: string format: date-time @@ -1804,11 +1836,13 @@ components: title: Metadata type: object required: + - mail_version - message_id - sender - recipients - subject - body + - tags - sent_at - metadata title: MAILMessage diff --git a/src/mail/client/docs/reference/cli.md b/src/mail/client/docs/reference/cli.md index 5aca9f0..b582234 100644 --- a/src/mail/client/docs/reference/cli.md +++ b/src/mail/client/docs/reference/cli.md @@ -12,8 +12,9 @@ mail [option]... [argument]... ### Core MAIL Operations -- `compose`: Draft a new MAIL message. -- `send`: Send an existing draft by ID to the specified recipient(s). +- `compose`: Draft a new MAIL message. Accepts `--tags TAG...` to attach slug tags. +- `send`: Send an existing draft by ID to the specified recipient(s). Accepts `--tags TAG...`, which are merged with the draft's tags. +- `reply`: Reply to an existing inbox message by ID. Addresses the reply to the original sender and defaults the subject to `Re: `. Accepts `--subject SUBJECT` and `--tags TAG...`. - `inbox`: Open your MAIL inbox. - `inbox-open`: Open a specific message by ID in your MAIL inbox. - `outbox`: Open your MAIL outbox. @@ -46,3 +47,20 @@ mail [option]... [argument]... - `-o`/`--output`: Choose the style of console output for this command. - **Default**: `text` - **Choices**: `text`, `json` + +## Examples + +Draft and send a message with tags: + +```bash +mail compose "Status update" "All systems nominal." --tags weekly status +mail send sage@chorus@localhost --tags urgent +``` + +Reply to a message in your inbox (replies to the original sender, subject +defaults to `Re: `): + +```bash +mail reply "Thanks, acknowledged." +mail reply "See attached." --subject "Follow-up" --tags project-x +``` diff --git a/src/mail/client/src/mail_client/cli.py b/src/mail/client/src/mail_client/cli.py index e940c5d..8dd69f1 100644 --- a/src/mail/client/src/mail_client/cli.py +++ b/src/mail/client/src/mail_client/cli.py @@ -22,6 +22,7 @@ cmd_outbox, cmd_outbox_open, cmd_ping, + cmd_reply, cmd_send, cmd_swarm_get, cmd_swarm_list, @@ -73,6 +74,7 @@ [ ("compose (c)", "Draft a new MAIL message."), ("send (s)", "Send a drafted message."), + ("reply (r)", "Reply to an inbox message by ID."), ("inbox (i)", "List your inbox messages."), ("inbox-open (open, o)", "Open an inbox message by ID."), ("outbox (O)", "List your sent messages."), @@ -106,9 +108,26 @@ 'mail compose "Status update" "The migration is complete."', "mail send user@example", "mail inbox-open ", + 'mail reply "Thanks, acknowledged."', ] +def _add_tags_arg(parser: argparse.ArgumentParser) -> None: + """ + Register the shared ``--tags`` flag for message-creating commands + (compose, send, reply). Tags are slug strings used to categorize a + message; the default is an empty list (no tags). + """ + + parser.add_argument( + "--tags", + nargs="*", + default=[], + metavar="TAG", + help="slug string tag(s) to attach to the message", + ) + + def _add_box_filter_args(box_parser: argparse.ArgumentParser) -> None: """ Register the shared query-param flags for the "GET box" commands @@ -203,6 +222,7 @@ def build_parser() -> argparse.ArgumentParser: ) compose_p.add_argument("subject", help="the subject line of the message to draft") compose_p.add_argument("body", help="the body of the message to draft") + _add_tags_arg(compose_p) compose_p.set_defaults(func=cmd_compose, cmd="compose") # command `send` @@ -218,8 +238,28 @@ def build_parser() -> argparse.ArgumentParser: send_p.add_argument( "to", nargs="+", help="the address(es) to deliver this message to" ) + _add_tags_arg(send_p) send_p.set_defaults(func=cmd_send, cmd="send") + # command `reply` + reply_d = "reply to an existing inbox message" + reply_p = subparsers.add_parser( + "reply", + aliases=["r"], + prog="mail reply", + help=reply_d, + description=reply_d, + ) + reply_p.add_argument("message_id", help="the ID of the inbox message to reply to") + reply_p.add_argument("body", help="the body of the reply") + reply_p.add_argument( + "--subject", + default=None, + help="the subject of the reply (default: 'Re: ')", + ) + _add_tags_arg(reply_p) + reply_p.set_defaults(func=cmd_reply, cmd="reply") + # command `inbox` inbox_d = "open your MAIL inbox" inbox_p = subparsers.add_parser( diff --git a/src/mail/client/src/mail_client/commands/__init__.py b/src/mail/client/src/mail_client/commands/__init__.py index ce04b88..e406d9f 100644 --- a/src/mail/client/src/mail_client/commands/__init__.py +++ b/src/mail/client/src/mail_client/commands/__init__.py @@ -26,6 +26,7 @@ from .outbox import cmd_outbox from .outbox_open import cmd_outbox_open from .ping import cmd_ping +from .reply import cmd_reply from .send import cmd_send from .swarm_delete import cmd_swarm_delete from .swarm_get import cmd_swarm_get @@ -73,6 +74,7 @@ "cmd_outbox", "cmd_outbox_open", "cmd_ping", + "cmd_reply", "cmd_send", "cmd_swarm_delete", "cmd_swarm_get", diff --git a/src/mail/client/src/mail_client/commands/compose.py b/src/mail/client/src/mail_client/commands/compose.py index cb77d96..6453e0a 100644 --- a/src/mail/client/src/mail_client/commands/compose.py +++ b/src/mail/client/src/mail_client/commands/compose.py @@ -27,6 +27,7 @@ def cmd_compose(args: Namespace) -> None: payload = DraftPostRequest( subject=args.subject, body=args.body, + tags=args.tags, ) response = httpx.post( url=f"{MAIL_SERVER}/drafts", @@ -70,6 +71,10 @@ def _print_text(response_obj: DraftPostResponse) -> None: print(f"Draft ID: {draft.draft_id}") print(f"Created At: {draft.created_at}") print(f"Subject: {draft.subject}") + if draft.reply_to is not None: + print(f"In Reply To: {draft.reply_to}") + if draft.tags: + print(f"Tags: {', '.join(draft.tags)}") print(f"Body:\n{draft.body}\n") print("=== Entry Data ===") print(f"Sent At: {entry.sent_at}") diff --git a/src/mail/client/src/mail_client/commands/inbox_open.py b/src/mail/client/src/mail_client/commands/inbox_open.py index 8fd7992..ae57869 100644 --- a/src/mail/client/src/mail_client/commands/inbox_open.py +++ b/src/mail/client/src/mail_client/commands/inbox_open.py @@ -67,6 +67,10 @@ def _print_text(response_obj: InboxMessageGetResponse) -> None: for recipient in message.recipients: print(f"- {recipient}") print(f"Subject: {message.subject}") + if message.reply_to is not None: + print(f"In Reply To: {message.reply_to}") + if message.tags: + print(f"Tags: {', '.join(message.tags)}") print(f"Body:\n{message.body}\n") print("=== Inbox Entry Data ===") print(f"Received At: {entry.received_at}") diff --git a/src/mail/client/src/mail_client/commands/outbox_open.py b/src/mail/client/src/mail_client/commands/outbox_open.py index 853d01c..2e52432 100644 --- a/src/mail/client/src/mail_client/commands/outbox_open.py +++ b/src/mail/client/src/mail_client/commands/outbox_open.py @@ -67,6 +67,10 @@ def _print_text(response_obj: OutboxMessageGetResponse) -> None: for recipient in message.recipients: print(f"- {recipient}") print(f"Subject: {message.subject}") + if message.reply_to is not None: + print(f"In Reply To: {message.reply_to}") + if message.tags: + print(f"Tags: {', '.join(message.tags)}") print(f"Body:\n{message.body}\n") print("=== Outbox Entry Data ===") print(f"Delivered At: {entry.delivered_at}") diff --git a/src/mail/client/src/mail_client/commands/reply.py b/src/mail/client/src/mail_client/commands/reply.py new file mode 100644 index 0000000..c92545e --- /dev/null +++ b/src/mail/client/src/mail_client/commands/reply.py @@ -0,0 +1,133 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +import os +from argparse import Namespace + +import httpx +from mail_protocol.network.requests import DraftPostRequest, DraftSendPostRequest +from mail_protocol.network.responses import ( + DraftPostResponse, + DraftSendPostResponse, + InboxMessageGetResponse, +) +from pydantic import ValidationError + +USER_AGENT = "Multi-Agent-Interface-Layer-CLI-Client/2.0.0 (github.com/charonlabs/mail)" + + +def _reply_subject(original_subject: str) -> str: + """ + Derive the default subject for a reply. An existing `Re:` prefix + (case-insensitive) is preserved rather than duplicated. + """ + + if original_subject.lower().startswith("re:"): + return original_subject + return f"Re: {original_subject}" + + +def cmd_reply(args: Namespace) -> None: + """ + Reply to an existing inbox message for the current MAIL user. + + Fetches the original message to derive the reply recipient (its sender) + and a default `Re:` subject, creates a draft that references the original + via `reply_to`, then sends it. + """ + + # 1. check that the required env vars are provided + MAIL_SERVER = os.getenv("MAIL_SERVER") + if MAIL_SERVER is None: + raise ValueError("env var MAIL_SERVER is required") + MAIL_TOKEN = os.getenv("MAIL_TOKEN") + if MAIL_TOKEN is None: + raise ValueError("env var MAIL_TOKEN is required") + + headers = { + "Authorization": f"Bearer {MAIL_TOKEN}", + "User-Agent": USER_AGENT, + "Content-Type": "application/json", + } + + # 2. fetch the original message from the user's inbox + original_response = httpx.get( + url=f"{MAIL_SERVER}/inbox/{args.message_id}", + headers=headers, + ) + if original_response.status_code != 200: + raise RuntimeError( + f"get inbox entry request to {MAIL_SERVER} failed with status code " + f"{original_response.status_code}" + ) + try: + original_obj = InboxMessageGetResponse.model_validate(original_response.json()) + except ValidationError as e: + raise RuntimeError(f"response validation failed: {e}") + + original = original_obj.entry.message + subject = args.subject if args.subject else _reply_subject(original.subject) + + # 3. create the reply draft, referencing the original via `reply_to` + draft_payload = DraftPostRequest( + subject=subject, + body=args.body, + reply_to=original.message_id, + tags=args.tags, + ) + draft_response = httpx.post( + url=f"{MAIL_SERVER}/drafts", + headers=headers, + json=draft_payload.model_dump(), + ) + if draft_response.status_code != 200: + raise RuntimeError( + f"post draft request to {MAIL_SERVER} failed with status code " + f"{draft_response.status_code}" + ) + try: + draft_obj = DraftPostResponse.model_validate(draft_response.json()) + except ValidationError as e: + raise RuntimeError(f"response validation failed: {e}") + + draft_id = draft_obj.entry.draft.draft_id + + # 4. send the draft back to the original sender + send_payload = DraftSendPostRequest(recipients=[original.sender]) + send_response = httpx.post( + url=f"{MAIL_SERVER}/drafts/{draft_id}/send", + headers=headers, + json=send_payload.model_dump(), + ) + if send_response.status_code != 200: + raise RuntimeError( + f"send request to {MAIL_SERVER} failed with status code " + f"{send_response.status_code}" + ) + try: + send_obj = DraftSendPostResponse.model_validate(send_response.json()) + except ValidationError as e: + raise RuntimeError(f"response validation failed: {e}") + + # 5. print the sent reply + match args.output: + case "json": + print(send_obj.model_dump_json()) + case "text": + _print_text(send_obj) + + +def _print_text(response_obj: DraftSendPostResponse) -> None: + message = response_obj.message + print("=== Reply Sent ===") + print(f"Message ID: {message.message_id}") + print(f"In Reply To: {message.reply_to}") + print(f"Sent At: {message.sent_at}") + print(f"Sender: {message.sender}") + print("Recipient(s):") + for recipient in message.recipients: + print(f"- {recipient}") + print(f"Subject: {message.subject}") + if message.tags: + print(f"Tags: {', '.join(message.tags)}") + print(f"Body:\n{message.body}\n") diff --git a/src/mail/client/src/mail_client/commands/send.py b/src/mail/client/src/mail_client/commands/send.py index 853bf81..b0fa9e7 100644 --- a/src/mail/client/src/mail_client/commands/send.py +++ b/src/mail/client/src/mail_client/commands/send.py @@ -26,6 +26,7 @@ def cmd_send(args: Namespace) -> None: # 2. hit the server endpoint `POST /drafts/{draft_id}/send` payload = DraftSendPostRequest( recipients=args.to, + tags=args.tags, ) response = httpx.post( url=f"{MAIL_SERVER}/drafts/{args.draft_id}/send", @@ -72,4 +73,8 @@ def _print_text(response_obj: DraftSendPostResponse) -> None: for recipient in message.recipients: print(f"- {recipient}") print(f"Subject: {message.subject}") + if message.reply_to is not None: + print(f"In Reply To: {message.reply_to}") + if message.tags: + print(f"Tags: {', '.join(message.tags)}") print(f"Body:\n{message.body}\n") diff --git a/src/mail/protocol/src/mail_protocol/core/constants.py b/src/mail/protocol/src/mail_protocol/core/constants.py index 00e9fdf..dd3d4c9 100644 --- a/src/mail/protocol/src/mail_protocol/core/constants.py +++ b/src/mail/protocol/src/mail_protocol/core/constants.py @@ -7,6 +7,9 @@ MESSAGE_BODY_LEN_MIN = 1 MESSAGE_BODY_LEN_MAX = 65535 +MESSAGE_TAG_LEN_MIN = 1 +MESSAGE_TAG_LEN_MAX = 32 + AGENT_NAME_LEN_MIN = 1 AGENT_NAME_LEN_MAX = 31 diff --git a/src/mail/protocol/src/mail_protocol/core/drafts.py b/src/mail/protocol/src/mail_protocol/core/drafts.py index eaa3d18..974ec8f 100644 --- a/src/mail/protocol/src/mail_protocol/core/drafts.py +++ b/src/mail/protocol/src/mail_protocol/core/drafts.py @@ -10,6 +10,7 @@ validate_mail_address, validate_message_body, validate_message_subject, + validate_message_tags, validate_uuid, ) @@ -18,6 +19,10 @@ class MAILDraft(BaseModel): """ A draft of an individual MAIL message. Does not yet account for intended recipients. + + `reply_to` and `tags` carry forward onto the `MAILMessage` produced when + the draft is sent. Both default to "no value" so drafts persisted before + these fields existed remain loadable. """ draft_id: Annotated[str, AfterValidator(validate_uuid)] @@ -25,6 +30,8 @@ class MAILDraft(BaseModel): body: Annotated[str, AfterValidator(validate_message_body)] created_at: datetime updated_at: datetime | None = None + reply_to: Annotated[str, AfterValidator(validate_uuid)] | None = None + tags: Annotated[list[str], AfterValidator(validate_message_tags)] = [] class MAILDraftsEntrySummary(BaseModel): diff --git a/src/mail/protocol/src/mail_protocol/core/messages.py b/src/mail/protocol/src/mail_protocol/core/messages.py index e242712..50f465d 100644 --- a/src/mail/protocol/src/mail_protocol/core/messages.py +++ b/src/mail/protocol/src/mail_protocol/core/messages.py @@ -2,7 +2,7 @@ # Copyright (c) 2025-26 Addison Kline from datetime import datetime -from typing import Annotated, Any +from typing import Annotated, Any, Literal from pydantic import AfterValidator, BaseModel @@ -11,6 +11,7 @@ validate_message_body, validate_message_recipients, validate_message_subject, + validate_message_tags, validate_uuid, ) @@ -33,11 +34,14 @@ class MAILMessage(BaseModel): A constructed message to be delivered via MAIL. """ + mail_version: Literal["2.0"] message_id: Annotated[str, AfterValidator(validate_uuid)] + reply_to: Annotated[str, AfterValidator(validate_uuid)] | None = None sender: Annotated[str, AfterValidator(validate_mail_address)] recipients: Annotated[list[str], AfterValidator(validate_message_recipients)] subject: Annotated[str, AfterValidator(validate_message_subject)] body: Annotated[str, AfterValidator(validate_message_body)] + tags: Annotated[list[str], AfterValidator(validate_message_tags)] sent_at: datetime metadata: dict[str, Any] diff --git a/src/mail/protocol/src/mail_protocol/core/validators.py b/src/mail/protocol/src/mail_protocol/core/validators.py index 828cdcb..b88359e 100644 --- a/src/mail/protocol/src/mail_protocol/core/validators.py +++ b/src/mail/protocol/src/mail_protocol/core/validators.py @@ -18,6 +18,8 @@ MESSAGE_BODY_LEN_MIN, MESSAGE_SUBJECT_LEN_MAX, MESSAGE_SUBJECT_LEN_MIN, + MESSAGE_TAG_LEN_MAX, + MESSAGE_TAG_LEN_MIN, SWARM_DESCRIPTION_LEN_MAX, SWARM_DESCRIPTION_LEN_MIN, SWARM_KEYWORD_LEN_MAX, @@ -105,9 +107,7 @@ def validate_mail_address(address: str) -> str: case _ if prefix == LIST_ADDRESS_PREFIX: validate_list_name(identifier) case _: - raise ValueError( - f"invalid MAIL address structure: {address}" - ) + raise ValueError(f"invalid MAIL address structure: {address}") else: # No prefix → agent address. validate_agent_name(first) @@ -160,6 +160,37 @@ def validate_message_recipients(addresses: list[str]) -> list[str]: return validate_mail_addresses(addresses) +def validate_message_tag(tag: str) -> str: + """ + Ensure that the given string is a valid MAIL message tag. + """ + + if len(tag) < MESSAGE_TAG_LEN_MIN: + raise ValueError( + f"message tag must be at least {MESSAGE_TAG_LEN_MIN} characters long" + ) + if len(tag) > MESSAGE_TAG_LEN_MAX: + raise ValueError( + f"message tag must be no longer than {MESSAGE_TAG_LEN_MAX} characters" + ) + if not string_is_slug(tag): + raise ValueError("message tag must be a slug string") + + return tag + + +def validate_message_tags(tags: list[str]) -> list[str]: + """ + Ensure that the given list is a valid list of MAIL message tags. + Can be of length 0. + """ + + for tag in tags: + validate_message_tag(tag) + + return tags + + def validate_local_address(address: str) -> str: """ Ensure that the given string is a valid MAIL local agent address (agent@swarm). diff --git a/src/mail/protocol/src/mail_protocol/core/webhooks.py b/src/mail/protocol/src/mail_protocol/core/webhooks.py index ac4f891..78a1556 100644 --- a/src/mail/protocol/src/mail_protocol/core/webhooks.py +++ b/src/mail/protocol/src/mail_protocol/core/webhooks.py @@ -10,6 +10,7 @@ validate_mail_address, validate_message_body, validate_message_subject, + validate_message_tags, validate_swarm_name, validate_url, validate_webhook_event_types, @@ -37,10 +38,12 @@ class MAILMessageInWebhook(BaseModel): """ message_id: Annotated[str, AfterValidator(validate_webhook_message_id)] + reply_to: Annotated[str, AfterValidator(validate_webhook_message_id)] | None = None sender: Annotated[str, AfterValidator(validate_mail_address)] recipient: Annotated[str, AfterValidator(validate_mail_address)] subject: Annotated[str, AfterValidator(validate_message_subject)] body: Annotated[str, AfterValidator(validate_message_body)] + tags: Annotated[list[str], AfterValidator(validate_message_tags)] = [] sent_at: datetime swarm: Annotated[str, AfterValidator(validate_swarm_name)] metadata: dict[str, Any] diff --git a/src/mail/protocol/src/mail_protocol/network/requests.py b/src/mail/protocol/src/mail_protocol/network/requests.py index ed94767..1777264 100644 --- a/src/mail/protocol/src/mail_protocol/network/requests.py +++ b/src/mail/protocol/src/mail_protocol/network/requests.py @@ -16,11 +16,13 @@ validate_message_body, validate_message_recipients, validate_message_subject, + validate_message_tags, validate_swarm_description, validate_swarm_keywords, validate_swarm_name, validate_url, validate_user_name, + validate_uuid, validate_uuids, validate_webhook_event_types, ) @@ -54,19 +56,30 @@ class DraftPostRequest(BaseModel): """ Corresponds to `POST /drafts/`. Contains relevant information for creating a new MAIL message draft. + + `reply_to` optionally references the `message_id` of the message this + draft is replying to. `tags` is an optional list of sender-defined slug + strings used to categorize the eventual message. """ subject: Annotated[str, AfterValidator(validate_message_subject)] body: Annotated[str, AfterValidator(validate_message_body)] + reply_to: Annotated[str, AfterValidator(validate_uuid)] | None = None + tags: Annotated[list[str], AfterValidator(validate_message_tags)] = [] class DraftSendPostRequest(BaseModel): """ Corresponds to `POST /drafts/{draft_id}/send`. Contains relevant information for sending an existing draft as a MAIL message. + + `tags` is an optional list of sender-defined slug strings; any tags + supplied here are merged (union, order-preserving) with the tags already + stored on the draft. """ recipients: Annotated[list[str], AfterValidator(validate_message_recipients)] + tags: Annotated[list[str], AfterValidator(validate_message_tags)] = [] # diff --git a/src/mail/server/src/mail_server/backends/base.py b/src/mail/server/src/mail_server/backends/base.py index 42f2e36..a26d072 100644 --- a/src/mail/server/src/mail_server/backends/base.py +++ b/src/mail/server/src/mail_server/backends/base.py @@ -686,10 +686,15 @@ async def _webhook_delivered_post( # MAILMessage stores a bare UUID; the webhook payload's # message_id is the prefixed form per validate_webhook_message_id. message_id=f"msg_{message.message_id}", + # reply_to, when set, is carried in the same prefixed form. + reply_to=( + f"msg_{message.reply_to}" if message.reply_to is not None else None + ), sender=message.sender, recipient=recipient, subject=message.subject, body=message.body, + tags=message.tags, sent_at=message.sent_at, swarm=recipient.split("@")[1], metadata=metadata, diff --git a/src/mail/server/src/mail_server/backends/memory/api.py b/src/mail/server/src/mail_server/backends/memory/api.py index f6ba79e..92a548e 100644 --- a/src/mail/server/src/mail_server/backends/memory/api.py +++ b/src/mail/server/src/mail_server/backends/memory/api.py @@ -631,6 +631,8 @@ async def post_draft( body=payload.body, created_at=datetime.now(UTC), updated_at=None, + reply_to=payload.reply_to, + tags=payload.tags, ) draft_entry = MAILDraftsEntry(draft=draft, sent_at=None) @@ -695,12 +697,21 @@ async def send_draft( draft = draft_entry.draft message_id = str(uuid.uuid4()) # make this different from draft_id + # Tags on the draft and tags supplied at send time are merged as an + # order-preserving union: draft tags first, then any new send tags. + tags = list(draft.tags) + for tag in payload.tags: + if tag not in tags: + tags.append(tag) message = MAILMessage( + mail_version="2.0", message_id=message_id, + reply_to=draft.reply_to, sender=ua_address, recipients=payload.recipients, subject=draft.subject, body=draft.body, + tags=tags, sent_at=datetime.now(UTC), metadata={}, ) diff --git a/tests/contract/test_spec_delivery.py b/tests/contract/test_spec_delivery.py index bc82980..83cf035 100644 --- a/tests/contract/test_spec_delivery.py +++ b/tests/contract/test_spec_delivery.py @@ -68,11 +68,13 @@ async def test_undeliverable_message_is_preserved_and_logged( message_id = "66666666-6666-4666-8666-666666666666" message = MAILMessage( + mail_version="2.0", message_id=message_id, sender="user:alice@localhost", recipients=["ghost@nowhere@localhost"], subject="Undeliverable", body="No such recipient.", + tags=[], sent_at=datetime(2026, 6, 12, 9, 0, tzinfo=UTC), metadata={}, ) diff --git a/tests/contract/test_spec_messages.py b/tests/contract/test_spec_messages.py index 2d4581a..da2d5f1 100644 --- a/tests/contract/test_spec_messages.py +++ b/tests/contract/test_spec_messages.py @@ -20,11 +20,13 @@ def make_message(**overrides: Any) -> MAILMessage: fields: dict[str, Any] = { + "mail_version": "2.0", "message_id": "55555555-5555-4555-8555-555555555555", "sender": "user:alice@localhost", "recipients": ["sage@chorus@localhost"], "subject": "A subject", "body": "A body.", + "tags": [], "sent_at": datetime(2026, 6, 12, 9, 0, tzinfo=UTC), "metadata": {}, } @@ -142,11 +144,69 @@ def test_metadata_field_is_required_but_may_be_empty() -> None: assert make_message(metadata={}).metadata == {} with pytest.raises(ValidationError): MAILMessage( + mail_version="2.0", message_id="55555555-5555-4555-8555-555555555555", sender="user:alice@localhost", recipients=["sage@chorus@localhost"], subject="A subject", body="A body.", + tags=[], sent_at=datetime(2026, 6, 12, 9, 0, tzinfo=UTC), # metadata intentionally omitted ) + + +# ─── §7.8 Protocol Version ───────────────────────────────────────── + + +def test_mail_version_must_be_present_and_2_0() -> None: + """§7.8: every message MUST carry mail_version, pinned to "2.0".""" + + assert make_message().mail_version == "2.0" + with pytest.raises(ValidationError): + make_message(mail_version="1.0") + + +# ─── §7.9 Reply References ───────────────────────────────────────── + + +def test_reply_to_defaults_to_none() -> None: + """§7.9: reply_to is optional; absent means the message is not a reply.""" + + assert make_message().reply_to is None + + +def test_reply_to_accepts_uuid() -> None: + """§7.9: when present, reply_to MUST be the UUID of another message.""" + + original_id = "66666666-6666-4666-8666-666666666666" + assert make_message(reply_to=original_id).reply_to == original_id + + +def test_reply_to_rejects_non_uuid() -> None: + """§7.9: a malformed reply_to MUST be rejected.""" + + with pytest.raises(ValidationError): + make_message(reply_to="not-a-uuid") + + +# ─── §7.10 Tags ──────────────────────────────────────────────────── + + +def test_tags_may_be_empty() -> None: + """§7.10: tags MUST be present; it MAY be an empty list.""" + + assert make_message(tags=[]).tags == [] + + +def test_tags_accept_slug_strings() -> None: + """§7.10: each tag MUST be a slug string.""" + + assert make_message(tags=["urgent", "project-x"]).tags == ["urgent", "project-x"] + + +def test_tags_reject_non_slug() -> None: + """§7.10: non-slug tags (spaces, uppercase, etc.) MUST be rejected.""" + + with pytest.raises(ValidationError): + make_message(tags=["Not A Slug"]) diff --git a/tests/integration/test_mailboxes.py b/tests/integration/test_mailboxes.py index 588e6d4..d18ffa1 100644 --- a/tests/integration/test_mailboxes.py +++ b/tests/integration/test_mailboxes.py @@ -274,11 +274,13 @@ def test_send_unknown_draft_returns_404(app_client: TestClient, headers_for) -> def _seed_trash(backend: MemoryBackend, owner: str) -> str: message = MAILMessage( + mail_version="2.0", message_id="22222222-2222-4222-8222-222222222222", sender="sage@chorus@localhost", recipients=[owner], subject="Trashed", body="This message was moved to trash.", + tags=[], sent_at=datetime(2026, 6, 11, tzinfo=UTC), metadata={}, ) @@ -352,11 +354,13 @@ def _seed_trash_n(backend: MemoryBackend, owner: str, n: int) -> list[str]: for i in range(n): message_id = f"{i:08d}-2222-4222-8222-222222222222" message = MAILMessage( + mail_version="2.0", message_id=message_id, sender="sage@chorus@localhost", recipients=[owner], subject=f"Trashed {i}", body="body", + tags=[], sent_at=datetime(2026, 6, 1, 12, n - i, tzinfo=UTC), # decreasing metadata={}, ) diff --git a/tests/integration/webhooks/test_delivery.py b/tests/integration/webhooks/test_delivery.py index 782cf20..03353a8 100644 --- a/tests/integration/webhooks/test_delivery.py +++ b/tests/integration/webhooks/test_delivery.py @@ -44,11 +44,13 @@ @pytest.fixture def message() -> MAILMessage: return MAILMessage( + mail_version="2.0", message_id=MESSAGE_ID, sender=SENDER, recipients=[RECIPIENT], subject="Webhook test", body="A body worth signing.", + tags=[], sent_at=datetime(2026, 6, 12, 9, 0, tzinfo=UTC), metadata={}, ) @@ -109,6 +111,36 @@ async def test_delivery_posts_expected_payload( assert payload_message["body"] == "A body worth signing." assert payload_message["swarm"] == "chorus" assert payload_message["metadata"] == {} + # A non-reply, untagged message carries the empty defaults. + assert payload_message["reply_to"] is None + assert payload_message["tags"] == [] + + +@respx.mock +@pytest.mark.asyncio +async def test_delivery_payload_carries_reply_to_and_tags( + pipeline_backend: MemoryBackend, +) -> None: + original_id = "44444444-4444-4444-4444-444444444444" + message = MAILMessage( + mail_version="2.0", + message_id=MESSAGE_ID, + reply_to=original_id, + sender=SENDER, + recipients=[RECIPIENT], + subject="Re: Webhook test", + body="A reply worth signing.", + tags=["urgent", "project-x"], + sent_at=datetime(2026, 6, 12, 9, 0, tzinfo=UTC), + metadata={}, + ) + route = respx.post(WEBHOOK_URL).mock(return_value=httpx.Response(200)) + await _fire(pipeline_backend, message) + + payload_message = json.loads(route.calls[0].request.content)["message"] + # reply_to is surfaced in the same msg_-prefixed form as message_id. + assert payload_message["reply_to"] == f"msg_{original_id}" + assert payload_message["tags"] == ["urgent", "project-x"] @respx.mock @@ -180,9 +212,7 @@ async def test_5xx_retries_until_success( assert recorded_sleeps == RETRY_LADDER[:2] # The event id is stable across attempts (receiver-side dedup key). - event_ids = { - call.request.headers["X-MAIL-Event-Id"] for call in route.calls - } + event_ids = {call.request.headers["X-MAIL-Event-Id"] for call in route.calls} assert len(event_ids) == 1 @@ -271,11 +301,13 @@ async def test_daemon_deliver_local_fires_registered_webhook( ) backend.inboxes[RECIPIENT] = [] message = MAILMessage( + mail_version="2.0", message_id=MESSAGE_ID, sender=SENDER, recipients=[RECIPIENT], subject="Wired", body="Through the whole pipeline.", + tags=[], sent_at=datetime(2026, 6, 12, 9, 0, tzinfo=UTC), metadata={}, ) @@ -337,11 +369,13 @@ async def test_delivery_to_non_agent_recipient_fires_no_webhook( ) backend.inboxes[user_address] = [] message = MAILMessage( + mail_version="2.0", message_id=MESSAGE_ID, sender="user:bob@localhost", recipients=[user_address], subject="No hook", body="Delivered silently.", + tags=[], sent_at=datetime(2026, 6, 12, 9, 0, tzinfo=UTC), metadata={}, ) diff --git a/tests/unit/test_cli_help.py b/tests/unit/test_cli_help.py index d455f98..f7c188f 100644 --- a/tests/unit/test_cli_help.py +++ b/tests/unit/test_cli_help.py @@ -27,6 +27,7 @@ def test_mail_help_uses_categorized_command_sections() -> None: assert " Mailing Lists:" in help_text assert "{ping,p,login" not in help_text assert 'mail compose "Status update"' in help_text + assert "reply (r)" in help_text def test_mail_admin_help_uses_categorized_command_sections() -> None: diff --git a/tests/unit/test_client_commands.py b/tests/unit/test_client_commands.py index e1dc548..21e3cc0 100644 --- a/tests/unit/test_client_commands.py +++ b/tests/unit/test_client_commands.py @@ -21,6 +21,7 @@ cmd_inbox, cmd_login, cmd_ping, + cmd_reply, cmd_send, ) @@ -164,11 +165,16 @@ def test_compose_posts_draft_payload(client_env, capsys: pytest.CaptureFixture) }, ) ) - cmd_compose(Namespace(output="text", subject="A subject", body="A body.")) + cmd_compose(Namespace(output="text", subject="A subject", body="A body.", tags=[])) request = route.calls[0].request assert request.headers["Authorization"] == f"Bearer {TOKEN}" - assert json.loads(request.content) == {"subject": "A subject", "body": "A body."} + assert json.loads(request.content) == { + "subject": "A subject", + "body": "A body.", + "reply_to": None, + "tags": [], + } assert "Draft ID: 55555555-5555-4555-8555-555555555555" in capsys.readouterr().out @@ -177,7 +183,7 @@ def test_compose_rejects_invalid_subject_before_any_request(client_env) -> None: no request reaches the server (SPEC.md §8.1).""" with pytest.raises(Exception): # noqa: B017 — pydantic ValidationError - cmd_compose(Namespace(output="text", subject="", body="A body.")) + cmd_compose(Namespace(output="text", subject="", body="A body.", tags=[])) # ─── send ────────────────────────────────────────────────────────── @@ -189,21 +195,30 @@ def test_send_posts_recipients_to_draft_endpoint( ) -> None: draft_id = "55555555-5555-4555-8555-555555555555" message = { + "mail_version": "2.0", "message_id": "66666666-6666-4666-8666-666666666666", "sender": "user:alice@localhost", "recipients": ["sage@chorus@localhost"], "subject": "A subject", "body": "A body.", + "tags": [], "sent_at": "2026-06-12T09:00:00+00:00", "metadata": {}, } route = respx.post(f"{SERVER}/drafts/{draft_id}/send").mock( return_value=httpx.Response(200, json={"message": message, "metadata": {}}) ) - cmd_send(Namespace(output="text", draft_id=draft_id, to=["sage@chorus@localhost"])) + cmd_send( + Namespace( + output="text", draft_id=draft_id, to=["sage@chorus@localhost"], tags=[] + ) + ) request = route.calls[0].request - assert json.loads(request.content) == {"recipients": ["sage@chorus@localhost"]} + assert json.loads(request.content) == { + "recipients": ["sage@chorus@localhost"], + "tags": [], + } out = capsys.readouterr().out assert "- sage@chorus@localhost" in out @@ -216,5 +231,167 @@ def test_send_raises_on_non_200(client_env) -> None: ) with pytest.raises(RuntimeError, match="404"): cmd_send( - Namespace(output="text", draft_id=draft_id, to=["sage@chorus@localhost"]) + Namespace( + output="text", + draft_id=draft_id, + to=["sage@chorus@localhost"], + tags=[], + ) + ) + + +# ─── reply ───────────────────────────────────────────────────────── + + +def _inbox_entry(message: dict) -> dict: + return { + "entry": { + "message": message, + "received_at": "2026-06-12T09:05:00+00:00", + "delivered_by": "daemon:worker@localhost", + }, + "metadata": {}, + } + + +ORIGINAL_ID = "66666666-6666-4666-8666-666666666666" +ORIGINAL_MESSAGE = { + "mail_version": "2.0", + "message_id": ORIGINAL_ID, + "sender": "philosopher@chorus@localhost", + "recipients": ["user:alice@localhost"], + "subject": "Original subject", + "body": "The original body.", + "tags": [], + "sent_at": "2026-06-12T09:00:00+00:00", + "metadata": {}, +} + + +def _mock_reply_routes(draft_id: str, reply_message: dict): + """Register the three calls a reply makes: fetch, draft, send.""" + + inbox_route = respx.get(f"{SERVER}/inbox/{ORIGINAL_ID}").mock( + return_value=httpx.Response(200, json=_inbox_entry(ORIGINAL_MESSAGE)) + ) + draft = { + "draft_id": draft_id, + "subject": reply_message["subject"], + "body": reply_message["body"], + "created_at": "2026-06-12T09:10:00+00:00", + "updated_at": None, + "reply_to": ORIGINAL_ID, + "tags": reply_message["tags"], + } + draft_route = respx.post(f"{SERVER}/drafts").mock( + return_value=httpx.Response( + 200, + json={ + "entry": {"draft": draft, "sent_at": None, "sent_by": None}, + "metadata": {}, + }, + ) + ) + send_route = respx.post(f"{SERVER}/drafts/{draft_id}/send").mock( + return_value=httpx.Response( + 200, json={"message": reply_message, "metadata": {}} + ) + ) + return inbox_route, draft_route, send_route + + +@respx.mock +def test_reply_defaults_subject_recipient_and_reply_to( + client_env, capsys: pytest.CaptureFixture +) -> None: + draft_id = "55555555-5555-4555-8555-555555555555" + reply_message = { + "mail_version": "2.0", + "message_id": "77777777-7777-4777-8777-777777777777", + "reply_to": ORIGINAL_ID, + "sender": "user:alice@localhost", + "recipients": ["philosopher@chorus@localhost"], + "subject": "Re: Original subject", + "body": "My reply.", + "tags": [], + "sent_at": "2026-06-12T09:11:00+00:00", + "metadata": {}, + } + _, draft_route, send_route = _mock_reply_routes(draft_id, reply_message) + + cmd_reply( + Namespace( + output="text", + message_id=ORIGINAL_ID, + body="My reply.", + subject=None, + tags=[], + ) + ) + + # The draft references the original and defaults the subject to "Re: ...". + draft_body = json.loads(draft_route.calls[0].request.content) + assert draft_body == { + "subject": "Re: Original subject", + "body": "My reply.", + "reply_to": ORIGINAL_ID, + "tags": [], + } + # The reply is addressed back to the original sender. + send_body = json.loads(send_route.calls[0].request.content) + assert send_body == { + "recipients": ["philosopher@chorus@localhost"], + "tags": [], + } + out = capsys.readouterr().out + assert "Reply Sent" in out + assert f"In Reply To: {ORIGINAL_ID}" in out + + +@respx.mock +def test_reply_honors_explicit_subject_and_tags( + client_env, capsys: pytest.CaptureFixture +) -> None: + draft_id = "55555555-5555-4555-8555-555555555555" + reply_message = { + "mail_version": "2.0", + "message_id": "77777777-7777-4777-8777-777777777777", + "reply_to": ORIGINAL_ID, + "sender": "user:alice@localhost", + "recipients": ["philosopher@chorus@localhost"], + "subject": "Custom subject", + "body": "My reply.", + "tags": ["urgent", "project-x"], + "sent_at": "2026-06-12T09:11:00+00:00", + "metadata": {}, + } + _, draft_route, _ = _mock_reply_routes(draft_id, reply_message) + + cmd_reply( + Namespace( + output="text", + message_id=ORIGINAL_ID, + body="My reply.", + subject="Custom subject", + tags=["urgent", "project-x"], + ) + ) + + draft_body = json.loads(draft_route.calls[0].request.content) + assert draft_body["subject"] == "Custom subject" + assert draft_body["tags"] == ["urgent", "project-x"] + + +@respx.mock +def test_reply_raises_when_original_missing(client_env) -> None: + respx.get(f"{SERVER}/inbox/{ORIGINAL_ID}").mock(return_value=httpx.Response(404)) + with pytest.raises(RuntimeError, match="404"): + cmd_reply( + Namespace( + output="text", + message_id=ORIGINAL_ID, + body="My reply.", + subject=None, + tags=[], + ) ) diff --git a/tests/unit/test_draft_reply_tags.py b/tests/unit/test_draft_reply_tags.py new file mode 100644 index 0000000..203eac3 --- /dev/null +++ b/tests/unit/test_draft_reply_tags.py @@ -0,0 +1,96 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Server-side behavior for MAIL 2.0 reply/tag support: drafts carry reply_to +and tags through to the sent MAILMessage, and send-time tags merge with draft +tags as an order-preserving union. +""" + +import pytest +from mail_protocol.core.user_agents import MAILUser, MAILUserAgent +from mail_protocol.network.requests import DraftPostRequest, DraftSendPostRequest +from mail_server.backends.memory.api import MemoryBackend + +ORIGINAL_ID = "66666666-6666-4666-8666-666666666666" + + +def _make_user_agent() -> MAILUserAgent: + return MAILUserAgent( + user_agent=MAILUser(ua_type="user", user_id="alice", host="localhost") + ) + + +async def _seed_draft( + backend: MemoryBackend, user_agent: MAILUserAgent, payload: DraftPostRequest +) -> str: + address = user_agent.get_address() + backend.drafts[address] = [] + backend.outboxes[address] = [] + draft_entry = await backend.post_draft(user_agent, payload) + return draft_entry.draft.draft_id + + +@pytest.mark.asyncio +async def test_send_draft_propagates_mail_version_and_reply_to( + backend: MemoryBackend, +) -> None: + user_agent = _make_user_agent() + draft_id = await _seed_draft( + backend, + user_agent, + DraftPostRequest( + subject="Re: Hi", body="A reply.", reply_to=ORIGINAL_ID, tags=[] + ), + ) + + message = await backend.send_draft( + user_agent, + draft_id, + DraftSendPostRequest(recipients=["philosopher@chorus@localhost"]), + ) + + assert message.mail_version == "2.0" + assert message.reply_to == ORIGINAL_ID + + +@pytest.mark.asyncio +async def test_send_draft_merges_draft_and_send_tags(backend: MemoryBackend) -> None: + user_agent = _make_user_agent() + draft_id = await _seed_draft( + backend, + user_agent, + DraftPostRequest(subject="Tagged", body="Body.", tags=["alpha", "beta"]), + ) + + message = await backend.send_draft( + user_agent, + draft_id, + DraftSendPostRequest( + recipients=["philosopher@chorus@localhost"], tags=["beta", "gamma"] + ), + ) + + # Order-preserving union: draft tags first, then new send-time tags. + assert message.tags == ["alpha", "beta", "gamma"] + + +@pytest.mark.asyncio +async def test_send_draft_without_reply_or_tags_is_unmarked( + backend: MemoryBackend, +) -> None: + user_agent = _make_user_agent() + draft_id = await _seed_draft( + backend, + user_agent, + DraftPostRequest(subject="Plain", body="Body."), + ) + + message = await backend.send_draft( + user_agent, + draft_id, + DraftSendPostRequest(recipients=["philosopher@chorus@localhost"]), + ) + + assert message.reply_to is None + assert message.tags == [] diff --git a/tests/unit/test_mail_lists_send.py b/tests/unit/test_mail_lists_send.py index 411aa24..0cbbef6 100644 --- a/tests/unit/test_mail_lists_send.py +++ b/tests/unit/test_mail_lists_send.py @@ -67,11 +67,13 @@ def _seed_message( ) -> MAILMessage: now = datetime(2026, 6, 6, 12, 0, tzinfo=UTC) message = MAILMessage( + mail_version="2.0", message_id="22222222-2222-2222-2222-222222222222", sender=SENDER, recipients=recipients, subject="Daily briefing", body="Body text.", + tags=[], sent_at=now, metadata={}, ) @@ -295,7 +297,9 @@ async def test_handle_webhook_delivered_skips_non_agent_recipients( fired: list[tuple[str, MAILMessage]] = [] - async def fake_handle_webhook_delivered_for_url(*, url, recipient, message, secret, list_address=None): + async def fake_handle_webhook_delivered_for_url( + *, url, recipient, message, secret, list_address=None + ): fired.append((recipient, message)) monkeypatch.setattr( @@ -305,17 +309,23 @@ async def fake_handle_webhook_delivered_for_url(*, url, recipient, message, secr ) msg = MAILMessage( + mail_version="2.0", message_id="33333333-3333-3333-3333-333333333333", sender=SENDER, recipients=[ALICE], subject="s", body="b", + tags=[], sent_at=datetime(2026, 6, 12, tzinfo=UTC), metadata={}, ) # Non-agent recipients: no webhook task is created. - for non_agent in ["admin:ryan@chrn.ai", "user:dummy@chrn.ai", "daemon:first@chrn.ai"]: + for non_agent in [ + "admin:ryan@chrn.ai", + "user:dummy@chrn.ai", + "daemon:first@chrn.ai", + ]: await backend._handle_webhook_delivered(recipient=non_agent, message=msg) # Agent recipient: webhook task IS created. @@ -326,6 +336,7 @@ async def fake_handle_webhook_delivered_for_url(*, url, recipient, message, secr # backend's _handle_webhook_delivered scheduling them as Tasks. Give # the loop one tick. import asyncio as _asyncio + await _asyncio.sleep(0) fired_recipients = [r for r, _ in fired] diff --git a/tests/unit/test_mail_trash_store.py b/tests/unit/test_mail_trash_store.py index 6eacaa9..700ed62 100644 --- a/tests/unit/test_mail_trash_store.py +++ b/tests/unit/test_mail_trash_store.py @@ -22,11 +22,13 @@ def _make_user_agent() -> MAILUserAgent: def _make_message() -> MAILMessage: return MAILMessage( + mail_version="2.0", message_id="11111111-1111-4111-8111-111111111111", sender="philosopher@chorus@localhost", recipients=["user:ryan@localhost"], subject="Trash lookup", body="This message should be read from trash, not drafts.", + tags=[], sent_at=datetime(2026, 6, 10, tzinfo=UTC), metadata={}, ) diff --git a/tests/unit/test_memory_fs_roundtrip.py b/tests/unit/test_memory_fs_roundtrip.py index c4e5577..82874e9 100644 --- a/tests/unit/test_memory_fs_roundtrip.py +++ b/tests/unit/test_memory_fs_roundtrip.py @@ -33,11 +33,13 @@ def _message() -> MAILMessage: return MAILMessage( + mail_version="2.0", message_id=UUID, sender=USER, recipients=[AGENT], subject="Persisted", body="Survives a save/load cycle.", + tags=[], sent_at=NOW, metadata={}, ) diff --git a/tests/unit/test_migrate_messages_v2.py b/tests/unit/test_migrate_messages_v2.py new file mode 100644 index 0000000..5f91236 --- /dev/null +++ b/tests/unit/test_migrate_messages_v2.py @@ -0,0 +1,102 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Tests for scripts/migrate_messages_v2.py, the MAIL 2.0 message-schema +backfill migration. +""" + +import importlib.util +import json +import sys +from pathlib import Path + +import pytest +from mail_protocol.core.messages import MAILMessage + +_SCRIPT = Path(__file__).resolve().parents[2] / "scripts" / "migrate_messages_v2.py" +_spec = importlib.util.spec_from_file_location("migrate_messages_v2", _SCRIPT) +assert _spec is not None and _spec.loader is not None +migrate = importlib.util.module_from_spec(_spec) +# Register before exec so dataclass introspection can resolve the module. +sys.modules[_spec.name] = migrate +_spec.loader.exec_module(migrate) + + +PRE_V2_RECORD = { + "message_id": "55555555-5555-4555-8555-555555555555", + "sender": "user:alice@localhost", + "recipients": ["sage@chorus@localhost"], + "subject": "Legacy", + "body": "Persisted before MAIL 2.0.", + "sent_at": "2026-06-12T09:00:00+00:00", + "metadata": {}, +} + + +def _write(messages_dir: Path, record: dict) -> Path: + path = messages_dir / record["message_id"] + path.write_text(json.dumps(record), encoding="utf-8") + return path + + +def test_dry_run_reports_without_writing(tmp_path: Path) -> None: + messages = tmp_path / "messages" + messages.mkdir() + path = _write(messages, PRE_V2_RECORD) + + result = migrate.migrate_messages(messages, dry_run=True) + + assert result.scanned == 1 + assert result.migrated == 1 + assert result.already_current == 0 + # File on disk is untouched in dry-run mode. + assert json.loads(path.read_text()) == PRE_V2_RECORD + + +def test_migration_backfills_and_validates(tmp_path: Path) -> None: + messages = tmp_path / "messages" + messages.mkdir() + path = _write(messages, PRE_V2_RECORD) + + result = migrate.migrate_messages(messages, dry_run=False) + assert result.migrated == 1 + + upgraded = json.loads(path.read_text()) + assert upgraded["mail_version"] == "2.0" + assert upgraded["tags"] == [] + # The migrated record now passes the MAIL 2.0 model contract. + model = MAILMessage.model_validate(upgraded) + assert model.reply_to is None + + +def test_existing_fields_are_preserved(tmp_path: Path) -> None: + messages = tmp_path / "messages" + messages.mkdir() + record = dict(PRE_V2_RECORD) + record["mail_version"] = "2.0" + record["tags"] = ["already-tagged"] + path = _write(messages, record) + + result = migrate.migrate_messages(messages, dry_run=False) + + assert result.migrated == 0 + assert result.already_current == 1 + assert json.loads(path.read_text())["tags"] == ["already-tagged"] + + +def test_idempotent_second_run_is_noop(tmp_path: Path) -> None: + messages = tmp_path / "messages" + messages.mkdir() + _write(messages, PRE_V2_RECORD) + + migrate.migrate_messages(messages, dry_run=False) + second = migrate.migrate_messages(messages, dry_run=False) + + assert second.migrated == 0 + assert second.already_current == 1 + + +def test_missing_directory_raises(tmp_path: Path) -> None: + with pytest.raises(FileNotFoundError): + migrate.migrate_messages(tmp_path / "nope", dry_run=True) diff --git a/tests/unit/test_protocol_models.py b/tests/unit/test_protocol_models.py index d7bc284..2beda6f 100644 --- a/tests/unit/test_protocol_models.py +++ b/tests/unit/test_protocol_models.py @@ -22,11 +22,13 @@ def _message() -> MAILMessage: return MAILMessage( + mail_version="2.0", message_id=UUID, sender="user:alice@localhost", recipients=["sage@chorus@localhost"], subject="Hello", body="A body worth summarizing.", + tags=[], sent_at=NOW, metadata={"k": "v"}, ) diff --git a/tests/unit/test_protocol_validators.py b/tests/unit/test_protocol_validators.py index 6cfc373..920d620 100644 --- a/tests/unit/test_protocol_validators.py +++ b/tests/unit/test_protocol_validators.py @@ -67,6 +67,37 @@ def test_validate_mail_addresses_permits_empty_list() -> None: assert v.validate_mail_addresses([]) == [] +# ─── message tags (SPEC.md §7.10) ────────────────────────────────── + + +@pytest.mark.parametrize("value", ["urgent", "project-x", "a", "v2-0", "x" * 32]) +def test_validate_message_tag_accepts_slugs(value: str) -> None: + assert v.validate_message_tag(value) == value + + +@pytest.mark.parametrize( + "value", + ["", "Urgent", "has space", "trailing-", "-leading", "under_score", "x" * 33], +) +def test_validate_message_tag_rejects_non_slugs(value: str) -> None: + with pytest.raises(ValueError): + v.validate_message_tag(value) + + +def test_validate_message_tags_permits_empty_list() -> None: + assert v.validate_message_tags([]) == [] + + +def test_validate_message_tags_accepts_list_of_slugs() -> None: + tags = ["urgent", "project-x"] + assert v.validate_message_tags(tags) == tags + + +def test_validate_message_tags_rejects_any_invalid_member() -> None: + with pytest.raises(ValueError): + v.validate_message_tags(["urgent", "Not A Slug"]) + + # ─── local addresses (agent@swarm) ───────────────────────────────── From 1f9b3d02d1c8f9a7f25bab1204dc45344889b505 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Thu, 18 Jun 2026 16:18:58 -0400 Subject: [PATCH 09/28] feat: support message forwarding --- src/mail/client/docs/reference/cli.md | 10 ++ src/mail/client/docs/tutorials/quickstart.md | 24 +++ src/mail/client/src/mail_client/cli.py | 33 +++- .../src/mail_client/commands/__init__.py | 2 + .../src/mail_client/commands/forward.py | 155 ++++++++++++++++++ tests/unit/test_cli_help.py | 1 + tests/unit/test_client_commands.py | 148 +++++++++++++++++ 7 files changed, 372 insertions(+), 1 deletion(-) create mode 100644 src/mail/client/src/mail_client/commands/forward.py diff --git a/src/mail/client/docs/reference/cli.md b/src/mail/client/docs/reference/cli.md index b582234..a2b9134 100644 --- a/src/mail/client/docs/reference/cli.md +++ b/src/mail/client/docs/reference/cli.md @@ -15,6 +15,7 @@ mail [option]... [argument]... - `compose`: Draft a new MAIL message. Accepts `--tags TAG...` to attach slug tags. - `send`: Send an existing draft by ID to the specified recipient(s). Accepts `--tags TAG...`, which are merged with the draft's tags. - `reply`: Reply to an existing inbox message by ID. Addresses the reply to the original sender and defaults the subject to `Re: `. Accepts `--subject SUBJECT` and `--tags TAG...`. +- `forward`: Forward an existing inbox message by ID to one or more new recipient(s). Encodes the original message (sender, recipients, subject, and body) into the forwarded body and defaults the subject to `Fwd: `. Accepts `--note NOTE` to prepend a note above the forwarded message, plus `--subject SUBJECT` and `--tags TAG...`. - `inbox`: Open your MAIL inbox. - `inbox-open`: Open a specific message by ID in your MAIL inbox. - `outbox`: Open your MAIL outbox. @@ -64,3 +65,12 @@ defaults to `Re: `): mail reply "Thanks, acknowledged." mail reply "See attached." --subject "Follow-up" --tags project-x ``` + +Forward a message from your inbox to new recipient(s) (the original message is +encoded into the forwarded body, subject defaults to `Fwd: `): + +```bash +mail forward sage@chorus@localhost +mail forward sage@chorus@localhost philosopher@chorus@localhost \ + --note "Please take a look." --subject "Heads up" --tags fyi +``` diff --git a/src/mail/client/docs/tutorials/quickstart.md b/src/mail/client/docs/tutorials/quickstart.md index 13d1417..8eb3d29 100644 --- a/src/mail/client/docs/tutorials/quickstart.md +++ b/src/mail/client/docs/tutorials/quickstart.md @@ -79,6 +79,30 @@ uv run mail send supervisor@default@example.com You should then see the MAIL message created and sent to the `supervisor@default@example.com`. This includes the message's unique ID. +## Forward a Message + +If a message in your inbox is relevant to other user-agents, you can forward it +to one or more new recipients with the `forward` command. +MAIL encodes the original message (its sender, recipients, subject, and body) into +the forwarded message's body, and defaults the subject to `Fwd: `. +To forward an inbox message to `sage@chorus@example.com`: + +```bash +MAIL_SERVER=... \ +MAIL_TOKEN=... \ +uv run mail forward sage@chorus@example.com +``` + +You can supply multiple recipients, prepend your own note with `--note`, and +override the subject with `--subject`: + +```bash +MAIL_SERVER=... \ +MAIL_TOKEN=... \ +uv run mail forward sage@chorus@example.com philosopher@chorus@example.com \ + --note "Please take a look." +``` + ## See Also - `mail-swarms-client` CLI reference: [reference/cli.md](/docs/reference/cli.md) diff --git a/src/mail/client/src/mail_client/cli.py b/src/mail/client/src/mail_client/cli.py index 8dd69f1..34d660e 100644 --- a/src/mail/client/src/mail_client/cli.py +++ b/src/mail/client/src/mail_client/cli.py @@ -12,6 +12,7 @@ cmd_compose, cmd_drafts, cmd_drafts_open, + cmd_forward, cmd_inbox, cmd_inbox_open, cmd_list_get, @@ -75,6 +76,7 @@ ("compose (c)", "Draft a new MAIL message."), ("send (s)", "Send a drafted message."), ("reply (r)", "Reply to an inbox message by ID."), + ("forward (f)", "Forward an inbox message to new recipient(s)."), ("inbox (i)", "List your inbox messages."), ("inbox-open (open, o)", "Open an inbox message by ID."), ("outbox (O)", "List your sent messages."), @@ -109,13 +111,14 @@ "mail send user@example", "mail inbox-open ", 'mail reply "Thanks, acknowledged."', + "mail forward sage@chorus@localhost", ] def _add_tags_arg(parser: argparse.ArgumentParser) -> None: """ Register the shared ``--tags`` flag for message-creating commands - (compose, send, reply). Tags are slug strings used to categorize a + (compose, send, reply, forward). Tags are slug strings used to categorize a message; the default is an empty list (no tags). """ @@ -260,6 +263,34 @@ def build_parser() -> argparse.ArgumentParser: _add_tags_arg(reply_p) reply_p.set_defaults(func=cmd_reply, cmd="reply") + # command `forward` + forward_d = "forward an existing inbox message to new recipient(s)" + forward_p = subparsers.add_parser( + "forward", + aliases=["f"], + prog="mail forward", + help=forward_d, + description=forward_d, + ) + forward_p.add_argument( + "message_id", help="the ID of the inbox message to forward" + ) + forward_p.add_argument( + "to", nargs="+", help="the address(es) to forward this message to" + ) + forward_p.add_argument( + "--note", + default=None, + help="an optional note to prepend above the forwarded message", + ) + forward_p.add_argument( + "--subject", + default=None, + help="the subject of the forward (default: 'Fwd: ')", + ) + _add_tags_arg(forward_p) + forward_p.set_defaults(func=cmd_forward, cmd="forward") + # command `inbox` inbox_d = "open your MAIL inbox" inbox_p = subparsers.add_parser( diff --git a/src/mail/client/src/mail_client/commands/__init__.py b/src/mail/client/src/mail_client/commands/__init__.py index e406d9f..8914fd5 100644 --- a/src/mail/client/src/mail_client/commands/__init__.py +++ b/src/mail/client/src/mail_client/commands/__init__.py @@ -9,6 +9,7 @@ from .daemon_post import cmd_daemon_post from .drafts import cmd_drafts from .drafts_open import cmd_drafts_open +from .forward import cmd_forward from .inbox import cmd_inbox from .inbox_open import cmd_inbox_open from .list_delete import cmd_list_delete @@ -57,6 +58,7 @@ "cmd_daemon_post", "cmd_drafts", "cmd_drafts_open", + "cmd_forward", "cmd_inbox", "cmd_inbox_open", "cmd_list_delete", diff --git a/src/mail/client/src/mail_client/commands/forward.py b/src/mail/client/src/mail_client/commands/forward.py new file mode 100644 index 0000000..28aa5cd --- /dev/null +++ b/src/mail/client/src/mail_client/commands/forward.py @@ -0,0 +1,155 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +import os +from argparse import Namespace + +import httpx +from mail_protocol.core.messages import MAILMessage +from mail_protocol.network.requests import DraftPostRequest, DraftSendPostRequest +from mail_protocol.network.responses import ( + DraftPostResponse, + DraftSendPostResponse, + InboxMessageGetResponse, +) +from pydantic import ValidationError + +USER_AGENT = "Multi-Agent-Interface-Layer-CLI-Client/2.0.0 (github.com/charonlabs/mail)" + + +def _forward_subject(original_subject: str) -> str: + """ + Derive the default subject for a forward. An existing `Fwd:` prefix + (case-insensitive) is preserved rather than duplicated. + """ + + if original_subject.lower().startswith("fwd:"): + return original_subject + return f"Fwd: {original_subject}" + + +def _forward_body(original: MAILMessage, note: str | None) -> str: + """ + Encode the original message into the forwarded body, optionally prefixed + with the forwarding user-agent's own note. The quoted block mirrors the + familiar email "forwarded message" convention so the original sender, + recipients, subject, and body remain legible to the new recipient(s). + """ + + forwarded_block = ( + "---------- Forwarded message ----------\n" + f"From: {original.sender}\n" + f"Date: {original.sent_at}\n" + f"Subject: {original.subject}\n" + f"To: {', '.join(original.recipients)}\n" + "\n" + f"{original.body}" + ) + if note: + return f"{note}\n\n{forwarded_block}" + return forwarded_block + + +def cmd_forward(args: Namespace) -> None: + """ + Forward an existing inbox message to one or more new recipient(s). + + Fetches the original message, builds a draft whose body encodes the + original (sender, recipients, subject, and body) along with an optional + note, defaults the subject to `Fwd: `, then sends it to + the specified recipient(s). + """ + + # 1. check that the required env vars are provided + MAIL_SERVER = os.getenv("MAIL_SERVER") + if MAIL_SERVER is None: + raise ValueError("env var MAIL_SERVER is required") + MAIL_TOKEN = os.getenv("MAIL_TOKEN") + if MAIL_TOKEN is None: + raise ValueError("env var MAIL_TOKEN is required") + + headers = { + "Authorization": f"Bearer {MAIL_TOKEN}", + "User-Agent": USER_AGENT, + "Content-Type": "application/json", + } + + # 2. fetch the original message from the user's inbox + original_response = httpx.get( + url=f"{MAIL_SERVER}/inbox/{args.message_id}", + headers=headers, + ) + if original_response.status_code != 200: + raise RuntimeError( + f"get inbox entry request to {MAIL_SERVER} failed with status code " + f"{original_response.status_code}" + ) + try: + original_obj = InboxMessageGetResponse.model_validate(original_response.json()) + except ValidationError as e: + raise RuntimeError(f"response validation failed: {e}") + + original = original_obj.entry.message + subject = args.subject if args.subject else _forward_subject(original.subject) + + # 3. create the forward draft, encoding the original message in the body + draft_payload = DraftPostRequest( + subject=subject, + body=_forward_body(original, args.note), + tags=args.tags, + ) + draft_response = httpx.post( + url=f"{MAIL_SERVER}/drafts", + headers=headers, + json=draft_payload.model_dump(), + ) + if draft_response.status_code != 200: + raise RuntimeError( + f"post draft request to {MAIL_SERVER} failed with status code " + f"{draft_response.status_code}" + ) + try: + draft_obj = DraftPostResponse.model_validate(draft_response.json()) + except ValidationError as e: + raise RuntimeError(f"response validation failed: {e}") + + draft_id = draft_obj.entry.draft.draft_id + + # 4. send the draft to the specified recipient(s) + send_payload = DraftSendPostRequest(recipients=args.to) + send_response = httpx.post( + url=f"{MAIL_SERVER}/drafts/{draft_id}/send", + headers=headers, + json=send_payload.model_dump(), + ) + if send_response.status_code != 200: + raise RuntimeError( + f"send request to {MAIL_SERVER} failed with status code " + f"{send_response.status_code}" + ) + try: + send_obj = DraftSendPostResponse.model_validate(send_response.json()) + except ValidationError as e: + raise RuntimeError(f"response validation failed: {e}") + + # 5. print the forwarded message + match args.output: + case "json": + print(send_obj.model_dump_json()) + case "text": + _print_text(send_obj) + + +def _print_text(response_obj: DraftSendPostResponse) -> None: + message = response_obj.message + print("=== Message Forwarded ===") + print(f"Message ID: {message.message_id}") + print(f"Sent At: {message.sent_at}") + print(f"Sender: {message.sender}") + print("Recipient(s):") + for recipient in message.recipients: + print(f"- {recipient}") + print(f"Subject: {message.subject}") + if message.tags: + print(f"Tags: {', '.join(message.tags)}") + print(f"Body:\n{message.body}\n") diff --git a/tests/unit/test_cli_help.py b/tests/unit/test_cli_help.py index f7c188f..51dfff2 100644 --- a/tests/unit/test_cli_help.py +++ b/tests/unit/test_cli_help.py @@ -28,6 +28,7 @@ def test_mail_help_uses_categorized_command_sections() -> None: assert "{ping,p,login" not in help_text assert 'mail compose "Status update"' in help_text assert "reply (r)" in help_text + assert "forward (f)" in help_text def test_mail_admin_help_uses_categorized_command_sections() -> None: diff --git a/tests/unit/test_client_commands.py b/tests/unit/test_client_commands.py index 21e3cc0..3619df6 100644 --- a/tests/unit/test_client_commands.py +++ b/tests/unit/test_client_commands.py @@ -18,6 +18,7 @@ import respx from mail_client.commands import ( cmd_compose, + cmd_forward, cmd_inbox, cmd_login, cmd_ping, @@ -395,3 +396,150 @@ def test_reply_raises_when_original_missing(client_env) -> None: tags=[], ) ) + + +# ─── forward ─────────────────────────────────────────────────────── + + +def _mock_forward_routes(draft_id: str, forwarded_message: dict): + """Register the three calls a forward makes: fetch, draft, send.""" + + inbox_route = respx.get(f"{SERVER}/inbox/{ORIGINAL_ID}").mock( + return_value=httpx.Response(200, json=_inbox_entry(ORIGINAL_MESSAGE)) + ) + draft = { + "draft_id": draft_id, + "subject": forwarded_message["subject"], + "body": forwarded_message["body"], + "created_at": "2026-06-12T09:10:00+00:00", + "updated_at": None, + "reply_to": None, + "tags": forwarded_message["tags"], + } + draft_route = respx.post(f"{SERVER}/drafts").mock( + return_value=httpx.Response( + 200, + json={ + "entry": {"draft": draft, "sent_at": None, "sent_by": None}, + "metadata": {}, + }, + ) + ) + send_route = respx.post(f"{SERVER}/drafts/{draft_id}/send").mock( + return_value=httpx.Response( + 200, json={"message": forwarded_message, "metadata": {}} + ) + ) + return inbox_route, draft_route, send_route + + +@respx.mock +def test_forward_defaults_subject_and_encodes_original( + client_env, capsys: pytest.CaptureFixture +) -> None: + draft_id = "55555555-5555-4555-8555-555555555555" + forwarded_message = { + "mail_version": "2.0", + "message_id": "77777777-7777-4777-8777-777777777777", + "reply_to": None, + "sender": "user:alice@localhost", + "recipients": ["sage@chorus@localhost"], + "subject": "Fwd: Original subject", + "body": "encoded", + "tags": [], + "sent_at": "2026-06-12T09:11:00+00:00", + "metadata": {}, + } + _, draft_route, send_route = _mock_forward_routes(draft_id, forwarded_message) + + cmd_forward( + Namespace( + output="text", + message_id=ORIGINAL_ID, + to=["sage@chorus@localhost"], + note=None, + subject=None, + tags=[], + ) + ) + + # The draft defaults the subject to "Fwd: ..." and is NOT a reply. + draft_body = json.loads(draft_route.calls[0].request.content) + assert draft_body["subject"] == "Fwd: Original subject" + assert draft_body["reply_to"] is None + # The encoded body carries the original sender, recipients, and content. + assert "---------- Forwarded message ----------" in draft_body["body"] + assert f"From: {ORIGINAL_MESSAGE['sender']}" in draft_body["body"] + assert "To: user:alice@localhost" in draft_body["body"] + assert ORIGINAL_MESSAGE["body"] in draft_body["body"] + # No note was supplied, so the body starts with the forwarded block. + assert draft_body["body"].startswith("---------- Forwarded message ----------") + + # The forward is addressed to the user-specified recipient(s). + send_body = json.loads(send_route.calls[0].request.content) + assert send_body == { + "recipients": ["sage@chorus@localhost"], + "tags": [], + } + out = capsys.readouterr().out + assert "Message Forwarded" in out + + +@respx.mock +def test_forward_honors_note_subject_and_tags( + client_env, capsys: pytest.CaptureFixture +) -> None: + draft_id = "55555555-5555-4555-8555-555555555555" + forwarded_message = { + "mail_version": "2.0", + "message_id": "77777777-7777-4777-8777-777777777777", + "reply_to": None, + "sender": "user:alice@localhost", + "recipients": ["sage@chorus@localhost", "philosopher@chorus@localhost"], + "subject": "Custom subject", + "body": "encoded", + "tags": ["fyi", "project-x"], + "sent_at": "2026-06-12T09:11:00+00:00", + "metadata": {}, + } + _, draft_route, send_route = _mock_forward_routes(draft_id, forwarded_message) + + cmd_forward( + Namespace( + output="text", + message_id=ORIGINAL_ID, + to=["sage@chorus@localhost", "philosopher@chorus@localhost"], + note="Please take a look.", + subject="Custom subject", + tags=["fyi", "project-x"], + ) + ) + + draft_body = json.loads(draft_route.calls[0].request.content) + assert draft_body["subject"] == "Custom subject" + assert draft_body["tags"] == ["fyi", "project-x"] + # The note is prepended above the forwarded block. + assert draft_body["body"].startswith("Please take a look.\n\n") + assert "---------- Forwarded message ----------" in draft_body["body"] + + send_body = json.loads(send_route.calls[0].request.content) + assert send_body["recipients"] == [ + "sage@chorus@localhost", + "philosopher@chorus@localhost", + ] + + +@respx.mock +def test_forward_raises_when_original_missing(client_env) -> None: + respx.get(f"{SERVER}/inbox/{ORIGINAL_ID}").mock(return_value=httpx.Response(404)) + with pytest.raises(RuntimeError, match="404"): + cmd_forward( + Namespace( + output="text", + message_id=ORIGINAL_ID, + to=["sage@chorus@localhost"], + note=None, + subject=None, + tags=[], + ) + ) From bb5e5b39079e569db8e34b16c536d766765b5370 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Mon, 22 Jun 2026 16:08:29 -0400 Subject: [PATCH 10/28] chore: rebrand CLI copyright to MAIL Contributors and add --license Change the user-facing copyright in CLI --help footers and the backend-init epilog from "Addison Kline" to "2025-present MAIL Contributors". Add a shared --license flag (via cli_help) that prints the Apache-2.0 notice and exits, wired into all CLIs. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../protocol/src/mail_protocol/cli_help.py | 47 ++++++++++++++++++- .../server/src/mail_server/backend_init.py | 4 +- 2 files changed, 48 insertions(+), 3 deletions(-) diff --git a/src/mail/protocol/src/mail_protocol/cli_help.py b/src/mail/protocol/src/mail_protocol/cli_help.py index e83ce52..ffc4be8 100644 --- a/src/mail/protocol/src/mail_protocol/cli_help.py +++ b/src/mail/protocol/src/mail_protocol/cli_help.py @@ -9,6 +9,42 @@ CommandHelp = tuple[str, str] CommandGroup = tuple[str, Sequence[CommandHelp]] +LICENSE_NOTICE = """\ +MAIL - Multi-Agent Interface Layer +Copyright (c) 2025-present MAIL Contributors + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this software except in compliance with the License. +You may obtain a copy of the License at: + + http://www.apache.org/licenses/LICENSE-2.0 + +Distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the LICENSE and NOTICE files +distributed with this project for the full terms.""" + + +class _LicenseAction(argparse.Action): + """Print MAIL's license notice and exit, mirroring argparse's --version.""" + + def __init__( + self, + option_strings: Sequence[str], + dest: str = argparse.SUPPRESS, + default: str = argparse.SUPPRESS, + help: str = "show license information and exit", + ): + super().__init__( + option_strings=option_strings, + dest=dest, + default=default, + nargs=0, + help=help, + ) + + def __call__(self, parser, namespace, values, option_string=None): + parser.exit(message=LICENSE_NOTICE + "\n") + class MAILHelpFormatter( argparse.ArgumentDefaultsHelpFormatter, @@ -24,7 +60,7 @@ def build_epilog( *, command_groups: Sequence[CommandGroup] | None = None, examples: Sequence[str] | None = None, - footer: str | None = "Copyright (c) 2026 Addison Kline", + footer: str | None = "Copyright (c) 2025-present MAIL Contributors", ) -> str | None: sections: list[str] = [] @@ -51,6 +87,11 @@ def build_epilog( return "\n\n".join(sections) +def add_license_argument(parser: argparse.ArgumentParser) -> None: + """Add a `--license` flag that prints MAIL's license notice and exits.""" + parser.add_argument("--license", action=_LicenseAction) + + def make_arg_parser( *, prog: str, @@ -59,13 +100,15 @@ def make_arg_parser( command_groups: Sequence[CommandGroup] | None = None, examples: Sequence[str] | None = None, ) -> argparse.ArgumentParser: - return argparse.ArgumentParser( + parser = argparse.ArgumentParser( prog=prog, usage=usage, description=description, epilog=build_epilog(command_groups=command_groups, examples=examples), formatter_class=MAILHelpFormatter, ) + add_license_argument(parser) + return parser def add_hidden_subparsers(parser: argparse.ArgumentParser): diff --git a/src/mail/server/src/mail_server/backend_init.py b/src/mail/server/src/mail_server/backend_init.py index 487379d..9bcd888 100644 --- a/src/mail/server/src/mail_server/backend_init.py +++ b/src/mail/server/src/mail_server/backend_init.py @@ -3,6 +3,7 @@ import argparse +from mail_protocol.cli_help import add_license_argument from mail_protocol.core.validators import ( validate_agent_names, validate_daemon_worker_names, @@ -21,8 +22,9 @@ def main() -> None: prog="backend-init", usage="backend-init [option]...", description="Initialize the MAIL server backend for use by mail-server", - epilog="Copyright (c) 2026 Addison Kline", + epilog="Copyright (c) 2025-present MAIL Contributors", ) + add_license_argument(parser) parser.add_argument( "-t", "--type", From 447a9ec1b69749046029174f59a70e119a39ca08 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Mon, 22 Jun 2026 16:53:01 -0400 Subject: [PATCH 11/28] feat: support draft editing and file-sourced draft bodies Add two draft enhancements: 1. `mail compose` can now read a draft body from a file via `-F`/`--body-file PATH` as an alternative to the inline positional body. Exactly one of the two must be supplied. 2. A new `PATCH /drafts/{draft_id}` endpoint lets an authenticated user-agent update an existing draft's subject, body, reply_to, and tags. Only supplied fields change (tags: [] clears, omitted leaves unchanged) and updated_at is refreshed on any edit. Exposed in the CLI as `mail draft-edit` (alias `de`). Regenerates spec/openapi.yaml and updates the HTTP and CLI docs. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 3 + spec/openapi.yaml | 28 ++++ src/mail/client/docs/reference/cli.md | 3 +- src/mail/client/docs/tutorials/quickstart.md | 8 + src/mail/client/src/mail_client/cli.py | 64 +++++++- .../src/mail_client/commands/__init__.py | 2 + .../client/src/mail_client/commands/_body.py | 40 +++++ .../src/mail_client/commands/compose.py | 5 +- .../src/mail_client/commands/drafts_patch.py | 86 +++++++++++ .../src/mail_protocol/network/requests.py | 17 +++ .../src/mail_protocol/network/responses.py | 10 ++ src/mail/server/docs/reference/http.md | 1 + .../server/src/mail_server/backends/base.py | 17 +++ .../src/mail_server/backends/memory/api.py | 51 +++++++ .../server/src/mail_server/routers/drafts.py | 27 ++++ src/mail/server/src/mail_server/validators.py | 15 ++ tests/integration/test_mailboxes.py | 141 ++++++++++++++++++ tests/unit/test_client_commands.py | 126 +++++++++++++++- 18 files changed, 639 insertions(+), 5 deletions(-) create mode 100644 src/mail/client/src/mail_client/commands/_body.py create mode 100644 src/mail/client/src/mail_client/commands/drafts_patch.py diff --git a/.gitignore b/.gitignore index c4c4fcf..241aec9 100644 --- a/.gitignore +++ b/.gitignore @@ -224,3 +224,6 @@ test-swarm-registry.json # plan documents .plans/ .v2_plans/ + +# draft documents +.drafts/ diff --git a/spec/openapi.yaml b/spec/openapi.yaml index 207becf..d1a524b 100644 --- a/spec/openapi.yaml +++ b/spec/openapi.yaml @@ -209,6 +209,18 @@ paths: application/json: schema: $ref: '#/components/schemas/DraftDeleteResponse' + patch: + tags: + - drafts + summary: Update a specific message draft by ID + operationId: patch_draft_drafts__draft_id__patch + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/DraftPatchResponse' /drafts/{draft_id}/send: post: tags: @@ -1294,6 +1306,22 @@ components: description: 'Corresponds to `GET /drafts/{draft_id}`. Contains a specific message draft inside the user-agent''s drafts box.' + DraftPatchResponse: + properties: + entry: + $ref: '#/components/schemas/MAILDraftsEntry' + metadata: + additionalProperties: true + type: object + title: Metadata + type: object + required: + - entry + - metadata + title: DraftPatchResponse + description: 'Corresponds to `PATCH /drafts/{draft_id}`. + + Contains the updated message draft in the user-agent''s drafts box.' DraftPostResponse: properties: entry: diff --git a/src/mail/client/docs/reference/cli.md b/src/mail/client/docs/reference/cli.md index a2b9134..dcb210e 100644 --- a/src/mail/client/docs/reference/cli.md +++ b/src/mail/client/docs/reference/cli.md @@ -12,7 +12,7 @@ mail [option]... [argument]... ### Core MAIL Operations -- `compose`: Draft a new MAIL message. Accepts `--tags TAG...` to attach slug tags. +- `compose`: Draft a new MAIL message. The body may be passed inline or read from a file with `-F`/`--body-file PATH` (provide exactly one). Accepts `--tags TAG...` to attach slug tags. - `send`: Send an existing draft by ID to the specified recipient(s). Accepts `--tags TAG...`, which are merged with the draft's tags. - `reply`: Reply to an existing inbox message by ID. Addresses the reply to the original sender and defaults the subject to `Re: `. Accepts `--subject SUBJECT` and `--tags TAG...`. - `forward`: Forward an existing inbox message by ID to one or more new recipient(s). Encodes the original message (sender, recipients, subject, and body) into the forwarded body and defaults the subject to `Fwd: `. Accepts `--note NOTE` to prepend a note above the forwarded message, plus `--subject SUBJECT` and `--tags TAG...`. @@ -22,6 +22,7 @@ mail [option]... [argument]... - `outbox-open`: Open a specific message by ID in your MAIL outbox. - `drafts`: List your existing MAIL message drafts. - `drafts-open`: Open a specific existing draft by ID. +- `draft-edit`: Edit fields on an existing draft by ID. Accepts a new body inline or via `-F`/`--body-file PATH`, plus `--subject SUBJECT`, `--reply-to ID`, and `--tags TAG...` (pass `--tags` with no values to clear all tags). Omitted fields are left unchanged. - `trash`: Open your MAIL trash box. - `trash-open`: Open a specific message by ID in your MAIL trash box. diff --git a/src/mail/client/docs/tutorials/quickstart.md b/src/mail/client/docs/tutorials/quickstart.md index 8eb3d29..ae57bc8 100644 --- a/src/mail/client/docs/tutorials/quickstart.md +++ b/src/mail/client/docs/tutorials/quickstart.md @@ -61,6 +61,14 @@ MAIL_TOKEN=... \ uv run mail compose "Hello, world!" "This is a test message body" ``` +For a longer body, read it from a file instead of passing it inline with `-F`/`--body-file`: + +```bash +MAIL_SERVER=... \ +MAIL_TOKEN=... \ +uv run mail compose "Hello, world!" --body-file my-message-body.md +``` + You should then see the newly-created draft printed to the console. This includes the draft's unique ID; copy this for use in subsequent operations. diff --git a/src/mail/client/src/mail_client/cli.py b/src/mail/client/src/mail_client/cli.py index 34d660e..8e961e5 100644 --- a/src/mail/client/src/mail_client/cli.py +++ b/src/mail/client/src/mail_client/cli.py @@ -12,6 +12,7 @@ cmd_compose, cmd_drafts, cmd_drafts_open, + cmd_drafts_patch, cmd_forward, cmd_inbox, cmd_inbox_open, @@ -83,6 +84,7 @@ ("outbox-open (Oopen, Oo)", "Open an outbox message by ID."), ("drafts (d)", "List message drafts."), ("drafts-open (do)", "Open a draft by ID."), + ("draft-edit (de)", "Edit an existing draft by ID."), ("trash (t)", "List trashed messages."), ("trash-open (to)", "Open a trashed message by ID."), ], @@ -131,6 +133,23 @@ def _add_tags_arg(parser: argparse.ArgumentParser) -> None: ) +def _add_body_file_arg(parser: argparse.ArgumentParser) -> None: + """ + Register the shared ``-F``/``--body-file`` flag for draft-creating commands + (compose, draft-edit). The flag names a path whose UTF-8 contents become the + message body, as an alternative to passing the body inline. + """ + + parser.add_argument( + "-F", + "--body-file", + dest="body_file", + default=None, + metavar="PATH", + help="read the message body from the file at this path", + ) + + def _add_box_filter_args(box_parser: argparse.ArgumentParser) -> None: """ Register the shared query-param flags for the "GET box" commands @@ -224,7 +243,13 @@ def build_parser() -> argparse.ArgumentParser: description=compose_d, ) compose_p.add_argument("subject", help="the subject line of the message to draft") - compose_p.add_argument("body", help="the body of the message to draft") + compose_p.add_argument( + "body", + nargs="?", + default=None, + help="the body of the message to draft (omit when using --body-file)", + ) + _add_body_file_arg(compose_p) _add_tags_arg(compose_p) compose_p.set_defaults(func=cmd_compose, cmd="compose") @@ -355,6 +380,43 @@ def build_parser() -> argparse.ArgumentParser: drafts_open_p.add_argument("draft_id", help="the ID of the drafted message to open") drafts_open_p.set_defaults(func=cmd_drafts_open, cmd="drafts-open") + # command `draft-edit` + draft_edit_d = "edit fields on an existing message draft by ID" + draft_edit_p = subparsers.add_parser( + "draft-edit", + aliases=["de"], + prog="mail draft-edit", + help=draft_edit_d, + description=draft_edit_d, + ) + draft_edit_p.add_argument("draft_id", help="the ID of the draft to edit") + draft_edit_p.add_argument( + "body", + nargs="?", + default=None, + help="the new body of the draft (omit to leave it unchanged)", + ) + draft_edit_p.add_argument( + "--subject", + default=None, + help="the new subject of the draft (omit to leave it unchanged)", + ) + _add_body_file_arg(draft_edit_p) + draft_edit_p.add_argument( + "--reply-to", + dest="reply_to", + default=None, + help="the message ID this draft replies to (omit to leave it unchanged)", + ) + draft_edit_p.add_argument( + "--tags", + nargs="*", + default=None, + metavar="TAG", + help="replace the draft's tags (pass with no values to clear all tags)", + ) + draft_edit_p.set_defaults(func=cmd_drafts_patch, cmd="draft-edit") + # command `trash` trash_d = "list your existing trashed messages" trash_p = subparsers.add_parser( diff --git a/src/mail/client/src/mail_client/commands/__init__.py b/src/mail/client/src/mail_client/commands/__init__.py index 8914fd5..f5f5a1d 100644 --- a/src/mail/client/src/mail_client/commands/__init__.py +++ b/src/mail/client/src/mail_client/commands/__init__.py @@ -9,6 +9,7 @@ from .daemon_post import cmd_daemon_post from .drafts import cmd_drafts from .drafts_open import cmd_drafts_open +from .drafts_patch import cmd_drafts_patch from .forward import cmd_forward from .inbox import cmd_inbox from .inbox_open import cmd_inbox_open @@ -58,6 +59,7 @@ "cmd_daemon_post", "cmd_drafts", "cmd_drafts_open", + "cmd_drafts_patch", "cmd_forward", "cmd_inbox", "cmd_inbox_open", diff --git a/src/mail/client/src/mail_client/commands/_body.py b/src/mail/client/src/mail_client/commands/_body.py new file mode 100644 index 0000000..dc857b1 --- /dev/null +++ b/src/mail/client/src/mail_client/commands/_body.py @@ -0,0 +1,40 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +from pathlib import Path + + +def resolve_body(body: str | None, body_file: str | None) -> str: + """ + Resolve a message body from the mutually exclusive ``body`` (inline) and + ``body_file`` (path) CLI arguments used by ``compose``. + + Exactly one of the two must be supplied. When ``body_file`` is given, its + contents are read as UTF-8 text. Raises ``ValueError`` (surfaced by the + CLI as a command error) when neither or both are provided, or when the + file cannot be read. + """ + + if body is not None and body_file is not None: + raise ValueError("provide either a body argument or --body-file, not both") + if body is not None: + return body + if body_file is not None: + try: + return Path(body_file).read_text(encoding="utf-8") + except OSError as e: + raise ValueError(f"could not read body file {body_file!r}: {e}") + raise ValueError("a message body is required: pass it inline or via --body-file") + + +def resolve_optional_body(body: str | None, body_file: str | None) -> str | None: + """ + Variant of :func:`resolve_body` for partial-update commands (draft-edit) + where the body is optional. Returns ``None`` when neither argument is + supplied (meaning "leave the body unchanged"); otherwise behaves like + :func:`resolve_body`, including the "not both" guard. + """ + + if body is None and body_file is None: + return None + return resolve_body(body, body_file) diff --git a/src/mail/client/src/mail_client/commands/compose.py b/src/mail/client/src/mail_client/commands/compose.py index 6453e0a..06429db 100644 --- a/src/mail/client/src/mail_client/commands/compose.py +++ b/src/mail/client/src/mail_client/commands/compose.py @@ -9,6 +9,8 @@ from mail_protocol.network.responses import DraftPostResponse from pydantic import ValidationError +from mail_client.commands._body import resolve_body + def cmd_compose(args: Namespace) -> None: """ @@ -24,9 +26,10 @@ def cmd_compose(args: Namespace) -> None: raise ValueError("env var MAIL_TOKEN is required") # 2. hit the server endpoint `POST /drafts` + body = resolve_body(args.body, args.body_file) payload = DraftPostRequest( subject=args.subject, - body=args.body, + body=body, tags=args.tags, ) response = httpx.post( diff --git a/src/mail/client/src/mail_client/commands/drafts_patch.py b/src/mail/client/src/mail_client/commands/drafts_patch.py new file mode 100644 index 0000000..3193f7d --- /dev/null +++ b/src/mail/client/src/mail_client/commands/drafts_patch.py @@ -0,0 +1,86 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +import os +from argparse import Namespace + +import httpx +from mail_protocol.network.requests import DraftPatchRequest +from mail_protocol.network.responses import DraftPatchResponse +from pydantic import ValidationError + +from mail_client.commands._body import resolve_optional_body + + +def cmd_drafts_patch(args: Namespace) -> None: + """ + Update an existing message draft for the current MAIL user. + """ + + # 1. check that the required env vars are provided + MAIL_SERVER = os.getenv("MAIL_SERVER") + if MAIL_SERVER is None: + raise ValueError("env var MAIL_SERVER is required") + MAIL_TOKEN = os.getenv("MAIL_TOKEN") + if MAIL_TOKEN is None: + raise ValueError("env var MAIL_TOKEN is required") + + # 2. hit the server endpoint `PATCH /drafts/{draft_id}` + body = resolve_optional_body(args.body, args.body_file) + payload = DraftPatchRequest( + subject=args.subject, + body=body, + reply_to=args.reply_to, + tags=args.tags, + ) + response = httpx.patch( + url=f"{MAIL_SERVER}/drafts/{args.draft_id}", + headers={ + "Authorization": f"Bearer {MAIL_TOKEN}", + "User-Agent": "Multi-Agent-Interface-Layer-CLI-Client/2.0.0 (github.com/charonlabs/mail)", + "Content-Type": "application/json", + }, + json=payload.model_dump(), + ) + + # 3. parse and validate server response + if response.status_code != 200: + raise RuntimeError( + f"patch draft request to {MAIL_SERVER} failed with status code {response.status_code}" + ) + + response_json = response.json() + try: + response_obj = DraftPatchResponse.model_validate(response_json) + except ValidationError as e: + raise RuntimeError(f"response validation failed: {e}") + + # 4. print the updated draft + match args.output: + case "json": + _print_json(response_obj) + case "text": + _print_text(response_obj) + + +def _print_json(response_obj: DraftPatchResponse) -> None: + print(response_obj.model_dump_json()) + + +def _print_text(response_obj: DraftPatchResponse) -> None: + entry = response_obj.entry + draft = entry.draft + + print("=== Draft ===") + print(f"Draft ID: {draft.draft_id}") + print(f"Created At: {draft.created_at}") + print(f"Updated At: {draft.updated_at}") + print(f"Subject: {draft.subject}") + if draft.reply_to is not None: + print(f"In Reply To: {draft.reply_to}") + if draft.tags: + print(f"Tags: {', '.join(draft.tags)}") + print(f"Body:\n{draft.body}\n") + print("=== Entry Data ===") + print(f"Sent At: {entry.sent_at}") + print(f"Sent By: {entry.sent_by}") diff --git a/src/mail/protocol/src/mail_protocol/network/requests.py b/src/mail/protocol/src/mail_protocol/network/requests.py index 1777264..f30e945 100644 --- a/src/mail/protocol/src/mail_protocol/network/requests.py +++ b/src/mail/protocol/src/mail_protocol/network/requests.py @@ -68,6 +68,23 @@ class DraftPostRequest(BaseModel): tags: Annotated[list[str], AfterValidator(validate_message_tags)] = [] +class DraftPatchRequest(BaseModel): + """ + Corresponds to `PATCH /drafts/{draft_id}`. + Contains the fields to update on an existing MAIL message draft. + + Every field is optional: a field left unset (``None``) is not modified, + so callers can patch a single field without resending the rest. The one + asymmetry is ``tags`` — sending ``tags: []`` clears all tags, while + omitting ``tags`` leaves the existing tags untouched. + """ + + subject: Annotated[str, AfterValidator(validate_message_subject)] | None = None + body: Annotated[str, AfterValidator(validate_message_body)] | None = None + reply_to: Annotated[str, AfterValidator(validate_uuid)] | None = None + tags: Annotated[list[str], AfterValidator(validate_message_tags)] | None = None + + class DraftSendPostRequest(BaseModel): """ Corresponds to `POST /drafts/{draft_id}/send`. diff --git a/src/mail/protocol/src/mail_protocol/network/responses.py b/src/mail/protocol/src/mail_protocol/network/responses.py index 7028388..0489dd1 100644 --- a/src/mail/protocol/src/mail_protocol/network/responses.py +++ b/src/mail/protocol/src/mail_protocol/network/responses.py @@ -203,6 +203,16 @@ class DraftGetResponse(BaseModel): metadata: dict[str, Any] +class DraftPatchResponse(BaseModel): + """ + Corresponds to `PATCH /drafts/{draft_id}`. + Contains the updated message draft in the user-agent's drafts box. + """ + + entry: MAILDraftsEntry + metadata: dict[str, Any] + + class DraftDeleteResponse(BaseModel): """ Corresponds to `DELETE /drafts/{draft_id}`. diff --git a/src/mail/server/docs/reference/http.md b/src/mail/server/docs/reference/http.md index 2c9d5d1..17c8262 100644 --- a/src/mail/server/docs/reference/http.md +++ b/src/mail/server/docs/reference/http.md @@ -37,6 +37,7 @@ This document serves as a reference for the MAIL (Mult-Agent Interface Layer) HT - `GET /drafts/`: Get a list of message drafts in the logged-in user-agent's draft box. - `POST /drafts/`: Create a new message draft to be stored in the logged-in user-agent's draft box. - `GET /drafts/{draft_id}`: Get a specific message draft by ID from the logged-in user-agent's draft box. +- `PATCH /drafts/{draft_id}`: Update fields on a specific message draft by ID in the logged-in user-agent's draft box. - `DELETE /drafts/{draft_id}`: Delete a specific message draft by ID from the logged-in user-agent's draft box. - `POST /drafts/{draft_id}/send`: Send a message from a draft by ID in the logged-in user-agent's draft box. diff --git a/src/mail/server/src/mail_server/backends/base.py b/src/mail/server/src/mail_server/backends/base.py index a26d072..371193a 100644 --- a/src/mail/server/src/mail_server/backends/base.py +++ b/src/mail/server/src/mail_server/backends/base.py @@ -41,6 +41,7 @@ BoxFilterParams, DaemonDeliverLocalRequest, DaemonDeliverRemoteRequest, + DraftPatchRequest, DraftPostRequest, DraftSendPostRequest, ) @@ -234,6 +235,22 @@ async def get_draft( pass + @abstractmethod + async def patch_draft( + self, + user_agent: MAILUserAgent, + draft_id: str, + payload: DraftPatchRequest, + ) -> MAILDraftsEntry: + """ + Update mutable fields on an existing message draft for this user-agent. + + Only the fields supplied on ``payload`` are modified; the rest are + left untouched. ``updated_at`` is refreshed on any successful edit. + """ + + pass + @abstractmethod async def delete_draft( self, user_agent: MAILUserAgent, draft_id: str diff --git a/src/mail/server/src/mail_server/backends/memory/api.py b/src/mail/server/src/mail_server/backends/memory/api.py index 92a548e..600378b 100644 --- a/src/mail/server/src/mail_server/backends/memory/api.py +++ b/src/mail/server/src/mail_server/backends/memory/api.py @@ -40,6 +40,7 @@ BoxFilterParams, DaemonDeliverLocalRequest, DaemonDeliverRemoteRequest, + DraftPatchRequest, DraftPostRequest, DraftSendPostRequest, ) @@ -663,6 +664,56 @@ async def get_draft( return draft_entry + async def patch_draft( + self, + user_agent: MAILUserAgent, + draft_id: str, + payload: DraftPatchRequest, + ) -> MAILDraftsEntry: + """ + Update mutable fields on an existing message draft for this user-agent. + + Only the fields supplied on ``payload`` are modified; ``updated_at`` is + refreshed whenever a successful edit is applied. + """ + + ua_address = user_agent.get_address() + draft_ids = self.drafts.get(ua_address) + if draft_ids is None: + raise ValueError(f"no drafts box found for address {ua_address}") + if draft_id not in draft_ids: + raise ValueError( + f"draft with ID {draft_id} not found in draft box at address {ua_address}" + ) + + draft_entry = self.draft_entries.get(draft_id) + if draft_entry is None: + raise ValueError(f"draft with ID {draft_id} not found in draft box entries") + + # Only the fields explicitly supplied on the request are modified. A + # field left unset (``None``) is not part of the update — except + # ``tags``, where an empty list is a deliberate "clear all tags". + updated_fields: dict[str, Any] = {} + if payload.subject is not None: + updated_fields["subject"] = payload.subject + if payload.body is not None: + updated_fields["body"] = payload.body + if payload.reply_to is not None: + updated_fields["reply_to"] = payload.reply_to + if payload.tags is not None: + updated_fields["tags"] = payload.tags + + if not updated_fields: + return draft_entry + + updated_draft = draft_entry.draft.model_copy( + update={**updated_fields, "updated_at": datetime.now(UTC)} + ) + updated_entry = draft_entry.model_copy(update={"draft": updated_draft}) + self.draft_entries[draft_id] = updated_entry + + return updated_entry + async def delete_draft( self, user_agent: MAILUserAgent, draft_id: str ) -> MAILDraftsEntry: diff --git a/src/mail/server/src/mail_server/routers/drafts.py b/src/mail/server/src/mail_server/routers/drafts.py index ab6ba31..a3e7eb7 100644 --- a/src/mail/server/src/mail_server/routers/drafts.py +++ b/src/mail/server/src/mail_server/routers/drafts.py @@ -5,6 +5,7 @@ from mail_protocol.network.responses import ( DraftDeleteResponse, DraftGetResponse, + DraftPatchResponse, DraftPostResponse, DraftSendPostResponse, DraftsGetResponse, @@ -14,6 +15,7 @@ from mail_server.utils import build_box_metadata from mail_server.validators import ( validate_box_filter_params, + validate_patch_draft_request, validate_post_draft_request, validate_post_draft_send_request, ) @@ -81,6 +83,31 @@ async def get_draft(request: Request) -> DraftGetResponse: ) +@router.patch( + "/{draft_id}", + summary="Update a specific message draft by ID", + response_model=DraftPatchResponse, +) +async def patch_draft(request: Request) -> DraftPatchResponse: + backend = request.app.state.backend + user_agent = await validate_user_agent(backend=backend, request=request) + payload = await validate_patch_draft_request(request) + draft_id = request.path_params.get("draft_id") + try: + result = await backend.patch_draft( + user_agent=user_agent, draft_id=draft_id, payload=payload + ) + except ValueError: + raise HTTPException( + status_code=404, detail=f"draft with ID {draft_id} not found" + ) + + return DraftPatchResponse( + entry=result, + metadata={}, + ) + + @router.delete( "/{draft_id}", summary="Delete a specific message draft by ID", diff --git a/src/mail/server/src/mail_server/validators.py b/src/mail/server/src/mail_server/validators.py index a60e88d..4ce6f36 100644 --- a/src/mail/server/src/mail_server/validators.py +++ b/src/mail/server/src/mail_server/validators.py @@ -14,6 +14,7 @@ AuthPasswordResetRequest, BoxFilterParams, DaemonDeliverLocalRequest, + DraftPatchRequest, DraftPostRequest, DraftSendPostRequest, ListMemberPostRequest, @@ -62,6 +63,20 @@ async def validate_post_draft_request(request: Request) -> DraftPostRequest: ) +async def validate_patch_draft_request(request: Request) -> DraftPatchRequest: + """ + Ensure the request payload is valid for `PATCH /drafts/{draft_id}`. + """ + + try: + body = await request.json() + return DraftPatchRequest.model_validate(body) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"request body validation failed: {e}" + ) + + async def validate_post_draft_send_request(request: Request) -> DraftSendPostRequest: """ Ensure the request payload is valid for `POST /drafts/{draft_id}/send`. diff --git a/tests/integration/test_mailboxes.py b/tests/integration/test_mailboxes.py index d18ffa1..24964a0 100644 --- a/tests/integration/test_mailboxes.py +++ b/tests/integration/test_mailboxes.py @@ -269,6 +269,147 @@ def test_send_unknown_draft_returns_404(app_client: TestClient, headers_for) -> assert response.status_code == 404 +def test_patch_draft_updates_fields(app_client: TestClient, headers_for) -> None: + response = app_client.post( + "/drafts", + json={"subject": "Original", "body": "Original body", "tags": ["a"]}, + headers=headers_for(USER), + ) + draft = response.json()["entry"]["draft"] + draft_id = draft["draft_id"] + assert draft["updated_at"] is None + + response = app_client.patch( + f"/drafts/{draft_id}", + json={"subject": "Updated", "body": "Updated body", "tags": ["b", "c"]}, + headers=headers_for(USER), + ) + assert response.status_code == 200 + updated = response.json()["entry"]["draft"] + assert updated["subject"] == "Updated" + assert updated["body"] == "Updated body" + assert updated["tags"] == ["b", "c"] + assert updated["updated_at"] is not None + # the change is persisted, not just echoed back + response = app_client.get(f"/drafts/{draft_id}", headers=headers_for(USER)) + assert response.json()["entry"]["draft"]["subject"] == "Updated" + + +def test_patch_draft_partial_leaves_other_fields( + app_client: TestClient, headers_for +) -> None: + response = app_client.post( + "/drafts", + json={"subject": "Keep subject", "body": "Old body", "tags": ["x"]}, + headers=headers_for(USER), + ) + draft_id = response.json()["entry"]["draft"]["draft_id"] + + response = app_client.patch( + f"/drafts/{draft_id}", + json={"body": "New body"}, + headers=headers_for(USER), + ) + assert response.status_code == 200 + updated = response.json()["entry"]["draft"] + assert updated["body"] == "New body" + assert updated["subject"] == "Keep subject" + assert updated["tags"] == ["x"] + + +def test_patch_draft_empty_tags_clears_them( + app_client: TestClient, headers_for +) -> None: + response = app_client.post( + "/drafts", + json={"subject": "Subject", "body": "Body", "tags": ["x", "y"]}, + headers=headers_for(USER), + ) + draft_id = response.json()["entry"]["draft"]["draft_id"] + + response = app_client.patch( + f"/drafts/{draft_id}", + json={"tags": []}, + headers=headers_for(USER), + ) + assert response.status_code == 200 + assert response.json()["entry"]["draft"]["tags"] == [] + + +def test_patch_draft_rejects_overlong_subject( + app_client: TestClient, headers_for +) -> None: + response = app_client.post( + "/drafts", + json={"subject": "Subject", "body": "Body"}, + headers=headers_for(USER), + ) + draft_id = response.json()["entry"]["draft"]["draft_id"] + + response = app_client.patch( + f"/drafts/{draft_id}", + json={"subject": "x" * (MESSAGE_SUBJECT_LEN_MAX + 1)}, + headers=headers_for(USER), + ) + assert response.status_code == 422 + + +def test_patch_draft_unknown_id_returns_404( + app_client: TestClient, headers_for +) -> None: + response = app_client.patch( + "/drafts/11111111-1111-4111-8111-111111111111", + json={"subject": "Updated"}, + headers=headers_for(USER), + ) + assert response.status_code == 404 + + +def test_patch_draft_isolated_between_users( + app_client: TestClient, headers_for +) -> None: + response = app_client.post( + "/drafts", + json={"subject": "Private", "body": "Draft"}, + headers=headers_for(USER), + ) + draft_id = response.json()["entry"]["draft"]["draft_id"] + + response = app_client.patch( + f"/drafts/{draft_id}", + json={"subject": "Hijacked"}, + headers=headers_for(OTHER_USER), + ) + assert response.status_code == 404 + + +def test_patch_draft_then_send_uses_new_content( + app_client: TestClient, headers_for +) -> None: + response = app_client.post( + "/drafts", + json={"subject": "Original", "body": "Original body"}, + headers=headers_for(USER), + ) + draft_id = response.json()["entry"]["draft"]["draft_id"] + + app_client.patch( + f"/drafts/{draft_id}", + json={"subject": "Edited", "body": "Edited body"}, + headers=headers_for(USER), + ) + + response = app_client.post( + f"/drafts/{draft_id}/send", + json={"recipients": [OTHER_USER]}, + headers=headers_for(USER), + ) + assert response.status_code == 200 + message = response.json()["message"] + assert message["subject"] == "Edited" + assert message["body"] == "Edited body" + + # ─── Trash ───────────────────────────────────────────────────────── diff --git a/tests/unit/test_client_commands.py b/tests/unit/test_client_commands.py index 3619df6..bb5f309 100644 --- a/tests/unit/test_client_commands.py +++ b/tests/unit/test_client_commands.py @@ -18,6 +18,7 @@ import respx from mail_client.commands import ( cmd_compose, + cmd_drafts_patch, cmd_forward, cmd_inbox, cmd_login, @@ -166,7 +167,11 @@ def test_compose_posts_draft_payload(client_env, capsys: pytest.CaptureFixture) }, ) ) - cmd_compose(Namespace(output="text", subject="A subject", body="A body.", tags=[])) + cmd_compose( + Namespace( + output="text", subject="A subject", body="A body.", body_file=None, tags=[] + ) + ) request = route.calls[0].request assert request.headers["Authorization"] == f"Bearer {TOKEN}" @@ -179,12 +184,129 @@ def test_compose_posts_draft_payload(client_env, capsys: pytest.CaptureFixture) assert "Draft ID: 55555555-5555-4555-8555-555555555555" in capsys.readouterr().out +@respx.mock +def test_compose_reads_body_from_file( + client_env, tmp_path, capsys: pytest.CaptureFixture +) -> None: + body_path = tmp_path / "message.md" + body_path.write_text("Body from a file.", encoding="utf-8") + route = respx.post(f"{SERVER}/drafts").mock( + return_value=httpx.Response( + 200, + json={ + "entry": { + "draft": { + "draft_id": "55555555-5555-4555-8555-555555555555", + "subject": "A subject", + "body": "Body from a file.", + "created_at": "2026-06-12T09:00:00+00:00", + "updated_at": None, + }, + "sent_at": None, + "sent_by": None, + }, + "metadata": {}, + }, + ) + ) + cmd_compose( + Namespace( + output="text", + subject="A subject", + body=None, + body_file=str(body_path), + tags=[], + ) + ) + + assert json.loads(route.calls[0].request.content)["body"] == "Body from a file." + + +def test_compose_rejects_both_body_and_body_file(client_env, tmp_path) -> None: + body_path = tmp_path / "message.md" + body_path.write_text("From file.", encoding="utf-8") + with pytest.raises(Exception): # noqa: B017 — ValueError from resolve_body + cmd_compose( + Namespace( + output="text", + subject="A subject", + body="Inline.", + body_file=str(body_path), + tags=[], + ) + ) + + +def test_compose_rejects_missing_body(client_env) -> None: + with pytest.raises(Exception): # noqa: B017 — ValueError from resolve_body + cmd_compose( + Namespace( + output="text", subject="A subject", body=None, body_file=None, tags=[] + ) + ) + + def test_compose_rejects_invalid_subject_before_any_request(client_env) -> None: """A malformed subject fails DraftPostRequest validation locally — no request reaches the server (SPEC.md §8.1).""" with pytest.raises(Exception): # noqa: B017 — pydantic ValidationError - cmd_compose(Namespace(output="text", subject="", body="A body.", tags=[])) + cmd_compose( + Namespace( + output="text", subject="", body="A body.", body_file=None, tags=[] + ) + ) + + +# ─── draft-edit ──────────────────────────────────────────────────── + + +@respx.mock +def test_draft_edit_patches_supplied_fields( + client_env, capsys: pytest.CaptureFixture +) -> None: + draft_id = "55555555-5555-4555-8555-555555555555" + route = respx.patch(f"{SERVER}/drafts/{draft_id}").mock( + return_value=httpx.Response( + 200, + json={ + "entry": { + "draft": { + "draft_id": draft_id, + "subject": "New subject", + "body": "Old body.", + "created_at": "2026-06-12T09:00:00+00:00", + "updated_at": "2026-06-12T10:00:00+00:00", + "tags": ["x"], + }, + "sent_at": None, + "sent_by": None, + }, + "metadata": {}, + }, + ) + ) + cmd_drafts_patch( + Namespace( + output="text", + draft_id=draft_id, + subject="New subject", + body=None, + body_file=None, + reply_to=None, + tags=None, + ) + ) + + request = route.calls[0].request + assert request.method == "PATCH" + assert json.loads(request.content) == { + "subject": "New subject", + "body": None, + "reply_to": None, + "tags": None, + } + assert "Subject: New subject" in capsys.readouterr().out # ─── send ────────────────────────────────────────────────────────── From f18f730cc5b7d1a6d6cae98d3696de66dd5e8f2c Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 24 Jun 2026 17:26:00 -0400 Subject: [PATCH 12/28] docs: continuing rebase into main --- src/mail/server/docs/README.md | 1 + src/mail/server/docs/reference/backends.md | 114 ++ src/mail/server/docs/reference/cli.md | 15 +- src/mail/server/docs/tutorials/quickstart.md | 24 + src/mail/server/pyproject.toml | 2 + .../server/src/mail_server/backend_init.py | 44 +- .../mail_server/backends/sqlite/__init__.py | 2 + .../src/mail_server/backends/sqlite/api.py | 1162 +++++++++++++++++ .../mail_server/backends/sqlite/database.py | 152 +++ .../src/mail_server/backends/sqlite/init.py | 159 +++ .../mail_server/backends/sqlite/migrate.py | 223 ++++ .../backends/sqlite/repositories.py | 631 +++++++++ .../src/mail_server/backends/sqlite/schema.py | 250 ++++ .../backends/sqlite/serializers.py | 223 ++++ src/mail/server/src/mail_server/cli.py | 23 +- .../server/src/mail_server/routers/daemon.py | 14 +- .../server/src/mail_server/routers/drafts.py | 15 +- .../server/src/mail_server/routers/inbox.py | 17 +- .../server/src/mail_server/routers/trash.py | 32 +- src/mail/server/src/mail_server/server.py | 31 +- src/mail/server/src/mail_server/validators.py | 17 + tests/e2e/conftest.py | 21 +- tests/e2e/test_sqlite_durability.py | 44 + tests/integration/conftest.py | 193 ++- tests/integration/test_admin.py | 36 +- tests/integration/test_daemon.py | 25 +- tests/integration/test_gap_fill.py | 137 ++ tests/integration/test_lists.py | 127 +- tests/integration/test_mailboxes.py | 62 +- tests/integration/test_stubs.py | 14 + tests/unit/test_sqlite_backend.py | 339 +++++ tests/unit/test_sqlite_cli_wiring.py | 83 ++ tests/unit/test_sqlite_database.py | 123 ++ tests/unit/test_sqlite_init.py | 91 ++ tests/unit/test_sqlite_migrate.py | 187 +++ tests/unit/test_sqlite_repositories.py | 345 +++++ tests/unit/test_sqlite_serializers.py | 226 ++++ uv.lock | 21 + 38 files changed, 5096 insertions(+), 129 deletions(-) create mode 100644 src/mail/server/docs/reference/backends.md create mode 100644 src/mail/server/src/mail_server/backends/sqlite/__init__.py create mode 100644 src/mail/server/src/mail_server/backends/sqlite/api.py create mode 100644 src/mail/server/src/mail_server/backends/sqlite/database.py create mode 100644 src/mail/server/src/mail_server/backends/sqlite/init.py create mode 100644 src/mail/server/src/mail_server/backends/sqlite/migrate.py create mode 100644 src/mail/server/src/mail_server/backends/sqlite/repositories.py create mode 100644 src/mail/server/src/mail_server/backends/sqlite/schema.py create mode 100644 src/mail/server/src/mail_server/backends/sqlite/serializers.py create mode 100644 tests/e2e/test_sqlite_durability.py create mode 100644 tests/integration/test_gap_fill.py create mode 100644 tests/unit/test_sqlite_backend.py create mode 100644 tests/unit/test_sqlite_cli_wiring.py create mode 100644 tests/unit/test_sqlite_database.py create mode 100644 tests/unit/test_sqlite_init.py create mode 100644 tests/unit/test_sqlite_migrate.py create mode 100644 tests/unit/test_sqlite_repositories.py create mode 100644 tests/unit/test_sqlite_serializers.py diff --git a/src/mail/server/docs/README.md b/src/mail/server/docs/README.md index f1d46bb..96b3d9c 100644 --- a/src/mail/server/docs/README.md +++ b/src/mail/server/docs/README.md @@ -9,4 +9,5 @@ This document serves as the root documentation file for the `mail-swarms-server` ## Reference Docs - **`mail-server` CLI reference**: [reference/cli.md](reference/cli.md) +- **Server backends (`memory` vs `sqlite`)**: [reference/backends.md](reference/backends.md) - **MAIL HTTP API reference**: [reference/http.md](reference/http.md) diff --git a/src/mail/server/docs/reference/backends.md b/src/mail/server/docs/reference/backends.md new file mode 100644 index 0000000..dc1d1a4 --- /dev/null +++ b/src/mail/server/docs/reference/backends.md @@ -0,0 +1,114 @@ +# MAIL Server Backends + +`mail-server` stores all of its state — user-agents, swarms, messages, the four +boxes (inbox/outbox/drafts/trash), the delivery buffer, webhooks, and lists — +through a pluggable backend. Two backends ship today, selected with +`--backend` (see [cli.md](cli.md)): + +| | `memory` (default) | `sqlite` | +|---|---|---| +| Store | process-local dicts | SQLite file (SQLAlchemy async + `aiosqlite`) | +| Durability | periodic checkpoint + shutdown flush | per-commit (transactional) | +| Survives `kill -9` | only up to the last checkpoint | yes — committed writes are durable | +| Pagination / sorting | in Python over the whole box | pushed into SQL (`ORDER BY ... LIMIT`) | +| Method coverage | core; some endpoints are stubs | full parity (implements the stubs too) | +| Scaling | single process | single node | + +Both implement the same `MAILServerBackend` protocol, so the HTTP API is +identical regardless of which one is selected. + +## `memory` backend + +The default. Holds all state in process-local dictionaries and persists it to a +directory tree under `~/.mail-swarms/deployments//` on shutdown and +on a periodic checkpoint (`--memory-save-interval`, default 60s). It is the +reference proof-of-concept: simple and dependency-light, but durability is +bounded by the checkpoint interval — an abrupt `kill -9` loses everything +written since the last checkpoint. + +A handful of endpoints (`DELETE /inbox/{id}`, `DELETE /drafts/{id}`, +`DELETE /trash/{id}`, `POST /trash/clear`, `POST /daemon/deliver/remote`, +`PATCH /admin/webhooks/{id}`) raise `NotImplementedError` on this backend. + +## `sqlite` backend + +A durable, transactional backend over a single SQLite file. Every write commits +in its own short transaction, so a committed message survives an abrupt +`kill -9` — the window the memory backend's checkpoint cannot close. +Pagination, sorting, and filtering are pushed into SQL rather than loaded into +Python. It implements **every** protocol method, including the ones the memory +backend leaves as stubs, making it the more complete backend. + +### Database location + +Resolution precedence (highest first): + +1. `--database-url` / `MAIL_DATABASE_URL` — a full URL, e.g. + `sqlite:////absolute/path/mail.db`. +2. `--sqlite-path` / `MAIL_SQLITE_PATH` — a file path. +3. Default: `~/.mail-swarms/deployments/default/mail.db`. + +A `sqlite://` URL is normalized to the async `sqlite+aiosqlite://` driver +automatically, and the parent directory is created if missing. + +### Connection settings + +Each connection is opened with: + +- `journal_mode=WAL` — readers never block the writer. +- `foreign_keys=ON` — referential integrity is enforced (cascade deletes work). +- `busy_timeout=5000` — brief write contention retries for up to 5s instead of + immediately raising `database is locked`. + +### Initialization + +Provision a SQLite deployment with `backend-init --type sqlite` (same argument +surface as the memory initializer — deployment, swarm, agents, daemons, users, +admins, host): + +```bash +backend-init --type sqlite --swarm chorus --host localhost \ + --agents supervisor --users alice --admins root --daemons dummy +``` + +This creates the database file and schema and seeds the swarm and user-agents, +writing each generated password to `~/.mail-swarms/deployments//.secrets/
`. +Re-running is safe: existing swarms and user-agents are left untouched. No box +files are created — per-owner box membership is created lazily on first +delivery. + +Then run the server against the same database: + +```bash +mail-server --backend sqlite +``` + +(The startup lifespan creates the schema if it does not already exist, so +running `mail-server --backend sqlite` against a fresh path also works; use +`backend-init` when you want a seeded cast.) + +### Migrating an existing `memory` deployment + +If you already have a filesystem (`memory`) deployment, you can import it into a +new SQLite database of the same name instead of seeding a fresh cast: + +```bash +backend-init --type sqlite --import-fs +``` + +This reads the existing `~/.mail-swarms/deployments//` tree (user-agents, +swarms, messages, all four boxes with their ordering, the delivery buffer, +webhooks, and lists) and writes it into `/mail.db`. Existing +`.secrets/` files are untouched, so credentials carry over. The import runs in a +single transaction and **refuses to run against a non-empty database**, so it +can't clobber an existing SQLite deployment. + +### Single-node caveat + +SQLite serializes writers even in WAL mode, and `aiosqlite` runs each connection +on a thread, so concurrent requests can still contend on the write lock (the +`busy_timeout` retry absorbs brief contention). This makes the `sqlite` backend +a good fit for **single-node durable deployments**. Horizontal, multi-process +scaling wants a client/server database such as PostgreSQL; the URL-normalization +seam leaves the door open for a future Postgres backend, but that is not part of +this release. diff --git a/src/mail/server/docs/reference/cli.md b/src/mail/server/docs/reference/cli.md index 461ad97..c2b2de2 100644 --- a/src/mail/server/docs/reference/cli.md +++ b/src/mail/server/docs/reference/cli.md @@ -18,10 +18,21 @@ mail-server [option]... - **Example**: `mail-server --port 8000` - `-b`/`--backend`: The MAIL server backend to use. - **Default**: `memory` - - **Choices**: `memory` - - **Example**: `mail-server --backend memory` + - **Choices**: `memory`, `sqlite` + - **Example**: `mail-server --backend sqlite` + - See [backends.md](backends.md) for how the two backends differ. - `--memory-save-interval`: Seconds between memory backend filesystem checkpoints. - **Default**: `60` - **Environment**: `MAIL_MEMORY_SAVE_INTERVAL_SECONDS` - **Disable**: Set to `0` to rely only on startup/shutdown persistence. - **Example**: `mail-server --memory-save-interval 30` + - Ignored unless `--backend memory`. +- `--sqlite-path`: Path to the SQLite database file (`sqlite` backend only). + - **Default**: `~/.mail-swarms/deployments/default/mail.db` + - **Environment**: `MAIL_SQLITE_PATH` + - **Example**: `mail-server --backend sqlite --sqlite-path /var/lib/mail/mail.db` +- `--database-url`: Full database URL (`sqlite` backend only); takes precedence + over `--sqlite-path`. + - **Default**: unset (falls back to `--sqlite-path`, then the default path) + - **Environment**: `MAIL_DATABASE_URL` + - **Example**: `mail-server --backend sqlite --database-url sqlite:////abs/path/mail.db` diff --git a/src/mail/server/docs/tutorials/quickstart.md b/src/mail/server/docs/tutorials/quickstart.md index c331585..8b06c88 100644 --- a/src/mail/server/docs/tutorials/quickstart.md +++ b/src/mail/server/docs/tutorials/quickstart.md @@ -49,6 +49,30 @@ All four user-agents listed above have associated plain-text password stored in > Copy the generated passwords and keep them in a safe place. > Afterwards, remove the files generated from the backend filesystem. +## `sqlite` Backend Setup + +The `sqlite` backend is a durable, transactional alternative to `memory`: a +committed message survives an abrupt `kill -9`, not just a clean shutdown. To +initialize one, pass `--type sqlite`: + +```bash +uv run backend-init --type sqlite +``` + +This creates a SQLite database at +`~/.mail-swarms/deployments/default/mail.db` and seeds the same cast as the +`memory` initializer, writing each generated password to the printed +`.secrets/` paths. Then run the server against it: + +```bash +uv run mail-server --backend sqlite +``` + +The database path can be overridden with `--sqlite-path` / `MAIL_SQLITE_PATH` +or `--database-url` / `MAIL_DATABASE_URL`. See +[reference/backends.md](../reference/backends.md) for the full comparison, +connection settings, and the single-node caveat. + ## Running the Server With your environment variables configured, try running `mail-server`: diff --git a/src/mail/server/pyproject.toml b/src/mail/server/pyproject.toml index 46c65bf..d0733bd 100644 --- a/src/mail/server/pyproject.toml +++ b/src/mail/server/pyproject.toml @@ -9,6 +9,7 @@ authors = [ requires-python = ">=3.12" dependencies = [ "aiohttp>=3.12.15", + "aiosqlite>=0.20", "fastapi>=0.116.1", "mail-swarms-protocol==2.0.1", "pwdlib[argon2]>=0.3.0", @@ -16,6 +17,7 @@ dependencies = [ "pyjwt>=2.10.1", "python-dotenv>=1.1.1", "python-multipart>=0.0.20", + "sqlalchemy[asyncio]>=2.0", "uvicorn>=0.35.0", ] diff --git a/src/mail/server/src/mail_server/backend_init.py b/src/mail/server/src/mail_server/backend_init.py index 9bcd888..7ae5b98 100644 --- a/src/mail/server/src/mail_server/backend_init.py +++ b/src/mail/server/src/mail_server/backend_init.py @@ -2,6 +2,7 @@ # Copyright (c) 2026 Addison Kline import argparse +import asyncio from mail_protocol.cli_help import add_license_argument from mail_protocol.core.validators import ( @@ -29,7 +30,7 @@ def main() -> None: "-t", "--type", default="memory", - choices=["memory"], + choices=["memory", "sqlite"], help="the type of backend to initialize (default: %(default)s)", ) parser.add_argument( @@ -87,10 +88,20 @@ def main() -> None: default="example.com", help="the host domain or IP address to use (default: %(default)s)", ) + parser.add_argument( + "--import-fs", + action="store_true", + help=( + "for --type sqlite: import the existing filesystem (memory) " + "deployment of the same name into the new SQLite database instead " + "of seeding a fresh cast" + ), + ) # parse and handle args args = parser.parse_args() be_type = args.type + import_fs = args.import_fs deployment = args.deployment swarm = args.swarm swarm_description = args.swarm_description @@ -142,6 +153,9 @@ def main() -> None: except ValueError as e: print(f"invalid host {host}: {e}") exit(1) + if import_fs and be_type != "sqlite": + print("--import-fs is only valid with --type sqlite") + exit(1) # initialize backend match be_type: @@ -157,5 +171,33 @@ def main() -> None: admins=admins, host=host, ) + case "sqlite": + # Imported lazily so SQLAlchemy stays off the import path for + # memory-only initialization. + if import_fs: + from mail_server.backends.sqlite.migrate import ( + import_memory_deployment, + ) + + counts = asyncio.run( + import_memory_deployment(deployment=deployment) + ) + print(f"imported filesystem deployment {deployment}: {counts}") + else: + from mail_server.backends.sqlite.init import init_sqlite_backend + + asyncio.run( + init_sqlite_backend( + deployment=deployment, + swarm=swarm, + swarm_description=swarm_description, + swarm_keywords=swarm_keywords, + agents=agents, + daemons=daemons, + users=users, + admins=admins, + host=host, + ) + ) case _: raise ValueError(f"invalid backend type: {be_type}") diff --git a/src/mail/server/src/mail_server/backends/sqlite/__init__.py b/src/mail/server/src/mail_server/backends/sqlite/__init__.py new file mode 100644 index 0000000..dbdfa38 --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline diff --git a/src/mail/server/src/mail_server/backends/sqlite/api.py b/src/mail/server/src/mail_server/backends/sqlite/api.py new file mode 100644 index 0000000..7b76d3f --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/api.py @@ -0,0 +1,1162 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +``SQLiteBackend`` — a durable, transactional ``MAILServerBackend``. + +Each protocol method opens one ``Database.session()`` and delegates to the +repository layer; the session context manager owns the transaction, so +multi-write operations (``send_draft``, daemon delivery, inbox→trash moves) +commit atomically. Error semantics mirror the memory backend exactly — the same +human-readable ``ValueError`` messages the routers translate to HTTP errors and +the integration suite asserts on. + +Two deliberate refinements over the memory backend, both enabled by lazy +membership rows (no per-agent box dicts to pre-create): + +- Box reads treat "no membership rows" as an *empty box for a known agent* + rather than raising "no inbox found"; the agent is already authenticated. +- Delivery is idempotent per (owner, box, message): re-delivering the same + message to a recipient is a no-op instead of a duplicate row. + +Webhooks fire *after* the delivery transaction commits (never inside it), via +``asyncio.create_task`` — matching memory and keeping DB transactions short. + +The gap-fill methods (``delete_inbox_message``, ``delete_draft``, +``delete_trash_message``, ``clear_trash``, ``admin_webhook_patch``, +``daemon_deliver_remote``) that memory leaves as ``NotImplementedError`` are +fully implemented here. See ``src/mail/server/docs/reference/backends.md``. +""" + +from __future__ import annotations + +import asyncio +import logging +import uuid +from datetime import UTC, datetime +from typing import Any, NamedTuple + +from mail_protocol.core.constants import LIST_ADDRESS_PREFIX +from mail_protocol.core.drafts import MAILDraft, MAILDraftsEntry, MAILDraftsEntrySummary +from mail_protocol.core.inbox import MAILInboxEntry, MAILInboxEntrySummary +from mail_protocol.core.lists import MAILList, MAILListInBackend +from mail_protocol.core.messages import MAILMessage, MAILMessageSummary +from mail_protocol.core.outbox import MAILOutboxEntry, MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm, MAILSwarmSummary +from mail_protocol.core.trash import MAILTrashEntry, MAILTrashEntrySummary +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILAgent, + MAILDaemon, + MAILUser, + MAILUserAgent, + MAILUserAgentInBackend, +) +from mail_protocol.core.webhooks import MAILWebhook +from mail_protocol.network.requests import ( + AdminAgentPostRequest, + AdminDaemonPostRequest, + AdminListPatchRequest, + AdminListPostRequest, + AdminSwarmPostRequest, + AdminUserPostRequest, + AdminWebhooksPatchRequest, + AdminWebhooksPostRequest, + AuthPasswordResetRequest, + BoxFilterParams, + DaemonDeliverLocalRequest, + DaemonDeliverRemoteRequest, + DraftPatchRequest, + DraftPostRequest, + DraftSendPostRequest, +) + +from mail_server.auth import get_password_hash, verify_password +from mail_server.backends.base import MAILServerBackend +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.repositories import ( + BOX_DRAFTS, + BOX_INBOX, + BOX_OUTBOX, + BOX_TRASH, + MailStore, +) + +logger = logging.getLogger(__name__) + + +def _is_agent_recipient(address: str) -> bool: + """ + Return True iff ``address`` is an *agent* address (``name@swarm@host``). + + Mirrors the memory backend: webhooks fire only for agent recipients, never + for ``list:`` fan-out targets or 2-segment user/admin/daemon addresses. + """ + + if address.startswith(f"{LIST_ADDRESS_PREFIX}:"): + return False + return address.count("@") == 2 + + +class _WebhookFire(NamedTuple): + """A ``mail.delivered`` POST to schedule once the delivery txn commits.""" + + url: str + recipient: str + message: MAILMessage + secret: str + list_address: str | None + + +class SQLiteBackend(MAILServerBackend): + """A transactional ``MAILServerBackend`` over SQLite (SQLAlchemy async).""" + + def __init__(self, url: str) -> None: + self._db = Database(url) + # Set from the ``host`` kwarg on startup; admin CRUD builds full + # addresses as ``f"{local}@{self.host}"``, matching the memory backend. + self.host: str = "" + # Retain references to in-flight webhook tasks so they are not GC'd + # mid-flight; entries are discarded when each task completes. + self._delivery_tasks: set[asyncio.Task[None]] = set() + + # + # Lifecycle handlers + # + async def on_server_startup(self, **kwargs: Any) -> None: + logger.info("initializing sqlite backend...") + host = kwargs.get("host") + if isinstance(host, str): + self.host = host + await self._db.create_schema() + logger.info("sqlite backend initialization complete") + + async def on_server_shutdown(self, **kwargs: Any) -> None: + logger.info("shutting down sqlite backend...") + await self._db.dispose() + logger.info("sqlite backend shutdown complete") + + # + # User-agent handlers + # + async def get_user_agent(self, address: str) -> MAILUserAgentInBackend: + async with self._db.session() as session: + user_agent = await MailStore(session).user_agents.get(address) + if user_agent is None: + raise ValueError(f"user-agent with address {address} not found") + return user_agent + + async def user_agent_exists(self, address: str) -> bool: + async with self._db.session() as session: + return await MailStore(session).user_agents.exists(address) + + async def reset_password( + self, user_agent: MAILUserAgent, payload: AuthPasswordResetRequest + ) -> str: + ua_addr = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + ua_in_be = await store.user_agents.get(ua_addr) + if ua_in_be is None: + raise ValueError(f"user-agent with address {ua_addr} not found") + if not verify_password( + plain_password=payload.current_password, + hashed_password=ua_in_be.hashed_password, + ): + raise ValueError("incorrect password") + await store.user_agents.set_password( + ua_addr, get_password_hash(payload.new_password) + ) + return "success" + + # + # Swarm endpoint handlers + # + async def get_swarms(self) -> list[MAILSwarmSummary]: + async with self._db.session() as session: + swarms = await MailStore(session).swarms.list_all() + return [swarm.summarize() for swarm in swarms] + + async def get_swarm(self, swarm_name: str) -> MAILSwarm: + async with self._db.session() as session: + swarm = await MailStore(session).swarms.get(swarm_name) + if swarm is None: + raise ValueError(f"swarm with name {swarm_name} not found") + return swarm + + async def get_swarm_health(self, swarm_name: str) -> str: + async with self._db.session() as session: + swarm = await MailStore(session).swarms.get(swarm_name) + if swarm is None: + raise ValueError(f"swarm with name {swarm_name} not found") + return "ok" + + # + # Inbox endpoint handlers + # + async def get_inbox( + self, user_agent: MAILUserAgent, filters: BoxFilterParams + ) -> tuple[list[MAILInboxEntrySummary], int]: + async with self._db.session() as session: + return await MailStore(session).boxes.list_inbox( + user_agent.get_address(), filters + ) + + async def get_inbox_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILInboxEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_INBOX, message_id): + raise ValueError( + f"message with ID {message_id} not found in inbox at " + f"address {ua_address}" + ) + inbox_entry = await store.boxes.get_inbox_entry(message_id) + if inbox_entry is None: + raise ValueError( + f"message with ID {message_id} not found in inbox entries" + ) + message = await store.messages.get(message_id) + if message is None: + raise ValueError(f"message with ID {message_id} not found in messages") + return MAILInboxEntry( + message=message, + received_at=inbox_entry.received_at, + delivered_by=inbox_entry.delivered_by, + ) + + async def delete_inbox_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILInboxEntry: + """Move a message from the owner's inbox to their trash (one txn).""" + + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_INBOX, message_id): + raise ValueError( + f"message with ID {message_id} not found in inbox at " + f"address {ua_address}" + ) + inbox_entry = await store.boxes.get_inbox_entry(message_id) + if inbox_entry is None: + raise ValueError( + f"message with ID {message_id} not found in inbox entries" + ) + message = await store.messages.get(message_id) + if message is None: + raise ValueError(f"message with ID {message_id} not found in messages") + + result = MAILInboxEntry( + message=message, + received_at=inbox_entry.received_at, + delivered_by=inbox_entry.delivered_by, + ) + + trashed_at = datetime.now(UTC) + await store.boxes.remove_membership(ua_address, BOX_INBOX, message_id) + await store.boxes.add_membership( + ua_address, BOX_TRASH, message_id, trashed_at + ) + await store.boxes.upsert_trash_entry( + MAILTrashEntry(message=message, trashed_at=trashed_at) + ) + # Drop the shared inbox entry once no inbox references it anymore. + if await store.boxes.count_item_members(BOX_INBOX, message_id) == 0: + await store.boxes.delete_inbox_entry(message_id) + + return result + + # + # Outbox endpoint handlers + # + async def get_outbox( + self, user_agent: MAILUserAgent, filters: BoxFilterParams + ) -> tuple[list[MAILOutboxEntrySummary], int]: + async with self._db.session() as session: + return await MailStore(session).boxes.list_outbox( + user_agent.get_address(), filters + ) + + async def get_outbox_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILOutboxEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_OUTBOX, message_id): + raise ValueError( + f"message with ID {message_id} not found in outbox at " + f"address {ua_address}" + ) + outbox_entry = await store.boxes.get_outbox_entry(message_id) + if outbox_entry is None: + raise ValueError( + f"message with ID {message_id} not found in outbox entries" + ) + message = await store.messages.get(message_id) + if message is None: + raise ValueError(f"message with ID {message_id} not found in messages") + return MAILOutboxEntry( + message=message, + delivered_at=outbox_entry.delivered_at, + ) + + # + # Drafts box endpoints + # + async def get_drafts( + self, user_agent: MAILUserAgent, filters: BoxFilterParams + ) -> tuple[list[MAILDraftsEntrySummary], int]: + async with self._db.session() as session: + return await MailStore(session).boxes.list_drafts( + user_agent.get_address(), filters + ) + + async def post_draft( + self, user_agent: MAILUserAgent, payload: DraftPostRequest + ) -> MAILDraftsEntry: + ua_address = user_agent.get_address() + draft_id = str(uuid.uuid4()) + draft = MAILDraft( + draft_id=draft_id, + subject=payload.subject, + body=payload.body, + created_at=datetime.now(UTC), + updated_at=None, + reply_to=payload.reply_to, + tags=payload.tags, + ) + draft_entry = MAILDraftsEntry(draft=draft, sent_at=None) + async with self._db.session() as session: + store = MailStore(session) + await store.boxes.upsert_draft_entry(draft_entry) + await store.boxes.add_membership( + ua_address, BOX_DRAFTS, draft_id, draft.created_at + ) + return draft_entry + + async def get_draft( + self, user_agent: MAILUserAgent, draft_id: str + ) -> MAILDraftsEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_DRAFTS, draft_id): + raise ValueError( + f"draft with ID {draft_id} not found in draft box at " + f"address {ua_address}" + ) + draft_entry = await store.boxes.get_draft_entry(draft_id) + if draft_entry is None: + raise ValueError(f"draft with ID {draft_id} not found in draft box entries") + return draft_entry + + async def patch_draft( + self, + user_agent: MAILUserAgent, + draft_id: str, + payload: DraftPatchRequest, + ) -> MAILDraftsEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_DRAFTS, draft_id): + raise ValueError( + f"draft with ID {draft_id} not found in draft box at " + f"address {ua_address}" + ) + draft_entry = await store.boxes.get_draft_entry(draft_id) + if draft_entry is None: + raise ValueError( + f"draft with ID {draft_id} not found in draft box entries" + ) + + # Only fields explicitly supplied are modified. ``tags=[]`` is a + # deliberate "clear all tags"; an unset field (None) is left alone. + updated_fields: dict[str, Any] = {} + if payload.subject is not None: + updated_fields["subject"] = payload.subject + if payload.body is not None: + updated_fields["body"] = payload.body + if payload.reply_to is not None: + updated_fields["reply_to"] = payload.reply_to + if payload.tags is not None: + updated_fields["tags"] = payload.tags + + if not updated_fields: + return draft_entry + + updated_draft = draft_entry.draft.model_copy( + update={**updated_fields, "updated_at": datetime.now(UTC)} + ) + updated_entry = draft_entry.model_copy(update={"draft": updated_draft}) + await store.boxes.upsert_draft_entry(updated_entry) + return updated_entry + + async def delete_draft( + self, user_agent: MAILUserAgent, draft_id: str + ) -> MAILDraftsEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_DRAFTS, draft_id): + raise ValueError( + f"draft with ID {draft_id} not found in draft box at " + f"address {ua_address}" + ) + draft_entry = await store.boxes.get_draft_entry(draft_id) + if draft_entry is None: + raise ValueError( + f"draft with ID {draft_id} not found in draft box entries" + ) + await store.boxes.remove_membership(ua_address, BOX_DRAFTS, draft_id) + await store.boxes.delete_draft_entry(draft_id) + return draft_entry + + async def send_draft( + self, + user_agent: MAILUserAgent, + draft_id: str, + payload: DraftSendPostRequest, + ) -> MAILMessage: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_DRAFTS, draft_id): + raise ValueError( + f"draft with ID {draft_id} not found in draft box at " + f"address {ua_address}" + ) + draft_entry = await store.boxes.get_draft_entry(draft_id) + if draft_entry is None: + raise ValueError( + f"draft with ID {draft_id} not found in draft box entries" + ) + draft = draft_entry.draft + + message_id = str(uuid.uuid4()) # distinct from draft_id + # Draft tags + send-time tags, order-preserving union. + tags = list(draft.tags) + for tag in payload.tags: + if tag not in tags: + tags.append(tag) + now = datetime.now(UTC) + message = MAILMessage( + mail_version="2.0", + message_id=message_id, + reply_to=draft.reply_to, + sender=ua_address, + recipients=payload.recipients, + subject=draft.subject, + body=draft.body, + tags=tags, + sent_at=now, + metadata={}, + ) + outbox_entry = MAILOutboxEntrySummary( + message_id=message_id, + recipients=message.recipients, + subject=message.subject, + body_size=len(message.body), + sent_at=now, + delivered_at=None, + delivered_by=None, + ) + + await store.messages.add(message) + await store.boxes.upsert_outbox_entry(outbox_entry) + await store.boxes.add_membership(ua_address, BOX_OUTBOX, message_id, now) + await store.buffer.enqueue(message_id) + return message + + # + # Trash box endpoints + # + async def get_trash( + self, user_agent: MAILUserAgent, filters: BoxFilterParams + ) -> tuple[list[MAILTrashEntrySummary], int]: + async with self._db.session() as session: + return await MailStore(session).boxes.list_trash( + user_agent.get_address(), filters + ) + + async def get_trash_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILTrashEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_TRASH, message_id): + raise ValueError( + f"message with ID {message_id} not found in trash box at " + f"address {ua_address}" + ) + trash_entry = await store.boxes.get_trash_entry(message_id) + if trash_entry is None: + raise ValueError( + f"message with ID {message_id} not found in trash entries" + ) + return trash_entry + + async def delete_trash_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILTrashEntry: + """Hard-delete a trashed message from the owner's trash box.""" + + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_TRASH, message_id): + raise ValueError( + f"message with ID {message_id} not found in trash box at " + f"address {ua_address}" + ) + trash_entry = await store.boxes.get_trash_entry(message_id) + if trash_entry is None: + raise ValueError( + f"message with ID {message_id} not found in trash entries" + ) + await store.boxes.remove_membership(ua_address, BOX_TRASH, message_id) + # Drop the shared trash entry once no trash references it; the + # canonical ``messages`` row is retained (mirrors memory, which + # never deletes messages). + if await store.boxes.count_item_members(BOX_TRASH, message_id) == 0: + await store.boxes.delete_trash_entry(message_id) + return trash_entry + + async def clear_trash( + self, user_agent: MAILUserAgent + ) -> list[MAILTrashEntrySummary]: + ua_address = user_agent.get_address() + summaries: list[MAILTrashEntrySummary] = [] + async with self._db.session() as session: + store = MailStore(session) + for message_id in await store.boxes.list_item_ids(ua_address, BOX_TRASH): + trash_entry = await store.boxes.get_trash_entry(message_id) + if trash_entry is not None: + summaries.append(trash_entry.summarize()) + await store.boxes.remove_membership(ua_address, BOX_TRASH, message_id) + if await store.boxes.count_item_members(BOX_TRASH, message_id) == 0: + await store.boxes.delete_trash_entry(message_id) + return summaries + + # + # Daemon-only endpoints + # + async def daemon_clear_message_buffer(self, daemon: MAILDaemon) -> list[str]: + async with self._db.session() as session: + return await MailStore(session).buffer.drain() + + async def daemon_deliver_local( + self, daemon: MAILDaemon, payload: DaemonDeliverLocalRequest + ) -> list[MAILMessageSummary]: + delivered: list[MAILMessageSummary] = [] + fires: list[_WebhookFire] = [] + async with self._db.session() as session: + store = MailStore(session) + webhooks = await self._delivered_webhooks(store) + for message_id in payload.message_ids: + message = await store.messages.get(message_id) + if message is None: + logger.warning(f"failed to get message by ID {message_id}") + continue + + delivered_time = datetime.now(UTC) + # Mark the shared outbox entry delivered (it must already exist). + outbox_entry = await store.boxes.get_outbox_entry(message_id) + if outbox_entry is not None: + outbox_entry.delivered_at = delivered_time + outbox_entry.delivered_by = daemon.get_address() + await store.boxes.upsert_outbox_entry(outbox_entry) + + await self._deliver_one( + store, + daemon=daemon, + message=message, + delivered_time=delivered_time, + webhooks=webhooks, + fires=fires, + ) + delivered.append(message.summarize()) + self._schedule_webhooks(fires) + return delivered + + async def daemon_deliver_remote( + self, daemon: MAILDaemon, payload: DaemonDeliverRemoteRequest + ) -> list[MAILMessageSummary]: + """ + Deliver messages authored by *remote* agents to local recipients. + + Mirrors ``daemon_deliver_local`` but the messages arrive in full from + off-server, so they are persisted into the canonical ``messages`` store + first; there is no local outbox to update (the sender is remote). + + TODO: the ``/daemon/deliver/remote`` HTTP route is still a + router-level ``NotImplementedError`` stub, so this method is currently + reachable only via direct backend calls/tests, not over the wire. + """ + + delivered: list[MAILMessageSummary] = [] + fires: list[_WebhookFire] = [] + async with self._db.session() as session: + store = MailStore(session) + webhooks = await self._delivered_webhooks(store) + for message in payload.messages: + if await store.messages.get(message.message_id) is None: + await store.messages.add(message) + await self._deliver_one( + store, + daemon=daemon, + message=message, + delivered_time=datetime.now(UTC), + webhooks=webhooks, + fires=fires, + ) + delivered.append(message.summarize()) + self._schedule_webhooks(fires) + return delivered + + # + # Delivery helpers (session-scoped; webhooks collected, fired post-commit) + # + async def _delivered_webhooks(self, store: MailStore) -> list[MAILWebhook]: + return [ + wh + for wh in await store.webhooks.list_all() + if "mail.delivered" in wh.events + ] + + async def _deliver_one( + self, + store: MailStore, + *, + daemon: MAILDaemon, + message: MAILMessage, + delivered_time: datetime, + webhooks: list[MAILWebhook], + fires: list[_WebhookFire], + ) -> None: + """Upsert the shared inbox entry and deliver to each recipient.""" + + inbox_entry = MAILInboxEntrySummary( + message_id=message.message_id, + sender=message.sender, + subject=message.subject, + body_size=len(message.body), + received_at=delivered_time, + delivered_by=daemon.get_address(), + ) + await store.boxes.upsert_inbox_entry(inbox_entry) + + for recipient in message.recipients: + if recipient.startswith(f"{LIST_ADDRESS_PREFIX}:"): + await self._fan_out_to_list( + store, + list_address=recipient, + message=message, + delivered_time=delivered_time, + webhooks=webhooks, + fires=fires, + ) + continue + await self._deliver_to_address( + store, + address=recipient, + message=message, + delivered_time=delivered_time, + list_address=None, + webhooks=webhooks, + fires=fires, + ) + + async def _deliver_to_address( + self, + store: MailStore, + *, + address: str, + message: MAILMessage, + delivered_time: datetime, + list_address: str | None, + webhooks: list[MAILWebhook], + fires: list[_WebhookFire], + ) -> None: + user_agent = await store.user_agents.get(address) + if user_agent is None: + logger.warning(f"failed to validate recipient address {address}") + return + ua_address = user_agent.get_address() + # Idempotent: re-delivering the same message is a no-op. + if not await store.boxes.is_member(ua_address, BOX_INBOX, message.message_id): + await store.boxes.add_membership( + ua_address, BOX_INBOX, message.message_id, delivered_time + ) + + # ``mail.delivered`` is agent-scoped: only ``name@swarm@host`` recipients + # carry the swarm the webhook payload requires. + if user_agent.user_agent.ua_type != "agent" or not _is_agent_recipient( + address + ): + logger.debug( + f"skipping `mail.delivered` webhooks for non-agent recipient {address}" + ) + return + for webhook in webhooks: + fires.append( + _WebhookFire( + url=webhook.url, + recipient=address, + message=message, + secret=webhook.secret, + list_address=list_address, + ) + ) + + async def _fan_out_to_list( + self, + store: MailStore, + *, + list_address: str, + message: MAILMessage, + delivered_time: datetime, + webhooks: list[MAILWebhook], + fires: list[_WebhookFire], + ) -> None: + mail_list = await store.lists.get_by_address(list_address) + if mail_list is None: + logger.warning( + f"unknown list address in recipients; skipping: {list_address}" + ) + return + for member in mail_list.members: + if member.startswith(f"{LIST_ADDRESS_PREFIX}:"): + logger.warning( + f"nested list members are not supported in v1; " + f"skipping {member!r} in {list_address!r}" + ) + continue + await self._deliver_to_address( + store, + address=member, + message=message, + delivered_time=delivered_time, + list_address=list_address, + webhooks=webhooks, + fires=fires, + ) + + def _schedule_webhooks(self, fires: list[_WebhookFire]) -> None: + """Fire collected ``mail.delivered`` POSTs after the txn has committed.""" + + for fire in fires: + task = asyncio.create_task( + self.handle_webhook_delivered_for_url( + url=fire.url, + recipient=fire.recipient, + message=fire.message, + secret=fire.secret, + list_address=fire.list_address, + ) + ) + self._delivery_tasks.add(task) + task.add_done_callback(self._delivery_tasks.discard) + + # + # Administrator endpoints — agents + # + async def admin_get_agents(self, admin: MAILAdmin) -> list[str]: + async with self._db.session() as session: + agents = await MailStore(session).user_agents.list_by_type("agent") + local_addrs: list[str] = [] + for agent in agents: + name, swarm, _host = agent.get_address().split("@") + local_addrs.append(f"{name}@{swarm}") + return local_addrs + + async def admin_get_agent( + self, admin: MAILAdmin, agent_address: str + ) -> MAILAgent: + full_address = f"{agent_address}@{self.host}" + async with self._db.session() as session: + agent = await MailStore(session).user_agents.get(full_address) + if agent is None: + raise ValueError(f"no agent found with address {agent_address}") + inner = agent.user_agent + if not isinstance(inner, MAILAgent): + raise ValueError(f"invalid agent address: {agent_address}") + return inner + + async def admin_post_agent( + self, admin: MAILAdmin, payload: AdminAgentPostRequest + ) -> MAILAgent: + full_address = f"{payload.agent_name}@{payload.swarm_name}@{self.host}" + agent = MAILAgent( + ua_type="agent", + name=payload.agent_name, + swarm=payload.swarm_name, + host=self.host, + ) + async with self._db.session() as session: + store = MailStore(session) + if await store.user_agents.exists(full_address): + raise ValueError(f"agent address already taken: {full_address}") + await store.user_agents.add( + MAILUserAgentInBackend( + user_agent=agent, + hashed_password=get_password_hash(payload.agent_password), + ) + ) + return agent + + async def admin_delete_agent( + self, admin: MAILAdmin, agent_address: str + ) -> MAILAgent: + full_address = f"{agent_address}@{self.host}" + async with self._db.session() as session: + store = MailStore(session) + agent = await store.user_agents.get(full_address) + if agent is None: + raise ValueError(f"agent not found: {agent_address}") + inner = agent.user_agent + if not isinstance(inner, MAILAgent): + raise ValueError(f"invalid agent address: {agent_address}") + await store.user_agents.delete(full_address) + return inner + + # + # Administrator endpoints — daemons + # + async def admin_get_daemons(self, admin: MAILAdmin) -> list[str]: + async with self._db.session() as session: + daemons = await MailStore(session).user_agents.list_by_type("daemon") + worker_names: list[str] = [] + for daemon in daemons: + name, _host = daemon.get_address().split("@") + worker_names.append(name.removeprefix("daemon:")) + return worker_names + + async def admin_get_daemon( + self, admin: MAILAdmin, worker_name: str + ) -> MAILDaemon: + full_address = f"daemon:{worker_name}@{self.host}" + async with self._db.session() as session: + daemon = await MailStore(session).user_agents.get(full_address) + if daemon is None: + raise ValueError(f"no daemon found with worker name {worker_name}") + inner = daemon.user_agent + if not isinstance(inner, MAILDaemon): + raise ValueError(f"invalid worker name: {worker_name}") + return inner + + async def admin_post_daemon( + self, admin: MAILAdmin, payload: AdminDaemonPostRequest + ) -> MAILDaemon: + full_address = f"daemon:{payload.worker_name}@{self.host}" + daemon = MAILDaemon( + ua_type="daemon", + worker_name=payload.worker_name, + host=self.host, + ) + async with self._db.session() as session: + store = MailStore(session) + if await store.user_agents.exists(full_address): + raise ValueError(f"daemon address already taken: {full_address}") + await store.user_agents.add( + MAILUserAgentInBackend( + user_agent=daemon, + hashed_password=get_password_hash(payload.daemon_password), + ) + ) + return daemon + + async def admin_delete_daemon( + self, admin: MAILAdmin, worker_name: str + ) -> MAILDaemon: + full_address = f"daemon:{worker_name}@{self.host}" + async with self._db.session() as session: + store = MailStore(session) + daemon = await store.user_agents.get(full_address) + if daemon is None: + raise ValueError(f"daemon not found: {worker_name}") + inner = daemon.user_agent + if not isinstance(inner, MAILDaemon): + raise ValueError(f"invalid daemon worker name: {worker_name}") + await store.user_agents.delete(full_address) + return inner + + # + # Administrator endpoints — users + # + async def admin_get_users(self, admin: MAILAdmin) -> list[str]: + async with self._db.session() as session: + users = await MailStore(session).user_agents.list_by_type("user") + user_ids: list[str] = [] + for user in users: + name, _host = user.get_address().split("@") + user_ids.append(name.removeprefix("user:")) + return user_ids + + async def admin_get_user(self, admin: MAILAdmin, user_id: str) -> MAILUser: + full_address = f"user:{user_id}@{self.host}" + async with self._db.session() as session: + user = await MailStore(session).user_agents.get(full_address) + if user is None: + raise ValueError(f"no user found with ID {user_id}") + inner = user.user_agent + if not isinstance(inner, MAILUser): + raise ValueError(f"invalid user ID: {user_id}") + return inner + + async def admin_post_user( + self, admin: MAILAdmin, payload: AdminUserPostRequest + ) -> MAILUser: + full_address = f"user:{payload.user_id}@{self.host}" + user = MAILUser(ua_type="user", user_id=payload.user_id, host=self.host) + async with self._db.session() as session: + store = MailStore(session) + if await store.user_agents.exists(full_address): + raise ValueError(f"user address already taken: {full_address}") + await store.user_agents.add( + MAILUserAgentInBackend( + user_agent=user, + hashed_password=get_password_hash(payload.user_password), + ) + ) + return user + + async def admin_delete_user(self, admin: MAILAdmin, user_id: str) -> MAILUser: + full_address = f"user:{user_id}@{self.host}" + async with self._db.session() as session: + store = MailStore(session) + user = await store.user_agents.get(full_address) + if user is None: + raise ValueError(f"user not found: {user_id}") + inner = user.user_agent + if not isinstance(inner, MAILUser): + raise ValueError(f"invalid user ID: {user_id}") + await store.user_agents.delete(full_address) + return inner + + # + # Administrator endpoints — swarms + # + async def admin_post_swarm( + self, admin: MAILAdmin, payload: AdminSwarmPostRequest + ) -> MAILSwarm: + new_swarm = MAILSwarm( + name=payload.name, + description=payload.description, + keywords=payload.keywords, + agents=[], + metadata={}, + ) + async with self._db.session() as session: + store = MailStore(session) + if await store.swarms.get(payload.name) is not None: + raise ValueError(f"swarm with name {payload.name} already exists") + await store.swarms.add(new_swarm) + return new_swarm + + async def admin_delete_swarm( + self, admin: MAILAdmin, swarm_name: str + ) -> MAILSwarm: + async with self._db.session() as session: + store = MailStore(session) + swarm = await store.swarms.delete(swarm_name) + if swarm is None: + raise ValueError(f"swarm with name {swarm_name} not found") + return swarm + + # + # Webhook handlers + # + async def admin_webhooks_get(self, admin: MAILAdmin) -> list[str]: + async with self._db.session() as session: + webhooks = await MailStore(session).webhooks.list_all() + return [wh.webhook_id for wh in webhooks] + + async def admin_webhook_get( + self, admin: MAILAdmin, webhook_id: str + ) -> MAILWebhook: + async with self._db.session() as session: + webhook = await MailStore(session).webhooks.get_by_id(webhook_id) + if webhook is None: + raise ValueError(f"webhook with ID {webhook_id} not found") + return webhook + + async def admin_webhook_post( + self, admin: MAILAdmin, payload: AdminWebhooksPostRequest + ) -> MAILWebhook: + async with self._db.session() as session: + store = MailStore(session) + # Idempotent on URL: an existing webhook is returned unchanged. + existing = await store.webhooks.get_by_url(payload.url) + if existing is not None: + return existing + new_webhook = MAILWebhook( + webhook_id=f"wh_{uuid.uuid4()}", + url=payload.url, + events=payload.events, + secret=payload.secret, + ) + await store.webhooks.add(new_webhook) + return new_webhook + + async def admin_webhook_patch( + self, + admin: MAILAdmin, + webhook_id: str, + payload: AdminWebhooksPatchRequest, + ) -> MAILWebhook: + """Update an existing webhook's URL and/or secret, preserving its id.""" + + async with self._db.session() as session: + store = MailStore(session) + existing = await store.webhooks.get_by_id(webhook_id) + if existing is None: + raise ValueError(f"webhook with ID {webhook_id} not found") + updated = existing.model_copy( + update={"url": payload.url, "secret": payload.secret} + ) + # The table is keyed by URL; delete + re-add handles both an + # in-place secret change and a URL (PK) move uniformly. + await store.webhooks.delete_by_url(existing.url) + await store.webhooks.add(updated) + return updated + + async def admin_webhook_delete( + self, admin: MAILAdmin, webhook_id: str + ) -> MAILWebhook: + async with self._db.session() as session: + store = MailStore(session) + webhook = await store.webhooks.get_by_id(webhook_id) + if webhook is None: + raise ValueError(f"webhook with ID {webhook_id} not found") + await store.webhooks.delete_by_url(webhook.url) + return webhook + + # + # List endpoints + # + async def get_lists(self) -> list[MAILListInBackend]: + async with self._db.session() as session: + return await MailStore(session).lists.list_all() + + async def get_list(self, list_address: str) -> MAILListInBackend: + async with self._db.session() as session: + mail_list = await MailStore(session).lists.get_by_address(list_address) + if mail_list is None: + raise ValueError(f"list not found: {list_address}") + return mail_list + + async def admin_get_lists(self, admin: MAILAdmin) -> list[MAILListInBackend]: + return await self.get_lists() + + async def admin_get_list( + self, admin: MAILAdmin, list_address: str + ) -> MAILListInBackend: + return await self.get_list(list_address) + + async def admin_post_list( + self, admin: MAILAdmin, payload: AdminListPostRequest + ) -> MAILListInBackend: + mail_list = MAILList( + name=payload.name, + swarm=payload.swarm_name, + host=self.host, + owner=payload.owner, + members=payload.members, + policy=payload.policy, + ) + address = mail_list.get_address() + now = datetime.now(UTC) + record = MAILListInBackend( + **mail_list.model_dump(), + list_id=str(uuid.uuid4()), + created_at=now, + updated_at=now, + ) + async with self._db.session() as session: + store = MailStore(session) + if await store.lists.get_by_address(address) is not None: + raise ValueError(f"list address already taken: {address}") + await store.lists.add(record) + return record + + async def admin_patch_list( + self, + admin: MAILAdmin, + list_address: str, + payload: AdminListPatchRequest, + ) -> MAILListInBackend: + async with self._db.session() as session: + store = MailStore(session) + existing = await store.lists.get_by_address(list_address) + if existing is None: + raise ValueError(f"list not found: {list_address}") + if payload.policy is None: + return existing + updated = existing.model_copy( + update={"policy": payload.policy, "updated_at": datetime.now(UTC)} + ) + await store.lists.update(updated) + return updated + + async def admin_delete_list( + self, admin: MAILAdmin, list_address: str + ) -> MAILListInBackend: + async with self._db.session() as session: + mail_list = await MailStore(session).lists.delete(list_address) + if mail_list is None: + raise ValueError(f"list not found: {list_address}") + return mail_list + + async def add_list_member( + self, list_address: str, member_address: str + ) -> MAILListInBackend: + async with self._db.session() as session: + store = MailStore(session) + existing = await store.lists.get_by_address(list_address) + if existing is None: + raise ValueError(f"list not found: {list_address}") + if member_address in existing.members: + return existing + updated = existing.model_copy( + update={ + "members": [*existing.members, member_address], + "updated_at": datetime.now(UTC), + } + ) + await store.lists.update(updated) + return updated + + async def remove_list_member( + self, list_address: str, member_address: str + ) -> MAILListInBackend: + async with self._db.session() as session: + store = MailStore(session) + existing = await store.lists.get_by_address(list_address) + if existing is None: + raise ValueError(f"list not found: {list_address}") + if member_address not in existing.members: + return existing + updated = existing.model_copy( + update={ + "members": [m for m in existing.members if m != member_address], + "updated_at": datetime.now(UTC), + } + ) + await store.lists.update(updated) + return updated + + # + # Message endpoints + # + async def get_message(self, message_id: str) -> MAILMessage: + async with self._db.session() as session: + message = await MailStore(session).messages.get(message_id) + if message is None: + raise ValueError(f"undefined message ID: {message_id}") + return message diff --git a/src/mail/server/src/mail_server/backends/sqlite/database.py b/src/mail/server/src/mail_server/backends/sqlite/database.py new file mode 100644 index 0000000..f0cb552 --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/database.py @@ -0,0 +1,152 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Database plumbing for the MAIL SQLite backend. + +``Database`` owns the async engine, the session factory, and the SQLite +pragmas. It mirrors the SQLAlchemy async stack used in chorus' ``db`` package: +``create_async_engine`` over ``sqlite+aiosqlite``, an ``async_sessionmaker`` +with ``expire_on_commit=False``, WAL + ``foreign_keys=ON`` + ``busy_timeout`` +configured via a ``connect`` event listener, and a ``session()`` context +manager that commits on success and rolls back on error. + +All backend mutations are expected to run inside a single ``session()`` block so +that multi-step operations (e.g. ``send_draft``: insert message + outbox entry + +membership + buffer row) commit atomically. Keep session scopes short and never +hold a transaction open across non-DB ``await``s — in particular, webhook POSTs +must fire *after* the delivery transaction commits, never inside it. + +See ``src/mail/server/docs/reference/backends.md`` for the backend overview. +""" + +from __future__ import annotations + +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager +from pathlib import Path +from typing import Protocol + +from sqlalchemy import event +from sqlalchemy.engine import Connection, make_url +from sqlalchemy.ext.asyncio import ( + AsyncEngine, + AsyncSession, + async_sessionmaker, + create_async_engine, +) + +from mail_server.backends.sqlite.schema import Base + +# Milliseconds SQLite waits on a held write lock before raising +# ``database is locked``. WAL lets readers proceed without blocking the writer, +# but writers still serialize; a non-zero busy timeout turns brief contention +# into a retry instead of an immediate error. +_BUSY_TIMEOUT_MS = 5000 + + +class _Cursor(Protocol): + def execute(self, statement: str) -> object: ... + + def close(self) -> None: ... + + +class _SQLiteConnection(Protocol): + def cursor(self) -> _Cursor: ... + + +class Database: + """Async engine + session factory for the SQLite backend.""" + + def __init__(self, url: str): + self.url = normalize_database_url(url) + _ensure_sqlite_parent(self.url) + self.engine = create_async_engine(self.url) + _configure_sqlite(self.engine) + self._sessions = async_sessionmaker( + self.engine, + expire_on_commit=False, + ) + + async def create_schema(self) -> None: + """Create every table/index, then run additive forward-compat guards.""" + + async with self.engine.begin() as connection: + await connection.run_sync(Base.metadata.create_all) + await connection.run_sync(_ensure_schema_columns) + + @asynccontextmanager + async def session(self) -> AsyncIterator[AsyncSession]: + """Yield a session that commits on success and rolls back on error.""" + + async with self._sessions() as session: + try: + yield session + await session.commit() + except Exception: + await session.rollback() + raise + + async def dispose(self) -> None: + """Dispose the engine and its connection pool (server shutdown).""" + + await self.engine.dispose() + + +def normalize_database_url(url: str) -> str: + """ + Map a plain driver URL onto its async driver. + + ``sqlite://`` → ``sqlite+aiosqlite://``. The ``postgresql://`` → + ``postgresql+psycopg://`` rewrite is kept as the seam for a future Postgres + backend (out of scope here) so the same repositories can target it later. + """ + + if url.startswith("sqlite://") and not url.startswith("sqlite+"): + return url.replace("sqlite://", "sqlite+aiosqlite://", 1) + if url.startswith("postgresql://"): + return url.replace("postgresql://", "postgresql+psycopg://", 1) + return url + + +def _ensure_schema_columns(connection: Connection) -> None: + """ + Additive, ``create_all``-friendly schema guard. + + This is the forward-compatibility hook mirroring chorus' approach: when a + queryable column is added to a ``*Row`` table in a later release, add an + idempotent ``ALTER TABLE ... ADD COLUMN`` here so existing databases pick it + up without a migration framework. There are no such additions yet, so this + is currently a no-op. + """ + + del connection # no additive columns yet; hook retained for forward-compat + + +def _ensure_sqlite_parent(url: str) -> None: + """Create the parent directory for a file-backed SQLite database.""" + + parsed = make_url(url) + if not parsed.drivername.startswith("sqlite"): + return + if parsed.database is None or parsed.database == ":memory:": + return + Path(parsed.database).expanduser().parent.mkdir(parents=True, exist_ok=True) + + +def _configure_sqlite(engine: AsyncEngine) -> None: + """Apply WAL, foreign-key enforcement, and a busy timeout per connection.""" + + if not engine.url.drivername.startswith("sqlite"): + return + + @event.listens_for(engine.sync_engine, "connect") + def _set_sqlite_pragmas( + dbapi_connection: _SQLiteConnection, + _connection_record: object, + ) -> None: + cursor = dbapi_connection.cursor() + cursor.execute("PRAGMA foreign_keys=ON") + cursor.execute("PRAGMA journal_mode=WAL") + cursor.execute(f"PRAGMA busy_timeout={_BUSY_TIMEOUT_MS}") + cursor.close() diff --git a/src/mail/server/src/mail_server/backends/sqlite/init.py b/src/mail/server/src/mail_server/backends/sqlite/init.py new file mode 100644 index 0000000..89e98fb --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/init.py @@ -0,0 +1,159 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +One-time initialization for the SQLite backend, used by ``backend-init``. + +Creates the deployment's database file + schema and seeds the same initial +state ``init_memory_backend`` writes — one swarm plus the requested agents / +daemons / users / admins, each with a generated password (hashed via +``PasswordHash``) and its plaintext written to the existing +``/.secrets/
`` files so the rest of the tooling is +unchanged. No empty box files are needed: ``mailbox_items`` membership is +created lazily on first delivery. + +Re-running against an existing deployment is safe — the swarm and any +user-agents that already exist are left untouched (their stored password hash +and secret file are preserved) rather than duplicated. +""" + +from __future__ import annotations + +import secrets +from pathlib import Path + +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILAgent, + MAILDaemon, + MAILUser, + MAILUserAgentInBackend, +) +from pwdlib import PasswordHash + +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.repositories import MailStore + +# The concrete user-agent variants ``backend-init`` seeds (the members of the +# ``MAILUserAgent.user_agent`` discriminated union). +type _UserAgentVariant = MAILAgent | MAILUser | MAILAdmin | MAILDaemon + + +def default_sqlite_path(deployment: str = "default") -> Path: + """Default DB file for a deployment: ``~/.mail-swarms/...//mail.db``.""" + + return ( + Path.home() + .joinpath(".mail-swarms", "deployments", deployment, "mail.db") + ) + + +async def _seed_user_agent( + store: MailStore, + password_hash: PasswordHash, + secrets_path: Path, + user_agent: _UserAgentVariant, + label: str, +) -> None: + """Insert one user-agent (if absent) and write its plaintext secret.""" + + address = user_agent.get_address() + if await store.user_agents.exists(address): + print(f"{label} already exists, skipping: {address}") + return + + password = secrets.token_urlsafe(32) + await store.user_agents.add( + MAILUserAgentInBackend( + user_agent=user_agent, + hashed_password=password_hash.hash(password), + ) + ) + secrets_path.joinpath(address).write_text(password, encoding="utf-8") + print(f"wrote new {label}: {address} (secret in {secrets_path})") + + +async def init_sqlite_backend( + deployment: str = "default", + swarm: str = "default", + swarm_description: str = "A MAIL swarm", + swarm_keywords: list[str] = [], + agents: list[str] = ["supervisor"], + daemons: list[str] = ["dummy"], + users: list[str] = ["dummy"], + admins: list[str] = ["dummy"], + host: str = "example.com", + db_path: Path | None = None, +) -> None: + """Initialize a fresh SQLite backend for ``mail-server``.""" + + db_path = db_path or default_sqlite_path(deployment) + deployment_path = db_path.parent + secrets_path = deployment_path.joinpath(".secrets") + secrets_path.mkdir(parents=True, exist_ok=True) + print(f"ensured deployment path: {deployment_path}") + print(f"ensured secrets path: {secrets_path}") + + # ``Database`` creates the db file's parent dir and applies the schema. + db = Database(f"sqlite:///{db_path}") + await db.create_schema() + print(f"ensured sqlite database + schema: {db_path}") + + password_hash = PasswordHash.recommended() + try: + async with db.session() as session: + store = MailStore(session) + + if await store.swarms.get(swarm) is None: + await store.swarms.add( + MAILSwarm( + name=swarm, + description=swarm_description, + keywords=swarm_keywords, + agents=agents, + metadata={}, + ) + ) + print(f"wrote swarm: {swarm}") + else: + print(f"swarm already exists, skipping: {swarm}") + + for agent_name in agents: + await _seed_user_agent( + store, + password_hash, + secrets_path, + MAILAgent( + ua_type="agent", name=agent_name, swarm=swarm, host=host + ), + "agent", + ) + for daemon_name in daemons: + await _seed_user_agent( + store, + password_hash, + secrets_path, + MAILDaemon(ua_type="daemon", worker_name=daemon_name, host=host), + "daemon", + ) + for user_name in users: + await _seed_user_agent( + store, + password_hash, + secrets_path, + MAILUser(ua_type="user", user_id=user_name, host=host), + "user", + ) + for admin_name in admins: + await _seed_user_agent( + store, + password_hash, + secrets_path, + MAILAdmin(ua_type="admin", admin_id=admin_name, host=host), + "admin", + ) + finally: + await db.dispose() + + print(f"sqlite backend initialization complete: {db_path}") diff --git a/src/mail/server/src/mail_server/backends/sqlite/migrate.py b/src/mail/server/src/mail_server/backends/sqlite/migrate.py new file mode 100644 index 0000000..008aced --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/migrate.py @@ -0,0 +1,223 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +One-time import of a filesystem (memory-backend) deployment into SQLite. + +The memory backend persists its state as a directory tree under +``~/.mail-swarms/deployments//``. This module reads that tree with +the memory backend's own loaders (so the on-disk format can never drift) and +writes every collection into the SQLite store in a single transaction. + +The per-owner box ordering that the memory backend kept implicitly (Python list +order) is reconstructed by inserting ``mailbox_items`` membership rows in list +order, each stamped with the box-arrival timestamp drawn from its entry +(``received_at`` / ``sent_at`` / ``draft.created_at`` / ``trashed_at``), so the +autoincrement ``id`` tiebreaker reproduces the original order. + +The import refuses to run against a non-empty database, so it can't clobber an +existing SQLite deployment. See ``src/mail/server/docs/reference/backends.md``. +""" + +from __future__ import annotations + +import logging +from collections.abc import Callable +from datetime import datetime +from pathlib import Path +from typing import Any + +from sqlalchemy import func, select +from sqlalchemy.ext.asyncio import AsyncSession + +import mail_server.backends.memory.fs as memory_fs +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.init import default_sqlite_path +from mail_server.backends.sqlite.repositories import ( + BOX_DRAFTS, + BOX_INBOX, + BOX_OUTBOX, + BOX_TRASH, + MailStore, +) +from mail_server.backends.sqlite.schema import ( + ListRow, + MessageRow, + SwarmRow, + UserAgentRow, + WebhookRow, +) + +logger = logging.getLogger(__name__) + + +def _memory_deployment_dir(deployment: str) -> Path: + return Path.home().joinpath(".mail-swarms", "deployments", deployment) + + +async def _is_empty(session: AsyncSession) -> bool: + """True if no top-level collection has any rows yet.""" + + for row_cls in (UserAgentRow, SwarmRow, MessageRow, WebhookRow, ListRow): + count = await session.scalar(select(func.count()).select_from(row_cls)) + if count: + return False + return True + + +async def import_memory_deployment( + deployment: str = "default", + source_dir: Path | None = None, + db_path: Path | None = None, +) -> dict[str, int]: + """ + Import a filesystem deployment into a fresh SQLite database. + + ``source_dir`` defaults to ``~/.mail-swarms/deployments/`` (the + memory backend's tree); ``db_path`` defaults to ``/mail.db``. + Returns per-collection row counts. Raises if the source is missing or the + target database already holds data. + """ + + source_dir = source_dir or _memory_deployment_dir(deployment) + if not source_dir.is_dir(): + raise FileNotFoundError( + f"no filesystem deployment to import at {source_dir}" + ) + db_path = db_path or default_sqlite_path(deployment) + + # Load every collection via the memory backend's loaders by pointing them at + # the source tree (mirrors how the test harness redirects persistence). + previous_path = memory_fs.DEPLOYMENT_PATH + memory_fs.DEPLOYMENT_PATH = source_dir + try: + user_agents = await memory_fs.load_user_agents() + swarms = await memory_fs.load_swarms() + messages = await memory_fs.load_messages() + inbox_entries = await memory_fs.load_inbox_entries() + inboxes = await memory_fs.load_inboxes() + outbox_entries = await memory_fs.load_outbox_entries() + outboxes = await memory_fs.load_outboxes() + draft_entries = await memory_fs.load_draft_entries() + drafts = await memory_fs.load_drafts() + trash_entries = await memory_fs.load_trash_entries() + trashes = await memory_fs.load_trashes() + message_buffer = await memory_fs.load_message_buffer() + webhooks = await memory_fs.load_webhooks() + lists = await memory_fs.load_lists() + finally: + memory_fs.DEPLOYMENT_PATH = previous_path + + db = Database(f"sqlite:///{db_path}") + await db.create_schema() + try: + async with db.session() as session: + if not await _is_empty(session): + raise ValueError( + f"target sqlite database {db_path} is not empty; " + "refusing to import" + ) + store = MailStore(session) + + for ua_in_be in user_agents.values(): + await store.user_agents.add(ua_in_be) + for swarm in swarms.values(): + await store.swarms.add(swarm) + + # Canonical messages first (box entries FK onto them). Track which + # ids exist so we never insert an entry whose message is absent. + present: set[str] = set() + for message in messages.values(): + await store.messages.add(message) + present.add(message.message_id) + + for entry in inbox_entries.values(): + if entry.message_id in present: + await store.boxes.upsert_inbox_entry(entry) + for entry in outbox_entries.values(): + if entry.message_id in present: + await store.boxes.upsert_outbox_entry(entry) + for trash_entry in trash_entries.values(): + # Trash entries carry the full message; restore it if the + # canonical row was already gone. + if trash_entry.message.message_id not in present: + await store.messages.add(trash_entry.message) + present.add(trash_entry.message.message_id) + await store.boxes.upsert_trash_entry(trash_entry) + for draft_entry in draft_entries.values(): + await store.boxes.upsert_draft_entry(draft_entry) + + await _import_membership( + store, BOX_INBOX, inboxes, inbox_entries, lambda e: e.received_at + ) + await _import_membership( + store, BOX_OUTBOX, outboxes, outbox_entries, lambda e: e.sent_at + ) + await _import_membership( + store, + BOX_DRAFTS, + drafts, + draft_entries, + lambda e: e.draft.created_at, + ) + await _import_membership( + store, BOX_TRASH, trashes, trash_entries, lambda e: e.trashed_at + ) + + for message_id in message_buffer: + await store.buffer.enqueue(message_id) + for webhook in webhooks.values(): + await store.webhooks.add(webhook) + for mail_list in lists.values(): + await store.lists.add(mail_list) + finally: + await db.dispose() + + counts = { + "user_agents": len(user_agents), + "swarms": len(swarms), + "messages": len(messages), + "inbox_entries": len(inbox_entries), + "outbox_entries": len(outbox_entries), + "draft_entries": len(draft_entries), + "trash_entries": len(trash_entries), + "webhooks": len(webhooks), + "lists": len(lists), + "buffered": len(message_buffer), + } + logger.info("imported filesystem deployment %s into %s: %s", deployment, db_path, counts) + return counts + + +async def _import_membership( + store: MailStore, + box: str, + boxes: dict[str, list[str]], + entries: dict[str, Any], + entered_at_of: Callable[[Any], datetime], +) -> None: + """ + Recreate ``mailbox_items`` rows for one box from its per-owner id lists. + + Insertion follows list order, so the autoincrement ``id`` reproduces the + memory backend's insertion-order tiebreaker. The arrival timestamp is read + from each item's entry via ``entered_at_of``; items missing an entry (or + already present) are skipped. + """ + + for owner, item_ids in boxes.items(): + for item_id in item_ids: + entry = entries.get(item_id) + if entry is None: + logger.warning( + "skipping %s membership for %s: no entry for item %s", + box, + owner, + item_id, + ) + continue + if await store.boxes.is_member(owner, box, item_id): + continue + await store.boxes.add_membership( + owner, box, item_id, entered_at_of(entry) + ) diff --git a/src/mail/server/src/mail_server/backends/sqlite/repositories.py b/src/mail/server/src/mail_server/backends/sqlite/repositories.py new file mode 100644 index 0000000..e5d2df7 --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/repositories.py @@ -0,0 +1,631 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Session-scoped repositories for the MAIL SQLite backend. + +Mirrors the chorus repository pattern: a top-level ``MailStore(session)`` frozen +dataclass exposes sub-repositories as properties, each a frozen dataclass +wrapping the *same* ``AsyncSession``. Repositories own **SQL only** — +``select`` / ``insert`` / ``update`` / ``delete``, pagination, and ordering — +and return MAIL Pydantic models (via ``serializers``), never ORM rows. They call +``session.flush()`` (never ``commit()``); the ``Database.session()`` context +manager owns the transaction boundary, so several repository calls compose into +one atomic operation. + +``MailboxRepository`` is the workhorse: it unifies the four boxes (inbox, +outbox, drafts, trash) over the shared ``mailbox_items`` membership table plus +the per-box entry tables, pushing sorting/pagination into ``ORDER BY ... LIMIT +/ OFFSET`` instead of loading whole boxes into Python. + +See ``src/mail/server/docs/reference/backends.md`` for the backend overview. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from typing import Any + +from mail_protocol.core.drafts import MAILDraftsEntry, MAILDraftsEntrySummary +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.outbox import MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.trash import MAILTrashEntry, MAILTrashEntrySummary +from mail_protocol.core.user_agents import MAILUserAgentInBackend +from mail_protocol.core.webhooks import MAILWebhook +from mail_protocol.network.requests import BoxFilterParams +from sqlalchemy import asc, delete, func, select +from sqlalchemy.ext.asyncio import AsyncSession + +from mail_server.backends.sqlite import serializers as ser +from mail_server.backends.sqlite.schema import ( + DraftEntryRow, + InboxEntryRow, + ListRow, + MailboxItemRow, + MessageBufferRow, + MessageRow, + OutboxEntryRow, + SwarmRow, + TrashEntryRow, + UserAgentRow, + WebhookRow, +) + +# Box discriminators stored in ``mailbox_items.box``. +BOX_INBOX = "inbox" +BOX_OUTBOX = "outbox" +BOX_DRAFTS = "drafts" +BOX_TRASH = "trash" + + +@dataclass(frozen=True) +class MailStore: + """Root handle wrapping a single session; hands out sub-repositories.""" + + session: AsyncSession + + @property + def user_agents(self) -> UserAgentRepository: + return UserAgentRepository(self.session) + + @property + def swarms(self) -> SwarmRepository: + return SwarmRepository(self.session) + + @property + def messages(self) -> MessageRepository: + return MessageRepository(self.session) + + @property + def boxes(self) -> MailboxRepository: + return MailboxRepository(self.session) + + @property + def buffer(self) -> MessageBufferRepository: + return MessageBufferRepository(self.session) + + @property + def webhooks(self) -> WebhookRepository: + return WebhookRepository(self.session) + + @property + def lists(self) -> ListRepository: + return ListRepository(self.session) + + +# --------------------------------------------------------------------------- # +# user_agents +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class UserAgentRepository: + session: AsyncSession + + async def get(self, address: str) -> MAILUserAgentInBackend | None: + row = await self.session.get(UserAgentRow, address) + if row is None: + return None + return ser.user_agent_from_row(row) + + async def exists(self, address: str) -> bool: + result = await self.session.scalar( + select(UserAgentRow.address).where(UserAgentRow.address == address) + ) + return result is not None + + async def list_by_type(self, ua_type: str) -> list[MAILUserAgentInBackend]: + rows = await self.session.scalars( + select(UserAgentRow) + .where(UserAgentRow.ua_type == ua_type) + .order_by(UserAgentRow.created_at, UserAgentRow.address) + ) + return [ser.user_agent_from_row(row) for row in rows] + + async def add(self, model: MAILUserAgentInBackend) -> MAILUserAgentInBackend: + self.session.add(UserAgentRow(**ser.user_agent_to_columns(model))) + await self.session.flush() + return model + + async def delete(self, address: str) -> MAILUserAgentInBackend | None: + row = await self.session.get(UserAgentRow, address) + if row is None: + return None + model = ser.user_agent_from_row(row) + await self.session.delete(row) + await self.session.flush() + return model + + async def set_password( + self, address: str, hashed_password: str + ) -> MAILUserAgentInBackend | None: + """Rewrite the password hash in both the typed column and the body.""" + + row = await self.session.get(UserAgentRow, address) + if row is None: + return None + model = ser.user_agent_from_row(row) + model.hashed_password = hashed_password + cols = ser.user_agent_to_columns(model) + row.hashed_password = cols["hashed_password"] + row.body = cols["body"] + await self.session.flush() + return model + + +# --------------------------------------------------------------------------- # +# swarms +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class SwarmRepository: + session: AsyncSession + + async def list_all(self) -> list[MAILSwarm]: + rows = await self.session.scalars( + select(SwarmRow).order_by(SwarmRow.created_at, SwarmRow.name) + ) + return [ser.swarm_from_row(row) for row in rows] + + async def get(self, name: str) -> MAILSwarm | None: + row = await self.session.get(SwarmRow, name) + if row is None: + return None + return ser.swarm_from_row(row) + + async def add(self, model: MAILSwarm) -> MAILSwarm: + self.session.add(SwarmRow(**ser.swarm_to_columns(model))) + await self.session.flush() + return model + + async def delete(self, name: str) -> MAILSwarm | None: + row = await self.session.get(SwarmRow, name) + if row is None: + return None + model = ser.swarm_from_row(row) + await self.session.delete(row) + await self.session.flush() + return model + + +# --------------------------------------------------------------------------- # +# messages (canonical store) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class MessageRepository: + session: AsyncSession + + async def get(self, message_id: str) -> MAILMessage | None: + row = await self.session.get(MessageRow, message_id) + if row is None: + return None + return ser.message_from_row(row) + + async def add(self, model: MAILMessage) -> MAILMessage: + self.session.add(MessageRow(**ser.message_to_columns(model))) + await self.session.flush() + return model + + async def delete(self, message_id: str) -> bool: + row = await self.session.get(MessageRow, message_id) + if row is None: + return False + await self.session.delete(row) + await self.session.flush() + return True + + +# --------------------------------------------------------------------------- # +# boxes — mailbox_items membership + the four entry tables +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class MailboxRepository: + session: AsyncSession + + # + # Membership (mailbox_items) + # + async def is_member(self, owner: str, box: str, item_id: str) -> bool: + result = await self.session.scalar( + select(MailboxItemRow.id).where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + MailboxItemRow.item_id == item_id, + ) + ) + return result is not None + + async def add_membership( + self, owner: str, box: str, item_id: str, entered_at: datetime + ) -> None: + self.session.add( + MailboxItemRow( + owner_address=owner, + box=box, + item_id=item_id, + entered_at=entered_at, + ) + ) + await self.session.flush() + + async def remove_membership(self, owner: str, box: str, item_id: str) -> bool: + """Delete one membership row; return whether it existed.""" + + if not await self.is_member(owner, box, item_id): + return False + await self.session.execute( + delete(MailboxItemRow).where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + MailboxItemRow.item_id == item_id, + ) + ) + await self.session.flush() + return True + + async def list_item_ids(self, owner: str, box: str) -> list[str]: + """Item ids in a box, in insertion order (used by ``clear_trash``).""" + + rows = await self.session.scalars( + select(MailboxItemRow.item_id) + .where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + ) + .order_by(asc(MailboxItemRow.id)) + ) + return list(rows) + + async def count_item_members(self, box: str, item_id: str) -> int: + """How many owners still reference ``item_id`` in ``box`` (orphan check).""" + + result = await self.session.scalar( + select(func.count()) + .select_from(MailboxItemRow) + .where(MailboxItemRow.box == box, MailboxItemRow.item_id == item_id) + ) + return result or 0 + + async def _count(self, owner: str, box: str) -> int: + result = await self.session.scalar( + select(func.count()) + .select_from(MailboxItemRow) + .where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + ) + ) + return result or 0 + + # + # Paginated reads + # + async def _page( + self, + *, + owner: str, + box: str, + entry_cls: Any, + entry_pk: Any, + filters: BoxFilterParams, + allow_message_sort: bool, + ) -> tuple[list[Any], int]: + """ + Return one ordered, sliced page of entry rows + the full box count. + + ``entered_at`` sorts by ``mailbox_items.entered_at`` (the box-arrival + time); ``sent_at`` joins ``messages`` and sorts by the original send + time — only valid for boxes whose items are real messages + (``allow_message_sort``). ``mailbox_items.id`` is the always-ascending + tiebreaker, reproducing the memory backend's stable insertion order. + """ + + total = await self._count(owner, box) + if total == 0: + return [], total + + stmt = ( + select(entry_cls) + .join(MailboxItemRow, MailboxItemRow.item_id == entry_pk) + .where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + ) + ) + + if allow_message_sort and filters.sort_by == "sent_at": + stmt = stmt.join( + MessageRow, MessageRow.message_id == MailboxItemRow.item_id + ) + sort_col: Any = MessageRow.sent_at + else: + sort_col = MailboxItemRow.entered_at + + sort_col = sort_col.desc() if filters.order == "desc" else sort_col.asc() + stmt = ( + stmt.order_by(sort_col, asc(MailboxItemRow.id)) + .limit(filters.limit) + .offset(filters.offset) + ) + + rows = list(await self.session.scalars(stmt)) + return rows, total + + async def list_inbox( + self, owner: str, filters: BoxFilterParams + ) -> tuple[list[MAILInboxEntrySummary], int]: + rows, total = await self._page( + owner=owner, + box=BOX_INBOX, + entry_cls=InboxEntryRow, + entry_pk=InboxEntryRow.message_id, + filters=filters, + allow_message_sort=True, + ) + return [ser.inbox_entry_from_row(row) for row in rows], total + + async def list_outbox( + self, owner: str, filters: BoxFilterParams + ) -> tuple[list[MAILOutboxEntrySummary], int]: + rows, total = await self._page( + owner=owner, + box=BOX_OUTBOX, + entry_cls=OutboxEntryRow, + entry_pk=OutboxEntryRow.message_id, + filters=filters, + allow_message_sort=True, + ) + return [ser.outbox_entry_from_row(row) for row in rows], total + + async def list_drafts( + self, owner: str, filters: BoxFilterParams + ) -> tuple[list[MAILDraftsEntrySummary], int]: + # Drafts have no send time; ``sort_by=sent_at`` is rejected at the + # router, so only ``entered_at`` (== created_at) ordering reaches here. + rows, total = await self._page( + owner=owner, + box=BOX_DRAFTS, + entry_cls=DraftEntryRow, + entry_pk=DraftEntryRow.draft_id, + filters=filters, + allow_message_sort=False, + ) + return [ser.draft_entry_from_row(row).summarize() for row in rows], total + + async def list_trash( + self, owner: str, filters: BoxFilterParams + ) -> tuple[list[MAILTrashEntrySummary], int]: + rows, total = await self._page( + owner=owner, + box=BOX_TRASH, + entry_cls=TrashEntryRow, + entry_pk=TrashEntryRow.message_id, + filters=filters, + allow_message_sort=True, + ) + return [ser.trash_entry_from_row(row).summarize() for row in rows], total + + # + # inbox_entries (shared, keyed by message id) + # + async def get_inbox_entry(self, message_id: str) -> MAILInboxEntrySummary | None: + row = await self.session.get(InboxEntryRow, message_id) + if row is None: + return None + return ser.inbox_entry_from_row(row) + + async def upsert_inbox_entry(self, summary: MAILInboxEntrySummary) -> None: + cols = ser.inbox_entry_to_columns(summary) + row = await self.session.get(InboxEntryRow, summary.message_id) + if row is None: + self.session.add(InboxEntryRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + + async def delete_inbox_entry(self, message_id: str) -> None: + row = await self.session.get(InboxEntryRow, message_id) + if row is not None: + await self.session.delete(row) + await self.session.flush() + + # + # outbox_entries (shared, keyed by message id) + # + async def get_outbox_entry(self, message_id: str) -> MAILOutboxEntrySummary | None: + row = await self.session.get(OutboxEntryRow, message_id) + if row is None: + return None + return ser.outbox_entry_from_row(row) + + async def upsert_outbox_entry(self, summary: MAILOutboxEntrySummary) -> None: + cols = ser.outbox_entry_to_columns(summary) + row = await self.session.get(OutboxEntryRow, summary.message_id) + if row is None: + self.session.add(OutboxEntryRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + + # + # draft_entries (keyed by draft id) + # + async def get_draft_entry(self, draft_id: str) -> MAILDraftsEntry | None: + row = await self.session.get(DraftEntryRow, draft_id) + if row is None: + return None + return ser.draft_entry_from_row(row) + + async def upsert_draft_entry(self, entry: MAILDraftsEntry) -> None: + cols = ser.draft_entry_to_columns(entry) + row = await self.session.get(DraftEntryRow, entry.draft.draft_id) + if row is None: + self.session.add(DraftEntryRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + + async def delete_draft_entry(self, draft_id: str) -> None: + row = await self.session.get(DraftEntryRow, draft_id) + if row is not None: + await self.session.delete(row) + await self.session.flush() + + # + # trash_entries (shared, keyed by message id) + # + async def get_trash_entry(self, message_id: str) -> MAILTrashEntry | None: + row = await self.session.get(TrashEntryRow, message_id) + if row is None: + return None + return ser.trash_entry_from_row(row) + + async def upsert_trash_entry(self, entry: MAILTrashEntry) -> None: + cols = ser.trash_entry_to_columns(entry) + row = await self.session.get(TrashEntryRow, entry.message.message_id) + if row is None: + self.session.add(TrashEntryRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + + async def delete_trash_entry(self, message_id: str) -> None: + row = await self.session.get(TrashEntryRow, message_id) + if row is not None: + await self.session.delete(row) + await self.session.flush() + + +# --------------------------------------------------------------------------- # +# message_buffer (FIFO delivery queue) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class MessageBufferRepository: + session: AsyncSession + + async def enqueue(self, message_id: str) -> None: + self.session.add(MessageBufferRow(message_id=message_id)) + await self.session.flush() + + async def list_ids(self) -> list[str]: + rows = await self.session.scalars( + select(MessageBufferRow.message_id).order_by(asc(MessageBufferRow.id)) + ) + return list(rows) + + async def drain(self) -> list[str]: + """Return every buffered id in FIFO order and clear the buffer atomically.""" + + ids = await self.list_ids() + if ids: + await self.session.execute(delete(MessageBufferRow)) + await self.session.flush() + return ids + + +# --------------------------------------------------------------------------- # +# webhooks (keyed by URL) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class WebhookRepository: + session: AsyncSession + + async def list_all(self) -> list[MAILWebhook]: + rows = await self.session.scalars( + select(WebhookRow).order_by(WebhookRow.created_at, WebhookRow.url) + ) + return [ser.webhook_from_row(row) for row in rows] + + async def get_by_url(self, url: str) -> MAILWebhook | None: + row = await self.session.get(WebhookRow, url) + if row is None: + return None + return ser.webhook_from_row(row) + + async def get_by_id(self, webhook_id: str) -> MAILWebhook | None: + row = await self.session.scalar( + select(WebhookRow).where(WebhookRow.webhook_id == webhook_id) + ) + if row is None: + return None + return ser.webhook_from_row(row) + + async def add(self, model: MAILWebhook) -> MAILWebhook: + self.session.add(WebhookRow(**ser.webhook_to_columns(model))) + await self.session.flush() + return model + + async def delete_by_url(self, url: str) -> MAILWebhook | None: + row = await self.session.get(WebhookRow, url) + if row is None: + return None + model = ser.webhook_from_row(row) + await self.session.delete(row) + await self.session.flush() + return model + + +# --------------------------------------------------------------------------- # +# lists (members live inside the body) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class ListRepository: + session: AsyncSession + + async def list_all(self) -> list[MAILListInBackend]: + rows = await self.session.scalars( + select(ListRow).order_by(ListRow.created_at, ListRow.address) + ) + return [ser.list_from_row(row) for row in rows] + + async def get_by_address(self, address: str) -> MAILListInBackend | None: + row = await self.session.get(ListRow, address) + if row is None: + return None + return ser.list_from_row(row) + + async def add(self, model: MAILListInBackend) -> MAILListInBackend: + self.session.add(ListRow(**ser.list_to_columns(model))) + await self.session.flush() + return model + + async def update(self, model: MAILListInBackend) -> MAILListInBackend: + """Persist a mutated list (member edits, policy patch) by address.""" + + cols = ser.list_to_columns(model) + row = await self.session.get(ListRow, model.get_address()) + if row is None: + self.session.add(ListRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + return model + + async def delete(self, address: str) -> MAILListInBackend | None: + row = await self.session.get(ListRow, address) + if row is None: + return None + model = ser.list_from_row(row) + await self.session.delete(row) + await self.session.flush() + return model diff --git a/src/mail/server/src/mail_server/backends/sqlite/schema.py b/src/mail/server/src/mail_server/backends/sqlite/schema.py new file mode 100644 index 0000000..9c469fc --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/schema.py @@ -0,0 +1,250 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Declarative SQLAlchemy schema for the MAIL SQLite backend. + +Every entity row follows the *hybrid* convention: typed, indexed columns for +the handful of fields that the backend filters, sorts, or paginates on, plus a +``body`` JSON column holding the full MAIL Pydantic model +(``model.model_dump(mode="json")``). Reads always rehydrate the model from +``body`` via ``Model.model_validate(...)`` — the typed columns exist only for +``WHERE`` / ``ORDER BY``. Adding a field to a MAIL model therefore needs a +schema change only when that field must become queryable. + +See ``src/mail/server/docs/reference/backends.md`` for the backend overview. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any + +from sqlalchemy import ( + JSON, + DateTime, + ForeignKey, + Index, + Integer, + String, + Text, + UniqueConstraint, +) +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column + + +def utc_now() -> datetime: + """Return the current time as a timezone-aware UTC ``datetime``.""" + + return datetime.now(UTC) + + +class Base(DeclarativeBase): + pass + + +class UserAgentRow(Base): + """A MAIL user-agent (agent / user / admin / daemon).""" + + __tablename__ = "user_agents" + + # Full MAIL address: ``name@swarm@host`` for agents, + # ``prefix:name@host`` for users / admins / daemons. + address: Mapped[str] = mapped_column(String(512), primary_key=True) + ua_type: Mapped[str] = mapped_column(String(16), index=True) + # Swarm is only meaningful for agents; null for user / admin / daemon. + swarm: Mapped[str | None] = mapped_column(String(128), nullable=True, index=True) + host: Mapped[str] = mapped_column(String(255), index=True) + hashed_password: Mapped[str] = mapped_column(Text) + # Full ``MAILUserAgentInBackend``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now, onupdate=utc_now + ) + + +class SwarmRow(Base): + """A MAIL swarm exposed by this server.""" + + __tablename__ = "swarms" + + name: Mapped[str] = mapped_column(String(128), primary_key=True) + # Full ``MAILSwarm``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now, onupdate=utc_now + ) + + +class MessageRow(Base): + """The canonical store of every MAIL message known to this server.""" + + __tablename__ = "messages" + + # Bare UUID, as stored on ``MAILMessage.message_id`` (no ``msg_`` prefix). + message_id: Mapped[str] = mapped_column(String(64), primary_key=True) + sender: Mapped[str] = mapped_column(String(512), index=True) + subject: Mapped[str] = mapped_column(Text) + reply_to: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True) + sent_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + # Full ``MAILMessage`` (recipients, tags, metadata, body text). + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class InboxEntryRow(Base): + """ + A shared inbox-entry summary, keyed globally by message id. + + Mirrors the memory backend: when a message fans out to N recipients they + share one entry row; the per-owner data is the *membership* + (``mailbox_items``), not the entry. + """ + + __tablename__ = "inbox_entries" + + message_id: Mapped[str] = mapped_column( + String(64), + ForeignKey("messages.message_id", ondelete="CASCADE"), + primary_key=True, + ) + sender: Mapped[str] = mapped_column(String(512)) + subject: Mapped[str] = mapped_column(Text) + body_size: Mapped[int] = mapped_column(Integer) + received_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + delivered_by: Mapped[str | None] = mapped_column(String(512), nullable=True) + # Full ``MAILInboxEntrySummary``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class OutboxEntryRow(Base): + """A shared outbox-entry summary, keyed globally by message id.""" + + __tablename__ = "outbox_entries" + + message_id: Mapped[str] = mapped_column( + String(64), + ForeignKey("messages.message_id", ondelete="CASCADE"), + primary_key=True, + ) + sent_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + delivered_at: Mapped[datetime | None] = mapped_column( + DateTime(timezone=True), nullable=True + ) + delivered_by: Mapped[str | None] = mapped_column(String(512), nullable=True) + # Full ``MAILOutboxEntrySummary``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class DraftEntryRow(Base): + """A draft-box entry, keyed by draft id.""" + + __tablename__ = "draft_entries" + + draft_id: Mapped[str] = mapped_column(String(64), primary_key=True) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + updated_at: Mapped[datetime | None] = mapped_column( + DateTime(timezone=True), nullable=True + ) + # Full ``MAILDraftsEntry``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class TrashEntryRow(Base): + """A shared trash-entry, keyed globally by message id.""" + + __tablename__ = "trash_entries" + + message_id: Mapped[str] = mapped_column( + String(64), + ForeignKey("messages.message_id", ondelete="CASCADE"), + primary_key=True, + ) + trashed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + # Full ``MAILTrashEntry``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class MailboxItemRow(Base): + """ + Unified per-owner box membership + ordering for all four boxes. + + One row links an owner's box to an entry. ``box`` discriminates between + ``inbox`` / ``outbox`` / ``drafts`` / ``trash``; ``item_id`` is a message + id (inbox / outbox / trash) or a draft id (drafts). The autoincrement + ``id`` reproduces the insertion order that the memory backend got for free + from Python lists, and serves as the stable tiebreaker when two entries + share an ``entered_at``. + """ + + __tablename__ = "mailbox_items" + __table_args__ = ( + UniqueConstraint( + "owner_address", + "box", + "item_id", + name="uq_mailbox_items_owner_box_item", + ), + Index("ix_mailbox_items_owner_box", "owner_address", "box"), + ) + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + owner_address: Mapped[str] = mapped_column( + String(512), + ForeignKey("user_agents.address", ondelete="CASCADE"), + index=True, + ) + box: Mapped[str] = mapped_column(String(8)) + item_id: Mapped[str] = mapped_column(String(64)) + entered_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + + +class MessageBufferRow(Base): + """The FIFO message delivery queue. Autoincrement ``id`` preserves order.""" + + __tablename__ = "message_buffer" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + message_id: Mapped[str] = mapped_column(String(64), unique=True, index=True) + + +class WebhookRow(Base): + """A server webhook, keyed by URL (matching the memory backend).""" + + __tablename__ = "webhooks" + + url: Mapped[str] = mapped_column(String(512), primary_key=True) + webhook_id: Mapped[str] = mapped_column(String(128), unique=True, index=True) + # Full ``MAILWebhook`` (events, secret). + body: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now, onupdate=utc_now + ) + + +class ListRow(Base): + """A MAIL list. Members live inside ``body``, mirroring the memory backend.""" + + __tablename__ = "lists" + + # ``list:@@``. + address: Mapped[str] = mapped_column(String(512), primary_key=True) + list_id: Mapped[str] = mapped_column(String(64), unique=True, index=True) + swarm: Mapped[str] = mapped_column(String(128), index=True) + host: Mapped[str] = mapped_column(String(255)) + # Full ``MAILListInBackend`` (members, policy, owner). + body: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now, onupdate=utc_now + ) diff --git a/src/mail/server/src/mail_server/backends/sqlite/serializers.py b/src/mail/server/src/mail_server/backends/sqlite/serializers.py new file mode 100644 index 0000000..30bffea --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/serializers.py @@ -0,0 +1,223 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Row <-> MAIL Pydantic model conversion for the SQLite backend. + +This module centralizes the *hybrid* invariant in one place: every entity row +carries a ``body`` JSON column equal to ``model.model_dump(mode="json")`` plus a +handful of typed columns used only for ``WHERE`` / ``ORDER BY``. Each table gets +a pair of helpers: + +- ``_to_columns(model)`` — the keyword arguments to construct (or update) + a ``*Row``: the derived typed columns *and* ``body``. The typed columns are + always computed from the model, never the other way around, so they can never + drift from the body. +- ``
_from_row(row)`` — rehydrate the model. This reads **only** + ``row.body`` via ``Model.model_validate(...)``; the typed columns are never + consulted on read. That is what makes the schema tolerant of model evolution: + adding a field to a MAIL model needs a schema change only if the new field + must become queryable. + +``mailbox_items`` and ``message_buffer`` carry no JSON body and have no MAIL +model — they are pure membership / ordering rows constructed directly by the +repositories, so they have no serializer here. + +See ``src/mail/server/docs/reference/backends.md`` for the backend overview. +""" + +from __future__ import annotations + +from typing import Any + +from mail_protocol.core.drafts import MAILDraftsEntry +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.outbox import MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.trash import MAILTrashEntry +from mail_protocol.core.user_agents import MAILUserAgentInBackend +from mail_protocol.core.webhooks import MAILWebhook + +from mail_server.backends.sqlite.schema import ( + DraftEntryRow, + InboxEntryRow, + ListRow, + MessageRow, + OutboxEntryRow, + SwarmRow, + TrashEntryRow, + UserAgentRow, + WebhookRow, +) + +# --------------------------------------------------------------------------- # +# user_agents +# --------------------------------------------------------------------------- # + + +def user_agent_to_columns(model: MAILUserAgentInBackend) -> dict[str, Any]: + ua = model.user_agent + return { + "address": model.get_address(), + "ua_type": ua.ua_type, + # ``swarm`` is meaningful only for agents; users/admins/daemons have none. + "swarm": getattr(ua, "swarm", None), + "host": ua.host, + "hashed_password": model.hashed_password, + "body": model.model_dump(mode="json"), + } + + +def user_agent_from_row(row: UserAgentRow) -> MAILUserAgentInBackend: + return MAILUserAgentInBackend.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# swarms +# --------------------------------------------------------------------------- # + + +def swarm_to_columns(model: MAILSwarm) -> dict[str, Any]: + return { + "name": model.name, + "body": model.model_dump(mode="json"), + } + + +def swarm_from_row(row: SwarmRow) -> MAILSwarm: + return MAILSwarm.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# messages +# --------------------------------------------------------------------------- # + + +def message_to_columns(model: MAILMessage) -> dict[str, Any]: + return { + "message_id": model.message_id, + "sender": model.sender, + "subject": model.subject, + "reply_to": model.reply_to, + "sent_at": model.sent_at, + "body": model.model_dump(mode="json"), + } + + +def message_from_row(row: MessageRow) -> MAILMessage: + return MAILMessage.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# inbox_entries (shared summary, keyed by message id) +# --------------------------------------------------------------------------- # + + +def inbox_entry_to_columns(model: MAILInboxEntrySummary) -> dict[str, Any]: + return { + "message_id": model.message_id, + "sender": model.sender, + "subject": model.subject, + "body_size": model.body_size, + "received_at": model.received_at, + "delivered_by": model.delivered_by, + "body": model.model_dump(mode="json"), + } + + +def inbox_entry_from_row(row: InboxEntryRow) -> MAILInboxEntrySummary: + return MAILInboxEntrySummary.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# outbox_entries (shared summary, keyed by message id) +# --------------------------------------------------------------------------- # + + +def outbox_entry_to_columns(model: MAILOutboxEntrySummary) -> dict[str, Any]: + return { + "message_id": model.message_id, + "sent_at": model.sent_at, + "delivered_at": model.delivered_at, + "delivered_by": model.delivered_by, + "body": model.model_dump(mode="json"), + } + + +def outbox_entry_from_row(row: OutboxEntryRow) -> MAILOutboxEntrySummary: + return MAILOutboxEntrySummary.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# draft_entries +# --------------------------------------------------------------------------- # + + +def draft_entry_to_columns(model: MAILDraftsEntry) -> dict[str, Any]: + return { + "draft_id": model.draft.draft_id, + "created_at": model.draft.created_at, + "updated_at": model.draft.updated_at, + "body": model.model_dump(mode="json"), + } + + +def draft_entry_from_row(row: DraftEntryRow) -> MAILDraftsEntry: + return MAILDraftsEntry.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# trash_entries (shared, keyed by message id) +# --------------------------------------------------------------------------- # + + +def trash_entry_to_columns(model: MAILTrashEntry) -> dict[str, Any]: + return { + "message_id": model.message.message_id, + "trashed_at": model.trashed_at, + "body": model.model_dump(mode="json"), + } + + +def trash_entry_from_row(row: TrashEntryRow) -> MAILTrashEntry: + return MAILTrashEntry.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# webhooks (keyed by URL) +# --------------------------------------------------------------------------- # + + +def webhook_to_columns(model: MAILWebhook) -> dict[str, Any]: + return { + "url": model.url, + "webhook_id": model.webhook_id, + "body": model.model_dump(mode="json"), + } + + +def webhook_from_row(row: WebhookRow) -> MAILWebhook: + return MAILWebhook.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# lists (members live inside the body) +# --------------------------------------------------------------------------- # + + +def list_to_columns(model: MAILListInBackend) -> dict[str, Any]: + return { + "address": model.get_address(), + "list_id": model.list_id, + "swarm": model.swarm, + "host": model.host, + "created_at": model.created_at, + "updated_at": model.updated_at, + "body": model.model_dump(mode="json"), + } + + +def list_from_row(row: ListRow) -> MAILListInBackend: + return MAILListInBackend.model_validate(row.body) diff --git a/src/mail/server/src/mail_server/cli.py b/src/mail/server/src/mail_server/cli.py index ae9c1a7..3fc1bf8 100644 --- a/src/mail/server/src/mail_server/cli.py +++ b/src/mail/server/src/mail_server/cli.py @@ -12,6 +12,9 @@ "mail-server --host 0.0.0.0 --port 8865", "mail-server --backend memory", "mail-server --memory-save-interval 30", + "mail-server --backend sqlite", + "mail-server --backend sqlite --sqlite-path /var/lib/mail/mail.db", + "mail-server --backend sqlite --database-url sqlite:////abs/path/mail.db", ] DEFAULT_MEMORY_SAVE_INTERVAL_SECONDS = 60.0 @@ -60,7 +63,7 @@ def build_parser() -> argparse.ArgumentParser: "-b", "--backend", metavar="BACKEND", - choices=["memory"], + choices=["memory", "sqlite"], default="memory", help="the MAIL server backend to use (default: %(default)s)", ) @@ -74,6 +77,24 @@ def build_parser() -> argparse.ArgumentParser: "set 0 to disable (default: %(default)s)" ), ) + parser.add_argument( + "--sqlite-path", + metavar="PATH", + default=os.getenv("MAIL_SQLITE_PATH"), + help=( + "sqlite backend database file (env: MAIL_SQLITE_PATH; default: " + "~/.mail-swarms/deployments/default/mail.db)" + ), + ) + parser.add_argument( + "--database-url", + metavar="URL", + default=os.getenv("MAIL_DATABASE_URL"), + help=( + "sqlite backend database URL; takes precedence over --sqlite-path " + "(env: MAIL_DATABASE_URL)" + ), + ) return parser diff --git a/src/mail/server/src/mail_server/routers/daemon.py b/src/mail/server/src/mail_server/routers/daemon.py index 85ac0f6..8d2d772 100644 --- a/src/mail/server/src/mail_server/routers/daemon.py +++ b/src/mail/server/src/mail_server/routers/daemon.py @@ -9,7 +9,10 @@ ) from mail_server.auth import validate_daemon -from mail_server.validators import validate_deliver_local_request +from mail_server.validators import ( + validate_deliver_local_request, + validate_deliver_remote_request, +) router = APIRouter(prefix="/daemon", tags=["daemon"]) @@ -53,4 +56,11 @@ async def deliver_local_messages(request: Request) -> DaemonDeliverLocalResponse response_model=DaemonDeliverRemoteResponse, ) async def deliver_remote_messages(request: Request) -> DaemonDeliverRemoteResponse: - raise NotImplementedError + backend = request.app.state.backend + daemon = await validate_daemon(backend=backend, request=request) + payload = await validate_deliver_remote_request(request=request) + result = await backend.daemon_deliver_remote(daemon=daemon, payload=payload) + return DaemonDeliverRemoteResponse( + messages=result, + metadata={}, + ) diff --git a/src/mail/server/src/mail_server/routers/drafts.py b/src/mail/server/src/mail_server/routers/drafts.py index a3e7eb7..9a1f670 100644 --- a/src/mail/server/src/mail_server/routers/drafts.py +++ b/src/mail/server/src/mail_server/routers/drafts.py @@ -114,7 +114,20 @@ async def patch_draft(request: Request) -> DraftPatchResponse: response_model=DraftDeleteResponse, ) async def delete_draft(request: Request) -> DraftDeleteResponse: - raise NotImplementedError + backend = request.app.state.backend + user_agent = await validate_user_agent(backend=backend, request=request) + draft_id = request.path_params.get("draft_id") + try: + result = await backend.delete_draft(user_agent=user_agent, draft_id=draft_id) + except ValueError: + raise HTTPException( + status_code=404, detail=f"draft with ID {draft_id} not found" + ) + + return DraftDeleteResponse( + entry=result, + metadata={}, + ) @router.post( diff --git a/src/mail/server/src/mail_server/routers/inbox.py b/src/mail/server/src/mail_server/routers/inbox.py index e754a5c..3702737 100644 --- a/src/mail/server/src/mail_server/routers/inbox.py +++ b/src/mail/server/src/mail_server/routers/inbox.py @@ -68,4 +68,19 @@ async def open_inbox_message(request: Request) -> InboxMessageGetResponse: response_model=InboxMessageDeleteResponse, ) async def delete_inbox_message(request: Request) -> InboxMessageDeleteResponse: - raise NotImplementedError + backend = request.app.state.backend + user_agent = await validate_user_agent(backend=backend, request=request) + message_id = request.path_params.get("message_id") + try: + result = await backend.delete_inbox_message( + user_agent=user_agent, message_id=message_id + ) + except ValueError: + raise HTTPException( + status_code=404, detail=f"message with ID {message_id} not found in inbox" + ) + + return InboxMessageDeleteResponse( + entry=result, + metadata={}, + ) diff --git a/src/mail/server/src/mail_server/routers/trash.py b/src/mail/server/src/mail_server/routers/trash.py index 2a2f854..6451b8d 100644 --- a/src/mail/server/src/mail_server/routers/trash.py +++ b/src/mail/server/src/mail_server/routers/trash.py @@ -67,8 +67,25 @@ async def get_trashed_message(request: Request) -> TrashMessageGetResponse: summary="Delete a specific trashed message by ID", response_model=TrashMessageDeleteResponse, ) -async def delete_trashed_message(message_id: str) -> TrashMessageDeleteResponse: - raise NotImplementedError +async def delete_trashed_message( + request: Request, message_id: str +) -> TrashMessageDeleteResponse: + backend = request.app.state.backend + user_agent = await validate_user_agent(backend=backend, request=request) + try: + result = await backend.delete_trash_message( + user_agent=user_agent, message_id=message_id + ) + except ValueError: + raise HTTPException( + status_code=404, + detail=f"no message with ID {message_id} found in trash box", + ) + + return TrashMessageDeleteResponse( + entry=result, + metadata={}, + ) @router.post( @@ -76,5 +93,12 @@ async def delete_trashed_message(message_id: str) -> TrashMessageDeleteResponse: summary="Remove all exisisting messages from trash", response_model=TrashClearPostResponse, ) -async def post_trash_clear() -> TrashClearPostResponse: - raise NotImplementedError +async def post_trash_clear(request: Request) -> TrashClearPostResponse: + backend = request.app.state.backend + user_agent = await validate_user_agent(backend=backend, request=request) + result = await backend.clear_trash(user_agent=user_agent) + + return TrashClearPostResponse( + entries=result, + metadata={}, + ) diff --git a/src/mail/server/src/mail_server/server.py b/src/mail/server/src/mail_server/server.py index 27b2372..8659393 100644 --- a/src/mail/server/src/mail_server/server.py +++ b/src/mail/server/src/mail_server/server.py @@ -6,6 +6,7 @@ import time from argparse import Namespace from contextlib import asynccontextmanager +from pathlib import Path import uvicorn from fastapi import FastAPI @@ -125,17 +126,45 @@ async def get_health() -> HealthGetResponse: # +def _resolve_sqlite_url(args: Namespace) -> str: + """ + Resolve the sqlite backend database URL from CLI args / env. + + Precedence: ``--database-url`` (full URL) > ``--sqlite-path`` (file path) > + the per-deployment default ``~/.mail-swarms/deployments/default/mail.db``. + Env fallbacks (``MAIL_DATABASE_URL`` / ``MAIL_SQLITE_PATH``) are applied as + the argparse defaults in ``cli.py``. + """ + + url = getattr(args, "database_url", None) + if url: + return url + path = getattr(args, "sqlite_path", None) + if path: + return f"sqlite:///{Path(path).expanduser()}" + + # Lazy import keeps SQLAlchemy off the import path for memory-only runs. + from mail_server.backends.sqlite.init import default_sqlite_path + + return f"sqlite:///{default_sqlite_path()}" + + def run_server(args: Namespace) -> None: """ Run the MAIL server from the CLI. """ + global _backend match args.backend: case "memory" | "mem": - global _backend _backend = MemoryBackend( persistence_interval_seconds=getattr(args, "memory_save_interval", 0) ) + case "sqlite": + # Lazy import so SQLAlchemy is only loaded when actually selected. + from mail_server.backends.sqlite.api import SQLiteBackend + + _backend = SQLiteBackend(url=_resolve_sqlite_url(args)) case _: raise ValueError(f"invalid backend type: {args.backend}") diff --git a/src/mail/server/src/mail_server/validators.py b/src/mail/server/src/mail_server/validators.py index 4ce6f36..9b182bb 100644 --- a/src/mail/server/src/mail_server/validators.py +++ b/src/mail/server/src/mail_server/validators.py @@ -14,6 +14,7 @@ AuthPasswordResetRequest, BoxFilterParams, DaemonDeliverLocalRequest, + DaemonDeliverRemoteRequest, DraftPatchRequest, DraftPostRequest, DraftSendPostRequest, @@ -110,6 +111,22 @@ async def validate_deliver_local_request( ) +async def validate_deliver_remote_request( + request: Request, +) -> DaemonDeliverRemoteRequest: + """ + Ensure that the request payload is valid for `POST /daemon/deliver/remote`. + """ + + try: + body = await request.json() + return DaemonDeliverRemoteRequest.model_validate(body) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"request body validation failed: {e}" + ) + + # # Admin endpoint validators # diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py index 0c7a9d9..2d6d573 100644 --- a/tests/e2e/conftest.py +++ b/tests/e2e/conftest.py @@ -42,6 +42,7 @@ class E2EStack: def __init__(self, home: Path) -> None: self.home = home + self.backend = "memory" self.port = _free_port() self.base_url = f"http://127.0.0.1:{self.port}" self.env = { @@ -57,10 +58,13 @@ def __init__(self, home: Path) -> None: # ─── provisioning and lifecycle ──────────────────────────────── - def provision(self) -> None: + def provision(self, backend: str = "memory") -> None: + self.backend = backend subprocess.run( [ str(VENV_BIN / "backend-init"), + "--type", + backend, "--swarm", SWARM, "--host", @@ -93,6 +97,8 @@ def start_server( ) -> None: command = [ str(VENV_BIN / "mail-server"), + "--backend", + self.backend, "--host", "127.0.0.1", "--port", @@ -217,3 +223,16 @@ def e2e_stack(tmp_path: Path) -> E2EStack: stack.start_server() yield stack stack.stop_server() + + +@pytest.fixture +def sqlite_e2e_stack(tmp_path: Path) -> E2EStack: + """An e2e stack provisioned and served on the sqlite backend.""" + + home = tmp_path / "home" + home.mkdir() + stack = E2EStack(home) + stack.provision(backend="sqlite") + stack.start_server() + yield stack + stack.stop_server() diff --git a/tests/e2e/test_sqlite_durability.py b/tests/e2e/test_sqlite_durability.py new file mode 100644 index 0000000..e5ffaeb --- /dev/null +++ b/tests/e2e/test_sqlite_durability.py @@ -0,0 +1,44 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +The durability property the sqlite backend exists to provide: a committed +message survives an abrupt ``kill -9`` with no graceful shutdown — the window +the memory backend's checkpoint cannot close. +""" + +HOST = "localhost" +USER = f"user:alice@{HOST}" +OTHER_USER = f"user:bob@{HOST}" + + +def test_sqlite_committed_message_survives_sigkill(sqlite_e2e_stack) -> None: + stack = sqlite_e2e_stack + alice = stack.login(USER) + bob = stack.login(OTHER_USER) + + draft = stack.cli_json( + "compose", "Durable", "Survives kill -9.", token=alice + ) + sent = stack.cli_json( + "send", draft["entry"]["draft"]["draft_id"], OTHER_USER, token=alice + ) + message_id = sent["message"]["message_id"] + + def bob_has_mail() -> bool: + inbox = stack.cli_json("inbox", token=bob) + return any(e["message_id"] == message_id for e in inbox["entries"]) + + with stack.daemon_running(): + stack.wait_for(bob_has_mail) + + # SIGKILL: no lifespan shutdown, no checkpoint — only what is already + # committed to the sqlite file can survive. + stack.kill_server() + stack.start_server() + + # JWTs are stateless, so the tokens remain valid across the restart. + opened = stack.cli_json("inbox-open", message_id, token=bob) + assert opened["entry"]["message"]["body"] == "Survives kill -9." + outbox = stack.cli_json("outbox", token=alice) + assert any(e["message_id"] == message_id for e in outbox["entries"]) diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 5c94c85..707c210 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -1,7 +1,10 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Charon Labs (contribution PR) +import asyncio import os +from collections.abc import Awaitable, Callable, Iterator +from datetime import datetime from pathlib import Path # mail_server.server reads MAIL_HOST and mail_server.routers.auth reads @@ -12,7 +15,10 @@ import pytest # noqa: E402 from fastapi.testclient import TestClient # noqa: E402 +from mail_protocol.core.lists import MAILListInBackend # noqa: E402 +from mail_protocol.core.messages import MAILMessage # noqa: E402 from mail_protocol.core.swarms import MAILSwarm # noqa: E402 +from mail_protocol.core.trash import MAILTrashEntry # noqa: E402 from mail_protocol.core.user_agents import ( # noqa: E402 MAILAdmin, MAILAgent, @@ -22,7 +28,14 @@ ) from mail_server import server as mail_server_module # noqa: E402 from mail_server.auth import get_password_hash # noqa: E402 +from mail_server.backends.base import MAILServerBackend # noqa: E402 from mail_server.backends.memory.api import MemoryBackend # noqa: E402 +from mail_server.backends.sqlite.api import SQLiteBackend # noqa: E402 +from mail_server.backends.sqlite.database import Database # noqa: E402 +from mail_server.backends.sqlite.repositories import ( # noqa: E402 + BOX_TRASH, + MailStore, +) HOST = "localhost" SWARM = "chorus" @@ -39,32 +52,33 @@ # per session instead of once per seeded user-agent per test. PASSWORD_HASH = get_password_hash(PASSWORD) +# The integration suite runs against every backend in this list. Backend +# internals are never touched directly — seeding/assertions go through the +# public API or the backend-agnostic ``seed_*`` fixtures below — so each test +# exercises identical behavior on memory and sqlite. +BACKENDS = ["memory", "sqlite"] -def _seed_cast(backend: MemoryBackend) -> None: - """ - Seed the standard cast: one admin, two users, one agent, one daemon, - and one swarm — all sharing PASSWORD. Mirrors what `backend-init` - plus admin CRUD calls would provision. - """ - cast = { +def _cast() -> dict[str, MAILUserAgentInBackend]: + """The standard cast (one admin, two users, one agent, one daemon).""" + + members = { ADMIN: MAILAdmin(ua_type="admin", admin_id="ryan", host=HOST), USER: MAILUser(ua_type="user", user_id="alice", host=HOST), OTHER_USER: MAILUser(ua_type="user", user_id="bob", host=HOST), AGENT: MAILAgent(ua_type="agent", name="sage", swarm=SWARM, host=HOST), DAEMON: MAILDaemon(ua_type="daemon", worker_name="dummy", host=HOST), } - for address, user_agent in cast.items(): - backend.user_agents[address] = MAILUserAgentInBackend( - user_agent=user_agent, - hashed_password=PASSWORD_HASH, + return { + address: MAILUserAgentInBackend( + user_agent=user_agent, hashed_password=PASSWORD_HASH ) - backend.inboxes[address] = [] - backend.outboxes[address] = [] - backend.drafts[address] = [] - backend.trashes[address] = [] + for address, user_agent in members.items() + } + - backend.swarms[SWARM] = MAILSwarm( +def _swarm() -> MAILSwarm: + return MAILSwarm( name=SWARM, description="integration test swarm", keywords=["testing"], @@ -73,28 +87,159 @@ def _seed_cast(backend: MemoryBackend) -> None: ) +def _seed_memory_cast(backend: MemoryBackend) -> None: + for address, ua_in_be in _cast().items(): + backend.user_agents[address] = ua_in_be + backend.inboxes[address] = [] + backend.outboxes[address] = [] + backend.drafts[address] = [] + backend.trashes[address] = [] + backend.swarms[SWARM] = _swarm() + + +def _run_sqlite_write( + url: str, + mutate: Callable[[MailStore], Awaitable[object]], + *, + create_schema: bool = False, +) -> None: + """ + Apply a write to a file-backed sqlite db via a throwaway engine. + + Per-test seeding can't reuse the app's engine (it is bound to the + TestClient's event loop), so we open a short-lived ``Database`` on the same + file in a fresh loop. WAL makes the committed rows visible to the app, and + seeding is sequential with the HTTP calls, so there is no write contention. + """ + + async def _run() -> None: + db = Database(url) + try: + if create_schema: + await db.create_schema() + async with db.session() as session: + await mutate(MailStore(session)) + finally: + await db.dispose() + + asyncio.run(_run()) + + +async def _seed_sqlite_cast(store: MailStore) -> None: + for ua_in_be in _cast().values(): + await store.user_agents.add(ua_in_be) + await store.swarms.add(_swarm()) + + +@pytest.fixture(params=BACKENDS) +def backend_kind(request: pytest.FixtureRequest) -> str: + """The backend under test for this parametrization (``memory``/``sqlite``).""" + + return request.param + + @pytest.fixture def app_client( + backend_kind: str, deployment_dir: Path, + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, -) -> TestClient: +) -> Iterator[TestClient]: """ - The real composed FastAPI app over ASGI with a fresh MemoryBackend, - seeded with the standard cast. Auth is NOT monkeypatched — requests - must carry real JWTs (see ``token_for`` / ``headers_for``). + The real composed FastAPI app over ASGI, seeded with the standard cast on + the selected backend. Auth is NOT monkeypatched — requests must carry real + JWTs (see ``token_for`` / ``headers_for``). """ - monkeypatch.setattr(mail_server_module, "_backend", MemoryBackend()) + if backend_kind == "memory": + monkeypatch.setattr(mail_server_module, "_backend", MemoryBackend()) + with TestClient(mail_server_module.app) as client: + _seed_memory_cast(mail_server_module.app.state.backend) + yield client + return + + # sqlite: seed the cast (and create the schema) before the app starts, so + # the rows are already committed when the lifespan runs. + db_url = f"sqlite:///{tmp_path / 'mail.db'}" + _run_sqlite_write(db_url, _seed_sqlite_cast, create_schema=True) + monkeypatch.setattr(mail_server_module, "_backend", SQLiteBackend(url=db_url)) with TestClient(mail_server_module.app) as client: - _seed_cast(client.app.state.backend) yield client @pytest.fixture -def backend(app_client: TestClient) -> MemoryBackend: +def backend(app_client: TestClient) -> MAILServerBackend: """The backend behind ``app_client`` (overrides the root fixture).""" - return app_client.app.state.backend + backend: MAILServerBackend = mail_server_module.app.state.backend + return backend + + +@pytest.fixture +def seed_trash(backend: MAILServerBackend) -> Callable[..., str]: + """ + Backend-agnostic: place ``message`` directly in ``owner``'s trash with the + given ``trashed_at`` (and register the message in the canonical store, which + the ``sent_at`` sort resolves against). Returns the message id. + + Bypassing the API is deliberate — it lets a test pin ``trashed_at`` and the + message's ``sent_at`` independently, which the natural inbox→trash flow + (both stamped with the wall clock) cannot. + """ + + def _seed(owner: str, *, message: MAILMessage, trashed_at: datetime) -> str: + entry = MAILTrashEntry(message=message, trashed_at=trashed_at) + if isinstance(backend, MemoryBackend): + backend.messages[message.message_id] = message + backend.trash_entries[message.message_id] = entry + backend.trashes.setdefault(owner, []).append(message.message_id) + else: + assert isinstance(backend, SQLiteBackend) + + async def mutate(store: MailStore) -> None: + await store.messages.add(message) + await store.boxes.upsert_trash_entry(entry) + await store.boxes.add_membership( + owner, BOX_TRASH, message.message_id, trashed_at + ) + + _run_sqlite_write(backend._db.url, mutate) + return message.message_id + + return _seed + + +@pytest.fixture +def seed_list(backend: MAILServerBackend) -> Callable[[MAILListInBackend], str]: + """Backend-agnostic: persist a prebuilt list. Returns its address.""" + + def _seed(record: MAILListInBackend) -> str: + if isinstance(backend, MemoryBackend): + backend.lists[record.get_address()] = record + else: + assert isinstance(backend, SQLiteBackend) + + async def mutate(store: MailStore) -> None: + await store.lists.add(record) + + _run_sqlite_write(backend._db.url, mutate) + return record.get_address() + + return _seed + + +@pytest.fixture +def list_members( + app_client: TestClient, headers_for: Callable[..., dict[str, str]] +) -> Callable[..., list[str]]: + """Read a list's members through the public API (backend-agnostic).""" + + def _members(address: str, viewer: str = USER) -> list[str]: + response = app_client.get(f"/lists/{address}", headers=headers_for(viewer)) + assert response.status_code == 200, response.text + return response.json()["mail_list"]["members"] + + return _members @pytest.fixture diff --git a/tests/integration/test_admin.py b/tests/integration/test_admin.py index 3213264..b1f0d7f 100644 --- a/tests/integration/test_admin.py +++ b/tests/integration/test_admin.py @@ -2,7 +2,6 @@ # Copyright (c) 2026 Charon Labs (contribution PR) from fastapi.testclient import TestClient -from mail_server.backends.memory.api import MemoryBackend ADMIN = "admin:ryan@localhost" SWARM = "chorus" @@ -81,18 +80,20 @@ def test_post_agent_invalid_name_returns_422( def test_delete_agent_removes_account_and_boxes( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: address = f"sage@{SWARM}@localhost" response = app_client.delete( f"/admin/agents/sage@{SWARM}", headers=headers_for(ADMIN) ) assert response.status_code == 200 - assert address not in backend.user_agents - assert address not in backend.inboxes - assert address not in backend.outboxes - assert address not in backend.drafts - assert address not in backend.trashes + # The account is gone: a follow-up admin read 404s. + assert ( + app_client.get( + f"/admin/agents/sage@{SWARM}", headers=headers_for(ADMIN) + ).status_code + == 404 + ) # Credentials no longer authenticate. response = app_client.post( @@ -152,11 +153,17 @@ def test_post_daemon_duplicate_returns_409(app_client: TestClient, headers_for) def test_delete_daemon_removes_account( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: response = app_client.delete("/admin/daemons/dummy", headers=headers_for(ADMIN)) assert response.status_code == 200 - assert "daemon:dummy@localhost" not in backend.user_agents + # The account is gone: a follow-up admin read 404s. + assert ( + app_client.get( + "/admin/daemons/dummy", headers=headers_for(ADMIN) + ).status_code + == 404 + ) def test_delete_daemon_unknown_returns_404(app_client: TestClient, headers_for) -> None: @@ -208,12 +215,17 @@ def test_post_user_duplicate_returns_409(app_client: TestClient, headers_for) -> def test_delete_user_removes_account_and_boxes( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: response = app_client.delete("/admin/users/bob", headers=headers_for(ADMIN)) assert response.status_code == 200 - assert "user:bob@localhost" not in backend.user_agents - assert "user:bob@localhost" not in backend.inboxes + # The account is gone: a follow-up admin read 404s. + assert ( + app_client.get( + "/admin/users/bob", headers=headers_for(ADMIN) + ).status_code + == 404 + ) def test_delete_user_unknown_returns_404(app_client: TestClient, headers_for) -> None: diff --git a/tests/integration/test_daemon.py b/tests/integration/test_daemon.py index 89e101e..dea499a 100644 --- a/tests/integration/test_daemon.py +++ b/tests/integration/test_daemon.py @@ -2,7 +2,6 @@ # Copyright (c) 2026 Charon Labs (contribution PR) from fastapi.testclient import TestClient -from mail_server.backends.memory.api import MemoryBackend USER = "user:alice@localhost" OTHER_USER = "user:bob@localhost" @@ -42,7 +41,7 @@ def test_clear_message_buffer_returns_pending_ids_once( def test_deliver_local_updates_recipient_inbox( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: message_id = _compose_and_send(app_client, headers_for(USER)) daemon_headers = headers_for(DAEMON) @@ -58,10 +57,15 @@ def test_deliver_local_updates_recipient_inbox( assert len(summaries) == 1 assert summaries[0]["message_id"] == message_id - assert message_id in backend.inboxes[OTHER_USER] - outbox_entry = backend.outbox_entries[message_id] - assert outbox_entry.delivered_at is not None - assert outbox_entry.delivered_by == DAEMON + # The recipient can now open it from their inbox... + opened = app_client.get( + f"/inbox/{message_id}", headers=headers_for(OTHER_USER) + ) + assert opened.status_code == 200 + # ...and the sender's outbox entry is marked delivered. + outbox = app_client.get(f"/outbox/{message_id}", headers=headers_for(USER)) + assert outbox.status_code == 200 + assert outbox.json()["entry"]["delivered_at"] is not None def test_deliver_local_skips_unknown_message_ids( @@ -88,7 +92,7 @@ def test_deliver_local_rejects_non_uuid_ids( def test_deliver_local_skips_unknown_recipient( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: """ A recipient address that doesn't resolve to a registered user-agent @@ -117,5 +121,8 @@ def test_deliver_local_skips_unknown_recipient( headers=daemon_headers, ) assert response.status_code == 200 - assert message_id in backend.inboxes[OTHER_USER] - assert "user:ghost@localhost" not in backend.inboxes + # The known recipient received it; the unknown one was skipped without error. + opened = app_client.get( + f"/inbox/{message_id}", headers=headers_for(OTHER_USER) + ) + assert opened.status_code == 200 diff --git a/tests/integration/test_gap_fill.py b/tests/integration/test_gap_fill.py new file mode 100644 index 0000000..189f0c9 --- /dev/null +++ b/tests/integration/test_gap_fill.py @@ -0,0 +1,137 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +End-to-end HTTP coverage for the endpoints that are ``NotImplementedError`` +stubs on the memory backend but fully implemented on sqlite (the routers were +wired to delegate to the backend). Pinned to the sqlite backend; the memory +backend's stub behavior is asserted in ``test_stubs.py``. +""" + +import pytest +from fastapi.testclient import TestClient + +USER = "user:alice@localhost" +OTHER_USER = "user:bob@localhost" +DAEMON = "daemon:dummy@localhost" +ADMIN = "admin:ryan@localhost" + + +@pytest.fixture +def backend_kind() -> str: + """Pin this suite to sqlite, where these endpoints are implemented.""" + + return "sqlite" + + +def test_delete_inbox_message_moves_to_trash( + app_client: TestClient, headers_for, deliver_message +) -> None: + message_id = deliver_message(USER, [OTHER_USER]) + bob = headers_for(OTHER_USER) + + response = app_client.delete(f"/inbox/{message_id}", headers=bob) + assert response.status_code == 200 + + # Gone from the inbox, now readable from trash. + assert app_client.get(f"/inbox/{message_id}", headers=bob).status_code == 404 + assert app_client.get(f"/trash/{message_id}", headers=bob).status_code == 200 + + +def test_delete_draft_removes_it(app_client: TestClient, headers_for) -> None: + headers = headers_for(USER) + response = app_client.post( + "/drafts", + json={"subject": "Disposable", "body": "Delete me"}, + headers=headers, + ) + draft_id = response.json()["entry"]["draft"]["draft_id"] + + response = app_client.delete(f"/drafts/{draft_id}", headers=headers) + assert response.status_code == 200 + assert app_client.get(f"/drafts/{draft_id}", headers=headers).status_code == 404 + + +def test_delete_trashed_message_removes_it( + app_client: TestClient, headers_for, deliver_message +) -> None: + message_id = deliver_message(USER, [OTHER_USER]) + bob = headers_for(OTHER_USER) + app_client.delete(f"/inbox/{message_id}", headers=bob) # inbox -> trash + + response = app_client.delete(f"/trash/{message_id}", headers=bob) + assert response.status_code == 200 + assert app_client.get(f"/trash/{message_id}", headers=bob).status_code == 404 + + +def test_trash_clear_empties_box( + app_client: TestClient, headers_for, deliver_message +) -> None: + bob = headers_for(OTHER_USER) + for _ in range(2): + message_id = deliver_message(USER, [OTHER_USER]) + app_client.delete(f"/inbox/{message_id}", headers=bob) + + response = app_client.post("/trash/clear", headers=bob) + assert response.status_code == 200 + assert len(response.json()["entries"]) == 2 + assert app_client.get("/trash", headers=bob).json()["entries"] == [] + + +def test_daemon_deliver_remote_delivers_to_local_inbox( + app_client: TestClient, headers_for +) -> None: + message_id = "99999999-9999-4999-8999-999999999999" + response = app_client.post( + "/daemon/deliver/remote", + json={ + "messages": [ + { + "mail_version": "2.0", + "message_id": message_id, + "sender": "echo@otherswarm@remote.example.com", + "recipients": [OTHER_USER], + "subject": "Remote", + "body": "from afar", + "tags": [], + "sent_at": "2026-06-24T00:00:00Z", + "metadata": {}, + } + ] + }, + headers=headers_for(DAEMON), + ) + assert response.status_code == 200 + assert response.json()["messages"][0]["message_id"] == message_id + + opened = app_client.get(f"/inbox/{message_id}", headers=headers_for(OTHER_USER)) + assert opened.status_code == 200 + assert opened.json()["entry"]["message"]["body"] == "from afar" + + +def test_patch_webhook_updates(app_client: TestClient, headers_for) -> None: + headers = headers_for(ADMIN) + response = app_client.post( + "/admin/webhooks", + json={ + "url": "https://example.com/mail-events", + "events": ["mail.delivered"], + "secret": "shhh", + }, + headers=headers, + ) + webhook_id = response.json()["webhook"]["webhook_id"] + + response = app_client.patch( + f"/admin/webhooks/{webhook_id}", + json={"url": "https://example.com/elsewhere", "secret": "new-secret"}, + headers=headers, + ) + assert response.status_code == 200 + body = response.json()["webhook"] + assert body["webhook_id"] == webhook_id # id preserved across URL move + assert body["url"] == "https://example.com/elsewhere" + + # Refetch by id reflects the new URL. + refetched = app_client.get(f"/admin/webhooks/{webhook_id}", headers=headers) + assert refetched.json()["webhook"]["url"] == "https://example.com/elsewhere" diff --git a/tests/integration/test_lists.py b/tests/integration/test_lists.py index 02d77ff..2a8d13d 100644 --- a/tests/integration/test_lists.py +++ b/tests/integration/test_lists.py @@ -8,11 +8,11 @@ minimal app with monkeypatched auth; the assertions are unchanged. """ +from collections.abc import Callable from datetime import UTC, datetime from fastapi.testclient import TestClient from mail_protocol.core.lists import MAILListInBackend, MAILListPolicy -from mail_server.backends.memory.api import MemoryBackend ADMIN_ADDRESS = "admin:ryan@localhost" USER_ADDRESS = "user:alice@localhost" @@ -35,7 +35,11 @@ def _make_post_body( return body -def _seed_list(backend: MemoryBackend, *, members: list[str] | None = None) -> str: +def _seed_list( + seed_list: Callable[[MAILListInBackend], str], + *, + members: list[str] | None = None, +) -> str: now = datetime(2026, 6, 4, 0, 0, tzinfo=UTC) record = MAILListInBackend( name="welfare-discourse", @@ -48,8 +52,7 @@ def _seed_list(backend: MemoryBackend, *, members: list[str] | None = None) -> s created_at=now, updated_at=now, ) - backend.lists[record.get_address()] = record - return record.get_address() + return seed_list(record) # ─── Admin endpoints ─────────────────────────────────────────────── @@ -115,18 +118,22 @@ def test_admin_post_list_duplicate_returns_409( def test_admin_get_lists_returns_all( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - _seed_list(backend) + _seed_list(seed_list) response = app_client.get("/admin/lists", headers=headers_for(ADMIN_ADDRESS)) assert response.status_code == 200 assert len(response.json()["lists"]) == 1 def test_admin_get_list_returns_specific( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.get( f"/admin/lists/{address}", headers=headers_for(ADMIN_ADDRESS) ) @@ -145,9 +152,11 @@ def test_admin_get_list_missing_returns_404( def test_admin_patch_list_updates_policy_no_op_for_open( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.patch( f"/admin/lists/{address}", json={"policy": MAILListPolicy().model_dump()}, @@ -157,9 +166,11 @@ def test_admin_patch_list_updates_policy_no_op_for_open( def test_admin_patch_list_rejects_closed_policy( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.patch( f"/admin/lists/{address}", json={"policy": MAILListPolicy(visibility="private").model_dump()}, @@ -169,14 +180,22 @@ def test_admin_patch_list_rejects_closed_policy( def test_admin_delete_list_removes( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.delete( f"/admin/lists/{address}", headers=headers_for(ADMIN_ADDRESS) ) assert response.status_code == 200 - assert address not in backend.lists + # The list is gone: a follow-up read 404s. + assert ( + app_client.get( + f"/lists/{address}", headers=headers_for(ADMIN_ADDRESS) + ).status_code + == 404 + ) def test_admin_delete_list_missing_returns_404( @@ -190,9 +209,11 @@ def test_admin_delete_list_missing_returns_404( def test_admin_add_member( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.post( f"/admin/lists/{address}/members", json={"member_address": "philosopher@chorus@localhost"}, @@ -203,21 +224,26 @@ def test_admin_add_member( def test_admin_remove_member( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: - address = _seed_list(backend, members=["philosopher@chorus@localhost"]) + address = _seed_list(seed_list, members=["philosopher@chorus@localhost"]) response = app_client.delete( f"/admin/lists/{address}/members/philosopher@chorus@localhost", headers=headers_for(ADMIN_ADDRESS), ) assert response.status_code == 200 - assert backend.lists[address].members == [] + assert list_members(address) == [] def test_admin_lists_reject_non_admin( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - _seed_list(backend) + _seed_list(seed_list) response = app_client.get( "/admin/lists", headers=headers_for(USER_ADDRESS) ) @@ -233,18 +259,22 @@ def test_get_lists_requires_auth(app_client: TestClient) -> None: def test_get_lists_returns_visible_lists( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - _seed_list(backend) + _seed_list(seed_list) response = app_client.get("/lists", headers=headers_for(USER_ADDRESS)) assert response.status_code == 200 assert len(response.json()["lists"]) == 1 def test_get_list_specific( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.get( f"/lists/{address}", headers=headers_for(USER_ADDRESS) ) @@ -263,18 +293,24 @@ def test_get_list_missing_returns_404( def test_subscribe_self( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.post( f"/lists/{address}/subscribe", headers=headers_for(USER_ADDRESS) ) assert response.status_code == 200 - assert USER_ADDRESS in backend.lists[address].members + assert USER_ADDRESS in list_members(address) def test_subscribe_ignores_supplied_member_address( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: """ Subscribe is body-less: only the authenticated caller is ever @@ -282,15 +318,16 @@ def test_subscribe_ignores_supplied_member_address( no effect on the member list. """ - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.post( f"/lists/{address}/subscribe", json={"member_address": OTHER_USER_ADDRESS}, headers=headers_for(USER_ADDRESS), ) assert response.status_code == 200 - assert USER_ADDRESS in backend.lists[address].members - assert OTHER_USER_ADDRESS not in backend.lists[address].members + members = list_members(address) + assert USER_ADDRESS in members + assert OTHER_USER_ADDRESS not in members def test_subscribe_missing_list_returns_404( @@ -304,24 +341,32 @@ def test_subscribe_missing_list_returns_404( def test_unsubscribe_self( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: - address = _seed_list(backend, members=[USER_ADDRESS, OTHER_USER_ADDRESS]) + address = _seed_list(seed_list, members=[USER_ADDRESS, OTHER_USER_ADDRESS]) response = app_client.post( f"/lists/{address}/unsubscribe", headers=headers_for(USER_ADDRESS) ) assert response.status_code == 200 - assert USER_ADDRESS not in backend.lists[address].members - assert OTHER_USER_ADDRESS in backend.lists[address].members + members = list_members(address) + assert USER_ADDRESS not in members + assert OTHER_USER_ADDRESS in members def test_unsubscribe_uses_authenticated_user( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: - address = _seed_list(backend, members=[USER_ADDRESS, OTHER_USER_ADDRESS]) + address = _seed_list(seed_list, members=[USER_ADDRESS, OTHER_USER_ADDRESS]) response = app_client.post( f"/lists/{address}/unsubscribe", headers=headers_for(OTHER_USER_ADDRESS) ) assert response.status_code == 200 - assert USER_ADDRESS in backend.lists[address].members - assert OTHER_USER_ADDRESS not in backend.lists[address].members + members = list_members(address) + assert USER_ADDRESS in members + assert OTHER_USER_ADDRESS not in members diff --git a/tests/integration/test_mailboxes.py b/tests/integration/test_mailboxes.py index 24964a0..301cf68 100644 --- a/tests/integration/test_mailboxes.py +++ b/tests/integration/test_mailboxes.py @@ -1,17 +1,17 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Charon Labs (contribution PR) +from collections.abc import Callable from datetime import UTC, datetime import pytest from fastapi.testclient import TestClient from mail_protocol.core.constants import MESSAGE_SUBJECT_LEN_MAX from mail_protocol.core.messages import MAILMessage -from mail_protocol.core.trash import MAILTrashEntry -from mail_server.backends.memory.api import MemoryBackend USER = "user:alice@localhost" OTHER_USER = "user:bob@localhost" +DAEMON = "daemon:dummy@localhost" # ─── Inbox ───────────────────────────────────────────────────────── @@ -199,7 +199,7 @@ def test_drafts_isolated_between_users(app_client: TestClient, headers_for) -> N def test_send_draft_creates_message_and_buffers_it( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: response = app_client.post( "/drafts", @@ -219,7 +219,13 @@ def test_send_draft_creates_message_and_buffers_it( assert message["recipients"] == [OTHER_USER] assert message["subject"] == "Outgoing" assert message["message_id"] != draft_id - assert message["message_id"] in backend.message_buffer + + # The message is queued for delivery: the daemon's buffer-clear returns it. + buffer = app_client.post( + "/daemon/message-buffer/clear", headers=headers_for(DAEMON) + ) + assert buffer.status_code == 200 + assert message["message_id"] in buffer.json()["message_ids"] def test_send_draft_rejects_empty_recipients( @@ -413,7 +419,7 @@ def test_patch_draft_then_send_uses_new_content( # ─── Trash ───────────────────────────────────────────────────────── -def _seed_trash(backend: MemoryBackend, owner: str) -> str: +def _seed_trash(seed_trash: Callable[..., str], owner: str) -> str: message = MAILMessage( mail_version="2.0", message_id="22222222-2222-4222-8222-222222222222", @@ -425,12 +431,11 @@ def _seed_trash(backend: MemoryBackend, owner: str) -> str: sent_at=datetime(2026, 6, 11, tzinfo=UTC), metadata={}, ) - backend.trash_entries[message.message_id] = MAILTrashEntry( + return seed_trash( + owner, message=message, trashed_at=datetime(2026, 6, 11, 12, 0, tzinfo=UTC), ) - backend.trashes[owner].append(message.message_id) - return message.message_id def test_trash_starts_empty(app_client: TestClient, headers_for) -> None: @@ -440,9 +445,9 @@ def test_trash_starts_empty(app_client: TestClient, headers_for) -> None: def test_trash_lists_seeded_entry( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - message_id = _seed_trash(backend, USER) + message_id = _seed_trash(seed_trash, USER) response = app_client.get("/trash", headers=headers_for(USER)) assert response.status_code == 200 entries = response.json()["entries"] @@ -451,9 +456,9 @@ def test_trash_lists_seeded_entry( def test_trash_open_returns_entry( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - message_id = _seed_trash(backend, USER) + message_id = _seed_trash(seed_trash, USER) response = app_client.get(f"/trash/{message_id}", headers=headers_for(USER)) assert response.status_code == 200 assert response.json()["entry"]["message"]["subject"] == "Trashed" @@ -468,9 +473,9 @@ def test_trash_open_unknown_id_returns_404(app_client: TestClient, headers_for) def test_trash_isolated_between_users( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - message_id = _seed_trash(backend, USER) + message_id = _seed_trash(seed_trash, USER) response = app_client.get(f"/trash/{message_id}", headers=headers_for(OTHER_USER)) assert response.status_code == 404 @@ -478,7 +483,7 @@ def test_trash_isolated_between_users( # ─── Box query parameters ────────────────────────────────────────── -def _seed_trash_n(backend: MemoryBackend, owner: str, n: int) -> list[str]: +def _seed_trash_n(seed_trash: Callable[..., str], owner: str, n: int) -> list[str]: """ Seed ``n`` trash entries for ``owner``. ``trashed_at`` *increases* with insertion order while the underlying message's ``sent_at`` *decreases*, so @@ -486,9 +491,9 @@ def _seed_trash_n(backend: MemoryBackend, owner: str, n: int) -> list[str]: orders — letting a single seed exercise both. Returns the message IDs in insertion order (oldest-trashed first), so ``ids[-1]`` is the newest. - The message is also registered in ``backend.messages`` because the - ``sent_at`` sort resolves send time via that store (as the real local - delivery path populates it). + The message is registered in the canonical store too (the ``sent_at`` sort + resolves send time via that store, as the real local delivery path does); + the ``seed_trash`` fixture handles that for whichever backend is active. """ ids: list[str] = [] @@ -505,12 +510,11 @@ def _seed_trash_n(backend: MemoryBackend, owner: str, n: int) -> list[str]: sent_at=datetime(2026, 6, 1, 12, n - i, tzinfo=UTC), # decreasing metadata={}, ) - backend.messages[message_id] = message - backend.trash_entries[message_id] = MAILTrashEntry( + seed_trash( + owner, message=message, trashed_at=datetime(2026, 6, 11, 12, i, tzinfo=UTC), # increasing ) - backend.trashes[owner].append(message_id) ids.append(message_id) return ids @@ -558,9 +562,9 @@ def test_box_rejects_invalid_query_params( def test_box_pagination_slices_and_counts( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - ids = _seed_trash_n(backend, USER, 5) # oldest → newest + ids = _seed_trash_n(seed_trash, USER, 5) # oldest → newest response = app_client.get("/trash?limit=2&offset=0", headers=headers_for(USER)) assert response.status_code == 200 @@ -575,9 +579,9 @@ def test_box_pagination_slices_and_counts( def test_box_sort_order_ascending( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - ids = _seed_trash_n(backend, USER, 3) + ids = _seed_trash_n(seed_trash, USER, 3) response = app_client.get("/trash?order=asc", headers=headers_for(USER)) assert response.status_code == 200 @@ -585,9 +589,9 @@ def test_box_sort_order_ascending( def test_box_offset_past_end_returns_empty_page( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - _seed_trash_n(backend, USER, 3) + _seed_trash_n(seed_trash, USER, 3) response = app_client.get("/trash?offset=10", headers=headers_for(USER)) assert response.status_code == 200 @@ -598,14 +602,14 @@ def test_box_offset_past_end_returns_empty_page( def test_box_sort_by_sent_at_uses_message_send_time( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: """ `sort_by=sent_at` orders by the underlying message's send time, which the seed makes the exact reverse of the default `entered_at` (trashed_at) order. """ - ids = _seed_trash_n(backend, USER, 3) + ids = _seed_trash_n(seed_trash, USER, 3) default = app_client.get("/trash", headers=headers_for(USER)) assert [e["message_id"] for e in default.json()["entries"]] == [ diff --git a/tests/integration/test_stubs.py b/tests/integration/test_stubs.py index 3b44b01..9d8ddd8 100644 --- a/tests/integration/test_stubs.py +++ b/tests/integration/test_stubs.py @@ -24,6 +24,20 @@ ) +@pytest.fixture +def backend_kind() -> str: + """ + Pin this suite to the memory backend. + + These endpoints are ``NotImplementedError`` stubs on the *memory* backend + only; the sqlite backend implements them all (its routes are exercised in + ``test_gap_fill.py``). Overriding the parametrized fixture from + ``conftest.py`` keeps the strict-xfail assertions valid. + """ + + return "memory" + + @stub def test_delete_inbox_message_moves_to_trash( app_client: TestClient, headers_for, deliver_message diff --git a/tests/unit/test_sqlite_backend.py b/tests/unit/test_sqlite_backend.py new file mode 100644 index 0000000..ee6b371 --- /dev/null +++ b/tests/unit/test_sqlite_backend.py @@ -0,0 +1,339 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Direct end-to-end coverage for ``SQLiteBackend``, exercised through the public +protocol methods against a real temp-file database. + +Emphasis on the gap-fill methods the memory backend leaves as +``NotImplementedError`` (``delete_inbox_message``, ``delete_draft``, +``delete_trash_message``, ``clear_trash``, ``admin_webhook_patch``, +``daemon_deliver_remote``), since the shared integration suite cannot reach +them — their HTTP routes are still router-level stubs. +""" + +from collections.abc import AsyncIterator +from datetime import UTC, datetime +from pathlib import Path + +import pytest +from mail_protocol.core.lists import MAILListPolicy +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILAgent, + MAILDaemon, + MAILUser, + MAILUserAgent, +) +from mail_protocol.network.requests import ( + AdminAgentPostRequest, + AdminDaemonPostRequest, + AdminListPatchRequest, + AdminListPostRequest, + AdminSwarmPostRequest, + AdminUserPostRequest, + AdminWebhooksPatchRequest, + AdminWebhooksPostRequest, + AuthPasswordResetRequest, + BoxFilterParams, + DaemonDeliverLocalRequest, + DaemonDeliverRemoteRequest, + DraftPatchRequest, + DraftPostRequest, + DraftSendPostRequest, +) +from mail_server.backends.sqlite.api import SQLiteBackend + +ADMIN = MAILAdmin(ua_type="admin", admin_id="ryan", host="localhost") +DAEMON = MAILDaemon(ua_type="daemon", worker_name="dummy", host="localhost") +ALICE = MAILUserAgent( + user_agent=MAILUser(ua_type="user", user_id="alice", host="localhost") +) +SAGE = MAILUserAgent( + user_agent=MAILAgent( + ua_type="agent", name="sage", swarm="chorus", host="localhost" + ) +) +SAGE_ADDR = "sage@chorus@localhost" + + +@pytest.fixture +async def backend(tmp_path: Path) -> AsyncIterator[SQLiteBackend]: + be = SQLiteBackend(f"sqlite:///{tmp_path / 'mail.db'}") + await be.on_server_startup(host="localhost") + # A cast of one user (sender) and one agent (recipient). + await be.admin_post_user( + ADMIN, AdminUserPostRequest(user_id="alice", user_password="pw") + ) + await be.admin_post_agent( + ADMIN, + AdminAgentPostRequest( + agent_name="sage", swarm_name="chorus", agent_password="pw" + ), + ) + await be.admin_post_daemon( + ADMIN, AdminDaemonPostRequest(worker_name="dummy", daemon_password="pw") + ) + yield be + await be.on_server_shutdown() + + +async def _send(backend: SQLiteBackend, recipients: list[str]) -> str: + """Alice drafts and sends a message; return its message id.""" + + entry = await backend.post_draft( + ALICE, DraftPostRequest(subject="Hi", body="hello there") + ) + message = await backend.send_draft( + ALICE, + entry.draft.draft_id, + DraftSendPostRequest(recipients=recipients), + ) + return message.message_id + + +# --------------------------------------------------------------------------- # +# Admin CRUD + lifecycle +# --------------------------------------------------------------------------- # + + +async def test_admin_agent_crud_and_duplicates(backend: SQLiteBackend) -> None: + assert await backend.admin_get_agents(ADMIN) == ["sage@chorus"] + assert (await backend.admin_get_agent(ADMIN, "sage@chorus")).name == "sage" + + with pytest.raises(ValueError, match="already taken"): + await backend.admin_post_agent( + ADMIN, + AdminAgentPostRequest( + agent_name="sage", swarm_name="chorus", agent_password="pw" + ), + ) + + deleted = await backend.admin_delete_agent(ADMIN, "sage@chorus") + assert deleted.name == "sage" + assert await backend.admin_get_agents(ADMIN) == [] + + +async def test_reset_password(backend: SQLiteBackend) -> None: + assert await backend.user_agent_exists("user:alice@localhost") + result = await backend.reset_password( + ALICE.user_agent, + AuthPasswordResetRequest(current_password="pw", new_password="pw2"), + ) + assert result == "success" + with pytest.raises(ValueError, match="incorrect password"): + await backend.reset_password( + ALICE.user_agent, + AuthPasswordResetRequest(current_password="wrong", new_password="pw3"), + ) + + +# --------------------------------------------------------------------------- # +# Draft -> send -> deliver -> inbox lifecycle +# --------------------------------------------------------------------------- # + + +async def test_full_local_delivery_lifecycle(backend: SQLiteBackend) -> None: + message_id = await _send(backend, [SAGE_ADDR]) + + # The send lands in alice's outbox and the delivery buffer. + outbox, total = await backend.get_outbox(ALICE, BoxFilterParams()) + assert total == 1 and outbox[0].message_id == message_id + + buffered = await backend.daemon_clear_message_buffer(DAEMON) + assert buffered == [message_id] + # Buffer is drained; a second clear is empty. + assert await backend.daemon_clear_message_buffer(DAEMON) == [] + + delivered = await backend.daemon_deliver_local( + DAEMON, DaemonDeliverLocalRequest(message_ids=[message_id]) + ) + assert [m.message_id for m in delivered] == [message_id] + + # The agent recipient now has it in their inbox. + inbox, total = await backend.get_inbox(SAGE, BoxFilterParams()) + assert total == 1 and inbox[0].message_id == message_id + full = await backend.get_inbox_message(SAGE, message_id) + assert full.message.body == "hello there" + assert full.delivered_by == DAEMON.get_address() + + # The sender's outbox entry is marked delivered. + out_msg = await backend.get_outbox_message(ALICE, message_id) + assert out_msg.delivered_at is not None + + +async def test_draft_patch_and_delete(backend: SQLiteBackend) -> None: + entry = await backend.post_draft( + ALICE, DraftPostRequest(subject="Draft", body="body") + ) + draft_id = entry.draft.draft_id + + patched = await backend.patch_draft( + ALICE, draft_id, DraftPatchRequest(subject="Edited") + ) + assert patched.draft.subject == "Edited" + assert patched.draft.updated_at is not None + + deleted = await backend.delete_draft(ALICE, draft_id) + assert deleted.draft.draft_id == draft_id + with pytest.raises(ValueError, match="not found in draft box"): + await backend.get_draft(ALICE, draft_id) + + +# --------------------------------------------------------------------------- # +# Gap-fill: inbox -> trash move, trash delete, clear +# --------------------------------------------------------------------------- # + + +async def test_delete_inbox_message_moves_to_trash(backend: SQLiteBackend) -> None: + message_id = await _send(backend, [SAGE_ADDR]) + await backend.daemon_deliver_local( + DAEMON, DaemonDeliverLocalRequest(message_ids=[message_id]) + ) + + moved = await backend.delete_inbox_message(SAGE, message_id) + assert moved.message.message_id == message_id + + # Gone from inbox, present in trash. + inbox, inbox_total = await backend.get_inbox(SAGE, BoxFilterParams()) + assert inbox_total == 0 and inbox == [] + trash, trash_total = await backend.get_trash(SAGE, BoxFilterParams()) + assert trash_total == 1 and trash[0].message_id == message_id + + fetched = await backend.get_trash_message(SAGE, message_id) + assert fetched.message.message_id == message_id + + +async def test_delete_trash_message_and_clear(backend: SQLiteBackend) -> None: + first = await _send(backend, [SAGE_ADDR]) + second = await _send(backend, [SAGE_ADDR]) + await backend.daemon_deliver_local( + DAEMON, DaemonDeliverLocalRequest(message_ids=[first, second]) + ) + await backend.delete_inbox_message(SAGE, first) + await backend.delete_inbox_message(SAGE, second) + + removed = await backend.delete_trash_message(SAGE, first) + assert removed.message.message_id == first + _, total = await backend.get_trash(SAGE, BoxFilterParams()) + assert total == 1 + + cleared = await backend.clear_trash(SAGE) + assert [s.message_id for s in cleared] == [second] + _, total_after = await backend.get_trash(SAGE, BoxFilterParams()) + assert total_after == 0 + + +# --------------------------------------------------------------------------- # +# Gap-fill: webhook patch + remote delivery +# --------------------------------------------------------------------------- # + + +async def test_webhook_crud_and_patch(backend: SQLiteBackend) -> None: + created = await backend.admin_webhook_post( + ADMIN, + AdminWebhooksPostRequest( + url="https://hooks.example.com/a", + events=["mail.delivered"], + secret="s1", + ), + ) + # Idempotent on URL. + again = await backend.admin_webhook_post( + ADMIN, + AdminWebhooksPostRequest( + url="https://hooks.example.com/a", + events=["mail.delivered"], + secret="ignored", + ), + ) + assert again.webhook_id == created.webhook_id + + patched = await backend.admin_webhook_patch( + ADMIN, + created.webhook_id, + AdminWebhooksPatchRequest(url="https://hooks.example.com/b", secret="s2"), + ) + assert patched.webhook_id == created.webhook_id # id preserved across URL move + assert patched.url == "https://hooks.example.com/b" + assert patched.secret == "s2" + + # Refetch by id reflects the move; the old URL is gone. + refetched = await backend.admin_webhook_get(ADMIN, created.webhook_id) + assert refetched.url == "https://hooks.example.com/b" + assert await backend.admin_webhooks_get(ADMIN) == [created.webhook_id] + + with pytest.raises(ValueError, match="not found"): + await backend.admin_webhook_patch( + ADMIN, + "wh_missing", + AdminWebhooksPatchRequest(url="https://x.example.com", secret="s"), + ) + + +async def test_daemon_deliver_remote(backend: SQLiteBackend) -> None: + remote = MAILMessage( + mail_version="2.0", + message_id="99999999-9999-4999-8999-999999999999", + sender="echo@otherswarm@remote.example.com", + recipients=[SAGE_ADDR], + subject="Remote", + body="from afar", + tags=[], + sent_at=datetime.now(UTC), + metadata={}, + ) + delivered = await backend.daemon_deliver_remote( + DAEMON, DaemonDeliverRemoteRequest(messages=[remote]) + ) + assert [m.message_id for m in delivered] == [remote.message_id] + + inbox, total = await backend.get_inbox(SAGE, BoxFilterParams()) + assert total == 1 and inbox[0].message_id == remote.message_id + # The remote message was persisted into the canonical store. + assert (await backend.get_message(remote.message_id)).body == "from afar" + + +# --------------------------------------------------------------------------- # +# Swarms + lists +# --------------------------------------------------------------------------- # + + +async def test_swarm_and_list_crud(backend: SQLiteBackend) -> None: + await backend.admin_post_swarm( + ADMIN, + AdminSwarmPostRequest(name="newswarm", description="d", keywords=["k"]), + ) + assert (await backend.get_swarm("newswarm")).name == "newswarm" + assert await backend.get_swarm_health("newswarm") == "ok" + + created = await backend.admin_post_list( + ADMIN, + AdminListPostRequest( + name="team", + swarm_name="chorus", + owner="user:alice@localhost", + members=[], + ), + ) + address = created.get_address() + + with_member = await backend.add_list_member(address, SAGE_ADDR) + assert with_member.members == [SAGE_ADDR] + # Idempotent re-add. + assert (await backend.add_list_member(address, SAGE_ADDR)).members == [SAGE_ADDR] + + without = await backend.remove_list_member(address, SAGE_ADDR) + assert without.members == [] + + patched = await backend.admin_patch_list( + ADMIN, + address, + AdminListPatchRequest(policy=MAILListPolicy(visibility="private")), + ) + assert patched.policy.visibility == "private" + + await backend.admin_delete_list(ADMIN, address) + with pytest.raises(ValueError, match="list not found"): + await backend.get_list(address) diff --git a/tests/unit/test_sqlite_cli_wiring.py b/tests/unit/test_sqlite_cli_wiring.py new file mode 100644 index 0000000..0c22db0 --- /dev/null +++ b/tests/unit/test_sqlite_cli_wiring.py @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +CLI / server wiring for the sqlite backend: the ``--backend sqlite`` flag, its +``--sqlite-path`` / ``--database-url`` options (with env fallbacks), the URL +resolution precedence, and that ``run_server`` actually constructs a +``SQLiteBackend``. +""" + +import os +from argparse import Namespace +from pathlib import Path + +import pytest + +# server.py reads MAIL_HOST and routers.auth reads MAIL_JWT_EXPIRE_MINUTES at +# import time. +os.environ.setdefault("MAIL_HOST", "localhost") +os.environ.setdefault("MAIL_JWT_EXPIRE_MINUTES", "15") + +from mail_server import server as server_module # noqa: E402 +from mail_server.backends.sqlite.api import SQLiteBackend # noqa: E402 +from mail_server.cli import build_parser # noqa: E402 + + +def test_parser_accepts_sqlite_backend_and_options() -> None: + parser = build_parser() + args = parser.parse_args( + ["--backend", "sqlite", "--sqlite-path", "/tmp/mail.db"] + ) + assert args.backend == "sqlite" + assert args.sqlite_path == "/tmp/mail.db" + assert args.database_url is None + + +def test_parser_reads_env_fallbacks(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("MAIL_SQLITE_PATH", "/env/path.db") + monkeypatch.setenv("MAIL_DATABASE_URL", "sqlite:////env/url.db") + args = build_parser().parse_args([]) + assert args.sqlite_path == "/env/path.db" + assert args.database_url == "sqlite:////env/url.db" + + +def test_resolve_url_prefers_database_url() -> None: + args = Namespace( + database_url="sqlite:////abs/custom.db", sqlite_path="/ignored.db" + ) + assert server_module._resolve_sqlite_url(args) == "sqlite:////abs/custom.db" + + +def test_resolve_url_uses_sqlite_path() -> None: + args = Namespace(database_url=None, sqlite_path="/var/lib/mail/mail.db") + assert ( + server_module._resolve_sqlite_url(args) + == "sqlite:////var/lib/mail/mail.db" + ) + + +def test_resolve_url_falls_back_to_default() -> None: + args = Namespace(database_url=None, sqlite_path=None) + url = server_module._resolve_sqlite_url(args) + assert url.startswith("sqlite:///") + assert url.endswith("deployments/default/mail.db") + + +def test_run_server_constructs_sqlite_backend( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # Stub out the blocking calls so run_server just performs backend selection. + monkeypatch.setattr(server_module, "init_logger", lambda: None) + monkeypatch.setattr(server_module.uvicorn, "run", lambda *a, **k: None) + + args = Namespace( + backend="sqlite", + host="localhost", + port=8000, + sqlite_path=str(tmp_path / "mail.db"), + database_url=None, + ) + server_module.run_server(args) + + assert isinstance(server_module._backend, SQLiteBackend) diff --git a/tests/unit/test_sqlite_database.py b/tests/unit/test_sqlite_database.py new file mode 100644 index 0000000..d106f4e --- /dev/null +++ b/tests/unit/test_sqlite_database.py @@ -0,0 +1,123 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Database-level guarantees for the sqlite backend: the connection pragmas, the +all-or-nothing transaction boundary around a multi-write operation, and that +concurrent writers don't trip ``database is locked`` (WAL + busy_timeout). +""" + +import asyncio +from collections.abc import AsyncIterator +from pathlib import Path + +import pytest +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILDaemon, + MAILUser, + MAILUserAgent, +) +from mail_protocol.network.requests import ( + AdminUserPostRequest, + BoxFilterParams, + DraftPostRequest, + DraftSendPostRequest, +) +from mail_server.backends.sqlite.api import SQLiteBackend +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.repositories import ( + MailStore, + MessageBufferRepository, +) +from sqlalchemy import text + +ADMIN = MAILAdmin(ua_type="admin", admin_id="ryan", host="localhost") +ALICE = MAILUserAgent( + user_agent=MAILUser(ua_type="user", user_id="alice", host="localhost") +) + + +@pytest.fixture +async def backend(tmp_path: Path) -> AsyncIterator[SQLiteBackend]: + be = SQLiteBackend(f"sqlite:///{tmp_path / 'mail.db'}") + await be.on_server_startup(host="localhost") + await be.admin_post_user( + ADMIN, AdminUserPostRequest(user_id="alice", user_password="pw") + ) + yield be + await be.on_server_shutdown() + + +async def test_connection_pragmas_applied(tmp_path: Path) -> None: + db = Database(f"sqlite:///{tmp_path / 'mail.db'}") + await db.create_schema() + try: + async with db.session() as session: + journal_mode = (await session.execute(text("PRAGMA journal_mode"))).scalar() + foreign_keys = (await session.execute(text("PRAGMA foreign_keys"))).scalar() + busy_timeout = (await session.execute(text("PRAGMA busy_timeout"))).scalar() + finally: + await db.dispose() + + assert journal_mode == "wal" + assert foreign_keys == 1 + assert busy_timeout == 5000 + + +async def test_send_draft_rolls_back_on_failure( + backend: SQLiteBackend, monkeypatch: pytest.MonkeyPatch +) -> None: + """A failure on the final write of ``send_draft`` leaves no partial rows.""" + + entry = await backend.post_draft( + ALICE, DraftPostRequest(subject="Tx", body="atomic") + ) + draft_id = entry.draft.draft_id + + async def _boom(self: MessageBufferRepository, message_id: str) -> None: + raise RuntimeError("buffer write failed") + + # Fail on the last step (buffer enqueue), after message/outbox/membership. + monkeypatch.setattr(MessageBufferRepository, "enqueue", _boom) + + with pytest.raises(RuntimeError, match="buffer write failed"): + await backend.send_draft( + ALICE, draft_id, DraftSendPostRequest(recipients=["user:alice@localhost"]) + ) + + # Nothing from the aborted send survived: no outbox entry, empty buffer. + _, outbox_total = await backend.get_outbox(ALICE, BoxFilterParams()) + assert outbox_total == 0 + async with backend._db.session() as session: + assert await MailStore(session).buffer.list_ids() == [] + # ...and the draft is untouched, so the send can be retried. + assert (await backend.get_draft(ALICE, draft_id)).draft.draft_id == draft_id + + +async def test_concurrent_sends_do_not_lock(backend: SQLiteBackend) -> None: + """WAL + busy_timeout: concurrent committed sends don't raise locked.""" + + drafts = [ + await backend.post_draft( + ALICE, DraftPostRequest(subject=f"D{i}", body="body") + ) + for i in range(8) + ] + + messages = await asyncio.gather( + *( + backend.send_draft( + ALICE, + entry.draft.draft_id, + DraftSendPostRequest(recipients=["user:alice@localhost"]), + ) + for entry in drafts + ) + ) + + # Every send committed a distinct message into the delivery buffer. + assert len({m.message_id for m in messages}) == 8 + daemon = MAILDaemon(ua_type="daemon", worker_name="dummy", host="localhost") + buffered = await backend.daemon_clear_message_buffer(daemon) + assert sorted(buffered) == sorted(m.message_id for m in messages) diff --git a/tests/unit/test_sqlite_init.py b/tests/unit/test_sqlite_init.py new file mode 100644 index 0000000..df84d97 --- /dev/null +++ b/tests/unit/test_sqlite_init.py @@ -0,0 +1,91 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Coverage for ``init_sqlite_backend`` — the ``backend-init --type sqlite`` path. + +Verifies it seeds the swarm + every user-agent, writes plaintext secrets whose +hash verifies, and is safely idempotent on re-run (no duplicates, secrets +preserved). +""" + +from pathlib import Path + +from mail_protocol.core.user_agents import MAILAdmin +from mail_server.auth import verify_password +from mail_server.backends.sqlite.api import SQLiteBackend +from mail_server.backends.sqlite.init import default_sqlite_path, init_sqlite_backend + +_ADMIN = MAILAdmin(ua_type="admin", admin_id="ryan", host="localhost") + + +def test_default_sqlite_path_layout() -> None: + path = default_sqlite_path("mydep") + assert path.name == "mail.db" + assert path.parent.name == "mydep" + assert path.parent.parent.name == "deployments" + + +async def test_init_seeds_swarm_agents_and_secrets(tmp_path: Path) -> None: + db_path = tmp_path / "mail.db" + await init_sqlite_backend( + swarm="chorus", + agents=["supervisor"], + daemons=["dummy"], + users=["alice"], + admins=["ryan"], + host="localhost", + db_path=db_path, + ) + + assert db_path.exists() + secrets_dir = tmp_path / ".secrets" + addresses = { + "supervisor@chorus@localhost", + "daemon:dummy@localhost", + "user:alice@localhost", + "admin:ryan@localhost", + } + assert {p.name for p in secrets_dir.iterdir()} == addresses + + backend = SQLiteBackend(f"sqlite:///{db_path}") + await backend.on_server_startup(host="localhost") + try: + assert (await backend.get_swarm("chorus")).agents == ["supervisor"] + for address in addresses: + ua = await backend.get_user_agent(address) + secret = (secrets_dir / address).read_text(encoding="utf-8") + # The plaintext written to disk verifies against the stored hash. + assert verify_password( + plain_password=secret, hashed_password=ua.hashed_password + ) + finally: + await backend.on_server_shutdown() + + +async def test_init_is_idempotent(tmp_path: Path) -> None: + db_path = tmp_path / "mail.db" + kwargs = dict( + swarm="chorus", + agents=["supervisor"], + daemons=["dummy"], + users=["alice"], + admins=["ryan"], + host="localhost", + db_path=db_path, + ) + await init_sqlite_backend(**kwargs) # type: ignore[arg-type] + secret_before = (tmp_path / ".secrets" / "user:alice@localhost").read_text() + + # Re-running must not duplicate rows or rotate the existing secret. + await init_sqlite_backend(**kwargs) # type: ignore[arg-type] + secret_after = (tmp_path / ".secrets" / "user:alice@localhost").read_text() + assert secret_before == secret_after + + backend = SQLiteBackend(f"sqlite:///{db_path}") + await backend.on_server_startup(host="localhost") + try: + assert await backend.admin_get_users(_ADMIN) == ["alice"] + assert await backend.admin_get_agents(_ADMIN) == ["supervisor@chorus"] + finally: + await backend.on_server_shutdown() diff --git a/tests/unit/test_sqlite_migrate.py b/tests/unit/test_sqlite_migrate.py new file mode 100644 index 0000000..804184c --- /dev/null +++ b/tests/unit/test_sqlite_migrate.py @@ -0,0 +1,187 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +One-time filesystem (memory) -> sqlite import. + +Writes a small deployment with the memory backend's own ``save_*`` loaders, +imports it, and verifies every collection — and the reconstructed per-owner box +membership — is readable through ``SQLiteBackend``. +""" + +from datetime import UTC, datetime +from pathlib import Path + +import pytest +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILList, MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.outbox import MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.user_agents import ( + MAILAgent, + MAILDaemon, + MAILUser, + MAILUserAgent, + MAILUserAgentInBackend, +) +from mail_protocol.core.webhooks import MAILWebhook +from mail_protocol.network.requests import BoxFilterParams +from mail_server.backends.memory import fs as memory_fs +from mail_server.backends.sqlite.api import SQLiteBackend +from mail_server.backends.sqlite.migrate import import_memory_deployment + +NOW = datetime(2026, 6, 12, 9, 0, tzinfo=UTC) +MID = "55555555-5555-4555-8555-555555555555" +UALICE = "user:alice@localhost" +SAGE = "sage@chorus@localhost" +DAEMON = "daemon:dummy@localhost" +LIST_ADDR = "list:team@chorus@localhost" + +ALICE_UA = MAILUserAgent( + user_agent=MAILUser(ua_type="user", user_id="alice", host="localhost") +) +SAGE_UA = MAILUserAgent( + user_agent=MAILAgent(ua_type="agent", name="sage", swarm="chorus", host="localhost") +) + + +async def _write_fs_deployment() -> None: + """Populate the (monkeypatched) memory deployment tree.""" + + await memory_fs.save_user_agents( + { + UALICE: MAILUserAgentInBackend( + user_agent=ALICE_UA.user_agent, hashed_password="h1" + ), + SAGE: MAILUserAgentInBackend( + user_agent=SAGE_UA.user_agent, hashed_password="h2" + ), + } + ) + await memory_fs.save_swarms( + { + "chorus": MAILSwarm( + name="chorus", + description="d", + keywords=["k"], + agents=["sage"], + metadata={}, + ) + } + ) + await memory_fs.save_messages( + { + MID: MAILMessage( + mail_version="2.0", + message_id=MID, + sender=UALICE, + recipients=[SAGE], + subject="Imported", + body="survives migration", + tags=[], + sent_at=NOW, + metadata={}, + ) + } + ) + await memory_fs.save_inbox_entries( + { + MID: MAILInboxEntrySummary( + message_id=MID, + sender=UALICE, + subject="Imported", + body_size=18, + received_at=NOW, + delivered_by=DAEMON, + ) + } + ) + await memory_fs.save_inboxes({SAGE: [MID], UALICE: []}) + await memory_fs.save_outbox_entries( + { + MID: MAILOutboxEntrySummary( + message_id=MID, + recipients=[SAGE], + subject="Imported", + body_size=18, + sent_at=NOW, + delivered_at=NOW, + delivered_by=DAEMON, + ) + } + ) + await memory_fs.save_outboxes({UALICE: [MID]}) + await memory_fs.save_message_buffer([MID]) + await memory_fs.save_webhooks( + { + "https://hooks.example.com/mail": MAILWebhook( + webhook_id=f"wh_{MID}", + url="https://hooks.example.com/mail", + events=["mail.delivered"], + secret="shh", + ) + } + ) + await memory_fs.save_lists( + { + LIST_ADDR: MAILListInBackend( + **MAILList( + name="team", swarm="chorus", host="localhost", owner=UALICE + ).model_dump(), + list_id=MID, + created_at=NOW, + updated_at=NOW, + ) + } + ) + + +async def test_import_filesystem_deployment(deployment_dir: Path) -> None: + await _write_fs_deployment() + db_path = deployment_dir / "mail.db" + + counts = await import_memory_deployment(source_dir=deployment_dir, db_path=db_path) + assert counts["user_agents"] == 2 + assert counts["messages"] == 1 + assert counts["buffered"] == 1 + + backend = SQLiteBackend(f"sqlite:///{db_path}") + await backend.on_server_startup(host="localhost") + try: + # User-agents + swarm imported. + assert (await backend.get_user_agent(SAGE)).hashed_password == "h2" + assert (await backend.get_swarm("chorus")).agents == ["sage"] + + # The recipient's inbox membership + entry survived... + inbox, total = await backend.get_inbox(SAGE_UA, BoxFilterParams()) + assert total == 1 and inbox[0].message_id == MID + opened = await backend.get_inbox_message(SAGE_UA, MID) + assert opened.message.body == "survives migration" + assert opened.delivered_by == DAEMON + + # ...as did the sender's outbox, the buffer, the webhook, and the list. + _, outbox_total = await backend.get_outbox(ALICE_UA, BoxFilterParams()) + assert outbox_total == 1 + daemon = MAILDaemon(ua_type="daemon", worker_name="dummy", host="localhost") + assert await backend.daemon_clear_message_buffer(daemon) == [MID] + assert (await backend.get_lists())[0].get_address() == LIST_ADDR + finally: + await backend.on_server_shutdown() + + +async def test_import_refuses_nonempty_database(deployment_dir: Path) -> None: + await _write_fs_deployment() + db_path = deployment_dir / "mail.db" + await import_memory_deployment(source_dir=deployment_dir, db_path=db_path) + + # A second import would clobber existing data; it must refuse. + with pytest.raises(ValueError, match="not empty"): + await import_memory_deployment(source_dir=deployment_dir, db_path=db_path) + + +async def test_import_missing_source_raises(tmp_path: Path) -> None: + with pytest.raises(FileNotFoundError, match="no filesystem deployment"): + await import_memory_deployment( + source_dir=tmp_path / "nope", db_path=tmp_path / "mail.db" + ) diff --git a/tests/unit/test_sqlite_repositories.py b/tests/unit/test_sqlite_repositories.py new file mode 100644 index 0000000..d29ff23 --- /dev/null +++ b/tests/unit/test_sqlite_repositories.py @@ -0,0 +1,345 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Repository-layer tests for the SQLite backend, run against a real temp-file +database so foreign keys, datetime columns, and ordering behave as in prod. + +The bulk pins ``MailboxRepository`` pagination: ``entered_at`` vs ``sent_at`` +ordering, ``asc`` / ``desc``, ``limit`` / ``offset``, ``total`` correctness, and +the always-ascending ``id`` tiebreaker that reproduces the memory backend's +stable insertion order. The rest cover CRUD round-trips and the FIFO buffer. +""" + +import uuid +from collections.abc import AsyncIterator +from datetime import UTC, datetime +from pathlib import Path + +import pytest +from mail_protocol.core.drafts import MAILDraft, MAILDraftsEntry +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILList, MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.trash import MAILTrashEntry +from mail_protocol.core.user_agents import MAILAgent, MAILUserAgentInBackend +from mail_protocol.core.webhooks import MAILWebhook +from mail_protocol.network.requests import BoxFilterParams +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.repositories import ( + BOX_INBOX, + BOX_TRASH, + MailStore, +) + +OWNER = "sage@chorus@localhost" +SENDER = "user:alice@localhost" +DAEMON = "daemon:dummy@localhost" + +T1 = datetime(2026, 6, 1, 12, 0, tzinfo=UTC) +T2 = datetime(2026, 6, 2, 12, 0, tzinfo=UTC) +T3 = datetime(2026, 6, 3, 12, 0, tzinfo=UTC) + + +@pytest.fixture +async def db(tmp_path: Path) -> AsyncIterator[Database]: + database = Database(f"sqlite:///{tmp_path / 'mail.db'}") + await database.create_schema() + yield database + await database.dispose() + + +def _uuid() -> str: + return str(uuid.uuid4()) + + +def _agent(owner: str = OWNER) -> MAILUserAgentInBackend: + name, swarm, host = owner.split("@") + return MAILUserAgentInBackend( + user_agent=MAILAgent(ua_type="agent", name=name, swarm=swarm, host=host), + hashed_password="hash", + ) + + +def _message(message_id: str, sent_at: datetime) -> MAILMessage: + return MAILMessage( + mail_version="2.0", + message_id=message_id, + sender=SENDER, + recipients=[OWNER], + subject="subject", + body="body", + tags=[], + sent_at=sent_at, + metadata={}, + ) + + +async def _seed_inbox( + store: MailStore, *, sent_at: datetime, entered_at: datetime +) -> str: + """Create message + shared inbox entry + membership for OWNER; return id.""" + + message_id = _uuid() + await store.messages.add(_message(message_id, sent_at)) + await store.boxes.upsert_inbox_entry( + MAILInboxEntrySummary( + message_id=message_id, + sender=SENDER, + subject="subject", + body_size=4, + received_at=entered_at, + delivered_by=DAEMON, + ) + ) + await store.boxes.add_membership(OWNER, BOX_INBOX, message_id, entered_at) + return message_id + + +# --------------------------------------------------------------------------- # +# MailboxRepository pagination / ordering +# --------------------------------------------------------------------------- # + + +async def test_inbox_default_orders_entered_at_desc(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + first = await _seed_inbox(store, sent_at=T1, entered_at=T1) + second = await _seed_inbox(store, sent_at=T2, entered_at=T2) + third = await _seed_inbox(store, sent_at=T3, entered_at=T3) + + page, total = await store.boxes.list_inbox(OWNER, BoxFilterParams()) + + assert total == 3 + assert [e.message_id for e in page] == [third, second, first] + + +async def test_inbox_entered_at_asc(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + first = await _seed_inbox(store, sent_at=T1, entered_at=T1) + second = await _seed_inbox(store, sent_at=T2, entered_at=T2) + + page, _ = await store.boxes.list_inbox( + OWNER, BoxFilterParams(order="asc") + ) + + assert [e.message_id for e in page] == [first, second] + + +async def test_inbox_sort_by_sent_at_differs_from_entered_at(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + # Arrival order is the inverse of send order. + late_arrival_old_send = await _seed_inbox(store, sent_at=T1, entered_at=T3) + early_arrival_new_send = await _seed_inbox(store, sent_at=T3, entered_at=T1) + + by_entered, _ = await store.boxes.list_inbox(OWNER, BoxFilterParams()) + by_sent, _ = await store.boxes.list_inbox( + OWNER, BoxFilterParams(sort_by="sent_at") + ) + + # entered_at desc -> the late arrival is first. + assert [e.message_id for e in by_entered] == [ + late_arrival_old_send, + early_arrival_new_send, + ] + # sent_at desc -> the newer send is first, flipping the order. + assert [e.message_id for e in by_sent] == [ + early_arrival_new_send, + late_arrival_old_send, + ] + + +async def test_inbox_limit_offset_and_total(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + ids = [ + await _seed_inbox(store, sent_at=t, entered_at=t) + for t in (T1, T2, T3) + ] + + page, total = await store.boxes.list_inbox( + OWNER, BoxFilterParams(limit=1, offset=1, order="asc") + ) + + assert total == 3 # count is the whole box, not the page + assert [e.message_id for e in page] == [ids[1]] + + +async def test_inbox_tiebreak_is_insertion_order(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + # Identical entered_at; desc must still fall back to ascending id + # (insertion order), matching the memory backend's stable sort. + first = await _seed_inbox(store, sent_at=T1, entered_at=T1) + second = await _seed_inbox(store, sent_at=T1, entered_at=T1) + + page, _ = await store.boxes.list_inbox(OWNER, BoxFilterParams()) + + assert [e.message_id for e in page] == [first, second] + + +async def test_empty_box_returns_empty_page(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + + page, total = await store.boxes.list_inbox(OWNER, BoxFilterParams()) + + assert page == [] + assert total == 0 + + +# --------------------------------------------------------------------------- # +# Membership / orphan accounting +# --------------------------------------------------------------------------- # + + +async def test_membership_add_remove_and_orphan_count(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + other = "echo@chorus@localhost" + await store.user_agents.add(_agent(other)) + + message_id = await _seed_inbox(store, sent_at=T1, entered_at=T1) + # Fan the same shared entry out to a second owner. + await store.boxes.add_membership(other, BOX_INBOX, message_id, T1) + + assert await store.boxes.count_item_members(BOX_INBOX, message_id) == 2 + assert await store.boxes.is_member(OWNER, BOX_INBOX, message_id) + + assert await store.boxes.remove_membership(OWNER, BOX_INBOX, message_id) + assert not await store.boxes.is_member(OWNER, BOX_INBOX, message_id) + assert await store.boxes.count_item_members(BOX_INBOX, message_id) == 1 + # Removing a non-member is a no-op returning False. + assert not await store.boxes.remove_membership(OWNER, BOX_INBOX, message_id) + + +# --------------------------------------------------------------------------- # +# CRUD round-trips through real SQLite +# --------------------------------------------------------------------------- # + + +async def test_user_agent_crud(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + agent = _agent() + await store.user_agents.add(agent) + + assert await store.user_agents.exists(OWNER) + assert await store.user_agents.get(OWNER) == agent + assert await store.user_agents.list_by_type("agent") == [agent] + + updated = await store.user_agents.set_password(OWNER, "new-hash") + assert updated is not None and updated.hashed_password == "new-hash" + reloaded = await store.user_agents.get(OWNER) + assert reloaded is not None and reloaded.hashed_password == "new-hash" + + deleted = await store.user_agents.delete(OWNER) + assert deleted is not None + assert not await store.user_agents.exists(OWNER) + + +async def test_swarm_and_webhook_and_list_crud(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + + swarm = MAILSwarm( + name="chorus", + description="d", + keywords=["k"], + agents=[], + metadata={}, + ) + await store.swarms.add(swarm) + assert await store.swarms.get("chorus") == swarm + assert await store.swarms.list_all() == [swarm] + + webhook = MAILWebhook( + webhook_id=f"wh_{_uuid()}", + url="https://hooks.example.com/mail", + events=["mail.delivered"], + secret="shh", + ) + await store.webhooks.add(webhook) + assert await store.webhooks.get_by_id(webhook.webhook_id) == webhook + assert await store.webhooks.get_by_url(webhook.url) == webhook + + mail_list = MAILListInBackend( + **MAILList( + name="team", swarm="chorus", host="localhost", owner=SENDER + ).model_dump(), + list_id=_uuid(), + created_at=T1, + updated_at=T1, + ) + await store.lists.add(mail_list) + # Member edit rewrites the JSON body wholesale, mirroring memory. + mutated = mail_list.model_copy( + update={"members": [OWNER], "updated_at": T2} + ) + await store.lists.update(mutated) + reloaded = await store.lists.get_by_address(mail_list.get_address()) + assert reloaded is not None and reloaded.members == [OWNER] + + +async def test_message_buffer_is_fifo_and_drains(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + ids = [_uuid() for _ in range(3)] + for message_id in ids: + await store.buffer.enqueue(message_id) + + assert await store.buffer.list_ids() == ids + assert await store.buffer.drain() == ids + # Buffer is empty after draining. + assert await store.buffer.list_ids() == [] + assert await store.buffer.drain() == [] + + +async def test_draft_entry_crud(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + draft_id = _uuid() + entry = MAILDraftsEntry( + draft=MAILDraft( + draft_id=draft_id, + subject="s", + body="b", + created_at=T1, + updated_at=None, + ), + sent_at=None, + sent_by=None, + ) + await store.boxes.upsert_draft_entry(entry) + assert await store.boxes.get_draft_entry(draft_id) == entry + + await store.boxes.delete_draft_entry(draft_id) + assert await store.boxes.get_draft_entry(draft_id) is None + + +async def test_trash_summary_listing(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + message_id = _uuid() + await store.messages.add(_message(message_id, T1)) + await store.boxes.upsert_trash_entry( + MAILTrashEntry(message=_message(message_id, T1), trashed_at=T2) + ) + await store.boxes.add_membership(OWNER, BOX_TRASH, message_id, T2) + + page, total = await store.boxes.list_trash(OWNER, BoxFilterParams()) + + assert total == 1 + assert page[0].message_id == message_id + assert page[0].trashed_at == T2 diff --git a/tests/unit/test_sqlite_serializers.py b/tests/unit/test_sqlite_serializers.py new file mode 100644 index 0000000..d62c0c4 --- /dev/null +++ b/tests/unit/test_sqlite_serializers.py @@ -0,0 +1,226 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Round-trips for every model-bearing SQLite row serializer: +``model -> to_columns -> Row(**columns) -> from_row`` must be identity. + +This pins the hybrid invariant — ``from_row`` rehydrates solely from the +``body`` JSON column, so a serialization change can't silently corrupt a +collection. The two body-less tables (``mailbox_items``, ``message_buffer``) +have no serializer and are covered by repository tests instead. + +The typed columns are asserted separately, since those (not ``body``) are what +``WHERE`` / ``ORDER BY`` rely on. +""" + +from datetime import UTC, datetime + +from mail_protocol.core.drafts import MAILDraft, MAILDraftsEntry +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILList, MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.outbox import MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.trash import MAILTrashEntry +from mail_protocol.core.user_agents import ( + MAILAgent, + MAILUser, + MAILUserAgentInBackend, +) +from mail_protocol.core.webhooks import MAILWebhook +from mail_server.backends.sqlite import serializers as s +from mail_server.backends.sqlite.schema import ( + DraftEntryRow, + InboxEntryRow, + ListRow, + MessageRow, + OutboxEntryRow, + SwarmRow, + TrashEntryRow, + UserAgentRow, + WebhookRow, +) + +NOW = datetime(2026, 6, 12, 9, 0, tzinfo=UTC) +UUID = "55555555-5555-4555-8555-555555555555" +USER = "user:alice@localhost" +AGENT = "sage@chorus@localhost" +DAEMON = "daemon:dummy@localhost" + + +def _message() -> MAILMessage: + return MAILMessage( + mail_version="2.0", + message_id=UUID, + sender=USER, + recipients=[AGENT], + subject="Persisted", + body="Survives a round-trip.", + tags=[], + sent_at=NOW, + metadata={}, + ) + + +def test_user_agent_roundtrip() -> None: + model = MAILUserAgentInBackend( + user_agent=MAILUser(ua_type="user", user_id="alice", host="localhost"), + hashed_password="a-stored-hash", + ) + cols = s.user_agent_to_columns(model) + assert cols["address"] == USER + assert cols["ua_type"] == "user" + assert cols["swarm"] is None + assert cols["host"] == "localhost" + assert cols["hashed_password"] == "a-stored-hash" + assert s.user_agent_from_row(UserAgentRow(**cols)) == model + + +def test_user_agent_agent_carries_swarm_column() -> None: + model = MAILUserAgentInBackend( + user_agent=MAILAgent( + ua_type="agent", name="sage", swarm="chorus", host="localhost" + ), + hashed_password="hash", + ) + cols = s.user_agent_to_columns(model) + assert cols["address"] == AGENT + assert cols["swarm"] == "chorus" + assert s.user_agent_from_row(UserAgentRow(**cols)) == model + + +def test_swarm_roundtrip() -> None: + model = MAILSwarm( + name="chorus", + description="A test swarm.", + keywords=["testing"], + agents=["sage"], + metadata={}, + ) + cols = s.swarm_to_columns(model) + assert cols["name"] == "chorus" + assert s.swarm_from_row(SwarmRow(**cols)) == model + + +def test_message_roundtrip() -> None: + model = _message() + cols = s.message_to_columns(model) + assert cols["message_id"] == UUID + assert cols["sender"] == USER + assert cols["subject"] == "Persisted" + assert cols["reply_to"] is None + assert cols["sent_at"] == NOW + assert s.message_from_row(MessageRow(**cols)) == model + + +def test_inbox_entry_roundtrip() -> None: + model = MAILInboxEntrySummary( + message_id=UUID, + sender=USER, + subject="Persisted", + body_size=10, + received_at=NOW, + delivered_by=DAEMON, + ) + cols = s.inbox_entry_to_columns(model) + assert cols["message_id"] == UUID + assert cols["body_size"] == 10 + assert cols["received_at"] == NOW + assert cols["delivered_by"] == DAEMON + assert s.inbox_entry_from_row(InboxEntryRow(**cols)) == model + + +def test_outbox_entry_roundtrip() -> None: + model = MAILOutboxEntrySummary( + message_id=UUID, + recipients=[AGENT], + subject="Persisted", + body_size=10, + sent_at=NOW, + delivered_at=NOW, + delivered_by=DAEMON, + ) + cols = s.outbox_entry_to_columns(model) + assert cols["message_id"] == UUID + assert cols["sent_at"] == NOW + assert cols["delivered_at"] == NOW + assert cols["delivered_by"] == DAEMON + assert s.outbox_entry_from_row(OutboxEntryRow(**cols)) == model + + +def test_outbox_entry_undelivered_roundtrip() -> None: + model = MAILOutboxEntrySummary( + message_id=UUID, + recipients=[AGENT], + subject="Persisted", + body_size=10, + sent_at=NOW, + ) + cols = s.outbox_entry_to_columns(model) + assert cols["delivered_at"] is None + assert cols["delivered_by"] is None + assert s.outbox_entry_from_row(OutboxEntryRow(**cols)) == model + + +def test_draft_entry_roundtrip() -> None: + model = MAILDraftsEntry( + draft=MAILDraft( + draft_id=UUID, + subject="Persisted", + body="A draft body.", + created_at=NOW, + updated_at=None, + ), + sent_at=None, + sent_by=None, + ) + cols = s.draft_entry_to_columns(model) + assert cols["draft_id"] == UUID + assert cols["created_at"] == NOW + assert cols["updated_at"] is None + assert s.draft_entry_from_row(DraftEntryRow(**cols)) == model + + +def test_trash_entry_roundtrip() -> None: + model = MAILTrashEntry(message=_message(), trashed_at=NOW) + cols = s.trash_entry_to_columns(model) + assert cols["message_id"] == UUID + assert cols["trashed_at"] == NOW + assert s.trash_entry_from_row(TrashEntryRow(**cols)) == model + + +def test_webhook_roundtrip() -> None: + model = MAILWebhook( + webhook_id=f"wh_{UUID}", + url="https://hooks.example.com/mail", + events=["mail.delivered"], + secret="shhh", + ) + cols = s.webhook_to_columns(model) + assert cols["url"] == "https://hooks.example.com/mail" + assert cols["webhook_id"] == f"wh_{UUID}" + assert s.webhook_from_row(WebhookRow(**cols)) == model + + +def test_list_roundtrip() -> None: + model = MAILListInBackend( + **MAILList( + name="team", + swarm="chorus", + host="localhost", + owner=USER, + members=[AGENT], + ).model_dump(), + list_id=UUID, + created_at=NOW, + updated_at=NOW, + ) + cols = s.list_to_columns(model) + assert cols["address"] == "list:team@chorus@localhost" + assert cols["list_id"] == UUID + assert cols["swarm"] == "chorus" + assert cols["host"] == "localhost" + assert cols["created_at"] == NOW + assert cols["updated_at"] == NOW + assert s.list_from_row(ListRow(**cols)) == model diff --git a/uv.lock b/uv.lock index 97b401f..80f26b6 100644 --- a/uv.lock +++ b/uv.lock @@ -88,6 +88,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" }, ] +[[package]] +name = "aiosqlite" +version = "0.22.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4e/8a/64761f4005f17809769d23e518d915db74e6310474e733e3593cfc854ef1/aiosqlite-0.22.1.tar.gz", hash = "sha256:043e0bd78d32888c0a9ca90fc788b38796843360c855a7262a532813133a0650", size = 14821, upload-time = "2025-12-23T19:25:43.997Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/00/b7/e3bf5133d697a08128598c8d0abc5e16377b51465a33756de24fa7dee953/aiosqlite-0.22.1-py3-none-any.whl", hash = "sha256:21c002eb13823fad740196c5a2e9d8e62f6243bd9e7e4a1f87fb5e44ecb4fceb", size = 17405, upload-time = "2025-12-23T19:25:42.139Z" }, +] + [[package]] name = "alabaster" version = "1.0.0" @@ -747,6 +756,7 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/44/69/9b804adb5fd0671f367781560eb5eb586c4d495277c93bde4307b9e28068/greenlet-3.2.4-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:3b67ca49f54cede0186854a008109d6ee71f66bd57bb36abd6d0a0267b540cdd", size = 274079, upload-time = "2025-08-07T13:15:45.033Z" }, { url = "https://files.pythonhosted.org/packages/46/e9/d2a80c99f19a153eff70bc451ab78615583b8dac0754cfb942223d2c1a0d/greenlet-3.2.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ddf9164e7a5b08e9d22511526865780a576f19ddd00d62f8a665949327fde8bb", size = 640997, upload-time = "2025-08-07T13:42:56.234Z" }, { url = "https://files.pythonhosted.org/packages/3b/16/035dcfcc48715ccd345f3a93183267167cdd162ad123cd93067d86f27ce4/greenlet-3.2.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f28588772bb5fb869a8eb331374ec06f24a83a9c25bfa1f38b6993afe9c1e968", size = 655185, upload-time = "2025-08-07T13:45:27.624Z" }, + { url = "https://files.pythonhosted.org/packages/31/da/0386695eef69ffae1ad726881571dfe28b41970173947e7c558d9998de0f/greenlet-3.2.4-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:5c9320971821a7cb77cfab8d956fa8e39cd07ca44b6070db358ceb7f8797c8c9", size = 649926, upload-time = "2025-08-07T13:53:15.251Z" }, { url = "https://files.pythonhosted.org/packages/68/88/69bf19fd4dc19981928ceacbc5fd4bb6bc2215d53199e367832e98d1d8fe/greenlet-3.2.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c60a6d84229b271d44b70fb6e5fa23781abb5d742af7b808ae3f6efd7c9c60f6", size = 651839, upload-time = "2025-08-07T13:18:30.281Z" }, { url = "https://files.pythonhosted.org/packages/19/0d/6660d55f7373b2ff8152401a83e02084956da23ae58cddbfb0b330978fe9/greenlet-3.2.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3b3812d8d0c9579967815af437d96623f45c0f2ae5f04e366de62a12d83a8fb0", size = 607586, upload-time = "2025-08-07T13:18:28.544Z" }, { url = "https://files.pythonhosted.org/packages/8e/1a/c953fdedd22d81ee4629afbb38d2f9d71e37d23caace44775a3a969147d4/greenlet-3.2.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:abbf57b5a870d30c4675928c37278493044d7c14378350b3aa5d484fa65575f0", size = 1123281, upload-time = "2025-08-07T13:42:39.858Z" }, @@ -757,6 +767,7 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/49/e8/58c7f85958bda41dafea50497cbd59738c5c43dbbea5ee83d651234398f4/greenlet-3.2.4-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:1a921e542453fe531144e91e1feedf12e07351b1cf6c9e8a3325ea600a715a31", size = 272814, upload-time = "2025-08-07T13:15:50.011Z" }, { url = "https://files.pythonhosted.org/packages/62/dd/b9f59862e9e257a16e4e610480cfffd29e3fae018a68c2332090b53aac3d/greenlet-3.2.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cd3c8e693bff0fff6ba55f140bf390fa92c994083f838fece0f63be121334945", size = 641073, upload-time = "2025-08-07T13:42:57.23Z" }, { url = "https://files.pythonhosted.org/packages/f7/0b/bc13f787394920b23073ca3b6c4a7a21396301ed75a655bcb47196b50e6e/greenlet-3.2.4-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:710638eb93b1fa52823aa91bf75326f9ecdfd5e0466f00789246a5280f4ba0fc", size = 655191, upload-time = "2025-08-07T13:45:29.752Z" }, + { url = "https://files.pythonhosted.org/packages/f2/d6/6adde57d1345a8d0f14d31e4ab9c23cfe8e2cd39c3baf7674b4b0338d266/greenlet-3.2.4-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:c5111ccdc9c88f423426df3fd1811bfc40ed66264d35aa373420a34377efc98a", size = 649516, upload-time = "2025-08-07T13:53:16.314Z" }, { url = "https://files.pythonhosted.org/packages/7f/3b/3a3328a788d4a473889a2d403199932be55b1b0060f4ddd96ee7cdfcad10/greenlet-3.2.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d76383238584e9711e20ebe14db6c88ddcedc1829a9ad31a584389463b5aa504", size = 652169, upload-time = "2025-08-07T13:18:32.861Z" }, { url = "https://files.pythonhosted.org/packages/ee/43/3cecdc0349359e1a527cbf2e3e28e5f8f06d3343aaf82ca13437a9aa290f/greenlet-3.2.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:23768528f2911bcd7e475210822ffb5254ed10d71f4028387e5a99b4c6699671", size = 610497, upload-time = "2025-08-07T13:18:31.636Z" }, { url = "https://files.pythonhosted.org/packages/b8/19/06b6cf5d604e2c382a6f31cafafd6f33d5dea706f4db7bdab184bad2b21d/greenlet-3.2.4-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:00fadb3fedccc447f517ee0d3fd8fe49eae949e1cd0f6a611818f4f6fb7dc83b", size = 1121662, upload-time = "2025-08-07T13:42:41.117Z" }, @@ -767,6 +778,7 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/22/5c/85273fd7cc388285632b0498dbbab97596e04b154933dfe0f3e68156c68c/greenlet-3.2.4-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:49a30d5fda2507ae77be16479bdb62a660fa51b1eb4928b524975b3bde77b3c0", size = 273586, upload-time = "2025-08-07T13:16:08.004Z" }, { url = "https://files.pythonhosted.org/packages/d1/75/10aeeaa3da9332c2e761e4c50d4c3556c21113ee3f0afa2cf5769946f7a3/greenlet-3.2.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:299fd615cd8fc86267b47597123e3f43ad79c9d8a22bebdce535e53550763e2f", size = 686346, upload-time = "2025-08-07T13:42:59.944Z" }, { url = "https://files.pythonhosted.org/packages/c0/aa/687d6b12ffb505a4447567d1f3abea23bd20e73a5bed63871178e0831b7a/greenlet-3.2.4-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:c17b6b34111ea72fc5a4e4beec9711d2226285f0386ea83477cbb97c30a3f3a5", size = 699218, upload-time = "2025-08-07T13:45:30.969Z" }, + { url = "https://files.pythonhosted.org/packages/dc/8b/29aae55436521f1d6f8ff4e12fb676f3400de7fcf27fccd1d4d17fd8fecd/greenlet-3.2.4-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:b4a1870c51720687af7fa3e7cda6d08d801dae660f75a76f3845b642b4da6ee1", size = 694659, upload-time = "2025-08-07T13:53:17.759Z" }, { url = "https://files.pythonhosted.org/packages/92/2e/ea25914b1ebfde93b6fc4ff46d6864564fba59024e928bdc7de475affc25/greenlet-3.2.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:061dc4cf2c34852b052a8620d40f36324554bc192be474b9e9770e8c042fd735", size = 695355, upload-time = "2025-08-07T13:18:34.517Z" }, { url = "https://files.pythonhosted.org/packages/72/60/fc56c62046ec17f6b0d3060564562c64c862948c9d4bc8aa807cf5bd74f4/greenlet-3.2.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:44358b9bf66c8576a9f57a590d5f5d6e72fa4228b763d0e43fee6d3b06d3a337", size = 657512, upload-time = "2025-08-07T13:18:33.969Z" }, { url = "https://files.pythonhosted.org/packages/23/6e/74407aed965a4ab6ddd93a7ded3180b730d281c77b765788419484cdfeef/greenlet-3.2.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:2917bdf657f5859fbf3386b12d68ede4cf1f04c90c3a6bc1f013dd68a22e2269", size = 1612508, upload-time = "2025-11-04T12:42:23.427Z" }, @@ -1424,6 +1436,7 @@ version = "2.0.1" source = { editable = "src/mail/server" } dependencies = [ { name = "aiohttp" }, + { name = "aiosqlite" }, { name = "fastapi" }, { name = "mail-swarms-protocol" }, { name = "pwdlib", extra = ["argon2"] }, @@ -1431,12 +1444,14 @@ dependencies = [ { name = "pyjwt" }, { name = "python-dotenv" }, { name = "python-multipart" }, + { name = "sqlalchemy", extra = ["asyncio"] }, { name = "uvicorn" }, ] [package.metadata] requires-dist = [ { name = "aiohttp", specifier = ">=3.12.15" }, + { name = "aiosqlite", specifier = ">=0.20" }, { name = "fastapi", specifier = ">=0.116.1" }, { name = "mail-swarms-protocol", editable = "src/mail/protocol" }, { name = "pwdlib", extras = ["argon2"], specifier = ">=0.3.0" }, @@ -1444,6 +1459,7 @@ requires-dist = [ { name = "pyjwt", specifier = ">=2.10.1" }, { name = "python-dotenv", specifier = ">=1.1.1" }, { name = "python-multipart", specifier = ">=0.0.20" }, + { name = "sqlalchemy", extras = ["asyncio"], specifier = ">=2.0" }, { name = "uvicorn", specifier = ">=0.35.0" }, ] @@ -2540,6 +2556,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b8/d9/13bdde6521f322861fab67473cec4b1cc8999f3871953531cf61945fad92/sqlalchemy-2.0.43-py3-none-any.whl", hash = "sha256:1681c21dd2ccee222c2fe0bef671d1aef7c504087c9c4e800371cfcc8ac966fc", size = 1924759, upload-time = "2025-08-11T15:39:53.024Z" }, ] +[package.optional-dependencies] +asyncio = [ + { name = "greenlet" }, +] + [[package]] name = "sse-starlette" version = "3.0.2" From 038af013fe46c1d9ac49289b67b902c09e35bb10 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Thu, 25 Jun 2026 16:51:40 -0400 Subject: [PATCH 13/28] refactor: standardize admin endpoint address path params MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Admin (and list) endpoints addressed resources by three inconsistent slices of the canonical MAIL address: agents used a partial address (name@swarm), daemons/users a bare id, and lists the full list:name@swarm@host address. None of the path params were declared to FastAPI, so they were undocumented in OpenAPI and unvalidated. Standardize on a single rule: the path param is the resource's local identifier, with the user-agent prefix implied by the route and the host implied by the server (agents name@swarm, daemons worker_name, users user_id, swarms swarm_name, lists name@swarm, webhooks wh_). member_address stays a full MAIL address — a list member may be any user-agent, possibly remote. - Add typed Path(...) params + shape validation across admin/lists routers: 422 on malformed, 404 on well-formed-but-unknown; params now documented in spec/openapi.yaml. - Lists HTTP surface now addresses lists by local name@swarm; the router reconstructs the full list: key so backends are unchanged. Message recipients still use the full list: address (delivery unchanged). - Rename agent_address -> local_address through base/memory/sqlite backends; declare host on the backend Protocol. - Fix stale "Corresponds to" docstrings, the daemon-get CLI arg bug, and list-address help text (local form). - Update tests + fixtures to local addresses; add 422 path-param tests. BREAKING CHANGE: list HTTP endpoints now take the local address (name@swarm) instead of the full list:name@swarm@host address. Co-Authored-By: Claude Opus 4.8 (1M context) --- spec/openapi.yaml | 420 ++++++++++++++++-- .../client/src/mail_client/admin_panel.py | 21 +- src/mail/client/src/mail_client/cli.py | 8 +- .../src/mail_client/commands/daemon_get.py | 6 +- .../src/mail_protocol/network/requests.py | 6 +- .../src/mail_protocol/network/responses.py | 22 +- .../server/src/mail_server/backends/base.py | 9 +- .../src/mail_server/backends/memory/api.py | 18 +- .../src/mail_server/backends/sqlite/api.py | 16 +- .../server/src/mail_server/routers/admin.py | 91 +++- .../server/src/mail_server/routers/lists.py | 112 +++-- src/mail/server/src/mail_server/validators.py | 102 ++++- tests/e2e/test_journeys.py | 5 +- tests/integration/conftest.py | 8 +- tests/integration/test_admin.py | 25 ++ tests/integration/test_admin_webhooks.py | 17 +- tests/integration/test_flows.py | 6 +- tests/integration/test_lists.py | 20 +- 18 files changed, 761 insertions(+), 151 deletions(-) diff --git a/spec/openapi.yaml b/spec/openapi.yaml index d1a524b..8305ec2 100644 --- a/spec/openapi.yaml +++ b/spec/openapi.yaml @@ -362,12 +362,23 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminAgentPostResponse' - /admin/agents/{agent_address}: + /admin/agents/{local_address}: get: tags: - admin summary: Get a specific registered agent by local address (name@swarm) - operationId: get_agent_admin_agents__agent_address__get + operationId: get_agent_admin_agents__local_address__get + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: Agent local address (name@swarm); host is implied. + examples: + - researcher@acme + title: Local Address + description: Agent local address (name@swarm); host is implied. responses: '200': description: Successful Response @@ -375,11 +386,28 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminAgentGetResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' delete: tags: - admin summary: Delete an existing MAIL agent on this server - operationId: delete_agent_admin_agents__agent_address__delete + operationId: delete_agent_admin_agents__local_address__delete + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: Agent local address (name@swarm); host is implied. + examples: + - researcher@acme + title: Local Address + description: Agent local address (name@swarm); host is implied. responses: '200': description: Successful Response @@ -387,6 +415,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminAgentDeleteResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/daemons: get: tags: @@ -418,6 +452,17 @@ paths: - admin summary: Get a specific registered daemon by worker name operationId: get_daemon_admin_daemons__worker_name__get + parameters: + - name: worker_name + in: path + required: true + schema: + type: string + description: Daemon worker name; prefix and host are implied. + examples: + - indexer + title: Worker Name + description: Daemon worker name; prefix and host are implied. responses: '200': description: Successful Response @@ -425,11 +470,28 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminDaemonGetResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' delete: tags: - admin summary: Delete an existing MAIL daemon on this server operationId: delete_daemon_admin_daemons__worker_name__delete + parameters: + - name: worker_name + in: path + required: true + schema: + type: string + description: Daemon worker name; prefix and host are implied. + examples: + - indexer + title: Worker Name + description: Daemon worker name; prefix and host are implied. responses: '200': description: Successful Response @@ -437,6 +499,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminDaemonDeleteResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/users: get: tags: @@ -468,6 +536,17 @@ paths: - admin summary: Get a specific registered user by ID operationId: get_user_admin_users__user_id__get + parameters: + - name: user_id + in: path + required: true + schema: + type: string + description: User id; prefix and host are implied. + examples: + - addison + title: User Id + description: User id; prefix and host are implied. responses: '200': description: Successful Response @@ -475,11 +554,28 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminUserGetResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' delete: tags: - admin summary: Delete an existing MAIL user on this server operationId: delete_user_admin_users__user_id__delete + parameters: + - name: user_id + in: path + required: true + schema: + type: string + description: User id; prefix and host are implied. + examples: + - addison + title: User Id + description: User id; prefix and host are implied. responses: '200': description: Successful Response @@ -487,6 +583,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminUserDeleteResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/swarms: post: tags: @@ -506,6 +608,17 @@ paths: - admin summary: Delete an existing MAIL swarm on this server by name operationId: delete_swarm_admin_swarms__swarm_name__delete + parameters: + - name: swarm_name + in: path + required: true + schema: + type: string + description: Swarm name. + examples: + - acme + title: Swarm Name + description: Swarm name. responses: '200': description: Successful Response @@ -513,6 +626,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminSwarmDeleteResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/webhooks: get: tags: @@ -544,6 +663,17 @@ paths: - admin summary: Get a specific existing server webhook by ID operationId: get_webhook_admin_webhooks__webhook_id__get + parameters: + - name: webhook_id + in: path + required: true + schema: + type: string + description: Webhook id (wh_). + examples: + - wh_123e4567-e89b-12d3-a456-426614174000 + title: Webhook Id + description: Webhook id (wh_). responses: '200': description: Successful Response @@ -551,30 +681,70 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminWebhookGetResponse' - delete: + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + patch: tags: - admin - summary: Delete an existing webhook by ID from this server - operationId: delete_webhook_admin_webhooks__webhook_id__delete + summary: Update an existing webhook by ID on this server + operationId: patch_webhook_admin_webhooks__webhook_id__patch + parameters: + - name: webhook_id + in: path + required: true + schema: + type: string + description: Webhook id (wh_). + examples: + - wh_123e4567-e89b-12d3-a456-426614174000 + title: Webhook Id + description: Webhook id (wh_). responses: '200': description: Successful Response content: application/json: schema: - $ref: '#/components/schemas/AdminWebhooksDeleteResponse' - patch: + $ref: '#/components/schemas/AdminWebhooksPatchResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + delete: tags: - admin - summary: Update an existing webhook by ID on this server - operationId: patch_webhook_admin_webhooks__webhook_id__patch + summary: Delete an existing webhook by ID from this server + operationId: delete_webhook_admin_webhooks__webhook_id__delete + parameters: + - name: webhook_id + in: path + required: true + schema: + type: string + description: Webhook id (wh_). + examples: + - wh_123e4567-e89b-12d3-a456-426614174000 + title: Webhook Id + description: Webhook id (wh_). responses: '200': description: Successful Response content: application/json: schema: - $ref: '#/components/schemas/AdminWebhooksPatchResponse' + $ref: '#/components/schemas/AdminWebhooksDeleteResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/lists: get: tags: @@ -600,12 +770,25 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminListPostResponse' - /admin/lists/{list_address}: + /admin/lists/{local_address}: get: tags: - admin-lists - summary: Get a specific MAIL list by address - operationId: admin_get_list_admin_lists__list_address__get + summary: Get a specific MAIL list by local address (name@swarm) + operationId: admin_get_list_admin_lists__local_address__get + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: 'List local address (name@swarm); the list: prefix and host + are implied.' + examples: + - announce@acme + title: Local Address + description: 'List local address (name@swarm); the list: prefix and host are + implied.' responses: '200': description: Successful Response @@ -613,36 +796,93 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminListGetResponse' - delete: + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + patch: tags: - admin-lists - summary: Delete a MAIL list - operationId: admin_delete_list_admin_lists__list_address__delete + summary: Update a MAIL list's policy + operationId: admin_patch_list_admin_lists__local_address__patch + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: 'List local address (name@swarm); the list: prefix and host + are implied.' + examples: + - announce@acme + title: Local Address + description: 'List local address (name@swarm); the list: prefix and host are + implied.' responses: '200': description: Successful Response content: application/json: schema: - $ref: '#/components/schemas/AdminListDeleteResponse' - patch: + $ref: '#/components/schemas/AdminListPatchResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + delete: tags: - admin-lists - summary: Update a MAIL list's policy - operationId: admin_patch_list_admin_lists__list_address__patch + summary: Delete a MAIL list + operationId: admin_delete_list_admin_lists__local_address__delete + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: 'List local address (name@swarm); the list: prefix and host + are implied.' + examples: + - announce@acme + title: Local Address + description: 'List local address (name@swarm); the list: prefix and host are + implied.' responses: '200': description: Successful Response content: application/json: schema: - $ref: '#/components/schemas/AdminListPatchResponse' - /admin/lists/{list_address}/members: + $ref: '#/components/schemas/AdminListDeleteResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /admin/lists/{local_address}/members: post: tags: - admin-lists summary: Admin-add a member to a MAIL list - operationId: admin_add_list_member_admin_lists__list_address__members_post + operationId: admin_add_list_member_admin_lists__local_address__members_post + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: 'List local address (name@swarm); the list: prefix and host + are implied.' + examples: + - announce@acme + title: Local Address + description: 'List local address (name@swarm); the list: prefix and host are + implied.' responses: '200': description: Successful Response @@ -650,12 +890,41 @@ paths: application/json: schema: $ref: '#/components/schemas/ListMemberPostResponse' - /admin/lists/{list_address}/members/{member_address}: + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /admin/lists/{local_address}/members/{member_address}: delete: tags: - admin-lists summary: Admin-remove a member from a MAIL list - operationId: admin_remove_list_member_admin_lists__list_address__members__member_address__delete + operationId: admin_remove_list_member_admin_lists__local_address__members__member_address__delete + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: 'List local address (name@swarm); the list: prefix and host + are implied.' + examples: + - announce@acme + title: Local Address + description: 'List local address (name@swarm); the list: prefix and host are + implied.' + - name: member_address + in: path + required: true + schema: + type: string + description: Full MAIL address of the member to remove (may be remote). + examples: + - user:bob@other-host + title: Member Address + description: Full MAIL address of the member to remove (may be remote). responses: '200': description: Successful Response @@ -663,6 +932,12 @@ paths: application/json: schema: $ref: '#/components/schemas/ListMemberDeleteResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /lists: get: tags: @@ -676,12 +951,25 @@ paths: application/json: schema: $ref: '#/components/schemas/ListsGetResponse' - /lists/{list_address}: + /lists/{local_address}: get: tags: - lists - summary: Get a specific list - operationId: get_list_lists__list_address__get + summary: Get a specific list by local address (name@swarm) + operationId: get_list_lists__local_address__get + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: 'List local address (name@swarm); the list: prefix and host + are implied.' + examples: + - announce@acme + title: Local Address + description: 'List local address (name@swarm); the list: prefix and host are + implied.' responses: '200': description: Successful Response @@ -689,12 +977,31 @@ paths: application/json: schema: $ref: '#/components/schemas/ListGetResponse' - /lists/{list_address}/subscribe: + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /lists/{local_address}/subscribe: post: tags: - lists summary: Subscribe to a MAIL list - operationId: subscribe_lists__list_address__subscribe_post + operationId: subscribe_lists__local_address__subscribe_post + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: 'List local address (name@swarm); the list: prefix and host + are implied.' + examples: + - announce@acme + title: Local Address + description: 'List local address (name@swarm); the list: prefix and host are + implied.' responses: '200': description: Successful Response @@ -702,12 +1009,31 @@ paths: application/json: schema: $ref: '#/components/schemas/ListMemberPostResponse' - /lists/{list_address}/unsubscribe: + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' + /lists/{local_address}/unsubscribe: post: tags: - lists summary: Unsubscribe from a MAIL list - operationId: unsubscribe_lists__list_address__unsubscribe_post + operationId: unsubscribe_lists__local_address__unsubscribe_post + parameters: + - name: local_address + in: path + required: true + schema: + type: string + description: 'List local address (name@swarm); the list: prefix and host + are implied.' + examples: + - announce@acme + title: Local Address + description: 'List local address (name@swarm); the list: prefix and host are + implied.' responses: '200': description: Successful Response @@ -715,6 +1041,12 @@ paths: application/json: schema: $ref: '#/components/schemas/ListMemberDeleteResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /: get: summary: Get basic server information and metadata @@ -752,7 +1084,7 @@ components: - agent - metadata title: AdminAgentDeleteResponse - description: 'Corresponds to `DELETE /admin/agent/{agent_address}`. + description: 'Corresponds to `DELETE /admin/agents/{local_address}`. Contains a newly-deleted MAIL agent registered on this server.' AdminAgentGetResponse: @@ -768,7 +1100,7 @@ components: - agent - metadata title: AdminAgentGetResponse - description: 'Corresponds to `GET /admin/agents/{agent_address}`. + description: 'Corresponds to `GET /admin/agents/{local_address}`. Contains a specific MAIL agent registered on this server.' AdminAgentPostResponse: @@ -784,7 +1116,7 @@ components: - agent - metadata title: AdminAgentPostResponse - description: 'Corresponds to `POST /admin/agent`. + description: 'Corresponds to `POST /admin/agents`. Contains the new MAIL agent registered on this server.' AdminAgentsGetResponse: @@ -887,7 +1219,7 @@ components: - mail_list - metadata title: AdminListDeleteResponse - description: Corresponds to `DELETE /admin/lists/{list_address}`. + description: Corresponds to `DELETE /admin/lists/{local_address}`. AdminListGetResponse: properties: mail_list: @@ -901,7 +1233,7 @@ components: - mail_list - metadata title: AdminListGetResponse - description: Corresponds to `GET /admin/lists/{list_address}`. + description: Corresponds to `GET /admin/lists/{local_address}`. AdminListPatchResponse: properties: mail_list: @@ -915,7 +1247,7 @@ components: - mail_list - metadata title: AdminListPatchResponse - description: Corresponds to `PATCH /admin/lists/{list_address}`. + description: Corresponds to `PATCH /admin/lists/{local_address}`. AdminListPostResponse: properties: mail_list: @@ -1460,7 +1792,7 @@ components: - mail_list - metadata title: ListGetResponse - description: Corresponds to `GET /lists/{list_address}`. + description: Corresponds to `GET /lists/{local_address}`. ListMemberDeleteResponse: properties: mail_list: @@ -1474,9 +1806,9 @@ components: - mail_list - metadata title: ListMemberDeleteResponse - description: 'Corresponds to `POST /lists/{list_address}/unsubscribe` + description: 'Corresponds to `POST /lists/{local_address}/unsubscribe` - and `DELETE /lists/{list_address}/members/{member_address}`. + and `DELETE /admin/lists/{local_address}/members/{member_address}`. The updated list with the member removed (idempotent — removing a @@ -1494,9 +1826,9 @@ components: - mail_list - metadata title: ListMemberPostResponse - description: 'Corresponds to `POST /lists/{list_address}/subscribe` and + description: 'Corresponds to `POST /lists/{local_address}/subscribe` and - `POST /admin/lists/{list_address}/members`. The updated list with + `POST /admin/lists/{local_address}/members`. The updated list with the member appended (idempotent — re-adding an existing member is diff --git a/src/mail/client/src/mail_client/admin_panel.py b/src/mail/client/src/mail_client/admin_panel.py index 0954b33..403e180 100644 --- a/src/mail/client/src/mail_client/admin_panel.py +++ b/src/mail/client/src/mail_client/admin_panel.py @@ -59,7 +59,7 @@ "Daemons", [ ("daemon-list (dl)", "List daemons."), - ("daemon-get (dg)", "Get a daemon by local address."), + ("daemon-get (dg)", "Get a daemon by worker name."), ("daemon-post (dp)", "Create daemon credentials."), ("daemon-delete (dd)", "Delete daemon credentials."), ], @@ -235,7 +235,7 @@ def build_parser() -> argparse.ArgumentParser: daemon_list_p.set_defaults(func=cmd_daemon_list, cmd="daemon-list") # command `daemon-get` - daemon_get_d = "get a specific daemon by local address on the MAIL server" + daemon_get_d = "get a specific daemon by worker name on the MAIL server" daemon_get_p = subparsers.add_parser( "daemon-get", aliases=["dg"], @@ -244,7 +244,7 @@ def build_parser() -> argparse.ArgumentParser: description=daemon_get_d, ) daemon_get_p.add_argument( - "local_address", help="the local address of the daemon to get (daemon@swarm)" + "worker_name", help="the worker name of the daemon to get" ) daemon_get_p.set_defaults(func=cmd_daemon_get, cmd="daemon-get") @@ -459,7 +459,8 @@ def build_parser() -> argparse.ArgumentParser: description=list_get_d, ) list_get_p.add_argument( - "list_address", help="the address of the mailing list to get" + "list_address", + help="the local address of the mailing list to get (name@swarm)", ) list_get_p.set_defaults(func=cmd_list_get_admin, cmd="list-get") @@ -507,7 +508,8 @@ def build_parser() -> argparse.ArgumentParser: description=list_delete_d, ) list_delete_p.add_argument( - "list_address", help="the address of the mailing list to delete" + "list_address", + help="the local address of the mailing list to delete (name@swarm)", ) list_delete_p.set_defaults(func=cmd_list_delete, cmd="list-delete") @@ -523,11 +525,12 @@ def build_parser() -> argparse.ArgumentParser: description=list_member_post_d, ) list_member_post_p.add_argument( - "list_address", help="the MAIL address of the mailing list to add a member to" + "list_address", + help="the local address of the mailing list to add a member to (name@swarm)", ) list_member_post_p.add_argument( "member_address", - help="the MAIL address of the member to add to this mailing list", + help="the full MAIL address of the member to add to this mailing list", ) list_member_post_p.set_defaults(func=cmd_list_member_post, cmd="list-member-post") @@ -544,11 +547,11 @@ def build_parser() -> argparse.ArgumentParser: ) list_member_delete_p.add_argument( "list_address", - help="the MAIL address of the mailing list to remove a member from", + help="the local address of the mailing list to remove a member from (name@swarm)", ) list_member_delete_p.add_argument( "member_address", - help="the MAIL address of the member to remove from this mailing list", + help="the full MAIL address of the member to remove from this mailing list", ) list_member_delete_p.set_defaults( func=cmd_list_member_delete, cmd="list-member-delete" diff --git a/src/mail/client/src/mail_client/cli.py b/src/mail/client/src/mail_client/cli.py index 8e961e5..f4c390c 100644 --- a/src/mail/client/src/mail_client/cli.py +++ b/src/mail/client/src/mail_client/cli.py @@ -488,7 +488,7 @@ def build_parser() -> argparse.ArgumentParser: description=list_get_d, ) list_get_p.add_argument( - "list_address", help="the address of the mailing list to get" + "list_address", help="the local address of the mailing list to get (name@swarm)" ) list_get_p.set_defaults(func=cmd_list_get, cmd="list-get") @@ -502,7 +502,8 @@ def build_parser() -> argparse.ArgumentParser: description=list_subscribe_d, ) list_subscribe_p.add_argument( - "list_address", help="the address of the mailing list to subscribe to" + "list_address", + help="the local address of the mailing list to subscribe to (name@swarm)", ) list_subscribe_p.set_defaults(func=cmd_list_subscribe, cmd="list-subscribe") @@ -516,7 +517,8 @@ def build_parser() -> argparse.ArgumentParser: description=list_unsubscribe_d, ) list_unsubscribe_p.add_argument( - "list_address", help="the address of the mailing list to unsubscribe from" + "list_address", + help="the local address of the mailing list to unsubscribe from (name@swarm)", ) list_unsubscribe_p.set_defaults(func=cmd_list_unsubscribe, cmd="list-unsubscribe") diff --git a/src/mail/client/src/mail_client/commands/daemon_get.py b/src/mail/client/src/mail_client/commands/daemon_get.py index c1933ad..00e5b91 100644 --- a/src/mail/client/src/mail_client/commands/daemon_get.py +++ b/src/mail/client/src/mail_client/commands/daemon_get.py @@ -13,7 +13,7 @@ def cmd_daemon_get(args: Namespace) -> None: """ - Get a specific daemon by local address on the MAIL server. + Get a specific daemon by worker name on the MAIL server. """ # 1. check that required env vars are provided @@ -24,9 +24,9 @@ def cmd_daemon_get(args: Namespace) -> None: if MAIL_TOKEN is None: raise ValueError("environment variable MAIL_TOKEN is required") - # 2. Attempt to get the specific daemon by local address on the MAIL server + # 2. Attempt to get the specific daemon by worker name on the MAIL server response = httpx.get( - url=f"{MAIL_SERVER}/admin/daemons/{args.local_address}", + url=f"{MAIL_SERVER}/admin/daemons/{args.worker_name}", headers={ "User-Agent": "Multi-Agent-Interface-Layer-CLI-Client/2.0.0 (github.com/charonlabs/mail)", "Authorization": f"Bearer {MAIL_TOKEN}", diff --git a/src/mail/protocol/src/mail_protocol/network/requests.py b/src/mail/protocol/src/mail_protocol/network/requests.py index f30e945..c679ea0 100644 --- a/src/mail/protocol/src/mail_protocol/network/requests.py +++ b/src/mail/protocol/src/mail_protocol/network/requests.py @@ -213,7 +213,7 @@ class AdminListPostRequest(BaseModel): class AdminListPatchRequest(BaseModel): """ - Corresponds to `PATCH /admin/lists/{list_address}`. + Corresponds to `PATCH /admin/lists/{local_address}`. All fields are optional; only the policy is mutable at v1. The canonical address (name, swarm, host) is immutable for the life of @@ -225,8 +225,8 @@ class AdminListPatchRequest(BaseModel): class ListMemberPostRequest(BaseModel): """ - Corresponds to ``POST /lists/{list_address}/members`` - and ``POST /admin/lists/{list_address}/members``. + Corresponds to ``POST /lists/{local_address}/subscribe`` + and ``POST /admin/lists/{local_address}/members``. ``member_address`` is the address being added. For the public subscribe path, this must match the authenticated bearer diff --git a/src/mail/protocol/src/mail_protocol/network/responses.py b/src/mail/protocol/src/mail_protocol/network/responses.py index 0489dd1..ae3e194 100644 --- a/src/mail/protocol/src/mail_protocol/network/responses.py +++ b/src/mail/protocol/src/mail_protocol/network/responses.py @@ -324,7 +324,7 @@ class AdminAgentsGetResponse(BaseModel): class AdminAgentGetResponse(BaseModel): """ - Corresponds to `GET /admin/agents/{agent_address}`. + Corresponds to `GET /admin/agents/{local_address}`. Contains a specific MAIL agent registered on this server. """ @@ -334,7 +334,7 @@ class AdminAgentGetResponse(BaseModel): class AdminAgentPostResponse(BaseModel): """ - Corresponds to `POST /admin/agent`. + Corresponds to `POST /admin/agents`. Contains the new MAIL agent registered on this server. """ @@ -344,7 +344,7 @@ class AdminAgentPostResponse(BaseModel): class AdminAgentDeleteResponse(BaseModel): """ - Corresponds to `DELETE /admin/agent/{agent_address}`. + Corresponds to `DELETE /admin/agents/{local_address}`. Contains a newly-deleted MAIL agent registered on this server. """ @@ -516,7 +516,7 @@ class AdminListsGetResponse(BaseModel): class AdminListGetResponse(BaseModel): """ - Corresponds to `GET /admin/lists/{list_address}`. + Corresponds to `GET /admin/lists/{local_address}`. """ mail_list: MAILListInBackend @@ -534,7 +534,7 @@ class AdminListPostResponse(BaseModel): class AdminListPatchResponse(BaseModel): """ - Corresponds to `PATCH /admin/lists/{list_address}`. + Corresponds to `PATCH /admin/lists/{local_address}`. """ mail_list: MAILListInBackend @@ -543,7 +543,7 @@ class AdminListPatchResponse(BaseModel): class AdminListDeleteResponse(BaseModel): """ - Corresponds to `DELETE /admin/lists/{list_address}`. + Corresponds to `DELETE /admin/lists/{local_address}`. """ mail_list: MAILListInBackend @@ -563,7 +563,7 @@ class ListsGetResponse(BaseModel): class ListGetResponse(BaseModel): """ - Corresponds to `GET /lists/{list_address}`. + Corresponds to `GET /lists/{local_address}`. """ mail_list: MAILListInBackend @@ -572,8 +572,8 @@ class ListGetResponse(BaseModel): class ListMemberPostResponse(BaseModel): """ - Corresponds to `POST /lists/{list_address}/subscribe` and - `POST /admin/lists/{list_address}/members`. The updated list with + Corresponds to `POST /lists/{local_address}/subscribe` and + `POST /admin/lists/{local_address}/members`. The updated list with the member appended (idempotent — re-adding an existing member is a no-op). """ @@ -584,8 +584,8 @@ class ListMemberPostResponse(BaseModel): class ListMemberDeleteResponse(BaseModel): """ - Corresponds to `POST /lists/{list_address}/unsubscribe` - and `DELETE /lists/{list_address}/members/{member_address}`. + Corresponds to `POST /lists/{local_address}/unsubscribe` + and `DELETE /admin/lists/{local_address}/members/{member_address}`. The updated list with the member removed (idempotent — removing a non-member is a no-op). """ diff --git a/src/mail/server/src/mail_server/backends/base.py b/src/mail/server/src/mail_server/backends/base.py index 371193a..2e6a090 100644 --- a/src/mail/server/src/mail_server/backends/base.py +++ b/src/mail/server/src/mail_server/backends/base.py @@ -60,6 +60,11 @@ class MAILServerBackend(Protocol): A generic base class for the MAIL server backend. """ + # The host this server answers for (e.g. ``mail.example.com``). Set + # during ``on_server_startup``. Routers read this to reconstruct a + # full canonical address from a local path-param identifier. + host: str + # # Lifecyle handlers # @@ -378,7 +383,7 @@ async def admin_get_agents( async def admin_get_agent( self, admin: MAILAdmin, - agent_address: str, + local_address: str, ) -> MAILAgent: """ Get a specific registered agent by local address (agent@swarm). @@ -402,7 +407,7 @@ async def admin_post_agent( async def admin_delete_agent( self, admin: MAILAdmin, - agent_address: str, + local_address: str, ) -> MAILAgent: """ Delete an existing MAIL agent by local address (agent@swarm). diff --git a/src/mail/server/src/mail_server/backends/memory/api.py b/src/mail/server/src/mail_server/backends/memory/api.py index 600378b..3c441cd 100644 --- a/src/mail/server/src/mail_server/backends/memory/api.py +++ b/src/mail/server/src/mail_server/backends/memory/api.py @@ -969,18 +969,18 @@ async def admin_get_agents( async def admin_get_agent( self, admin: MAILAdmin, - agent_address: str, + local_address: str, ) -> MAILAgent: """ Get a specific registered agent by local address (agent@swarm). """ - full_address = f"{agent_address}@{self.host}" + full_address = f"{local_address}@{self.host}" agent = self.user_agents.get(full_address) if agent is None: - raise ValueError(f"no agent found with address {agent_address}") + raise ValueError(f"no agent found with address {local_address}") if agent.user_agent.ua_type != "agent": - raise ValueError(f"invalid agent address: {agent_address}") + raise ValueError(f"invalid agent address: {local_address}") return agent.user_agent @@ -1022,25 +1022,25 @@ async def admin_post_agent( return agent async def admin_delete_agent( - self, admin: MAILAdmin, agent_address: str + self, admin: MAILAdmin, local_address: str ) -> MAILAgent: """ Delete an existing MAIL agent by local address (agent@swarm). """ - full_address = f"{agent_address}@{self.host}" + full_address = f"{local_address}@{self.host}" user_agent = self.user_agents.get(full_address) if user_agent is None: - raise ValueError(f"agent not found: {agent_address}") + raise ValueError(f"agent not found: {local_address}") if user_agent.user_agent.ua_type != "agent": - raise ValueError(f"invalid agent address: {agent_address}") + raise ValueError(f"invalid agent address: {local_address}") agent = self.user_agents.pop(full_address) if not isinstance(agent.user_agent, MAILAgent): self.user_agents.update( {full_address: agent} ) # re-add if invalid this far in - raise ValueError(f"invalid agent address: {agent_address}") + raise ValueError(f"invalid agent address: {local_address}") # remove inbox from self.inboxes self.inboxes.pop(full_address) diff --git a/src/mail/server/src/mail_server/backends/sqlite/api.py b/src/mail/server/src/mail_server/backends/sqlite/api.py index 7b76d3f..cefe3c3 100644 --- a/src/mail/server/src/mail_server/backends/sqlite/api.py +++ b/src/mail/server/src/mail_server/backends/sqlite/api.py @@ -776,16 +776,16 @@ async def admin_get_agents(self, admin: MAILAdmin) -> list[str]: return local_addrs async def admin_get_agent( - self, admin: MAILAdmin, agent_address: str + self, admin: MAILAdmin, local_address: str ) -> MAILAgent: - full_address = f"{agent_address}@{self.host}" + full_address = f"{local_address}@{self.host}" async with self._db.session() as session: agent = await MailStore(session).user_agents.get(full_address) if agent is None: - raise ValueError(f"no agent found with address {agent_address}") + raise ValueError(f"no agent found with address {local_address}") inner = agent.user_agent if not isinstance(inner, MAILAgent): - raise ValueError(f"invalid agent address: {agent_address}") + raise ValueError(f"invalid agent address: {local_address}") return inner async def admin_post_agent( @@ -811,17 +811,17 @@ async def admin_post_agent( return agent async def admin_delete_agent( - self, admin: MAILAdmin, agent_address: str + self, admin: MAILAdmin, local_address: str ) -> MAILAgent: - full_address = f"{agent_address}@{self.host}" + full_address = f"{local_address}@{self.host}" async with self._db.session() as session: store = MailStore(session) agent = await store.user_agents.get(full_address) if agent is None: - raise ValueError(f"agent not found: {agent_address}") + raise ValueError(f"agent not found: {local_address}") inner = agent.user_agent if not isinstance(inner, MAILAgent): - raise ValueError(f"invalid agent address: {agent_address}") + raise ValueError(f"invalid agent address: {local_address}") await store.user_agents.delete(full_address) return inner diff --git a/src/mail/server/src/mail_server/routers/admin.py b/src/mail/server/src/mail_server/routers/admin.py index fb24cd9..38146e8 100644 --- a/src/mail/server/src/mail_server/routers/admin.py +++ b/src/mail/server/src/mail_server/routers/admin.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Addison Kline -from fastapi import APIRouter, HTTPException, Request +from fastapi import APIRouter, HTTPException, Path, Request from mail_protocol.network.responses import ( AdminAgentDeleteResponse, AdminAgentGetResponse, @@ -32,6 +32,11 @@ validate_admin_post_user_request, validate_admin_webhook_patch_request, validate_admin_webhook_post_request, + validate_local_address_param, + validate_swarm_name_param, + validate_user_id_param, + validate_webhook_id_param, + validate_worker_name_param, ) router = APIRouter(prefix="/admin", tags=["admin"]) @@ -59,18 +64,22 @@ async def get_agents( @router.get( - "/agents/{agent_address}", + "/agents/{local_address}", summary="Get a specific registered agent by local address (name@swarm)", response_model=AdminAgentGetResponse, ) async def get_agent( request: Request, + local_address: str = Path( + description="Agent local address (name@swarm); host is implied.", + examples=["researcher@acme"], + ), ) -> AdminAgentGetResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - agent_address = request.path_params.get("agent_address") + local_address = validate_local_address_param(local_address) try: - result = await backend.admin_get_agent(admin=admin, agent_address=agent_address) + result = await backend.admin_get_agent(admin=admin, local_address=local_address) except ValueError: raise HTTPException(status_code=404, detail="agent not found") @@ -103,19 +112,23 @@ async def post_agent( @router.delete( - "/agents/{agent_address}", + "/agents/{local_address}", summary="Delete an existing MAIL agent on this server", response_model=AdminAgentDeleteResponse, ) async def delete_agent( request: Request, + local_address: str = Path( + description="Agent local address (name@swarm); host is implied.", + examples=["researcher@acme"], + ), ) -> AdminAgentDeleteResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - agent_address = request.path_params.get("agent_address") + local_address = validate_local_address_param(local_address) try: result = await backend.admin_delete_agent( - admin=admin, agent_address=agent_address + admin=admin, local_address=local_address ) except ValueError: raise HTTPException(status_code=404, detail="agent not found") @@ -154,10 +167,14 @@ async def get_daemons( ) async def get_daemon( request: Request, + worker_name: str = Path( + description="Daemon worker name; prefix and host are implied.", + examples=["indexer"], + ), ) -> AdminDaemonGetResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - worker_name = request.path_params.get("worker_name") + worker_name = validate_worker_name_param(worker_name) try: result = await backend.admin_get_daemon(admin=admin, worker_name=worker_name) except ValueError: @@ -198,10 +215,14 @@ async def post_daemon( ) async def delete_daemon( request: Request, + worker_name: str = Path( + description="Daemon worker name; prefix and host are implied.", + examples=["indexer"], + ), ) -> AdminDaemonDeleteResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - worker_name = request.path_params.get("worker_name") + worker_name = validate_worker_name_param(worker_name) try: result = await backend.admin_delete_daemon(admin=admin, worker_name=worker_name) except ValueError: @@ -241,10 +262,14 @@ async def get_users( ) async def get_user( request: Request, + user_id: str = Path( + description="User id; prefix and host are implied.", + examples=["addison"], + ), ) -> AdminUserGetResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - user_id = request.path_params.get("user_id") + user_id = validate_user_id_param(user_id) try: result = await backend.admin_get_user(admin=admin, user_id=user_id) except ValueError: @@ -285,10 +310,14 @@ async def post_user( ) async def delete_user( request: Request, + user_id: str = Path( + description="User id; prefix and host are implied.", + examples=["addison"], + ), ) -> AdminUserDeleteResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - user_id = request.path_params.get("user_id") + user_id = validate_user_id_param(user_id) try: result = await backend.admin_delete_user(admin=admin, user_id=user_id) except ValueError: @@ -328,10 +357,16 @@ async def post_swarm(request: Request) -> AdminSwarmPostResponse: summary="Delete an existing MAIL swarm on this server by name", response_model=AdminSwarmDeleteResponse, ) -async def delete_swarm(request: Request) -> AdminSwarmDeleteResponse: +async def delete_swarm( + request: Request, + swarm_name: str = Path( + description="Swarm name.", + examples=["acme"], + ), +) -> AdminSwarmDeleteResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - swarm_name = request.path_params.get("swarm_name") + swarm_name = validate_swarm_name_param(swarm_name) try: result = await backend.admin_delete_swarm(admin=admin, swarm_name=swarm_name) except ValueError: @@ -367,10 +402,16 @@ async def get_webhooks(request: Request) -> AdminWebhooksGetResponse: summary="Get a specific existing server webhook by ID", response_model=AdminWebhookGetResponse, ) -async def get_webhook(request: Request) -> AdminWebhookGetResponse: +async def get_webhook( + request: Request, + webhook_id: str = Path( + description="Webhook id (wh_).", + examples=["wh_123e4567-e89b-12d3-a456-426614174000"], + ), +) -> AdminWebhookGetResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - webhook_id = request.path_params.get("webhook_id") + webhook_id = validate_webhook_id_param(webhook_id) try: result = await backend.admin_webhook_get(admin=admin, webhook_id=webhook_id) except ValueError: @@ -404,11 +445,17 @@ async def post_webhook(request: Request) -> AdminWebhooksPostResponse: summary="Update an existing webhook by ID on this server", response_model=AdminWebhooksPatchResponse, ) -async def patch_webhook(request: Request) -> AdminWebhooksPatchResponse: +async def patch_webhook( + request: Request, + webhook_id: str = Path( + description="Webhook id (wh_).", + examples=["wh_123e4567-e89b-12d3-a456-426614174000"], + ), +) -> AdminWebhooksPatchResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) payload = await validate_admin_webhook_patch_request(request=request) - webhook_id = request.path_params.get("webhook_id") + webhook_id = validate_webhook_id_param(webhook_id) try: result = await backend.admin_webhook_patch( admin=admin, webhook_id=webhook_id, payload=payload @@ -427,10 +474,16 @@ async def patch_webhook(request: Request) -> AdminWebhooksPatchResponse: summary="Delete an existing webhook by ID from this server", response_model=AdminWebhooksDeleteResponse, ) -async def delete_webhook(request: Request) -> AdminWebhooksDeleteResponse: +async def delete_webhook( + request: Request, + webhook_id: str = Path( + description="Webhook id (wh_).", + examples=["wh_123e4567-e89b-12d3-a456-426614174000"], + ), +) -> AdminWebhooksDeleteResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - webhook_id = request.path_params.get("webhook_id") + webhook_id = validate_webhook_id_param(webhook_id) try: result = await backend.admin_webhook_delete(admin=admin, webhook_id=webhook_id) except ValueError: diff --git a/src/mail/server/src/mail_server/routers/lists.py b/src/mail/server/src/mail_server/routers/lists.py index de963df..7eba721 100644 --- a/src/mail/server/src/mail_server/routers/lists.py +++ b/src/mail/server/src/mail_server/routers/lists.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Charon Labs (contribution PR) -from fastapi import APIRouter, HTTPException, Request +from fastapi import APIRouter, HTTPException, Path, Request from mail_protocol.core.lists import MAILListPolicy from mail_protocol.network.responses import ( AdminListDeleteResponse, @@ -16,15 +16,34 @@ ) from mail_server.auth import validate_admin, validate_user_agent +from mail_server.backends.base import MAILServerBackend from mail_server.validators import ( validate_admin_patch_list_request, validate_admin_post_list_request, validate_list_member_post_request, + validate_local_address_param, + validate_member_address_param, ) admin_router = APIRouter(prefix="/admin/lists", tags=["admin-lists"]) public_router = APIRouter(prefix="/lists", tags=["lists"]) +_LOCAL_ADDRESS_PATH = Path( + description="List local address (name@swarm); the list: prefix and host are implied.", + examples=["announce@acme"], +) + + +def _full_list_address(backend: MAILServerBackend, local_address: str) -> str: + """ + Reconstruct a list's full canonical ``list:name@swarm@host`` address + from the local ``name@swarm`` path param. The path param carries only + the local identifier; the ``list:`` prefix is implied by the route and + the host by the server. + """ + + return f"list:{local_address}@{backend.host}" + # # Policy guardrails — v1 ships only the open / public variants. The @@ -82,14 +101,18 @@ async def admin_get_lists(request: Request) -> AdminListsGetResponse: @admin_router.get( - "/{list_address}", - summary="Get a specific MAIL list by address", + "/{local_address}", + summary="Get a specific MAIL list by local address (name@swarm)", response_model=AdminListGetResponse, ) -async def admin_get_list(request: Request) -> AdminListGetResponse: +async def admin_get_list( + request: Request, local_address: str = _LOCAL_ADDRESS_PATH +) -> AdminListGetResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - list_address = request.path_params.get("list_address") + list_address = _full_list_address( + backend, validate_local_address_param(local_address) + ) try: result = await backend.admin_get_list(admin=admin, list_address=list_address) except ValueError: @@ -115,14 +138,18 @@ async def admin_post_list(request: Request) -> AdminListPostResponse: @admin_router.patch( - "/{list_address}", + "/{local_address}", summary="Update a MAIL list's policy", response_model=AdminListPatchResponse, ) -async def admin_patch_list(request: Request) -> AdminListPatchResponse: +async def admin_patch_list( + request: Request, local_address: str = _LOCAL_ADDRESS_PATH +) -> AdminListPatchResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - list_address = request.path_params.get("list_address") + list_address = _full_list_address( + backend, validate_local_address_param(local_address) + ) payload = await validate_admin_patch_list_request(request=request) _reject_unsupported_policy(payload.policy) try: @@ -135,14 +162,18 @@ async def admin_patch_list(request: Request) -> AdminListPatchResponse: @admin_router.delete( - "/{list_address}", + "/{local_address}", summary="Delete a MAIL list", response_model=AdminListDeleteResponse, ) -async def admin_delete_list(request: Request) -> AdminListDeleteResponse: +async def admin_delete_list( + request: Request, local_address: str = _LOCAL_ADDRESS_PATH +) -> AdminListDeleteResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - list_address = request.path_params.get("list_address") + list_address = _full_list_address( + backend, validate_local_address_param(local_address) + ) try: result = await backend.admin_delete_list(admin=admin, list_address=list_address) except ValueError: @@ -151,15 +182,19 @@ async def admin_delete_list(request: Request) -> AdminListDeleteResponse: @admin_router.post( - "/{list_address}/members", + "/{local_address}/members", summary="Admin-add a member to a MAIL list", response_model=ListMemberPostResponse, ) -async def admin_add_list_member(request: Request) -> ListMemberPostResponse: +async def admin_add_list_member( + request: Request, local_address: str = _LOCAL_ADDRESS_PATH +) -> ListMemberPostResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) _ = admin # auth-only; the backend method does not gate by admin - list_address = request.path_params.get("list_address") + list_address = _full_list_address( + backend, validate_local_address_param(local_address) + ) payload = await validate_list_member_post_request(request=request) try: result = await backend.add_list_member( @@ -172,16 +207,25 @@ async def admin_add_list_member(request: Request) -> ListMemberPostResponse: @admin_router.delete( - "/{list_address}/members/{member_address}", + "/{local_address}/members/{member_address}", summary="Admin-remove a member from a MAIL list", response_model=ListMemberDeleteResponse, ) -async def admin_remove_list_member(request: Request) -> ListMemberDeleteResponse: +async def admin_remove_list_member( + request: Request, + local_address: str = _LOCAL_ADDRESS_PATH, + member_address: str = Path( + description="Full MAIL address of the member to remove (may be remote).", + examples=["user:bob@other-host"], + ), +) -> ListMemberDeleteResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) _ = admin - list_address = request.path_params.get("list_address") - member_address = request.path_params.get("member_address") + list_address = _full_list_address( + backend, validate_local_address_param(local_address) + ) + member_address = validate_member_address_param(member_address) try: result = await backend.remove_list_member( list_address=list_address, @@ -212,14 +256,18 @@ async def get_lists(request: Request) -> ListsGetResponse: @public_router.get( - "/{list_address}", - summary="Get a specific list", + "/{local_address}", + summary="Get a specific list by local address (name@swarm)", response_model=ListGetResponse, ) -async def get_list(request: Request) -> ListGetResponse: +async def get_list( + request: Request, local_address: str = _LOCAL_ADDRESS_PATH +) -> ListGetResponse: backend = request.app.state.backend await validate_user_agent(backend=backend, request=request) - list_address = request.path_params.get("list_address") + list_address = _full_list_address( + backend, validate_local_address_param(local_address) + ) try: result = await backend.get_list(list_address=list_address) except ValueError: @@ -231,14 +279,18 @@ async def get_list(request: Request) -> ListGetResponse: @public_router.post( - "/{list_address}/subscribe", + "/{local_address}/subscribe", summary="Subscribe to a MAIL list", response_model=ListMemberPostResponse, ) -async def subscribe(request: Request) -> ListMemberPostResponse: +async def subscribe( + request: Request, local_address: str = _LOCAL_ADDRESS_PATH +) -> ListMemberPostResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - list_address = request.path_params.get("list_address") + list_address = _full_list_address( + backend, validate_local_address_param(local_address) + ) try: existing = await backend.get_list(list_address=list_address) @@ -262,14 +314,18 @@ async def subscribe(request: Request) -> ListMemberPostResponse: @public_router.post( - "/{list_address}/unsubscribe", + "/{local_address}/unsubscribe", summary="Unsubscribe from a MAIL list", response_model=ListMemberDeleteResponse, ) -async def unsubscribe(request: Request) -> ListMemberDeleteResponse: +async def unsubscribe( + request: Request, local_address: str = _LOCAL_ADDRESS_PATH +) -> ListMemberDeleteResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - list_address = request.path_params.get("list_address") + list_address = _full_list_address( + backend, validate_local_address_param(local_address) + ) member_address = user_agent.get_address() try: diff --git a/src/mail/server/src/mail_server/validators.py b/src/mail/server/src/mail_server/validators.py index 9b182bb..bbaafea 100644 --- a/src/mail/server/src/mail_server/validators.py +++ b/src/mail/server/src/mail_server/validators.py @@ -2,6 +2,14 @@ # Copyright (c) 2026 Addison Kline from fastapi import HTTPException, Request +from mail_protocol.core.validators import ( + validate_daemon_worker_name, + validate_local_address, + validate_mail_address, + validate_swarm_name, + validate_user_name, + validate_webhook_id, +) from mail_protocol.network.requests import ( AdminAgentPostRequest, AdminDaemonPostRequest, @@ -47,6 +55,96 @@ async def validate_box_filter_params(request: Request) -> BoxFilterParams: ) +# +# Path parameter validators +# +# Admin (and list) endpoints address resources by their *local* +# identifier — the host is implied by the server and the user-agent +# prefix (``daemon:``/``user:``/``list:``) is implied by the route. +# These helpers validate the shape of a path segment and 422 on +# malformed input, mirroring the body/query validators above. A +# well-formed-but-unknown id still 404s downstream. +# +def validate_local_address_param(value: str) -> str: + """ + Validate an agent or list local address path param (``name@swarm``). + """ + + try: + return validate_local_address(value) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"invalid local address path parameter: {e}" + ) + + +def validate_worker_name_param(value: str) -> str: + """ + Validate a daemon ``worker_name`` path param. + """ + + try: + return validate_daemon_worker_name(value) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"invalid worker name path parameter: {e}" + ) + + +def validate_user_id_param(value: str) -> str: + """ + Validate a ``user_id`` path param. + """ + + try: + return validate_user_name(value) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"invalid user id path parameter: {e}" + ) + + +def validate_swarm_name_param(value: str) -> str: + """ + Validate a ``swarm_name`` path param. + """ + + try: + return validate_swarm_name(value) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"invalid swarm name path parameter: {e}" + ) + + +def validate_webhook_id_param(value: str) -> str: + """ + Validate a ``webhook_id`` path param (``wh_``). + """ + + try: + return validate_webhook_id(value) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"invalid webhook id path parameter: {e}" + ) + + +def validate_member_address_param(value: str) -> str: + """ + Validate a list ``member_address`` path param. Unlike the resource + identifiers above, a member can be any user-agent — possibly on + another host — so this is a full MAIL address, not a local one. + """ + + try: + return validate_mail_address(value) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"invalid member address path parameter: {e}" + ) + + # # Draft endpoint validators # @@ -246,7 +344,7 @@ async def validate_admin_patch_list_request( request: Request, ) -> AdminListPatchRequest: """ - Ensure that the request payload is valid for `PATCH /admin/lists/{list_address}`. + Ensure that the request payload is valid for `PATCH /admin/lists/{local_address}`. """ try: @@ -263,7 +361,7 @@ async def validate_list_member_post_request( ) -> ListMemberPostRequest: """ Ensure that the request payload is valid for the member-add endpoints - (`POST /lists/{list_address}/members` and the admin variant). + (`POST /admin/lists/{local_address}/members` and the subscribe variant). """ try: diff --git a/tests/e2e/test_journeys.py b/tests/e2e/test_journeys.py index cfbe22b..9404ac5 100644 --- a/tests/e2e/test_journeys.py +++ b/tests/e2e/test_journeys.py @@ -17,7 +17,10 @@ USER = f"user:alice@{HOST}" OTHER_USER = f"user:bob@{HOST}" AGENT = f"sage@{SWARM}@{HOST}" +# Full address: used as a message recipient (delivery resolves the full +# ``list:`` form). The list HTTP/CLI surface addresses it by local form. LIST_ADDRESS = f"list:town-square@{SWARM}@{HOST}" +LIST_LOCAL_ADDRESS = f"town-square@{SWARM}" def test_send_deliver_read_journey(e2e_stack) -> None: @@ -67,7 +70,7 @@ def test_list_fan_out_journey(e2e_stack) -> None: assert response.status_code == 200, response.text # Bob subscribes himself through the CLI. - subscribed = e2e_stack.cli_json("list-subscribe", LIST_ADDRESS, token=bob) + subscribed = e2e_stack.cli_json("list-subscribe", LIST_LOCAL_ADDRESS, token=bob) assert OTHER_USER in subscribed["mail_list"]["members"] draft = e2e_stack.cli_json("compose", "To the square", "Hear ye.", token=alice) diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 707c210..37baa57 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -211,7 +211,11 @@ async def mutate(store: MailStore) -> None: @pytest.fixture def seed_list(backend: MAILServerBackend) -> Callable[[MAILListInBackend], str]: - """Backend-agnostic: persist a prebuilt list. Returns its address.""" + """ + Backend-agnostic: persist a prebuilt list. Returns its *local* + address (``name@swarm``) — the form the HTTP API addresses lists by. + The backend still keys lists by the full ``list:`` address. + """ def _seed(record: MAILListInBackend) -> str: if isinstance(backend, MemoryBackend): @@ -223,7 +227,7 @@ async def mutate(store: MailStore) -> None: await store.lists.add(record) _run_sqlite_write(backend._db.url, mutate) - return record.get_address() + return f"{record.name}@{record.swarm}" return _seed diff --git a/tests/integration/test_admin.py b/tests/integration/test_admin.py index b1f0d7f..cc63f70 100644 --- a/tests/integration/test_admin.py +++ b/tests/integration/test_admin.py @@ -29,6 +29,24 @@ def test_get_agent_unknown_returns_404(app_client: TestClient, headers_for) -> N assert response.status_code == 404 +def test_get_agent_malformed_local_address_returns_422( + app_client: TestClient, headers_for +) -> None: + # A full address (name@swarm@host) is no longer accepted here: the + # path param is the *local* address (name@swarm). + response = app_client.get( + f"/admin/agents/sage@{SWARM}@localhost", headers=headers_for(ADMIN) + ) + assert response.status_code == 422 + + +def test_get_agent_non_slug_returns_422(app_client: TestClient, headers_for) -> None: + response = app_client.get( + f"/admin/agents/Sage@{SWARM}", headers=headers_for(ADMIN) + ) + assert response.status_code == 422 + + def test_post_agent_creates_and_can_login( app_client: TestClient, headers_for, token_for ) -> None: @@ -130,6 +148,13 @@ def test_get_daemon_unknown_returns_404(app_client: TestClient, headers_for) -> assert response.status_code == 404 +def test_get_daemon_malformed_worker_name_returns_422( + app_client: TestClient, headers_for +) -> None: + response = app_client.get("/admin/daemons/Bad-Name", headers=headers_for(ADMIN)) + assert response.status_code == 422 + + def test_post_daemon_creates_and_can_login(app_client: TestClient, headers_for) -> None: response = app_client.post( "/admin/daemons", diff --git a/tests/integration/test_admin_webhooks.py b/tests/integration/test_admin_webhooks.py index 78e5597..3955155 100644 --- a/tests/integration/test_admin_webhooks.py +++ b/tests/integration/test_admin_webhooks.py @@ -59,12 +59,23 @@ def test_get_webhook_by_id(app_client: TestClient, headers_for) -> None: def test_get_webhook_unknown_returns_404( app_client: TestClient, headers_for ) -> None: + # Well-formed (wh_) but unknown id → 404. response = app_client.get( - "/admin/webhooks/wh_nonexistent", headers=headers_for(ADMIN) + "/admin/webhooks/wh_123e4567-e89b-12d3-a456-426614174000", + headers=headers_for(ADMIN), ) assert response.status_code == 404 +def test_get_webhook_malformed_id_returns_422( + app_client: TestClient, headers_for +) -> None: + response = app_client.get( + "/admin/webhooks/wh_nonexistent", headers=headers_for(ADMIN) + ) + assert response.status_code == 422 + + def test_delete_webhook_removes(app_client: TestClient, headers_for) -> None: headers = headers_for(ADMIN) webhook_id = _post_webhook(app_client, headers) @@ -80,8 +91,10 @@ def test_delete_webhook_removes(app_client: TestClient, headers_for) -> None: def test_delete_webhook_unknown_returns_404( app_client: TestClient, headers_for ) -> None: + # Well-formed (wh_) but unknown id → 404. response = app_client.delete( - "/admin/webhooks/wh_nonexistent", headers=headers_for(ADMIN) + "/admin/webhooks/wh_123e4567-e89b-12d3-a456-426614174000", + headers=headers_for(ADMIN), ) assert response.status_code == 404 diff --git a/tests/integration/test_flows.py b/tests/integration/test_flows.py index ee46392..399dd64 100644 --- a/tests/integration/test_flows.py +++ b/tests/integration/test_flows.py @@ -12,7 +12,11 @@ USER = "user:alice@localhost" OTHER_USER = "user:bob@localhost" AGENT = "sage@chorus@localhost" +# Full address: used as a message recipient (delivery resolves the full +# ``list:`` form). The HTTP list endpoints address the same list by its +# local form (``name@swarm``). LIST_ADDRESS = "list:welfare-discourse@chorus@localhost" +LIST_LOCAL_ADDRESS = "welfare-discourse@chorus" def test_send_and_read_journey( @@ -64,7 +68,7 @@ def test_list_fan_out_journey( # Bob subscribes himself through the public endpoint. response = app_client.post( - f"/lists/{LIST_ADDRESS}/subscribe", headers=headers_for(OTHER_USER) + f"/lists/{LIST_LOCAL_ADDRESS}/subscribe", headers=headers_for(OTHER_USER) ) assert response.status_code == 200 assert OTHER_USER in response.json()["mail_list"]["members"] diff --git a/tests/integration/test_lists.py b/tests/integration/test_lists.py index 2a8d13d..407c89a 100644 --- a/tests/integration/test_lists.py +++ b/tests/integration/test_lists.py @@ -145,12 +145,24 @@ def test_admin_get_list_missing_returns_404( app_client: TestClient, headers_for ) -> None: response = app_client.get( - "/admin/lists/list:nonexistent@chorus@localhost", + "/admin/lists/nonexistent@chorus", headers=headers_for(ADMIN_ADDRESS), ) assert response.status_code == 404 +def test_admin_get_list_malformed_local_address_returns_422( + app_client: TestClient, headers_for +) -> None: + # The full ``list:`` address is no longer accepted here — the path + # param is the list's local address (name@swarm). + response = app_client.get( + "/admin/lists/list:welfare-discourse@chorus@localhost", + headers=headers_for(ADMIN_ADDRESS), + ) + assert response.status_code == 422 + + def test_admin_patch_list_updates_policy_no_op_for_open( app_client: TestClient, headers_for, @@ -202,7 +214,7 @@ def test_admin_delete_list_missing_returns_404( app_client: TestClient, headers_for ) -> None: response = app_client.delete( - "/admin/lists/list:nonexistent@chorus@localhost", + "/admin/lists/nonexistent@chorus", headers=headers_for(ADMIN_ADDRESS), ) assert response.status_code == 404 @@ -286,7 +298,7 @@ def test_get_list_missing_returns_404( app_client: TestClient, headers_for ) -> None: response = app_client.get( - "/lists/list:nonexistent@chorus@localhost", + "/lists/nonexistent@chorus", headers=headers_for(USER_ADDRESS), ) assert response.status_code == 404 @@ -334,7 +346,7 @@ def test_subscribe_missing_list_returns_404( app_client: TestClient, headers_for ) -> None: response = app_client.post( - "/lists/list:nonexistent@chorus@localhost/subscribe", + "/lists/nonexistent@chorus/subscribe", headers=headers_for(USER_ADDRESS), ) assert response.status_code == 404 From de3b37e1128a95818c6200bec67f02f8ff7b878e Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Fri, 26 Jun 2026 18:38:50 -0400 Subject: [PATCH 14/28] feat: add refresh token support Add a stateful, rotating refresh-token mechanism so browsers (and other long-lived clients) can renew their access-token JWT without re-entering a password. Refresh tokens are opaque, high-entropy strings stored hashed (sha256) and grouped into families: login starts a family, each rotation keeps the family and carries its absolute `expires_at` forward unchanged (no sliding window). Presenting a revoked-or-rotated token is treated as reuse and revokes the whole family. Only interactive principals (users/admins) get refresh tokens; agents and daemons re-authenticate with their credentials. Delivery is dual: the token is returned in the response body and set as an `httpOnly; Secure; SameSite=Strict` cookie scoped to `/auth`, so browsers get silent renewal while the wider API stays header-only and CSRF-immune; the CLI sends the token back in the request body. - protocol: `RefreshTokenRecord`; `refresh_token`/`expires_in` on the token response; `AuthRefreshPostRequest`/`AuthRefreshPostResponse`/`AuthLogoutPostResponse` - backends: six refresh-token methods on the protocol, implemented for both the memory (persisted via fs checkpoints) and sqlite (new `refresh_tokens` table, FK cascade on user deletion) backends, with dual-backend conformance tests - server: `/auth/token` mints + sets the cookie, new `/auth/refresh` (rotate + reuse detection + fail-closed on deleted owner) and `/auth/logout`, and `/auth/password/reset` now revokes all of the principal's families - config: env-only `MAIL_REFRESH_TOKEN_EXPIRE_DAYS` (required), `MAIL_COOKIE_SECURE`, `MAIL_COOKIE_DOMAIN` - client: `mail refresh` command; `mail login` surfaces the refresh token - docs: HTTP API reference, quickstarts, .env.example, regenerated openapi.yaml Co-Authored-By: Claude Opus 4.8 (1M context) --- llms.txt | 4 +- spec/openapi.yaml | 100 +++++++++- src/mail/client/docs/reference/cli.md | 3 +- src/mail/client/docs/tutorials/quickstart.md | 20 ++ src/mail/client/src/mail_client/cli.py | 13 ++ .../src/mail_client/commands/__init__.py | 2 + .../client/src/mail_client/commands/login.py | 11 ++ .../src/mail_client/commands/refresh.py | 78 ++++++++ .../protocol/src/mail_protocol/core/auth.py | 34 ++++ .../src/mail_protocol/network/requests.py | 11 ++ .../src/mail_protocol/network/responses.py | 34 ++++ src/mail/server/.env.example | 13 ++ src/mail/server/docs/reference/http.md | 37 +++- src/mail/server/docs/tutorials/quickstart.md | 8 + src/mail/server/src/mail_server/auth.py | 104 +++++++++- .../server/src/mail_server/backends/base.py | 70 +++++++ .../src/mail_server/backends/memory/api.py | 99 ++++++++++ .../src/mail_server/backends/memory/fs.py | 51 +++++ .../src/mail_server/backends/memory/init.py | 5 + .../src/mail_server/backends/sqlite/api.py | 58 ++++++ .../backends/sqlite/repositories.py | 62 +++++- .../src/mail_server/backends/sqlite/schema.py | 31 +++ .../backends/sqlite/serializers.py | 48 +++++ .../server/src/mail_server/routers/auth.py | 138 ++++++++++++- src/mail/server/src/mail_server/validators.py | 23 +++ tests/conftest.py | 6 + tests/e2e/conftest.py | 1 + tests/integration/conftest.py | 41 +++- tests/integration/test_refresh_flow.py | 184 +++++++++++++++++ .../test_refresh_tokens_backend.py | 187 ++++++++++++++++++ tests/unit/test_client_commands.py | 55 +++++- tests/unit/test_daemon_api.py | 7 +- tests/unit/test_refresh_auth_helpers.py | 98 +++++++++ 33 files changed, 1621 insertions(+), 15 deletions(-) create mode 100644 src/mail/client/src/mail_client/commands/refresh.py create mode 100644 src/mail/protocol/src/mail_protocol/core/auth.py create mode 100644 tests/integration/test_refresh_flow.py create mode 100644 tests/integration/test_refresh_tokens_backend.py create mode 100644 tests/unit/test_refresh_auth_helpers.py diff --git a/llms.txt b/llms.txt index 99e0fb1..2f0a6f2 100644 --- a/llms.txt +++ b/llms.txt @@ -187,8 +187,8 @@ three files moves here, alongside: - `backend` — started `MemoryBackend` seeded with a standard cast: one admin, two users, one agent, one daemon, one swarm - `app_client` — `TestClient` over the **real** `mail_server.server.app` - (env vars `MAIL_HOST`, `MAIL_JWT_SECRET_KEY`, `MAIL_JWT_ALGORITHM` set - before import), wired to `backend` + (env vars `MAIL_HOST`, `MAIL_JWT_SECRET_KEY`, `MAIL_JWT_ALGORITHM`, + `MAIL_REFRESH_TOKEN_EXPIRE_DAYS` set before import), wired to `backend` - `token_for(address)` — factory issuing real JWTs via `POST /auth/token`, so integration tests exercise real auth instead of monkeypatching it - `webhook_receiver` — in-process ASGI app that records deliveries and can be diff --git a/spec/openapi.yaml b/spec/openapi.yaml index 8305ec2..4677c33 100644 --- a/spec/openapi.yaml +++ b/spec/openapi.yaml @@ -30,6 +30,33 @@ paths: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' + /auth/refresh: + post: + tags: + - authentication + summary: Exchange a refresh token for a new access token (rotates the refresh + token) + operationId: post_auth_refresh_auth_refresh_post + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/AuthRefreshPostResponse' + /auth/logout: + post: + tags: + - authentication + summary: Revoke the presented refresh token's family and clear the cookie + operationId: post_auth_logout_auth_logout_post + responses: + '200': + description: Successful Response + content: + application/json: + schema: + $ref: '#/components/schemas/AuthLogoutPostResponse' /auth/whoami: get: tags: @@ -1461,6 +1488,19 @@ components: description: 'Corresponds to `POST /admin/webhooks`. Contains information on the newly-created webhook.' + AuthLogoutPostResponse: + properties: + status: + type: string + const: success + title: Status + type: object + required: + - status + title: AuthLogoutPostResponse + description: 'Corresponds to `POST /auth/logout`. + + Contains a message indicating operation success.' AuthPasswordResetResponse: properties: status: @@ -1474,6 +1514,44 @@ components: description: 'Corresponds to `POST /auth/password/reset`. Contains a message indicating operation success.' + AuthRefreshPostResponse: + properties: + access_token: + type: string + title: Access Token + token_type: + type: string + const: bearer + title: Token Type + refresh_token: + anyOf: + - type: string + - type: 'null' + title: Refresh Token + expires_in: + type: integer + title: Expires In + metadata: + additionalProperties: true + type: object + title: Metadata + type: object + required: + - access_token + - token_type + - expires_in + - metadata + title: AuthRefreshPostResponse + description: 'Corresponds to `POST /auth/refresh`. + + Contains a freshly-minted access token and a rotated refresh token. + + + Mirrors `AuthTokenPostResponse`. The previous refresh token is invalidated + + on every successful refresh; ``refresh_token`` carries its replacement (also + + rotated in the ``httpOnly`` cookie for browser clients).' AuthTokenPostResponse: properties: access_token: @@ -1483,6 +1561,14 @@ components: type: string const: bearer title: Token Type + refresh_token: + anyOf: + - type: string + - type: 'null' + title: Refresh Token + expires_in: + type: integer + title: Expires In metadata: additionalProperties: true type: object @@ -1491,11 +1577,23 @@ components: required: - access_token - token_type + - expires_in - metadata title: AuthTokenPostResponse description: 'Corresponds to `POST /auth/token`. - Contains a temporary JWT and associated metadata.' + Contains a temporary JWT and associated metadata. + + + ``refresh_token`` is populated only for interactive principals (users and + + admins); agents and daemons re-authenticate with their credentials and + + receive ``None``. When present, the server also sets it as an ``httpOnly`` + + cookie for browser clients. ``expires_in`` is the access-token lifetime in + + seconds.' AuthWhoamiGetResponse: properties: user_agent: diff --git a/src/mail/client/docs/reference/cli.md b/src/mail/client/docs/reference/cli.md index dcb210e..97fb8bb 100644 --- a/src/mail/client/docs/reference/cli.md +++ b/src/mail/client/docs/reference/cli.md @@ -41,7 +41,8 @@ mail [option]... [argument]... ### Utility Commands - `ping`: Attempt to ping the MAIL server at the URL provided. -- `login`: Log into a MAIL server with valid credentials to obtain a temporary access token. +- `login`: Log into a MAIL server with valid credentials to obtain a temporary access token (and, for users/admins, a refresh token). +- `refresh`: Renew your access token from a refresh token (set `MAIL_REFRESH_TOKEN`), without logging in again. The refresh token is rotated, so update `MAIL_REFRESH_TOKEN` with the returned value. - `whoami`: View information on the logged-in MAIL user-agent. ## Top-level Options diff --git a/src/mail/client/docs/tutorials/quickstart.md b/src/mail/client/docs/tutorials/quickstart.md index ae57bc8..a1bb386 100644 --- a/src/mail/client/docs/tutorials/quickstart.md +++ b/src/mail/client/docs/tutorials/quickstart.md @@ -23,6 +23,26 @@ uv run mail login If your credentials are valid, you should see a JWT printed to the console. Copy this and use it as the value for environment variable `MAIL_TOKEN` in subsequent commands. +If you logged in as a user or admin, a **refresh token** is printed as well. +Save it as `MAIL_REFRESH_TOKEN` — it lets you renew your access token without +re-entering your password. (Agents and daemons don't get one; they simply log in +again.) + +## Renew Your Access Token + +Access tokens are short-lived. When yours expires, exchange your refresh token +for a new one with the `refresh` command instead of logging in again: + +```bash +MAIL_SERVER=... \ +MAIL_REFRESH_TOKEN=... \ +uv run mail refresh +``` + +A new access token is printed, along with a **rotated** refresh token — the old +refresh token is now invalid, so update `MAIL_REFRESH_TOKEN` with the new value +for next time. + ## Validate Client Identity To ensure your credentials are valid and as expected, use the `whoami` command: diff --git a/src/mail/client/src/mail_client/cli.py b/src/mail/client/src/mail_client/cli.py index f4c390c..2b4f0c9 100644 --- a/src/mail/client/src/mail_client/cli.py +++ b/src/mail/client/src/mail_client/cli.py @@ -24,6 +24,7 @@ cmd_outbox, cmd_outbox_open, cmd_ping, + cmd_refresh, cmd_reply, cmd_send, cmd_swarm_get, @@ -68,6 +69,7 @@ [ ("ping (p)", "Ping a MAIL server."), ("login (l)", "Log into a MAIL server."), + ("refresh (rt)", "Renew your access token with a refresh token."), ("whoami (me, id)", "Show authenticated user-agent info."), ], ), @@ -219,6 +221,17 @@ def build_parser() -> argparse.ArgumentParser: ) login_p.set_defaults(func=cmd_login, cmd="login") + # command `refresh` + refresh_d = "renew your access token using a refresh token" + refresh_p = subparsers.add_parser( + "refresh", + aliases=["rt"], + prog="mail refresh", + help=refresh_d, + description=refresh_d, + ) + refresh_p.set_defaults(func=cmd_refresh, cmd="refresh") + # command `whoami` whoami_d = "get authenticated user-agent info from a MAIL server" whoami_p = subparsers.add_parser( diff --git a/src/mail/client/src/mail_client/commands/__init__.py b/src/mail/client/src/mail_client/commands/__init__.py index f5f5a1d..fccf88f 100644 --- a/src/mail/client/src/mail_client/commands/__init__.py +++ b/src/mail/client/src/mail_client/commands/__init__.py @@ -28,6 +28,7 @@ from .outbox import cmd_outbox from .outbox_open import cmd_outbox_open from .ping import cmd_ping +from .refresh import cmd_refresh from .reply import cmd_reply from .send import cmd_send from .swarm_delete import cmd_swarm_delete @@ -78,6 +79,7 @@ "cmd_outbox", "cmd_outbox_open", "cmd_ping", + "cmd_refresh", "cmd_reply", "cmd_send", "cmd_swarm_delete", diff --git a/src/mail/client/src/mail_client/commands/login.py b/src/mail/client/src/mail_client/commands/login.py index 4b4d633..cfecbfe 100644 --- a/src/mail/client/src/mail_client/commands/login.py +++ b/src/mail/client/src/mail_client/commands/login.py @@ -73,3 +73,14 @@ def _print_text(response_obj: AuthTokenPostResponse) -> None: print(response_obj.access_token) print() print("Run subsequent commands with `MAIL_TOKEN={token}`") + # Interactive principals (users/admins) also receive a refresh token; agents + # and daemons do not. + if response_obj.refresh_token is not None: + print() + print("Got refresh token:") + print(response_obj.refresh_token) + print() + print( + "Renew your access token without logging in again by setting " + "`MAIL_REFRESH_TOKEN={refresh_token}` and running `mail refresh`" + ) diff --git a/src/mail/client/src/mail_client/commands/refresh.py b/src/mail/client/src/mail_client/commands/refresh.py new file mode 100644 index 0000000..58e0e71 --- /dev/null +++ b/src/mail/client/src/mail_client/commands/refresh.py @@ -0,0 +1,78 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +import os +from argparse import Namespace + +import httpx +from mail_protocol.network.responses import AuthRefreshPostResponse +from pydantic import ValidationError + + +def cmd_refresh(args: Namespace) -> None: + """ + Exchange a refresh token for a renewed access token. + + The refresh token is rotated server-side: the value in ``MAIL_REFRESH_TOKEN`` + is invalidated and a replacement is returned, which the caller should store + for the next refresh. + """ + + # 1. check that required env vars are provided + MAIL_SERVER = os.getenv("MAIL_SERVER") + if MAIL_SERVER is None: + raise ValueError("environment variable MAIL_SERVER is required") + MAIL_REFRESH_TOKEN = os.getenv("MAIL_REFRESH_TOKEN") + if MAIL_REFRESH_TOKEN is None: + raise ValueError("environment variable MAIL_REFRESH_TOKEN is required") + + # 2. hit the server endpoint `POST /auth/refresh`. The CLI can't use the + # httpOnly cookie browsers rely on, so the token is sent in the body. + response = httpx.post( + url=f"{MAIL_SERVER}/auth/refresh", + headers={ + "accept": "application/json", + "Content-Type": "application/json", + "User-Agent": "Multi-Agent-Interface-Layer-CLI-Client/2.0.0 (github.com/charonlabs/mail)", + }, + json={"refresh_token": MAIL_REFRESH_TOKEN}, + ) + + # 3. parse and validate server response + if response.status_code != 200: + raise RuntimeError( + f"refresh request to {MAIL_SERVER} failed with status code {response.status_code}" + ) + + response_json = response.json() + try: + response_obj = AuthRefreshPostResponse.model_validate(response_json) + except ValidationError as e: + raise RuntimeError(f"response validation failed: {e}") + + # 4. print the renewed token(s) + match args.output: + case "json": + _print_json(response_obj) + case "text": + _print_text(response_obj) + + +def _print_json(response_obj: AuthRefreshPostResponse) -> None: + print(response_obj.model_dump_json()) + + +def _print_text(response_obj: AuthRefreshPostResponse) -> None: + print("Got token:") + print(response_obj.access_token) + print() + print("Run subsequent commands with `MAIL_TOKEN={token}`") + if response_obj.refresh_token is not None: + print() + print("Got rotated refresh token:") + print(response_obj.refresh_token) + print() + print( + "Your previous refresh token is now invalid. " + "Update `MAIL_REFRESH_TOKEN` with this value." + ) diff --git a/src/mail/protocol/src/mail_protocol/core/auth.py b/src/mail/protocol/src/mail_protocol/core/auth.py new file mode 100644 index 0000000..00b648a --- /dev/null +++ b/src/mail/protocol/src/mail_protocol/core/auth.py @@ -0,0 +1,34 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +from datetime import datetime +from typing import Annotated + +from pydantic import AfterValidator, BaseModel + +from mail_protocol.core.validators import validate_mail_address + + +class RefreshTokenRecord(BaseModel): + """ + A stored refresh token, as persisted by a MAIL server backend. + + Backend-internal — this never crosses the wire. The plaintext token is + returned to the client exactly once at issuance; only its SHA-256 hash is + stored, so this record carries the hash rather than the token itself. + + Refresh tokens are grouped into *families*: the token minted at login starts + a family, and every rotation keeps the same ``family_id``. ``expires_at`` is + an absolute cap set at login and carried forward unchanged on rotation (it + does not slide). A token is unusable once ``revoked`` is ``True`` or + ``rotated_at`` is set — presenting such a token is treated as reuse and + revokes the whole family. + """ + + token_hash: str + family_id: str + owner_address: Annotated[str, AfterValidator(validate_mail_address)] + issued_at: datetime + expires_at: datetime + revoked: bool = False + rotated_at: datetime | None = None diff --git a/src/mail/protocol/src/mail_protocol/network/requests.py b/src/mail/protocol/src/mail_protocol/network/requests.py index c679ea0..dad39ef 100644 --- a/src/mail/protocol/src/mail_protocol/network/requests.py +++ b/src/mail/protocol/src/mail_protocol/network/requests.py @@ -39,6 +39,17 @@ class AuthTokenPostRequest(BaseModel): pass +class AuthRefreshPostRequest(BaseModel): + """ + Corresponds to `POST /auth/refresh`. + Body fallback carrying the refresh token for clients that cannot use the + ``httpOnly`` cookie (e.g. the CLI). Browsers send the token via cookie and + may omit the body entirely. + """ + + refresh_token: str | None = None + + class AuthPasswordResetRequest(BaseModel): """ Corresponds to `POST /auth/password/reset`. diff --git a/src/mail/protocol/src/mail_protocol/network/responses.py b/src/mail/protocol/src/mail_protocol/network/responses.py index ae3e194..f60ece0 100644 --- a/src/mail/protocol/src/mail_protocol/network/responses.py +++ b/src/mail/protocol/src/mail_protocol/network/responses.py @@ -56,13 +56,47 @@ class AuthTokenPostResponse(BaseModel): """ Corresponds to `POST /auth/token`. Contains a temporary JWT and associated metadata. + + ``refresh_token`` is populated only for interactive principals (users and + admins); agents and daemons re-authenticate with their credentials and + receive ``None``. When present, the server also sets it as an ``httpOnly`` + cookie for browser clients. ``expires_in`` is the access-token lifetime in + seconds. """ access_token: str token_type: Literal["bearer"] + refresh_token: str | None = None + expires_in: int metadata: dict[str, Any] +class AuthRefreshPostResponse(BaseModel): + """ + Corresponds to `POST /auth/refresh`. + Contains a freshly-minted access token and a rotated refresh token. + + Mirrors `AuthTokenPostResponse`. The previous refresh token is invalidated + on every successful refresh; ``refresh_token`` carries its replacement (also + rotated in the ``httpOnly`` cookie for browser clients). + """ + + access_token: str + token_type: Literal["bearer"] + refresh_token: str | None = None + expires_in: int + metadata: dict[str, Any] + + +class AuthLogoutPostResponse(BaseModel): + """ + Corresponds to `POST /auth/logout`. + Contains a message indicating operation success. + """ + + status: Literal["success"] + + class AuthWhoamiGetResponse(BaseModel): """ Corresponds to `GET /auth/whoami`. diff --git a/src/mail/server/.env.example b/src/mail/server/.env.example index ae88234..30a280a 100644 --- a/src/mail/server/.env.example +++ b/src/mail/server/.env.example @@ -5,3 +5,16 @@ MAIL_HOST="swarms.example.com" MAIL_JWT_SECRET_KEY="0d67b4cce591d1ff298fdbc8781f721b811d0483d9892c66dd7f430d15c42492" MAIL_JWT_ALGORITHM="HS256" MAIL_JWT_EXPIRE_MINUTES=30 + +# Refresh token configuration +# Absolute lifetime (in days) of a refresh-token family; carried forward +# unchanged across rotations (the window does not slide). Required. +MAIL_REFRESH_TOKEN_EXPIRE_DAYS=30 + +# Refresh-token cookie configuration (optional) +# MAIL_COOKIE_SECURE: send the refresh cookie only over HTTPS. Defaults to +# "true"; set "false" for local http:// development. +MAIL_COOKIE_SECURE="true" +# MAIL_COOKIE_DOMAIN: optional cookie Domain for cross-subdomain deployments. +# Leave unset for a host-only cookie. +# MAIL_COOKIE_DOMAIN="example.com" diff --git a/src/mail/server/docs/reference/http.md b/src/mail/server/docs/reference/http.md index 17c8262..f96c4c8 100644 --- a/src/mail/server/docs/reference/http.md +++ b/src/mail/server/docs/reference/http.md @@ -11,7 +11,9 @@ This document serves as a reference for the MAIL (Mult-Agent Interface Layer) HT ### Authentication -- `POST /auth/token`: Log in with a valid MAIL address and password to obtain a temporary access token. +- `POST /auth/token`: Log in with a valid MAIL address and password to obtain a temporary access token (and, for users/admins, a refresh token). +- `POST /auth/refresh`: Exchange a refresh token for a new access token, rotating the refresh token. +- `POST /auth/logout`: Revoke the presented refresh token's family and clear the refresh cookie. - `GET /auth/whoami`: Obtain information on the logged-in MAIL server user-agent. - `POST /auth/password/reset`: Reset the logged-in user-agent's password. @@ -75,3 +77,36 @@ This document serves as a reference for the MAIL (Mult-Agent Interface Layer) HT - `GET /admin/webhooks/{webhook_id}`: Get an existing server webhook by ID. - `DELETE /admin/webhooks/{webhook_id}`: Delete an existing server webhook by ID. - `PATCH /admin/webhooks/{webhook_id}`: Update an existing server webhook by ID. + +## Refresh tokens + +Access tokens are short-lived JWTs sent as `Authorization: Bearer `. +Interactive principals (users and admins) additionally receive a **refresh +token** at login, which renews the access token without re-entering a password. +Agents and daemons do not receive refresh tokens — they re-authenticate with +their credentials. + +- **Delivery.** `POST /auth/token` and `POST /auth/refresh` return the refresh + token in the response body **and** set it as an `httpOnly`, `Secure`, + `SameSite=Strict` cookie scoped to `/auth`. Browsers rely on the cookie (it is + not readable by JavaScript, which mitigates XSS token theft); non-browser + clients (e.g. the CLI) send the token back in the `POST /auth/refresh` body. + The cookie takes precedence over the body when both are present. +- **Rotation & reuse detection.** Every successful refresh invalidates the + presented token and issues a replacement in the same *family*. Presenting an + already-rotated (or revoked) token is treated as theft and revokes the entire + family. +- **Expiry.** A family has an absolute lifetime + (`MAIL_REFRESH_TOKEN_EXPIRE_DAYS`) that is carried forward unchanged across + rotations — the window does not slide. +- **Revocation.** `POST /auth/logout` revokes the family; a successful + `POST /auth/password/reset` revokes **all** of the principal's families. + +### Browser silent-refresh pattern + +Keep the access token in memory and let the browser hold the refresh cookie. On +a `401` from any API call, `POST /auth/refresh` once (no body needed — the +cookie is sent automatically) and retry the original request; optionally refresh +proactively shortly before `expires_in` elapses. To avoid two tabs racing and +tripping reuse detection, coalesce concurrent refreshes into a single in-flight +request (single-flight). diff --git a/src/mail/server/docs/tutorials/quickstart.md b/src/mail/server/docs/tutorials/quickstart.md index 8b06c88..9020d64 100644 --- a/src/mail/server/docs/tutorials/quickstart.md +++ b/src/mail/server/docs/tutorials/quickstart.md @@ -18,6 +18,14 @@ The following environment variables are required for `mail-server` to run: - **Example**: `0d67b4cce591d1ff298fdbc8781f721b811d0483d9892c66dd7f430d15c42492` - `MAIL_JWT_EXPIRE_MINUTES`: The lifetime in minutes of JWTs issues by the MAIL server. - **Example**: `30` +- `MAIL_REFRESH_TOKEN_EXPIRE_DAYS`: The absolute lifetime in days of a refresh-token family. Carried forward unchanged across rotations (the window does not slide). + - **Example**: `30` + +The following environment variables are optional: +- `MAIL_COOKIE_SECURE`: Whether the refresh-token cookie is marked `Secure` (HTTPS-only). Defaults to `true`; set `false` for local `http://` development. + - **Example**: `false` +- `MAIL_COOKIE_DOMAIN`: An optional cookie `Domain` for cross-subdomain deployments. Leave unset for a host-only cookie. + - **Example**: `example.com` > [!NOTE] > Refer to `.env.example` in the `mail-server` root for an example environment variable configuration to test with. diff --git a/src/mail/server/src/mail_server/auth.py b/src/mail/server/src/mail_server/auth.py index 39a02da..e31d04a 100644 --- a/src/mail/server/src/mail_server/auth.py +++ b/src/mail/server/src/mail_server/auth.py @@ -1,14 +1,21 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2025-26 Addison Kline +import hashlib import os +import secrets from datetime import UTC, datetime, timedelta import jwt -from fastapi import HTTPException, Request +from fastapi import HTTPException, Request, Response from fastapi.security import OAuth2PasswordBearer from jwt.exceptions import InvalidTokenError -from mail_protocol.core.user_agents import MAILAdmin, MAILDaemon, MAILUserAgent +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILDaemon, + MAILUser, + MAILUserAgent, +) from pwdlib import PasswordHash from pydantic import BaseModel @@ -21,6 +28,27 @@ if ALGORITHM is None: raise RuntimeError("env var MAIL_JWT_ALGORITHM must be set") +_REFRESH_TOKEN_EXPIRE_DAYS = os.getenv("MAIL_REFRESH_TOKEN_EXPIRE_DAYS") +if _REFRESH_TOKEN_EXPIRE_DAYS is None: + raise RuntimeError("env var MAIL_REFRESH_TOKEN_EXPIRE_DAYS must be set") +REFRESH_TOKEN_EXPIRE_DAYS = int(_REFRESH_TOKEN_EXPIRE_DAYS) + +# Refresh-token cookie configuration. +# +# The cookie is scoped to ``/auth`` so it is sent to ``/auth/refresh`` and +# ``/auth/logout`` (and only those auth endpoints) — never to the wider API, +# which authenticates exclusively via the ``Authorization`` header and so stays +# CSRF-immune. ``Secure`` defaults on; set ``MAIL_COOKIE_SECURE=false`` for +# local ``http://`` development. +REFRESH_COOKIE_NAME = "mail_refresh_token" +REFRESH_COOKIE_PATH = "/auth" +COOKIE_SECURE = os.getenv("MAIL_COOKIE_SECURE", "true").lower() != "false" +COOKIE_DOMAIN = os.getenv("MAIL_COOKIE_DOMAIN") + +# High-entropy opaque refresh tokens; the ``rt_`` prefix aids on-the-wire +# identification. Stored hashed (sha256) — never in plaintext. +REFRESH_TOKEN_PREFIX = "rt_" + class Token(BaseModel): access_token: str @@ -75,6 +103,78 @@ def create_access_token(data: dict, expires_delta: timedelta | None = None): return encoded_jwt +# +# Refresh token helpers +# +def is_interactive_principal(user_agent: MAILUserAgent) -> bool: + """ + Return True for principals that get a refresh token (users and admins). + + Agents and daemons run unattended and re-authenticate with their + credentials, so they are deliberately excluded — issuing them refresh + tokens would only widen the attack surface. + """ + + return isinstance(user_agent.user_agent, MAILUser | MAILAdmin) + + +def generate_refresh_token() -> str: + """ + Generate a new high-entropy opaque refresh token (the plaintext returned to + the client). At least 256 bits of entropy. + """ + + return f"{REFRESH_TOKEN_PREFIX}{secrets.token_urlsafe(32)}" + + +def hash_refresh_token(token: str) -> str: + """ + Hash a refresh token for storage/lookup. SHA-256 is appropriate (and fast) + because the token is already high-entropy — unlike passwords, it needs no + slow KDF. + """ + + return hashlib.sha256(token.encode()).hexdigest() + + +def refresh_token_expiry() -> datetime: + """ + The absolute expiry for a refresh-token family minted now. Carried forward + unchanged on rotation (the window does not slide). + """ + + return datetime.now(UTC) + timedelta(days=REFRESH_TOKEN_EXPIRE_DAYS) + + +def set_refresh_cookie(response: Response, token: str) -> None: + """ + Set the ``httpOnly`` refresh-token cookie for browser clients. + """ + + response.set_cookie( + key=REFRESH_COOKIE_NAME, + value=token, + max_age=REFRESH_TOKEN_EXPIRE_DAYS * 24 * 3600, + path=REFRESH_COOKIE_PATH, + domain=COOKIE_DOMAIN, + secure=COOKIE_SECURE, + httponly=True, + samesite="strict", + ) + + +def clear_refresh_cookie(response: Response) -> None: + """ + Clear the refresh-token cookie (logout). + """ + + response.delete_cookie( + key=REFRESH_COOKIE_NAME, + path=REFRESH_COOKIE_PATH, + domain=COOKIE_DOMAIN, + ) + + async def validate_user_agent( backend: MAILServerBackend, request: Request ) -> MAILUserAgent: diff --git a/src/mail/server/src/mail_server/backends/base.py b/src/mail/server/src/mail_server/backends/base.py index 2e6a090..a56c0f9 100644 --- a/src/mail/server/src/mail_server/backends/base.py +++ b/src/mail/server/src/mail_server/backends/base.py @@ -12,6 +12,7 @@ from uuid import uuid4 import httpx +from mail_protocol.core.auth import RefreshTokenRecord from mail_protocol.core.drafts import MAILDraftsEntry, MAILDraftsEntrySummary from mail_protocol.core.inbox import MAILInboxEntry, MAILInboxEntrySummary from mail_protocol.core.lists import MAILListInBackend @@ -113,6 +114,75 @@ async def reset_password( pass + # + # Refresh token handlers + # + # Refresh tokens are stored hashed (never plaintext). They are grouped into + # *families*: the token minted at login starts a family, and every rotation + # keeps the same ``family_id`` while carrying the family's original + # ``expires_at`` forward (absolute cap — rotation never extends it). A token + # is unusable once ``revoked`` is True or ``rotated_at`` is set; presenting + # such a token is reuse and the caller revokes the whole family. + # + @abstractmethod + async def create_refresh_token( + self, + owner_address: str, + token_hash: str, + family_id: str, + expires_at: datetime, + ) -> None: + """ + Persist a newly-issued refresh token (stamped ``issued_at`` = now, + ``revoked`` = False, ``rotated_at`` = None). + """ + + pass + + @abstractmethod + async def get_refresh_token(self, token_hash: str) -> RefreshTokenRecord | None: + """ + Get a stored refresh token by its hash, or None if it does not exist. + """ + + pass + + @abstractmethod + async def rotate_refresh_token(self, old_hash: str, new_hash: str) -> None: + """ + Rotate a refresh token atomically: mark ``old_hash`` revoked + rotated, + and insert ``new_hash`` into the same family carrying the old token's + ``expires_at`` forward unchanged. Raises ``ValueError`` if ``old_hash`` + is not found. + """ + + pass + + @abstractmethod + async def revoke_refresh_family(self, family_id: str) -> None: + """ + Revoke every refresh token in a family (logout, or reuse detection). + """ + + pass + + @abstractmethod + async def revoke_all_refresh_tokens(self, owner_address: str) -> None: + """ + Revoke every refresh token owned by an address (e.g. on password reset). + """ + + pass + + @abstractmethod + async def purge_expired_refresh_tokens(self) -> int: + """ + Delete every refresh token whose ``expires_at`` is in the past. + Returns the number of tokens removed. + """ + + pass + # # Swarm endpoint handlers # diff --git a/src/mail/server/src/mail_server/backends/memory/api.py b/src/mail/server/src/mail_server/backends/memory/api.py index 3c441cd..c258fc6 100644 --- a/src/mail/server/src/mail_server/backends/memory/api.py +++ b/src/mail/server/src/mail_server/backends/memory/api.py @@ -10,6 +10,7 @@ from datetime import UTC, datetime from typing import Any +from mail_protocol.core.auth import RefreshTokenRecord from mail_protocol.core.constants import LIST_ADDRESS_PREFIX from mail_protocol.core.drafts import MAILDraft, MAILDraftsEntry, MAILDraftsEntrySummary from mail_protocol.core.inbox import MAILInboxEntry, MAILInboxEntrySummary @@ -57,6 +58,7 @@ load_messages, load_outbox_entries, load_outboxes, + load_refresh_tokens, load_swarms, load_trash_entries, load_trashes, @@ -71,6 +73,7 @@ save_messages, save_outbox_entries, save_outboxes, + save_refresh_tokens, save_swarms, save_trash_entries, save_trashes, @@ -150,6 +153,7 @@ def _snapshot_persistence_state(self) -> dict[str, Any]: "message_buffer": list(self.message_buffer), "webhooks": dict(self.webhooks), "lists": dict(self.lists), + "refresh_tokens": dict(self.refresh_tokens), } async def persist(self, *, reason: str = "manual") -> None: @@ -176,6 +180,7 @@ async def persist(self, *, reason: str = "manual") -> None: await save_message_buffer(snapshot["message_buffer"]) await save_webhooks(snapshot["webhooks"]) await save_lists(snapshot["lists"]) + await save_refresh_tokens(snapshot["refresh_tokens"]) elapsed = time.monotonic() - started_at logger.info( @@ -348,6 +353,15 @@ async def on_server_startup(self, **kwargs: Any) -> None: Values: MAILListInBackend instances """ + self.refresh_tokens: dict[str, RefreshTokenRecord] = ( + await load_refresh_tokens() + ) + """ + A dict of all stored refresh tokens on this server. + Keys: token hashes (sha256 hex) + Values: RefreshTokenRecord instances + """ + host = kwargs.get("host") if host is not None: if isinstance(host, str): @@ -411,6 +425,91 @@ async def reset_password( return "success" + # + # Refresh token handlers + # + async def create_refresh_token( + self, + owner_address: str, + token_hash: str, + family_id: str, + expires_at: datetime, + ) -> None: + """ + Persist a newly-issued refresh token. + """ + + self.refresh_tokens[token_hash] = RefreshTokenRecord( + token_hash=token_hash, + family_id=family_id, + owner_address=owner_address, + issued_at=datetime.now(UTC), + expires_at=expires_at, + ) + + async def get_refresh_token(self, token_hash: str) -> RefreshTokenRecord | None: + """ + Get a stored refresh token by its hash, or None if it does not exist. + """ + + return self.refresh_tokens.get(token_hash) + + async def rotate_refresh_token(self, old_hash: str, new_hash: str) -> None: + """ + Rotate a refresh token: revoke the old one and mint a replacement in the + same family carrying the old token's ``expires_at`` forward. + """ + + old = self.refresh_tokens.get(old_hash) + if old is None: + raise ValueError(f"refresh token {old_hash} not found") + + now = datetime.now(UTC) + old.revoked = True + old.rotated_at = now + self.refresh_tokens[old_hash] = old + + self.refresh_tokens[new_hash] = RefreshTokenRecord( + token_hash=new_hash, + family_id=old.family_id, + owner_address=old.owner_address, + issued_at=now, + expires_at=old.expires_at, + ) + + async def revoke_refresh_family(self, family_id: str) -> None: + """ + Revoke every refresh token in a family. + """ + + for record in self.refresh_tokens.values(): + if record.family_id == family_id: + record.revoked = True + + async def revoke_all_refresh_tokens(self, owner_address: str) -> None: + """ + Revoke every refresh token owned by an address. + """ + + for record in self.refresh_tokens.values(): + if record.owner_address == owner_address: + record.revoked = True + + async def purge_expired_refresh_tokens(self) -> int: + """ + Delete every refresh token whose ``expires_at`` is in the past. + """ + + now = datetime.now(UTC) + expired = [ + token_hash + for token_hash, record in self.refresh_tokens.items() + if record.expires_at < now + ] + for token_hash in expired: + del self.refresh_tokens[token_hash] + return len(expired) + # # Swarm endpoint handlers # diff --git a/src/mail/server/src/mail_server/backends/memory/fs.py b/src/mail/server/src/mail_server/backends/memory/fs.py index 3911892..b98ff49 100644 --- a/src/mail/server/src/mail_server/backends/memory/fs.py +++ b/src/mail/server/src/mail_server/backends/memory/fs.py @@ -7,6 +7,7 @@ from os import scandir from pathlib import Path +from mail_protocol.core.auth import RefreshTokenRecord from mail_protocol.core.drafts import MAILDraftsEntry from mail_protocol.core.inbox import MAILInboxEntrySummary from mail_protocol.core.lists import MAILListInBackend @@ -584,6 +585,37 @@ async def load_webhooks() -> dict[str, MAILWebhook]: return webhooks +async def load_refresh_tokens() -> dict[str, RefreshTokenRecord]: + """ + Load saved refresh tokens from the local filesystem. + + The directory is created if absent so memory deployments provisioned before + refresh-token support start cleanly. Each file is named by its token hash + (sha256 hex) and holds the serialized ``RefreshTokenRecord``. + """ + + refresh_tokens_path = DEPLOYMENT_PATH.joinpath("refresh_tokens") + refresh_tokens_path.mkdir(parents=True, exist_ok=True) + logger.info(f"loading refresh_tokens: {refresh_tokens_path}...") + refresh_tokens: dict[str, RefreshTokenRecord] = {} + with scandir(refresh_tokens_path) as entries: + for entry in entries: + if entry.is_file(): + with open(entry) as rt_file: + content = rt_file.read() + try: + rt_model = RefreshTokenRecord.model_validate_json(content) + except Exception as e: + logger.warning(f"RefreshTokenRecord validation failed: {e}") + continue + + refresh_tokens.update({rt_model.token_hash: rt_model}) + + logger.info(f"found {len(refresh_tokens)} refresh_tokens") + + return refresh_tokens + + # # Save memory backend to the local filesystem # (on server shutdown and periodic checkpoints) @@ -805,3 +837,22 @@ async def save_webhooks(webhooks: dict[str, MAILWebhook]) -> None: for webhook in webhooks.values() }, ) + + +async def save_refresh_tokens( + refresh_tokens: dict[str, RefreshTokenRecord], +) -> None: + """ + Save refresh tokens from memory to the local filesystem, one file per token + keyed by its hash. + """ + + logger.info(f"saving {len(refresh_tokens)} refresh_tokens...") + + _save_directory_snapshot( + DEPLOYMENT_PATH.joinpath("refresh_tokens"), + { + token_hash: record.model_dump_json() + for token_hash, record in refresh_tokens.items() + }, + ) diff --git a/src/mail/server/src/mail_server/backends/memory/init.py b/src/mail/server/src/mail_server/backends/memory/init.py index df25fc1..5b60dce 100644 --- a/src/mail/server/src/mail_server/backends/memory/init.py +++ b/src/mail/server/src/mail_server/backends/memory/init.py @@ -128,6 +128,11 @@ def init_memory_backend( LISTS_PATH.mkdir(exist_ok=True) print(f"ensured deployment lists: {LISTS_PATH}") + # ~/.mail-swarms/deployments/{deployment}/refresh_tokens + REFRESH_TOKENS_PATH = DEPLOYMENT_PATH.joinpath("refresh_tokens") + REFRESH_TOKENS_PATH.mkdir(exist_ok=True) + print(f"ensured deployment refresh_tokens: {REFRESH_TOKENS_PATH}") + # write swarm file SWARM_PATH = SWARMS_PATH.joinpath(swarm) with open(SWARM_PATH, "w") as swarm_file: diff --git a/src/mail/server/src/mail_server/backends/sqlite/api.py b/src/mail/server/src/mail_server/backends/sqlite/api.py index cefe3c3..88c0411 100644 --- a/src/mail/server/src/mail_server/backends/sqlite/api.py +++ b/src/mail/server/src/mail_server/backends/sqlite/api.py @@ -36,6 +36,7 @@ from datetime import UTC, datetime from typing import Any, NamedTuple +from mail_protocol.core.auth import RefreshTokenRecord from mail_protocol.core.constants import LIST_ADDRESS_PREFIX from mail_protocol.core.drafts import MAILDraft, MAILDraftsEntry, MAILDraftsEntrySummary from mail_protocol.core.inbox import MAILInboxEntry, MAILInboxEntrySummary @@ -169,6 +170,63 @@ async def reset_password( ) return "success" + # + # Refresh token handlers + # + async def create_refresh_token( + self, + owner_address: str, + token_hash: str, + family_id: str, + expires_at: datetime, + ) -> None: + record = RefreshTokenRecord( + token_hash=token_hash, + family_id=family_id, + owner_address=owner_address, + issued_at=datetime.now(UTC), + expires_at=expires_at, + ) + async with self._db.session() as session: + await MailStore(session).refresh_tokens.add(record) + + async def get_refresh_token(self, token_hash: str) -> RefreshTokenRecord | None: + async with self._db.session() as session: + return await MailStore(session).refresh_tokens.get(token_hash) + + async def rotate_refresh_token(self, old_hash: str, new_hash: str) -> None: + async with self._db.session() as session: + store = MailStore(session) + old = await store.refresh_tokens.get(old_hash) + if old is None: + raise ValueError(f"refresh token {old_hash} not found") + now = datetime.now(UTC) + # Carry the family's original ``expires_at`` forward unchanged — the + # absolute cap does not slide on rotation. + new_record = RefreshTokenRecord( + token_hash=new_hash, + family_id=old.family_id, + owner_address=old.owner_address, + issued_at=now, + expires_at=old.expires_at, + ) + await store.refresh_tokens.mark_rotated(old_hash, now) + await store.refresh_tokens.add(new_record) + + async def revoke_refresh_family(self, family_id: str) -> None: + async with self._db.session() as session: + await MailStore(session).refresh_tokens.revoke_family(family_id) + + async def revoke_all_refresh_tokens(self, owner_address: str) -> None: + async with self._db.session() as session: + await MailStore(session).refresh_tokens.revoke_for_owner(owner_address) + + async def purge_expired_refresh_tokens(self) -> int: + async with self._db.session() as session: + return await MailStore(session).refresh_tokens.purge_expired( + datetime.now(UTC) + ) + # # Swarm endpoint handlers # diff --git a/src/mail/server/src/mail_server/backends/sqlite/repositories.py b/src/mail/server/src/mail_server/backends/sqlite/repositories.py index e5d2df7..dc2e8cc 100644 --- a/src/mail/server/src/mail_server/backends/sqlite/repositories.py +++ b/src/mail/server/src/mail_server/backends/sqlite/repositories.py @@ -27,6 +27,7 @@ from datetime import datetime from typing import Any +from mail_protocol.core.auth import RefreshTokenRecord from mail_protocol.core.drafts import MAILDraftsEntry, MAILDraftsEntrySummary from mail_protocol.core.inbox import MAILInboxEntrySummary from mail_protocol.core.lists import MAILListInBackend @@ -37,7 +38,7 @@ from mail_protocol.core.user_agents import MAILUserAgentInBackend from mail_protocol.core.webhooks import MAILWebhook from mail_protocol.network.requests import BoxFilterParams -from sqlalchemy import asc, delete, func, select +from sqlalchemy import asc, delete, func, select, update from sqlalchemy.ext.asyncio import AsyncSession from mail_server.backends.sqlite import serializers as ser @@ -49,6 +50,7 @@ MessageBufferRow, MessageRow, OutboxEntryRow, + RefreshTokenRow, SwarmRow, TrashEntryRow, UserAgentRow, @@ -96,6 +98,10 @@ def webhooks(self) -> WebhookRepository: def lists(self) -> ListRepository: return ListRepository(self.session) + @property + def refresh_tokens(self) -> RefreshTokenRepository: + return RefreshTokenRepository(self.session) + # --------------------------------------------------------------------------- # # user_agents @@ -629,3 +635,57 @@ async def delete(self, address: str) -> MAILListInBackend | None: await self.session.delete(row) await self.session.flush() return model + + +# --------------------------------------------------------------------------- # +# refresh_tokens (keyed by hash; no JSON body) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class RefreshTokenRepository: + session: AsyncSession + + async def get(self, token_hash: str) -> RefreshTokenRecord | None: + row = await self.session.get(RefreshTokenRow, token_hash) + if row is None: + return None + return ser.refresh_token_from_row(row) + + async def add(self, model: RefreshTokenRecord) -> RefreshTokenRecord: + self.session.add(RefreshTokenRow(**ser.refresh_token_to_columns(model))) + await self.session.flush() + return model + + async def mark_rotated(self, token_hash: str, rotated_at: datetime) -> None: + """Mark a token as revoked + rotated (the old half of a rotation).""" + + await self.session.execute( + update(RefreshTokenRow) + .where(RefreshTokenRow.token_hash == token_hash) + .values(revoked=True, rotated_at=rotated_at) + ) + await self.session.flush() + + async def revoke_family(self, family_id: str) -> None: + await self.session.execute( + update(RefreshTokenRow) + .where(RefreshTokenRow.family_id == family_id) + .values(revoked=True) + ) + await self.session.flush() + + async def revoke_for_owner(self, owner_address: str) -> None: + await self.session.execute( + update(RefreshTokenRow) + .where(RefreshTokenRow.owner_address == owner_address) + .values(revoked=True) + ) + await self.session.flush() + + async def purge_expired(self, now: datetime) -> int: + result = await self.session.execute( + delete(RefreshTokenRow).where(RefreshTokenRow.expires_at < now) + ) + await self.session.flush() + return result.rowcount or 0 diff --git a/src/mail/server/src/mail_server/backends/sqlite/schema.py b/src/mail/server/src/mail_server/backends/sqlite/schema.py index 9c469fc..5d7212f 100644 --- a/src/mail/server/src/mail_server/backends/sqlite/schema.py +++ b/src/mail/server/src/mail_server/backends/sqlite/schema.py @@ -230,6 +230,37 @@ class WebhookRow(Base): ) +class RefreshTokenRow(Base): + """ + A stored refresh token, keyed by its hash. + + Unlike the entity rows, this table has **no** ``body`` JSON column: the + ``RefreshTokenRecord`` model is tiny and every field is queried directly, so + all columns are typed (mirroring ``mailbox_items`` / ``message_buffer``). + ``owner_address`` cascades on user-agent deletion, so removing a principal + drops their refresh tokens for free. + """ + + __tablename__ = "refresh_tokens" + + # sha256 hex of the plaintext token. + token_hash: Mapped[str] = mapped_column(String(64), primary_key=True) + family_id: Mapped[str] = mapped_column(String(64), index=True) + owner_address: Mapped[str] = mapped_column( + String(512), + ForeignKey("user_agents.address", ondelete="CASCADE"), + index=True, + ) + issued_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now + ) + expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + revoked: Mapped[bool] = mapped_column(default=False) + rotated_at: Mapped[datetime | None] = mapped_column( + DateTime(timezone=True), nullable=True + ) + + class ListRow(Base): """A MAIL list. Members live inside ``body``, mirroring the memory backend.""" diff --git a/src/mail/server/src/mail_server/backends/sqlite/serializers.py b/src/mail/server/src/mail_server/backends/sqlite/serializers.py index 30bffea..18e5d19 100644 --- a/src/mail/server/src/mail_server/backends/sqlite/serializers.py +++ b/src/mail/server/src/mail_server/backends/sqlite/serializers.py @@ -28,8 +28,10 @@ from __future__ import annotations +from datetime import UTC, datetime from typing import Any +from mail_protocol.core.auth import RefreshTokenRecord from mail_protocol.core.drafts import MAILDraftsEntry from mail_protocol.core.inbox import MAILInboxEntrySummary from mail_protocol.core.lists import MAILListInBackend @@ -46,6 +48,7 @@ ListRow, MessageRow, OutboxEntryRow, + RefreshTokenRow, SwarmRow, TrashEntryRow, UserAgentRow, @@ -221,3 +224,48 @@ def list_to_columns(model: MAILListInBackend) -> dict[str, Any]: def list_from_row(row: ListRow) -> MAILListInBackend: return MAILListInBackend.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# refresh_tokens (no body column — every field is typed and queried) +# --------------------------------------------------------------------------- # + + +def refresh_token_to_columns(model: RefreshTokenRecord) -> dict[str, Any]: + return { + "token_hash": model.token_hash, + "family_id": model.family_id, + "owner_address": model.owner_address, + "issued_at": model.issued_at, + "expires_at": model.expires_at, + "revoked": model.revoked, + "rotated_at": model.rotated_at, + } + + +def _as_utc(value: datetime | None) -> datetime | None: + """ + Re-attach UTC to a datetime read back from SQLite. + + SQLite has no native datetime type, so ``DateTime(timezone=True)`` round-trips + as a tz-*naive* value even though we always persist UTC. The entity tables + avoid this by rehydrating from the JSON ``body`` (ISO strings keep the + offset); ``refresh_tokens`` has no body, so we normalize here to keep the + backend contract (tz-aware UTC) identical to the memory backend. + """ + + if value is not None and value.tzinfo is None: + return value.replace(tzinfo=UTC) + return value + + +def refresh_token_from_row(row: RefreshTokenRow) -> RefreshTokenRecord: + return RefreshTokenRecord( + token_hash=row.token_hash, + family_id=row.family_id, + owner_address=row.owner_address, + issued_at=_as_utc(row.issued_at), # type: ignore[arg-type] + expires_at=_as_utc(row.expires_at), # type: ignore[arg-type] + revoked=row.revoked, + rotated_at=_as_utc(row.rotated_at), + ) diff --git a/src/mail/server/src/mail_server/routers/auth.py b/src/mail/server/src/mail_server/routers/auth.py index 3c13828..e77ce98 100644 --- a/src/mail/server/src/mail_server/routers/auth.py +++ b/src/mail/server/src/mail_server/routers/auth.py @@ -2,23 +2,36 @@ # Copyright (c) 2026 Addison Kline import os -from datetime import timedelta +from datetime import UTC, datetime, timedelta from typing import Annotated +from uuid import uuid4 -from fastapi import APIRouter, Depends, HTTPException, Request +from fastapi import APIRouter, Depends, HTTPException, Request, Response from fastapi.security.oauth2 import OAuth2PasswordRequestForm from mail_protocol.network.responses import ( + AuthLogoutPostResponse, AuthPasswordResetResponse, + AuthRefreshPostResponse, AuthTokenPostResponse, AuthWhoamiGetResponse, ) from mail_server.auth import ( + REFRESH_COOKIE_NAME, authenticate_user_agent, + clear_refresh_cookie, create_access_token, + generate_refresh_token, + hash_refresh_token, + is_interactive_principal, + refresh_token_expiry, + set_refresh_cookie, validate_user_agent, ) -from mail_server.validators import validate_auth_password_reset_request +from mail_server.validators import ( + validate_auth_password_reset_request, + validate_auth_refresh_request, +) ACCESS_TOKEN_EXPIRE_MINUTES = os.getenv("MAIL_JWT_EXPIRE_MINUTES") if ACCESS_TOKEN_EXPIRE_MINUTES is None: @@ -35,6 +48,7 @@ ) async def create_auth_token( request: Request, + response: Response, form_data: Annotated[OAuth2PasswordRequestForm, Depends()], ) -> AuthTokenPostResponse: backend = request.app.state.backend @@ -51,13 +65,126 @@ async def create_auth_token( access_token = create_access_token( data={"sub": user_agent.get_address()}, expires_delta=access_token_expires ) + + # Interactive principals (users/admins) also get a refresh token, set as an + # httpOnly cookie for browsers and returned in the body for the CLI. Agents + # and daemons re-authenticate with their credentials and get none. + refresh_token: str | None = None + if is_interactive_principal(user_agent): + refresh_token = generate_refresh_token() + await backend.create_refresh_token( + owner_address=user_agent.get_address(), + token_hash=hash_refresh_token(refresh_token), + family_id=f"fam_{uuid4()}", + expires_at=refresh_token_expiry(), + ) + set_refresh_cookie(response, refresh_token) + return AuthTokenPostResponse( access_token=access_token, token_type="bearer", + refresh_token=refresh_token, + expires_in=default_token_limit * 60, metadata={}, ) +async def _read_refresh_token(request: Request) -> str | None: + """ + Extract the presented refresh token: the cookie (browsers) takes precedence, + falling back to the request body (CLI / non-cookie clients). + """ + + token = request.cookies.get(REFRESH_COOKIE_NAME) + if token is not None: + return token + payload = await validate_auth_refresh_request(request=request) + return payload.refresh_token + + +@router.post( + "/refresh", + summary="Exchange a refresh token for a new access token (rotates the refresh token)", + response_model=AuthRefreshPostResponse, +) +async def post_auth_refresh( + request: Request, + response: Response, +) -> AuthRefreshPostResponse: + backend = request.app.state.backend + credentials_exception = HTTPException( + status_code=401, + detail="could not validate refresh token", + headers={"WWW-Authenticate": "Bearer"}, + ) + + token = await _read_refresh_token(request) + if token is None: + raise credentials_exception + + record = await backend.get_refresh_token(hash_refresh_token(token)) + if record is None: + raise credentials_exception + + # Absolute cap: a family expires at a fixed time, carried forward unchanged + # across rotations. + if record.expires_at <= datetime.now(UTC): + raise credentials_exception + + # Reuse detection: a revoked or already-rotated token being presented means + # the token was stolen (or replayed) — revoke the whole family. + if record.revoked or record.rotated_at is not None: + await backend.revoke_refresh_family(record.family_id) + raise credentials_exception + + # Fail closed if the owner no longer exists (e.g. deleted by an admin). + if not await backend.user_agent_exists(record.owner_address): + await backend.revoke_refresh_family(record.family_id) + raise credentials_exception + + new_refresh_token = generate_refresh_token() + await backend.rotate_refresh_token( + hash_refresh_token(token), hash_refresh_token(new_refresh_token) + ) + + access_token = create_access_token( + data={"sub": record.owner_address}, + expires_delta=timedelta(minutes=default_token_limit), # type: ignore + ) + set_refresh_cookie(response, new_refresh_token) + + return AuthRefreshPostResponse( + access_token=access_token, + token_type="bearer", + refresh_token=new_refresh_token, + expires_in=default_token_limit * 60, + metadata={}, + ) + + +@router.post( + "/logout", + summary="Revoke the presented refresh token's family and clear the cookie", + response_model=AuthLogoutPostResponse, +) +async def post_auth_logout( + request: Request, + response: Response, +) -> AuthLogoutPostResponse: + backend = request.app.state.backend + + # Idempotent: revoke the family if the token resolves, but always succeed and + # clear the cookie so a stale/absent token still logs the client out. + token = await _read_refresh_token(request) + if token is not None: + record = await backend.get_refresh_token(hash_refresh_token(token)) + if record is not None: + await backend.revoke_refresh_family(record.family_id) + + clear_refresh_cookie(response) + return AuthLogoutPostResponse(status="success") + + @router.get( "/whoami", summary="Get MAIL user-agent info", response_model=AuthWhoamiGetResponse ) @@ -89,6 +216,11 @@ async def post_password_reset(request: Request) -> AuthPasswordResetResponse: ) if result != "success": raise HTTPException(status_code=400, detail="could not reset password") + + # A password change invalidates every existing session for this principal — + # revoke all of their refresh-token families, forcing re-login everywhere. + await backend.revoke_all_refresh_tokens(user_agent.get_address()) + return AuthPasswordResetResponse( status=result, ) diff --git a/src/mail/server/src/mail_server/validators.py b/src/mail/server/src/mail_server/validators.py index bbaafea..9a7c6e6 100644 --- a/src/mail/server/src/mail_server/validators.py +++ b/src/mail/server/src/mail_server/validators.py @@ -20,6 +20,7 @@ AdminWebhooksPatchRequest, AdminWebhooksPostRequest, AuthPasswordResetRequest, + AuthRefreshPostRequest, BoxFilterParams, DaemonDeliverLocalRequest, DaemonDeliverRemoteRequest, @@ -390,3 +391,25 @@ async def validate_auth_password_reset_request( raise HTTPException( status_code=422, detail=f"request body validation failed: {e}" ) + + +async def validate_auth_refresh_request( + request: Request, +) -> AuthRefreshPostRequest: + """ + Ensure that the request payload is valid for `POST /auth/refresh`. + + An empty body is allowed: browsers carry the refresh token in the + ``httpOnly`` cookie and may send no body at all. A non-empty body must still + be valid JSON for the model, otherwise 422. + """ + + raw = await request.body() + if not raw: + return AuthRefreshPostRequest() + try: + return AuthRefreshPostRequest.model_validate_json(raw) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"request body validation failed: {e}" + ) diff --git a/tests/conftest.py b/tests/conftest.py index ab76519..c445925 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -10,6 +10,11 @@ # mail_server.* imports succeed everywhere. os.environ.setdefault("MAIL_JWT_SECRET_KEY", "test-secret-not-used") os.environ.setdefault("MAIL_JWT_ALGORITHM", "HS256") +os.environ.setdefault("MAIL_REFRESH_TOKEN_EXPIRE_DAYS", "30") +# The TestClient speaks http://testserver; a ``Secure`` cookie would never be +# sent back over http, so disable the flag for the in-process suites. The +# secure-by-default behavior is covered by a dedicated unit test. +os.environ.setdefault("MAIL_COOKIE_SECURE", "false") import pytest # noqa: E402 from mail_server.backends.memory import fs as memory_fs # noqa: E402 @@ -61,6 +66,7 @@ def deployment_dir( "trashes", "webhooks", "lists", + "refresh_tokens", ): (deployment / subdir).mkdir(parents=True, exist_ok=True) (deployment / "message_buffer.lock").touch() diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py index 2d6d573..0ba0aff 100644 --- a/tests/e2e/conftest.py +++ b/tests/e2e/conftest.py @@ -52,6 +52,7 @@ def __init__(self, home: Path) -> None: "MAIL_JWT_SECRET_KEY": "e2e-secret", "MAIL_JWT_ALGORITHM": "HS256", "MAIL_JWT_EXPIRE_MINUTES": "15", + "MAIL_REFRESH_TOKEN_EXPIRE_DAYS": "30", } self.server: subprocess.Popen | None = None self.credentials: dict[str, str] = {} diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 37baa57..85dcb88 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -4,7 +4,7 @@ import asyncio import os from collections.abc import Awaitable, Callable, Iterator -from datetime import datetime +from datetime import UTC, datetime from pathlib import Path # mail_server.server reads MAIL_HOST and mail_server.routers.auth reads @@ -15,6 +15,7 @@ import pytest # noqa: E402 from fastapi.testclient import TestClient # noqa: E402 +from mail_protocol.core.auth import RefreshTokenRecord # noqa: E402 from mail_protocol.core.lists import MAILListInBackend # noqa: E402 from mail_protocol.core.messages import MAILMessage # noqa: E402 from mail_protocol.core.swarms import MAILSwarm # noqa: E402 @@ -27,7 +28,7 @@ MAILUserAgentInBackend, ) from mail_server import server as mail_server_module # noqa: E402 -from mail_server.auth import get_password_hash # noqa: E402 +from mail_server.auth import get_password_hash, hash_refresh_token # noqa: E402 from mail_server.backends.base import MAILServerBackend # noqa: E402 from mail_server.backends.memory.api import MemoryBackend # noqa: E402 from mail_server.backends.sqlite.api import SQLiteBackend # noqa: E402 @@ -232,6 +233,42 @@ async def mutate(store: MailStore) -> None: return _seed +@pytest.fixture +def seed_refresh_token(backend: MAILServerBackend) -> Callable[..., str]: + """ + Backend-agnostic: persist a refresh token directly so a test can pin its + ``expires_at`` / ``family_id`` (impossible through the login API, which + always stamps the configured absolute cap). Returns the plaintext token. + """ + + def _seed( + owner: str, + token: str, + *, + expires_at: datetime, + family_id: str = "fam_seed", + ) -> str: + record = RefreshTokenRecord( + token_hash=hash_refresh_token(token), + family_id=family_id, + owner_address=owner, + issued_at=datetime.now(UTC), + expires_at=expires_at, + ) + if isinstance(backend, MemoryBackend): + backend.refresh_tokens[record.token_hash] = record + else: + assert isinstance(backend, SQLiteBackend) + + async def mutate(store: MailStore) -> None: + await store.refresh_tokens.add(record) + + _run_sqlite_write(backend._db.url, mutate) + return token + + return _seed + + @pytest.fixture def list_members( app_client: TestClient, headers_for: Callable[..., dict[str, str]] diff --git a/tests/integration/test_refresh_flow.py b/tests/integration/test_refresh_flow.py new file mode 100644 index 0000000..bdd293a --- /dev/null +++ b/tests/integration/test_refresh_flow.py @@ -0,0 +1,184 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +HTTP-level tests for the refresh-token flow (``POST /auth/token`` issuance, +``POST /auth/refresh`` rotation + reuse detection, ``POST /auth/logout``, and +the password-reset cascade). Every test runs against both backends via the +parametrized ``app_client`` fixture. + +The ``Secure`` cookie flag is disabled for the suite (see ``tests/conftest.py``) +so the TestClient replays the cookie over http; ``test_login_cookie_attributes`` +asserts the remaining hardening attributes. +""" + +from collections.abc import Callable +from datetime import UTC, datetime, timedelta + +import pytest +from fastapi.testclient import TestClient + +ADMIN = "admin:ryan@localhost" +USER = "user:alice@localhost" +AGENT = "sage@chorus@localhost" +DAEMON = "daemon:dummy@localhost" +PASSWORD = "correct-horse-battery-staple" + + +def _login(client: TestClient, address: str, password: str = PASSWORD): + return client.post( + "/auth/token", data={"username": address, "password": password} + ) + + +def _refresh_body(client: TestClient, token: str): + """Refresh via the body path — cookies cleared so the cookie can't win.""" + + client.cookies.clear() + return client.post("/auth/refresh", json={"refresh_token": token}) + + +# ─── issuance ────────────────────────────────────────────────────── + + +@pytest.mark.parametrize("address", [USER, ADMIN]) +def test_login_issues_refresh_token_for_interactive( + app_client: TestClient, address: str +) -> None: + app_client.cookies.clear() + resp = _login(app_client, address) + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["refresh_token"] is not None + assert body["refresh_token"].startswith("rt_") + assert body["expires_in"] > 0 + assert "mail_refresh_token=" in (resp.headers.get("set-cookie") or "") + + +@pytest.mark.parametrize("address", [AGENT, DAEMON]) +def test_login_no_refresh_token_for_non_interactive( + app_client: TestClient, address: str +) -> None: + app_client.cookies.clear() + resp = _login(app_client, address) + assert resp.status_code == 200, resp.text + assert resp.json()["refresh_token"] is None + assert resp.headers.get("set-cookie") is None + + +def test_login_cookie_attributes(app_client: TestClient) -> None: + app_client.cookies.clear() + resp = _login(app_client, USER) + set_cookie = (resp.headers.get("set-cookie") or "").lower() + assert "httponly" in set_cookie + assert "samesite=strict" in set_cookie + assert "path=/auth" in set_cookie + + +# ─── refresh / rotation ──────────────────────────────────────────── + + +def test_refresh_via_cookie_rotates(app_client: TestClient) -> None: + app_client.cookies.clear() + old = _login(app_client, USER).json()["refresh_token"] + + # cookie from login is replayed automatically + resp = app_client.post("/auth/refresh") + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["access_token"] + assert body["refresh_token"] and body["refresh_token"] != old + + # the consumed token is now dead (reuse → 401) + assert _refresh_body(app_client, old).status_code == 401 + + +def test_refresh_via_body(app_client: TestClient) -> None: + app_client.cookies.clear() + token = _login(app_client, USER).json()["refresh_token"] + + resp = _refresh_body(app_client, token) + assert resp.status_code == 200, resp.text + assert resp.json()["refresh_token"] != token + + +def test_reuse_revokes_whole_family(app_client: TestClient) -> None: + app_client.cookies.clear() + t0 = _login(app_client, USER).json()["refresh_token"] + + t1 = _refresh_body(app_client, t0).json()["refresh_token"] + + # replaying the already-rotated t0 is reuse → 401 and nukes the family + assert _refresh_body(app_client, t0).status_code == 401 + # the live sibling t1 is now collateral-revoked + assert _refresh_body(app_client, t1).status_code == 401 + + +def test_refresh_unknown_token_401(app_client: TestClient) -> None: + assert _refresh_body(app_client, "rt_does-not-exist").status_code == 401 + + +def test_refresh_without_token_401(app_client: TestClient) -> None: + app_client.cookies.clear() + assert app_client.post("/auth/refresh").status_code == 401 + + +def test_refresh_expired_token_401( + app_client: TestClient, seed_refresh_token: Callable[..., str] +) -> None: + token = seed_refresh_token( + USER, "rt_expired", expires_at=datetime.now(UTC) - timedelta(seconds=1) + ) + assert _refresh_body(app_client, token).status_code == 401 + + +def test_refresh_after_owner_deleted_401( + app_client: TestClient, headers_for: Callable[..., dict[str, str]] +) -> None: + app_client.cookies.clear() + token = _login(app_client, USER).json()["refresh_token"] + + deleted = app_client.delete("/admin/users/alice", headers=headers_for(ADMIN)) + assert deleted.status_code == 200, deleted.text + + assert _refresh_body(app_client, token).status_code == 401 + + +# ─── logout ──────────────────────────────────────────────────────── + + +def test_logout_revokes_family_and_succeeds(app_client: TestClient) -> None: + app_client.cookies.clear() + token = _login(app_client, USER).json()["refresh_token"] + + out = app_client.post("/auth/logout") + assert out.status_code == 200, out.text + assert out.json()["status"] == "success" + + assert _refresh_body(app_client, token).status_code == 401 + + +def test_logout_without_token_is_idempotent(app_client: TestClient) -> None: + app_client.cookies.clear() + out = app_client.post("/auth/logout") + assert out.status_code == 200 + assert out.json()["status"] == "success" + + +# ─── password reset cascade ──────────────────────────────────────── + + +def test_password_reset_revokes_refresh_tokens( + app_client: TestClient, headers_for: Callable[..., dict[str, str]] +) -> None: + app_client.cookies.clear() + token = _login(app_client, USER).json()["refresh_token"] + + reset = app_client.post( + "/auth/password/reset", + json={"current_password": PASSWORD, "new_password": "new-passw0rd-here"}, + headers=headers_for(USER), + ) + assert reset.status_code == 200, reset.text + + assert _refresh_body(app_client, token).status_code == 401 diff --git a/tests/integration/test_refresh_tokens_backend.py b/tests/integration/test_refresh_tokens_backend.py new file mode 100644 index 0000000..4459a23 --- /dev/null +++ b/tests/integration/test_refresh_tokens_backend.py @@ -0,0 +1,187 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Conformance tests for the refresh-token backend protocol methods. + +Each test runs against **both** the memory and sqlite backends via the +``rt_backend`` fixture, so the two implementations are held to identical +behavior (create / get / rotate / revoke-family / revoke-for-owner / purge). +Unlike the rest of the integration suite these exercise the backend protocol +directly rather than the FastAPI app — the HTTP-level refresh flow is covered in +the auth router tests. +""" + +import hashlib +from collections.abc import AsyncIterator +from datetime import UTC, datetime, timedelta +from pathlib import Path + +import pytest +from mail_protocol.core.user_agents import MAILUser, MAILUserAgentInBackend +from mail_server.auth import get_password_hash +from mail_server.backends.base import MAILServerBackend +from mail_server.backends.memory.api import MemoryBackend +from mail_server.backends.sqlite.api import SQLiteBackend +from mail_server.backends.sqlite.repositories import MailStore + +HOST = "localhost" +OWNER = f"user:alice@{HOST}" +OTHER = f"user:bob@{HOST}" + +# Argon2 is deliberately slow; hash the throwaway password once. +_PWHASH = get_password_hash("pw") + + +def _user(user_id: str) -> MAILUserAgentInBackend: + return MAILUserAgentInBackend( + user_agent=MAILUser(ua_type="user", user_id=user_id, host=HOST), + hashed_password=_PWHASH, + ) + + +def _h(label: str) -> str: + """A deterministic, validly-shaped (64 hex) stand-in token hash.""" + + return hashlib.sha256(label.encode()).hexdigest() + + +def _exp(*, days: int = 30) -> datetime: + return datetime.now(UTC) + timedelta(days=days) + + +@pytest.fixture(params=["memory", "sqlite"]) +async def rt_backend( + request: pytest.FixtureRequest, + deployment_dir: Path, + tmp_path: Path, +) -> AsyncIterator[MAILServerBackend]: + """A started backend seeded with two users (FK owners for refresh tokens).""" + + if request.param == "memory": + backend: MAILServerBackend = MemoryBackend() + await backend.on_server_startup(host=HOST) + backend.user_agents[OWNER] = _user("alice") # type: ignore[attr-defined] + backend.user_agents[OTHER] = _user("bob") # type: ignore[attr-defined] + try: + yield backend + finally: + await backend.on_server_shutdown() + return + + db_url = f"sqlite:///{tmp_path / 'mail.db'}" + sqlite_backend = SQLiteBackend(url=db_url) + await sqlite_backend.on_server_startup(host=HOST) + async with sqlite_backend._db.session() as session: + store = MailStore(session) + await store.user_agents.add(_user("alice")) + await store.user_agents.add(_user("bob")) + try: + yield sqlite_backend + finally: + await sqlite_backend.on_server_shutdown() + + +async def test_create_and_get(rt_backend: MAILServerBackend) -> None: + exp = _exp() + await rt_backend.create_refresh_token(OWNER, _h("t1"), "fam1", exp) + + rec = await rt_backend.get_refresh_token(_h("t1")) + assert rec is not None + assert rec.token_hash == _h("t1") + assert rec.owner_address == OWNER + assert rec.family_id == "fam1" + assert rec.revoked is False + assert rec.rotated_at is None + assert abs((rec.expires_at - exp).total_seconds()) < 1 + assert rec.issued_at <= datetime.now(UTC) + + +async def test_get_missing_returns_none(rt_backend: MAILServerBackend) -> None: + assert await rt_backend.get_refresh_token(_h("nope")) is None + + +async def test_rotate_revokes_old_and_carries_expiry_forward( + rt_backend: MAILServerBackend, +) -> None: + exp = _exp() + await rt_backend.create_refresh_token(OWNER, _h("old"), "fam", exp) + + await rt_backend.rotate_refresh_token(_h("old"), _h("new")) + + old = await rt_backend.get_refresh_token(_h("old")) + new = await rt_backend.get_refresh_token(_h("new")) + assert old is not None and new is not None + # old half: revoked + rotated + assert old.revoked is True + assert old.rotated_at is not None + # new half: live, same family, absolute cap unchanged + assert new.revoked is False + assert new.rotated_at is None + assert new.family_id == "fam" + assert new.owner_address == OWNER + assert abs((new.expires_at - old.expires_at).total_seconds()) < 1 + + +async def test_rotate_missing_raises(rt_backend: MAILServerBackend) -> None: + with pytest.raises(ValueError): + await rt_backend.rotate_refresh_token(_h("ghost"), _h("x")) + + +async def test_revoke_family_only_targets_that_family( + rt_backend: MAILServerBackend, +) -> None: + await rt_backend.create_refresh_token(OWNER, _h("a1"), "A", _exp()) + await rt_backend.create_refresh_token(OWNER, _h("a2"), "A", _exp()) + await rt_backend.create_refresh_token(OWNER, _h("b1"), "B", _exp()) + + await rt_backend.revoke_refresh_family("A") + + a1 = await rt_backend.get_refresh_token(_h("a1")) + a2 = await rt_backend.get_refresh_token(_h("a2")) + b1 = await rt_backend.get_refresh_token(_h("b1")) + assert a1 is not None and a1.revoked is True + assert a2 is not None and a2.revoked is True + assert b1 is not None and b1.revoked is False + + +async def test_revoke_all_for_owner_spares_other_owners( + rt_backend: MAILServerBackend, +) -> None: + await rt_backend.create_refresh_token(OWNER, _h("o1"), "X", _exp()) + await rt_backend.create_refresh_token(OTHER, _h("p1"), "Y", _exp()) + + await rt_backend.revoke_all_refresh_tokens(OWNER) + + o1 = await rt_backend.get_refresh_token(_h("o1")) + p1 = await rt_backend.get_refresh_token(_h("p1")) + assert o1 is not None and o1.revoked is True + assert p1 is not None and p1.revoked is False + + +async def test_purge_expired_removes_only_expired( + rt_backend: MAILServerBackend, +) -> None: + await rt_backend.create_refresh_token( + OWNER, _h("dead"), "fam", datetime.now(UTC) - timedelta(seconds=1) + ) + await rt_backend.create_refresh_token(OWNER, _h("live"), "fam", _exp()) + + removed = await rt_backend.purge_expired_refresh_tokens() + + assert removed == 1 + assert await rt_backend.get_refresh_token(_h("dead")) is None + assert await rt_backend.get_refresh_token(_h("live")) is not None + + +async def test_sqlite_user_delete_cascades_refresh_tokens( + rt_backend: MAILServerBackend, +) -> None: + if not isinstance(rt_backend, SQLiteBackend): + pytest.skip("FK cascade-on-delete is a sqlite-specific guarantee") + + await rt_backend.create_refresh_token(OWNER, _h("c1"), "fam", _exp()) + async with rt_backend._db.session() as session: + await MailStore(session).user_agents.delete(OWNER) + + assert await rt_backend.get_refresh_token(_h("c1")) is None diff --git a/tests/unit/test_client_commands.py b/tests/unit/test_client_commands.py index bb5f309..533a959 100644 --- a/tests/unit/test_client_commands.py +++ b/tests/unit/test_client_commands.py @@ -23,6 +23,7 @@ cmd_inbox, cmd_login, cmd_ping, + cmd_refresh, cmd_reply, cmd_send, ) @@ -95,7 +96,12 @@ def test_login_posts_credentials_as_form_data( route = respx.post(f"{SERVER}/auth/token").mock( return_value=httpx.Response( 200, - json={"access_token": "issued-jwt", "token_type": "bearer", "metadata": {}}, + json={ + "access_token": "issued-jwt", + "token_type": "bearer", + "expires_in": 900, + "metadata": {}, + }, ) ) cmd_login(Namespace(output="text")) @@ -113,6 +119,53 @@ def test_login_requires_credentials_env(monkeypatch: pytest.MonkeyPatch) -> None cmd_login(Namespace(output="text")) +# ─── refresh ─────────────────────────────────────────────────────── + + +@respx.mock +def test_refresh_posts_token_in_body_and_prints_rotated( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture +) -> None: + monkeypatch.setenv("MAIL_SERVER", SERVER) + monkeypatch.setenv("MAIL_REFRESH_TOKEN", "rt_old") + + route = respx.post(f"{SERVER}/auth/refresh").mock( + return_value=httpx.Response( + 200, + json={ + "access_token": "fresh-jwt", + "token_type": "bearer", + "refresh_token": "rt_new", + "expires_in": 900, + "metadata": {}, + }, + ) + ) + cmd_refresh(Namespace(output="text")) + + sent = json.loads(route.calls[0].request.content.decode()) + assert sent == {"refresh_token": "rt_old"} + out = capsys.readouterr().out + assert "fresh-jwt" in out + assert "rt_new" in out + + +def test_refresh_requires_refresh_token_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("MAIL_SERVER", SERVER) + monkeypatch.delenv("MAIL_REFRESH_TOKEN", raising=False) + with pytest.raises(ValueError, match="MAIL_REFRESH_TOKEN"): + cmd_refresh(Namespace(output="text")) + + +@respx.mock +def test_refresh_raises_on_401(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("MAIL_SERVER", SERVER) + monkeypatch.setenv("MAIL_REFRESH_TOKEN", "rt_dead") + respx.post(f"{SERVER}/auth/refresh").mock(return_value=httpx.Response(401)) + with pytest.raises(RuntimeError, match="401"): + cmd_refresh(Namespace(output="text")) + + # ─── inbox ───────────────────────────────────────────────────────── diff --git a/tests/unit/test_daemon_api.py b/tests/unit/test_daemon_api.py index c4a64c8..d9e3fee 100644 --- a/tests/unit/test_daemon_api.py +++ b/tests/unit/test_daemon_api.py @@ -20,7 +20,12 @@ TOKEN = "daemon-jwt" ROOT_RESPONSE = {"protocol_name": "mail", "protocol_version": "2.0", "uptime": 1.5} -TOKEN_RESPONSE = {"access_token": TOKEN, "token_type": "bearer", "metadata": {}} +TOKEN_RESPONSE = { + "access_token": TOKEN, + "token_type": "bearer", + "expires_in": 900, + "metadata": {}, +} def _whoami_response(ua_type: str = "daemon") -> dict: diff --git a/tests/unit/test_refresh_auth_helpers.py b/tests/unit/test_refresh_auth_helpers.py new file mode 100644 index 0000000..e6889b9 --- /dev/null +++ b/tests/unit/test_refresh_auth_helpers.py @@ -0,0 +1,98 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Unit tests for the refresh-token helpers in ``mail_server.auth``: token +generation/hashing, the interactive-principal check, and cookie attributes +(including the ``Secure`` flag, which the integration suite disables so the +TestClient can replay cookies over http). +""" + +import hashlib + +import mail_server.auth as auth +import pytest +from fastapi import Response +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILAgent, + MAILDaemon, + MAILUser, + MAILUserAgent, + MAILUserAgentInBackend, +) + +HOST = "example.com" + + +def _set_cookie(resp: Response) -> str: + return resp.headers.get("set-cookie") or "" + + +def _wrap(ua: MAILUserAgent) -> MAILUserAgentInBackend: + return MAILUserAgentInBackend(user_agent=ua, hashed_password="x") + + +def test_generate_refresh_token_prefixed_and_unique() -> None: + a = auth.generate_refresh_token() + b = auth.generate_refresh_token() + assert a.startswith(auth.REFRESH_TOKEN_PREFIX) + assert b.startswith(auth.REFRESH_TOKEN_PREFIX) + assert a != b + + +def test_hash_refresh_token_is_sha256_hex() -> None: + token = "rt_example" + assert auth.hash_refresh_token(token) == hashlib.sha256(token.encode()).hexdigest() + + +def test_is_interactive_principal() -> None: + assert auth.is_interactive_principal( + _wrap(MAILUser(ua_type="user", user_id="alice", host=HOST)) + ) + assert auth.is_interactive_principal( + _wrap(MAILAdmin(ua_type="admin", admin_id="ryan", host=HOST)) + ) + assert not auth.is_interactive_principal( + _wrap(MAILAgent(ua_type="agent", name="sage", swarm="chorus", host=HOST)) + ) + assert not auth.is_interactive_principal( + _wrap(MAILDaemon(ua_type="daemon", worker_name="dummy", host=HOST)) + ) + + +def test_set_refresh_cookie_secure_when_configured( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(auth, "COOKIE_SECURE", True) + resp = Response() + auth.set_refresh_cookie(resp, "rt_value") + + header = _set_cookie(resp) + lowered = header.lower() + assert f"{auth.REFRESH_COOKIE_NAME}=rt_value" in header + assert "secure" in lowered + assert "httponly" in lowered + assert "samesite=strict" in lowered + assert "path=/auth" in lowered + + +def test_set_refresh_cookie_omits_secure_when_disabled( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr(auth, "COOKIE_SECURE", False) + resp = Response() + auth.set_refresh_cookie(resp, "rt_value") + + assert "secure" not in _set_cookie(resp).lower() + + +def test_clear_refresh_cookie_emits_deletion() -> None: + resp = Response() + auth.clear_refresh_cookie(resp) + + lowered = _set_cookie(resp).lower() + assert f"{auth.REFRESH_COOKIE_NAME}=" in lowered + assert "path=/auth" in lowered + # deletion is expressed as an immediate expiry + assert "max-age=0" in lowered or 'expires=thu, 01 jan 1970' in lowered From e1aac769025e4aa9e872e81949c8edc7779e7605 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Mon, 29 Jun 2026 15:57:48 -0400 Subject: [PATCH 15/28] fix: document request bodies and box query params in OpenAPI POST/PATCH handlers took only `request: Request` and parsed bodies by hand via validators.py, so FastAPI generated no requestBody schema and the bodies were absent from /docs and spec/openapi.yaml. Promote each request model to a typed handler parameter (Approach A) across the drafts, daemon, admin, lists, and auth routers; FastAPI now validates and documents all 17 bodies. /auth/refresh and /auth/logout take the refresh-token body as an optional Body(default=None) to preserve the cookie-only browser flow; _read_refresh_token is now sync and reads the parsed payload, keeping cookie-over-body precedence. Also declare BoxFilterParams as a typed Query() param on the inbox/outbox/ trash/drafts GET endpoints (same root cause, query side), so the limit/offset/ sort_by/order filters are documented too. Drop the now-unused body and query validators (path-param validators kept). Add the missing MAIL_REFRESH_TOKEN_EXPIRE_DAYS placeholder to generate_openapi.py so the spec regenerates standalone, and regenerate spec/openapi.yaml. Add a contract test asserting requestBody/query-param presence so the regression can't silently return. The 422 body shape for invalid bodies is now FastAPI's native structured detail instead of the previous string; status code is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) --- scripts/generate_openapi.py | 1 + spec/openapi.yaml | 714 ++++++++++++++++++ .../server/src/mail_server/routers/admin.py | 32 +- .../server/src/mail_server/routers/auth.py | 39 +- .../server/src/mail_server/routers/daemon.py | 18 +- .../server/src/mail_server/routers/drafts.py | 34 +- .../server/src/mail_server/routers/inbox.py | 11 +- .../server/src/mail_server/routers/lists.py | 23 +- .../server/src/mail_server/routers/outbox.py | 11 +- .../server/src/mail_server/routers/trash.py | 11 +- src/mail/server/src/mail_server/validators.py | 322 +------- tests/contract/test_openapi_request_bodies.py | 100 +++ 12 files changed, 931 insertions(+), 385 deletions(-) create mode 100644 tests/contract/test_openapi_request_bodies.py diff --git a/scripts/generate_openapi.py b/scripts/generate_openapi.py index 9836f45..3ac0f63 100644 --- a/scripts/generate_openapi.py +++ b/scripts/generate_openapi.py @@ -23,6 +23,7 @@ def _set_import_defaults() -> None: os.environ.setdefault("MAIL_JWT_SECRET_KEY", "openapi-generation-only") os.environ.setdefault("MAIL_JWT_ALGORITHM", "HS256") os.environ.setdefault("MAIL_JWT_EXPIRE_MINUTES", "15") + os.environ.setdefault("MAIL_REFRESH_TOKEN_EXPIRE_DAYS", "30") def _load_schema() -> dict[str, Any]: diff --git a/spec/openapi.yaml b/spec/openapi.yaml index 4677c33..632881b 100644 --- a/spec/openapi.yaml +++ b/spec/openapi.yaml @@ -36,7 +36,19 @@ paths: - authentication summary: Exchange a refresh token for a new access token (rotates the refresh token) + description: The refresh token is read from the httpOnly cookie when present + (browsers); the request body is the fallback for clients that cannot use the + cookie (e.g. the CLI). The body may be omitted entirely when the cookie carries + the token. operationId: post_auth_refresh_auth_refresh_post + requestBody: + content: + application/json: + schema: + anyOf: + - $ref: '#/components/schemas/AuthRefreshPostRequest' + - type: 'null' + title: Payload responses: '200': description: Successful Response @@ -44,12 +56,29 @@ paths: application/json: schema: $ref: '#/components/schemas/AuthRefreshPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /auth/logout: post: tags: - authentication summary: Revoke the presented refresh token's family and clear the cookie + description: The refresh token is read from the httpOnly cookie when present + (browsers), falling back to the request body. The body may be omitted entirely; + logout always succeeds and clears the cookie regardless. operationId: post_auth_logout_auth_logout_post + requestBody: + content: + application/json: + schema: + anyOf: + - $ref: '#/components/schemas/AuthRefreshPostRequest' + - type: 'null' + title: Payload responses: '200': description: Successful Response @@ -57,6 +86,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AuthLogoutPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /auth/whoami: get: tags: @@ -76,6 +111,12 @@ paths: - authentication summary: Reset the user-agent's password operationId: post_password_reset_auth_password_reset_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/AuthPasswordResetRequest' + required: true responses: '200': description: Successful Response @@ -83,6 +124,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AuthPasswordResetResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /swarms: get: tags: @@ -128,6 +175,44 @@ paths: - inbox summary: Get a list of inbox messages operationId: get_inbox_inbox_get + parameters: + - name: limit + in: query + required: false + schema: + type: integer + maximum: 100 + exclusiveMinimum: 0 + default: 20 + title: Limit + - name: offset + in: query + required: false + schema: + type: integer + minimum: 0 + default: 0 + title: Offset + - name: sort_by + in: query + required: false + schema: + enum: + - sent_at + - entered_at + type: string + default: entered_at + title: Sort By + - name: order + in: query + required: false + schema: + enum: + - asc + - desc + type: string + default: desc + title: Order responses: '200': description: Successful Response @@ -135,6 +220,12 @@ paths: application/json: schema: $ref: '#/components/schemas/InboxGetResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /inbox/{message_id}: get: tags: @@ -166,6 +257,44 @@ paths: - outbox summary: Get a list of outbox messages operationId: get_outbox_outbox_get + parameters: + - name: limit + in: query + required: false + schema: + type: integer + maximum: 100 + exclusiveMinimum: 0 + default: 20 + title: Limit + - name: offset + in: query + required: false + schema: + type: integer + minimum: 0 + default: 0 + title: Offset + - name: sort_by + in: query + required: false + schema: + enum: + - sent_at + - entered_at + type: string + default: entered_at + title: Sort By + - name: order + in: query + required: false + schema: + enum: + - asc + - desc + type: string + default: desc + title: Order responses: '200': description: Successful Response @@ -173,6 +302,12 @@ paths: application/json: schema: $ref: '#/components/schemas/OutboxGetResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /outbox/{message_id}: get: tags: @@ -192,6 +327,44 @@ paths: - drafts summary: Get a list of message drafts operationId: get_drafts_drafts_get + parameters: + - name: limit + in: query + required: false + schema: + type: integer + maximum: 100 + exclusiveMinimum: 0 + default: 20 + title: Limit + - name: offset + in: query + required: false + schema: + type: integer + minimum: 0 + default: 0 + title: Offset + - name: sort_by + in: query + required: false + schema: + enum: + - sent_at + - entered_at + type: string + default: entered_at + title: Sort By + - name: order + in: query + required: false + schema: + enum: + - asc + - desc + type: string + default: desc + title: Order responses: '200': description: Successful Response @@ -199,11 +372,23 @@ paths: application/json: schema: $ref: '#/components/schemas/DraftsGetResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' post: tags: - drafts summary: Create a new message draft operationId: post_draft_drafts_post + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/DraftPostRequest' responses: '200': description: Successful Response @@ -211,6 +396,12 @@ paths: application/json: schema: $ref: '#/components/schemas/DraftPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /drafts/{draft_id}: get: tags: @@ -241,6 +432,12 @@ paths: - drafts summary: Update a specific message draft by ID operationId: patch_draft_drafts__draft_id__patch + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/DraftPatchRequest' + required: true responses: '200': description: Successful Response @@ -248,12 +445,24 @@ paths: application/json: schema: $ref: '#/components/schemas/DraftPatchResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /drafts/{draft_id}/send: post: tags: - drafts summary: Send a message from an existing draft by ID operationId: post_draft_send_drafts__draft_id__send_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/DraftSendPostRequest' + required: true responses: '200': description: Successful Response @@ -261,12 +470,56 @@ paths: application/json: schema: $ref: '#/components/schemas/DraftSendPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /trash: get: tags: - trash summary: Get a list of messages in trash operationId: get_trashed_messages_trash_get + parameters: + - name: limit + in: query + required: false + schema: + type: integer + maximum: 100 + exclusiveMinimum: 0 + default: 20 + title: Limit + - name: offset + in: query + required: false + schema: + type: integer + minimum: 0 + default: 0 + title: Offset + - name: sort_by + in: query + required: false + schema: + enum: + - sent_at + - entered_at + type: string + default: entered_at + title: Sort By + - name: order + in: query + required: false + schema: + enum: + - asc + - desc + type: string + default: desc + title: Order responses: '200': description: Successful Response @@ -274,6 +527,12 @@ paths: application/json: schema: $ref: '#/components/schemas/TrashGetResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /trash/{message_id}: get: tags: @@ -344,6 +603,12 @@ paths: - daemon summary: Upload new messages to deliver from local agent(s) operationId: deliver_local_messages_daemon_deliver_local_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/DaemonDeliverLocalRequest' + required: true responses: '200': description: Successful Response @@ -351,12 +616,24 @@ paths: application/json: schema: $ref: '#/components/schemas/DaemonDeliverLocalResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /daemon/deliver/remote: post: tags: - daemon summary: Upload new messages to deliver from remote agent(s) operationId: deliver_remote_messages_daemon_deliver_remote_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/DaemonDeliverRemoteRequest' + required: true responses: '200': description: Successful Response @@ -364,6 +641,12 @@ paths: application/json: schema: $ref: '#/components/schemas/DaemonDeliverRemoteResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/agents: get: tags: @@ -382,6 +665,12 @@ paths: - admin summary: Create a new MAIL agent on this server operationId: post_agent_admin_agents_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/AdminAgentPostRequest' + required: true responses: '200': description: Successful Response @@ -389,6 +678,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminAgentPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/agents/{local_address}: get: tags: @@ -466,6 +761,12 @@ paths: - admin summary: Create a new MAIL daemon on this server operationId: post_daemon_admin_daemons_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/AdminDaemonPostRequest' + required: true responses: '200': description: Successful Response @@ -473,6 +774,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminDaemonPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/daemons/{worker_name}: get: tags: @@ -550,6 +857,12 @@ paths: - admin summary: Create a new MAIL user on this server operationId: post_user_admin_users_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/AdminUserPostRequest' + required: true responses: '200': description: Successful Response @@ -557,6 +870,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminUserPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/users/{user_id}: get: tags: @@ -622,6 +941,12 @@ paths: - admin summary: Create a new MAIL swarm on this server operationId: post_swarm_admin_swarms_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/AdminSwarmPostRequest' + required: true responses: '200': description: Successful Response @@ -629,6 +954,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminSwarmPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/swarms/{swarm_name}: delete: tags: @@ -677,6 +1008,12 @@ paths: - admin summary: Create a new webhook for this server operationId: post_webhook_admin_webhooks_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/AdminWebhooksPostRequest' + required: true responses: '200': description: Successful Response @@ -684,6 +1021,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminWebhooksPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/webhooks/{webhook_id}: get: tags: @@ -730,6 +1073,12 @@ paths: - wh_123e4567-e89b-12d3-a456-426614174000 title: Webhook Id description: Webhook id (wh_). + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/AdminWebhooksPatchRequest' responses: '200': description: Successful Response @@ -790,6 +1139,12 @@ paths: - admin-lists summary: Create a new MAIL list on this server operationId: admin_post_list_admin_lists_post + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/AdminListPostRequest' + required: true responses: '200': description: Successful Response @@ -797,6 +1152,12 @@ paths: application/json: schema: $ref: '#/components/schemas/AdminListPostResponse' + '422': + description: Validation Error + content: + application/json: + schema: + $ref: '#/components/schemas/HTTPValidationError' /admin/lists/{local_address}: get: tags: @@ -847,6 +1208,12 @@ paths: title: Local Address description: 'List local address (name@swarm); the list: prefix and host are implied.' + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/AdminListPatchRequest' responses: '200': description: Successful Response @@ -910,6 +1277,12 @@ paths: title: Local Address description: 'List local address (name@swarm); the list: prefix and host are implied.' + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/ListMemberPostRequest' responses: '200': description: Successful Response @@ -1130,6 +1503,26 @@ components: description: 'Corresponds to `GET /admin/agents/{local_address}`. Contains a specific MAIL agent registered on this server.' + AdminAgentPostRequest: + properties: + agent_name: + type: string + title: Agent Name + swarm_name: + type: string + title: Swarm Name + agent_password: + type: string + title: Agent Password + type: object + required: + - agent_name + - swarm_name + - agent_password + title: AdminAgentPostRequest + description: 'Corresponds to `POST /admin/agents`. + + Contains an agent name, swarm, and password to register with.' AdminAgentPostResponse: properties: agent: @@ -1198,6 +1591,22 @@ components: description: 'Corresponds to `GET /admin/daemons/{worker_name}`. Contains a specific MAIL daemon registered on this server.' + AdminDaemonPostRequest: + properties: + worker_name: + type: string + title: Worker Name + daemon_password: + type: string + title: Daemon Password + type: object + required: + - worker_name + - daemon_password + title: AdminDaemonPostRequest + description: 'Corresponds to `POST /admin/daemons`. + + Contains a worker name and password to register with.' AdminDaemonPostResponse: properties: daemon: @@ -1261,6 +1670,22 @@ components: - metadata title: AdminListGetResponse description: Corresponds to `GET /admin/lists/{local_address}`. + AdminListPatchRequest: + properties: + policy: + anyOf: + - $ref: '#/components/schemas/MAILListPolicy' + - type: 'null' + type: object + title: AdminListPatchRequest + description: 'Corresponds to `PATCH /admin/lists/{local_address}`. + + + All fields are optional; only the policy is mutable at v1. The + + canonical address (name, swarm, host) is immutable for the life of + + the list — re-create + cut over if a rename is required.' AdminListPatchResponse: properties: mail_list: @@ -1275,6 +1700,38 @@ components: - metadata title: AdminListPatchResponse description: Corresponds to `PATCH /admin/lists/{local_address}`. + AdminListPostRequest: + properties: + name: + type: string + title: Name + swarm_name: + type: string + title: Swarm Name + owner: + type: string + title: Owner + members: + items: + type: string + type: array + title: Members + default: [] + policy: + $ref: '#/components/schemas/MAILListPolicy' + default: + visibility: public + join_policy: open + send_policy: open + type: object + required: + - name + - swarm_name + - owner + title: AdminListPostRequest + description: 'Corresponds to `POST /admin/lists`. + + Contains the descriptive fields needed to create a new MAIL list.' AdminListPostResponse: properties: mail_list: @@ -1322,6 +1779,28 @@ components: description: 'Corresponds to `DELETE /admin/swarms/{swarm_name}`. Contains info on the newly-deleted MAIL swarm on this server.' + AdminSwarmPostRequest: + properties: + name: + type: string + title: Name + description: + type: string + title: Description + keywords: + items: + type: string + type: array + title: Keywords + type: object + required: + - name + - description + - keywords + title: AdminSwarmPostRequest + description: 'Corresponds to `POST /admin/swarms`. + + Contains basic info necessary for new MAIL swarm creation.' AdminSwarmPostResponse: properties: swarm: @@ -1370,6 +1849,22 @@ components: description: 'Corresponds to `GET /admin/users/{user_id}`. Contains a specific MAIL user registered on this server.' + AdminUserPostRequest: + properties: + user_id: + type: string + title: User Id + user_password: + type: string + title: User Password + type: object + required: + - user_id + - user_password + title: AdminUserPostRequest + description: 'Corresponds to `POST /admin/users`. + + Contains a user ID and password to register with.' AdminUserPostResponse: properties: user: @@ -1456,6 +1951,22 @@ components: description: 'Corresponds to `GET /admin/webhooks`. Contains a list of existing webhooks by ID.' + AdminWebhooksPatchRequest: + properties: + url: + type: string + title: Url + secret: + type: string + title: Secret + type: object + required: + - url + - secret + title: AdminWebhooksPatchRequest + description: 'Corresponds to `PATCH /admin/webhooks`. + + Allows client to change URL or secret for an existing webhook.' AdminWebhooksPatchResponse: properties: webhook: @@ -1472,6 +1983,28 @@ components: description: 'Corresponds to `PATCH /admin/webhooks/{webhook_id}`. Contains information on the patched webhook.' + AdminWebhooksPostRequest: + properties: + url: + type: string + title: Url + events: + items: + type: string + type: array + title: Events + secret: + type: string + title: Secret + type: object + required: + - url + - events + - secret + title: AdminWebhooksPostRequest + description: 'Corresponds to `POST /admin/webhooks`. + + Contains info required for webhook setup.' AdminWebhooksPostResponse: properties: webhook: @@ -1501,6 +2034,22 @@ components: description: 'Corresponds to `POST /auth/logout`. Contains a message indicating operation success.' + AuthPasswordResetRequest: + properties: + current_password: + type: string + title: Current Password + new_password: + type: string + title: New Password + type: object + required: + - current_password + - new_password + title: AuthPasswordResetRequest + description: 'Corresponds to `POST /auth/password/reset`. + + Contains user''s current password and desired new password.' AuthPasswordResetResponse: properties: status: @@ -1514,6 +2063,22 @@ components: description: 'Corresponds to `POST /auth/password/reset`. Contains a message indicating operation success.' + AuthRefreshPostRequest: + properties: + refresh_token: + anyOf: + - type: string + - type: 'null' + title: Refresh Token + type: object + title: AuthRefreshPostRequest + description: 'Corresponds to `POST /auth/refresh`. + + Body fallback carrying the refresh token for clients that cannot use the + + ``httpOnly`` cookie (e.g. the CLI). Browsers send the token via cookie and + + may omit the body entirely.' AuthRefreshPostResponse: properties: access_token: @@ -1645,6 +2210,20 @@ components: - username - password title: Body_create_auth_token_auth_token_post + DaemonDeliverLocalRequest: + properties: + message_ids: + items: + type: string + type: array + title: Message Ids + type: object + required: + - message_ids + title: DaemonDeliverLocalRequest + description: 'Corresponds to `POST /daemon/deliver/local`. + + Contains a list of local message IDs to deliver to their intended local targets.' DaemonDeliverLocalResponse: properties: messages: @@ -1664,6 +2243,21 @@ components: description: 'Corresponds to `POST /daemon/deliver/local`. Contains the list of messages successfully delivered to server-local user-agents.' + DaemonDeliverRemoteRequest: + properties: + messages: + items: + $ref: '#/components/schemas/MAILMessage' + type: array + title: Messages + type: object + required: + - messages + title: DaemonDeliverRemoteRequest + description: 'Corresponds to `POST /daemon/deliver/remote`. + + Contains a list of remote MAIL messages to deliver to their intended local + targets.' DaemonDeliverRemoteResponse: properties: messages: @@ -1736,6 +2330,44 @@ components: description: 'Corresponds to `GET /drafts/{draft_id}`. Contains a specific message draft inside the user-agent''s drafts box.' + DraftPatchRequest: + properties: + subject: + anyOf: + - type: string + - type: 'null' + title: Subject + body: + anyOf: + - type: string + - type: 'null' + title: Body + reply_to: + anyOf: + - type: string + - type: 'null' + title: Reply To + tags: + anyOf: + - items: + type: string + type: array + - type: 'null' + title: Tags + type: object + title: DraftPatchRequest + description: 'Corresponds to `PATCH /drafts/{draft_id}`. + + Contains the fields to update on an existing MAIL message draft. + + + Every field is optional: a field left unset (``None``) is not modified, + + so callers can patch a single field without resending the rest. The one + + asymmetry is ``tags`` — sending ``tags: []`` clears all tags, while + + omitting ``tags`` leaves the existing tags untouched.' DraftPatchResponse: properties: entry: @@ -1752,6 +2384,40 @@ components: description: 'Corresponds to `PATCH /drafts/{draft_id}`. Contains the updated message draft in the user-agent''s drafts box.' + DraftPostRequest: + properties: + subject: + type: string + title: Subject + body: + type: string + title: Body + reply_to: + anyOf: + - type: string + - type: 'null' + title: Reply To + tags: + items: + type: string + type: array + title: Tags + default: [] + type: object + required: + - subject + - body + title: DraftPostRequest + description: 'Corresponds to `POST /drafts/`. + + Contains relevant information for creating a new MAIL message draft. + + + `reply_to` optionally references the `message_id` of the message this + + draft is replying to. `tags` is an optional list of sender-defined slug + + strings used to categorize the eventual message.' DraftPostResponse: properties: entry: @@ -1768,6 +2434,33 @@ components: description: 'Corresponds to `POST /drafts`. Contains the new entry in the user-agent''s drafts box.' + DraftSendPostRequest: + properties: + recipients: + items: + type: string + type: array + title: Recipients + tags: + items: + type: string + type: array + title: Tags + default: [] + type: object + required: + - recipients + title: DraftSendPostRequest + description: 'Corresponds to `POST /drafts/{draft_id}/send`. + + Contains relevant information for sending an existing draft as a MAIL message. + + + `tags` is an optional list of sender-defined slug strings; any tags + + supplied here are merged (union, order-preserving) with the tags already + + stored on the draft.' DraftSendPostResponse: properties: message: @@ -1911,6 +2604,27 @@ components: The updated list with the member removed (idempotent — removing a non-member is a no-op).' + ListMemberPostRequest: + properties: + member_address: + type: string + title: Member Address + type: object + required: + - member_address + title: ListMemberPostRequest + description: 'Corresponds to ``POST /lists/{local_address}/subscribe`` + + and ``POST /admin/lists/{local_address}/members``. + + + ``member_address`` is the address being added. For the public + + subscribe path, this must match the authenticated bearer + + (self-subscribe); for the admin add path, any valid MAIL address + + is accepted.' ListMemberPostResponse: properties: mail_list: diff --git a/src/mail/server/src/mail_server/routers/admin.py b/src/mail/server/src/mail_server/routers/admin.py index 38146e8..6cffb34 100644 --- a/src/mail/server/src/mail_server/routers/admin.py +++ b/src/mail/server/src/mail_server/routers/admin.py @@ -2,6 +2,14 @@ # Copyright (c) 2026 Addison Kline from fastapi import APIRouter, HTTPException, Path, Request +from mail_protocol.network.requests import ( + AdminAgentPostRequest, + AdminDaemonPostRequest, + AdminSwarmPostRequest, + AdminUserPostRequest, + AdminWebhooksPatchRequest, + AdminWebhooksPostRequest, +) from mail_protocol.network.responses import ( AdminAgentDeleteResponse, AdminAgentGetResponse, @@ -26,12 +34,6 @@ from mail_server.auth import validate_admin from mail_server.validators import ( - validate_admin_post_agent_request, - validate_admin_post_daemon_request, - validate_admin_post_swarm_request, - validate_admin_post_user_request, - validate_admin_webhook_patch_request, - validate_admin_webhook_post_request, validate_local_address_param, validate_swarm_name_param, validate_user_id_param, @@ -96,10 +98,10 @@ async def get_agent( ) async def post_agent( request: Request, + payload: AdminAgentPostRequest, ) -> AdminAgentPostResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - payload = await validate_admin_post_agent_request(request=request) try: result = await backend.admin_post_agent(admin=admin, payload=payload) except ValueError: @@ -193,10 +195,10 @@ async def get_daemon( ) async def post_daemon( request: Request, + payload: AdminDaemonPostRequest, ) -> AdminDaemonPostResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - payload = await validate_admin_post_daemon_request(request=request) try: result = await backend.admin_post_daemon(admin=admin, payload=payload) except ValueError: @@ -288,10 +290,10 @@ async def get_user( ) async def post_user( request: Request, + payload: AdminUserPostRequest, ) -> AdminUserPostResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - payload = await validate_admin_post_user_request(request=request) try: result = await backend.admin_post_user(admin=admin, payload=payload) except ValueError: @@ -337,10 +339,11 @@ async def delete_user( summary="Create a new MAIL swarm on this server", response_model=AdminSwarmPostResponse, ) -async def post_swarm(request: Request) -> AdminSwarmPostResponse: +async def post_swarm( + request: Request, payload: AdminSwarmPostRequest +) -> AdminSwarmPostResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - payload = await validate_admin_post_swarm_request(request=request) try: result = await backend.admin_post_swarm(admin=admin, payload=payload) except ValueError: @@ -428,10 +431,11 @@ async def get_webhook( summary="Create a new webhook for this server", response_model=AdminWebhooksPostResponse, ) -async def post_webhook(request: Request) -> AdminWebhooksPostResponse: +async def post_webhook( + request: Request, payload: AdminWebhooksPostRequest +) -> AdminWebhooksPostResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - payload = await validate_admin_webhook_post_request(request=request) result = await backend.admin_webhook_post(admin=admin, payload=payload) return AdminWebhooksPostResponse( @@ -447,6 +451,7 @@ async def post_webhook(request: Request) -> AdminWebhooksPostResponse: ) async def patch_webhook( request: Request, + payload: AdminWebhooksPatchRequest, webhook_id: str = Path( description="Webhook id (wh_).", examples=["wh_123e4567-e89b-12d3-a456-426614174000"], @@ -454,7 +459,6 @@ async def patch_webhook( ) -> AdminWebhooksPatchResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - payload = await validate_admin_webhook_patch_request(request=request) webhook_id = validate_webhook_id_param(webhook_id) try: result = await backend.admin_webhook_patch( diff --git a/src/mail/server/src/mail_server/routers/auth.py b/src/mail/server/src/mail_server/routers/auth.py index e77ce98..7dfc8a2 100644 --- a/src/mail/server/src/mail_server/routers/auth.py +++ b/src/mail/server/src/mail_server/routers/auth.py @@ -6,8 +6,12 @@ from typing import Annotated from uuid import uuid4 -from fastapi import APIRouter, Depends, HTTPException, Request, Response +from fastapi import APIRouter, Body, Depends, HTTPException, Request, Response from fastapi.security.oauth2 import OAuth2PasswordRequestForm +from mail_protocol.network.requests import ( + AuthPasswordResetRequest, + AuthRefreshPostRequest, +) from mail_protocol.network.responses import ( AuthLogoutPostResponse, AuthPasswordResetResponse, @@ -28,10 +32,6 @@ set_refresh_cookie, validate_user_agent, ) -from mail_server.validators import ( - validate_auth_password_reset_request, - validate_auth_refresh_request, -) ACCESS_TOKEN_EXPIRE_MINUTES = os.getenv("MAIL_JWT_EXPIRE_MINUTES") if ACCESS_TOKEN_EXPIRE_MINUTES is None: @@ -89,7 +89,9 @@ async def create_auth_token( ) -async def _read_refresh_token(request: Request) -> str | None: +def _read_refresh_token( + request: Request, payload: AuthRefreshPostRequest | None +) -> str | None: """ Extract the presented refresh token: the cookie (browsers) takes precedence, falling back to the request body (CLI / non-cookie clients). @@ -98,18 +100,24 @@ async def _read_refresh_token(request: Request) -> str | None: token = request.cookies.get(REFRESH_COOKIE_NAME) if token is not None: return token - payload = await validate_auth_refresh_request(request=request) - return payload.refresh_token + return payload.refresh_token if payload is not None else None @router.post( "/refresh", summary="Exchange a refresh token for a new access token (rotates the refresh token)", + description=( + "The refresh token is read from the httpOnly cookie when present " + "(browsers); the request body is the fallback for clients that cannot " + "use the cookie (e.g. the CLI). The body may be omitted entirely when " + "the cookie carries the token." + ), response_model=AuthRefreshPostResponse, ) async def post_auth_refresh( request: Request, response: Response, + payload: AuthRefreshPostRequest | None = Body(default=None), ) -> AuthRefreshPostResponse: backend = request.app.state.backend credentials_exception = HTTPException( @@ -118,7 +126,7 @@ async def post_auth_refresh( headers={"WWW-Authenticate": "Bearer"}, ) - token = await _read_refresh_token(request) + token = _read_refresh_token(request, payload) if token is None: raise credentials_exception @@ -165,17 +173,23 @@ async def post_auth_refresh( @router.post( "/logout", summary="Revoke the presented refresh token's family and clear the cookie", + description=( + "The refresh token is read from the httpOnly cookie when present " + "(browsers), falling back to the request body. The body may be omitted " + "entirely; logout always succeeds and clears the cookie regardless." + ), response_model=AuthLogoutPostResponse, ) async def post_auth_logout( request: Request, response: Response, + payload: AuthRefreshPostRequest | None = Body(default=None), ) -> AuthLogoutPostResponse: backend = request.app.state.backend # Idempotent: revoke the family if the token resolves, but always succeed and # clear the cookie so a stale/absent token still logs the client out. - token = await _read_refresh_token(request) + token = _read_refresh_token(request, payload) if token is not None: record = await backend.get_refresh_token(hash_refresh_token(token)) if record is not None: @@ -202,10 +216,11 @@ async def get_token_info(request: Request) -> AuthWhoamiGetResponse: summary="Reset the user-agent's password", response_model=AuthPasswordResetResponse, ) -async def post_password_reset(request: Request) -> AuthPasswordResetResponse: +async def post_password_reset( + request: Request, payload: AuthPasswordResetRequest +) -> AuthPasswordResetResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - payload = await validate_auth_password_reset_request(request=request) try: result = await backend.reset_password(user_agent=user_agent, payload=payload) except ValueError: diff --git a/src/mail/server/src/mail_server/routers/daemon.py b/src/mail/server/src/mail_server/routers/daemon.py index 8d2d772..1820169 100644 --- a/src/mail/server/src/mail_server/routers/daemon.py +++ b/src/mail/server/src/mail_server/routers/daemon.py @@ -2,6 +2,10 @@ # Copyright (c) 2026 Addison Kline from fastapi import APIRouter, Request +from mail_protocol.network.requests import ( + DaemonDeliverLocalRequest, + DaemonDeliverRemoteRequest, +) from mail_protocol.network.responses import ( DaemonDeliverLocalResponse, DaemonDeliverRemoteResponse, @@ -9,10 +13,6 @@ ) from mail_server.auth import validate_daemon -from mail_server.validators import ( - validate_deliver_local_request, - validate_deliver_remote_request, -) router = APIRouter(prefix="/daemon", tags=["daemon"]) @@ -39,10 +39,11 @@ async def clear_message_buffer( summary="Upload new messages to deliver from local agent(s)", response_model=DaemonDeliverLocalResponse, ) -async def deliver_local_messages(request: Request) -> DaemonDeliverLocalResponse: +async def deliver_local_messages( + request: Request, payload: DaemonDeliverLocalRequest +) -> DaemonDeliverLocalResponse: backend = request.app.state.backend daemon = await validate_daemon(backend=backend, request=request) - payload = await validate_deliver_local_request(request=request) result = await backend.daemon_deliver_local(daemon=daemon, payload=payload) return DaemonDeliverLocalResponse( messages=result, @@ -55,10 +56,11 @@ async def deliver_local_messages(request: Request) -> DaemonDeliverLocalResponse summary="Upload new messages to deliver from remote agent(s)", response_model=DaemonDeliverRemoteResponse, ) -async def deliver_remote_messages(request: Request) -> DaemonDeliverRemoteResponse: +async def deliver_remote_messages( + request: Request, payload: DaemonDeliverRemoteRequest +) -> DaemonDeliverRemoteResponse: backend = request.app.state.backend daemon = await validate_daemon(backend=backend, request=request) - payload = await validate_deliver_remote_request(request=request) result = await backend.daemon_deliver_remote(daemon=daemon, payload=payload) return DaemonDeliverRemoteResponse( messages=result, diff --git a/src/mail/server/src/mail_server/routers/drafts.py b/src/mail/server/src/mail_server/routers/drafts.py index 9a1f670..e3055cf 100644 --- a/src/mail/server/src/mail_server/routers/drafts.py +++ b/src/mail/server/src/mail_server/routers/drafts.py @@ -1,7 +1,15 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Addison Kline -from fastapi import APIRouter, HTTPException, Request +from typing import Annotated + +from fastapi import APIRouter, HTTPException, Query, Request +from mail_protocol.network.requests import ( + BoxFilterParams, + DraftPatchRequest, + DraftPostRequest, + DraftSendPostRequest, +) from mail_protocol.network.responses import ( DraftDeleteResponse, DraftGetResponse, @@ -13,12 +21,6 @@ from mail_server.auth import validate_user_agent from mail_server.utils import build_box_metadata -from mail_server.validators import ( - validate_box_filter_params, - validate_patch_draft_request, - validate_post_draft_request, - validate_post_draft_send_request, -) router = APIRouter(prefix="/drafts", tags=["drafts"]) @@ -26,10 +28,11 @@ @router.get( "", summary="Get a list of message drafts", response_model=DraftsGetResponse ) -async def get_drafts(request: Request) -> DraftsGetResponse: +async def get_drafts( + request: Request, filters: Annotated[BoxFilterParams, Query()] +) -> DraftsGetResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - filters = await validate_box_filter_params(request) if filters.sort_by == "sent_at": raise HTTPException( status_code=422, @@ -49,10 +52,9 @@ async def get_drafts(request: Request) -> DraftsGetResponse: @router.post("", summary="Create a new message draft", response_model=DraftPostResponse) -async def post_draft(request: Request) -> DraftPostResponse: +async def post_draft(request: Request, payload: DraftPostRequest) -> DraftPostResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - payload = await validate_post_draft_request(request) result = await backend.post_draft(user_agent=user_agent, payload=payload) return DraftPostResponse( @@ -88,10 +90,11 @@ async def get_draft(request: Request) -> DraftGetResponse: summary="Update a specific message draft by ID", response_model=DraftPatchResponse, ) -async def patch_draft(request: Request) -> DraftPatchResponse: +async def patch_draft( + request: Request, payload: DraftPatchRequest +) -> DraftPatchResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - payload = await validate_patch_draft_request(request) draft_id = request.path_params.get("draft_id") try: result = await backend.patch_draft( @@ -135,10 +138,11 @@ async def delete_draft(request: Request) -> DraftDeleteResponse: summary="Send a message from an existing draft by ID", response_model=DraftSendPostResponse, ) -async def post_draft_send(request: Request) -> DraftSendPostResponse: +async def post_draft_send( + request: Request, payload: DraftSendPostRequest +) -> DraftSendPostResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - payload = await validate_post_draft_send_request(request) draft_id = request.path_params.get("draft_id") try: result = await backend.send_draft( diff --git a/src/mail/server/src/mail_server/routers/inbox.py b/src/mail/server/src/mail_server/routers/inbox.py index 3702737..5230850 100644 --- a/src/mail/server/src/mail_server/routers/inbox.py +++ b/src/mail/server/src/mail_server/routers/inbox.py @@ -1,7 +1,10 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Addison Kline -from fastapi import APIRouter, HTTPException, Request +from typing import Annotated + +from fastapi import APIRouter, HTTPException, Query, Request +from mail_protocol.network.requests import BoxFilterParams from mail_protocol.network.responses import ( InboxGetResponse, InboxMessageDeleteResponse, @@ -10,7 +13,6 @@ from mail_server.auth import validate_user_agent from mail_server.utils import build_box_metadata -from mail_server.validators import validate_box_filter_params router = APIRouter(prefix="/inbox", tags=["inbox"]) @@ -20,10 +22,11 @@ summary="Get a list of inbox messages", response_model=InboxGetResponse, ) -async def get_inbox(request: Request) -> InboxGetResponse: +async def get_inbox( + request: Request, filters: Annotated[BoxFilterParams, Query()] +) -> InboxGetResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - filters = await validate_box_filter_params(request) try: entries, total = await backend.get_inbox(user_agent, filters) except ValueError: diff --git a/src/mail/server/src/mail_server/routers/lists.py b/src/mail/server/src/mail_server/routers/lists.py index 7eba721..f6e05cd 100644 --- a/src/mail/server/src/mail_server/routers/lists.py +++ b/src/mail/server/src/mail_server/routers/lists.py @@ -3,6 +3,11 @@ from fastapi import APIRouter, HTTPException, Path, Request from mail_protocol.core.lists import MAILListPolicy +from mail_protocol.network.requests import ( + AdminListPatchRequest, + AdminListPostRequest, + ListMemberPostRequest, +) from mail_protocol.network.responses import ( AdminListDeleteResponse, AdminListGetResponse, @@ -18,9 +23,6 @@ from mail_server.auth import validate_admin, validate_user_agent from mail_server.backends.base import MAILServerBackend from mail_server.validators import ( - validate_admin_patch_list_request, - validate_admin_post_list_request, - validate_list_member_post_request, validate_local_address_param, validate_member_address_param, ) @@ -125,10 +127,11 @@ async def admin_get_list( summary="Create a new MAIL list on this server", response_model=AdminListPostResponse, ) -async def admin_post_list(request: Request) -> AdminListPostResponse: +async def admin_post_list( + request: Request, payload: AdminListPostRequest +) -> AdminListPostResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) - payload = await validate_admin_post_list_request(request=request) _reject_unsupported_policy(payload.policy) try: result = await backend.admin_post_list(admin=admin, payload=payload) @@ -143,14 +146,15 @@ async def admin_post_list(request: Request) -> AdminListPostResponse: response_model=AdminListPatchResponse, ) async def admin_patch_list( - request: Request, local_address: str = _LOCAL_ADDRESS_PATH + request: Request, + payload: AdminListPatchRequest, + local_address: str = _LOCAL_ADDRESS_PATH, ) -> AdminListPatchResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) list_address = _full_list_address( backend, validate_local_address_param(local_address) ) - payload = await validate_admin_patch_list_request(request=request) _reject_unsupported_policy(payload.policy) try: result = await backend.admin_patch_list( @@ -187,7 +191,9 @@ async def admin_delete_list( response_model=ListMemberPostResponse, ) async def admin_add_list_member( - request: Request, local_address: str = _LOCAL_ADDRESS_PATH + request: Request, + payload: ListMemberPostRequest, + local_address: str = _LOCAL_ADDRESS_PATH, ) -> ListMemberPostResponse: backend = request.app.state.backend admin = await validate_admin(backend=backend, request=request) @@ -195,7 +201,6 @@ async def admin_add_list_member( list_address = _full_list_address( backend, validate_local_address_param(local_address) ) - payload = await validate_list_member_post_request(request=request) try: result = await backend.add_list_member( list_address=list_address, diff --git a/src/mail/server/src/mail_server/routers/outbox.py b/src/mail/server/src/mail_server/routers/outbox.py index f4e1722..4e721b9 100644 --- a/src/mail/server/src/mail_server/routers/outbox.py +++ b/src/mail/server/src/mail_server/routers/outbox.py @@ -1,12 +1,14 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Addison Kline -from fastapi import APIRouter, HTTPException, Request +from typing import Annotated + +from fastapi import APIRouter, HTTPException, Query, Request +from mail_protocol.network.requests import BoxFilterParams from mail_protocol.network.responses import OutboxGetResponse, OutboxMessageGetResponse from mail_server.auth import validate_user_agent from mail_server.utils import build_box_metadata -from mail_server.validators import validate_box_filter_params router = APIRouter(prefix="/outbox", tags=["outbox"]) @@ -14,10 +16,11 @@ @router.get( "", summary="Get a list of outbox messages", response_model=OutboxGetResponse ) -async def get_outbox(request: Request) -> OutboxGetResponse: +async def get_outbox( + request: Request, filters: Annotated[BoxFilterParams, Query()] +) -> OutboxGetResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - filters = await validate_box_filter_params(request) try: entries, total = await backend.get_outbox(user_agent, filters) except ValueError: diff --git a/src/mail/server/src/mail_server/routers/trash.py b/src/mail/server/src/mail_server/routers/trash.py index 6451b8d..0be09d4 100644 --- a/src/mail/server/src/mail_server/routers/trash.py +++ b/src/mail/server/src/mail_server/routers/trash.py @@ -1,7 +1,10 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Addison Kline -from fastapi import APIRouter, HTTPException, Request +from typing import Annotated + +from fastapi import APIRouter, HTTPException, Query, Request +from mail_protocol.network.requests import BoxFilterParams from mail_protocol.network.responses import ( TrashClearPostResponse, TrashGetResponse, @@ -11,7 +14,6 @@ from mail_server.auth import validate_user_agent from mail_server.utils import build_box_metadata -from mail_server.validators import validate_box_filter_params router = APIRouter(prefix="/trash", tags=["trash"]) @@ -19,10 +21,11 @@ @router.get( "", summary="Get a list of messages in trash", response_model=TrashGetResponse ) -async def get_trashed_messages(request: Request) -> TrashGetResponse: +async def get_trashed_messages( + request: Request, filters: Annotated[BoxFilterParams, Query()] +) -> TrashGetResponse: backend = request.app.state.backend user_agent = await validate_user_agent(backend=backend, request=request) - filters = await validate_box_filter_params(request) try: entries, total = await backend.get_trash(user_agent, filters) except ValueError: diff --git a/src/mail/server/src/mail_server/validators.py b/src/mail/server/src/mail_server/validators.py index 9a7c6e6..9ebe531 100644 --- a/src/mail/server/src/mail_server/validators.py +++ b/src/mail/server/src/mail_server/validators.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Addison Kline -from fastapi import HTTPException, Request +from fastapi import HTTPException from mail_protocol.core.validators import ( validate_daemon_worker_name, validate_local_address, @@ -10,50 +10,11 @@ validate_user_name, validate_webhook_id, ) -from mail_protocol.network.requests import ( - AdminAgentPostRequest, - AdminDaemonPostRequest, - AdminListPatchRequest, - AdminListPostRequest, - AdminSwarmPostRequest, - AdminUserPostRequest, - AdminWebhooksPatchRequest, - AdminWebhooksPostRequest, - AuthPasswordResetRequest, - AuthRefreshPostRequest, - BoxFilterParams, - DaemonDeliverLocalRequest, - DaemonDeliverRemoteRequest, - DraftPatchRequest, - DraftPostRequest, - DraftSendPostRequest, - ListMemberPostRequest, -) -# NOTE: the except clauses below catch ValueError, which covers both -# pydantic.ValidationError and json.JSONDecodeError — an unparseable -# body must 422 the same way an invalid one does. - - -# -# Query parameter validators -# -async def validate_box_filter_params(request: Request) -> BoxFilterParams: - """ - Ensure the query string is valid for the "GET box" endpoints - (`GET /inbox`, `GET /outbox`, `GET /trash`, `GET /drafts`). - - `BoxFilterParams` declares `extra="forbid"`, so an unknown query - parameter 422s the same way an out-of-bounds `limit` does. Values - arrive as strings; pydantic coerces and bounds-checks them. - """ - - try: - return BoxFilterParams.model_validate(dict(request.query_params)) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"query parameter validation failed: {e}" - ) +# Request bodies and query strings are validated by FastAPI from the typed +# parameters declared on each handler (the models live in +# ``mail_protocol.network.requests``). The helpers below cover only path +# parameters, which the handlers still validate explicitly. # @@ -63,8 +24,8 @@ async def validate_box_filter_params(request: Request) -> BoxFilterParams: # identifier — the host is implied by the server and the user-agent # prefix (``daemon:``/``user:``/``list:``) is implied by the route. # These helpers validate the shape of a path segment and 422 on -# malformed input, mirroring the body/query validators above. A -# well-formed-but-unknown id still 404s downstream. +# malformed input, mirroring the body/query validation FastAPI performs. +# A well-formed-but-unknown id still 404s downstream. # def validate_local_address_param(value: str) -> str: """ @@ -144,272 +105,3 @@ def validate_member_address_param(value: str) -> str: raise HTTPException( status_code=422, detail=f"invalid member address path parameter: {e}" ) - - -# -# Draft endpoint validators -# -async def validate_post_draft_request(request: Request) -> DraftPostRequest: - """ - Ensure the request payload is valid for `POST /drafts`. - """ - - try: - body = await request.json() - return DraftPostRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_patch_draft_request(request: Request) -> DraftPatchRequest: - """ - Ensure the request payload is valid for `PATCH /drafts/{draft_id}`. - """ - - try: - body = await request.json() - return DraftPatchRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_post_draft_send_request(request: Request) -> DraftSendPostRequest: - """ - Ensure the request payload is valid for `POST /drafts/{draft_id}/send`. - """ - - try: - body = await request.json() - return DraftSendPostRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -# -# Daemon endpoint validators -# -async def validate_deliver_local_request( - request: Request, -) -> DaemonDeliverLocalRequest: - """ - Ensure that the request payload is valid for `POST /daemon/deliver/local`. - """ - - try: - body = await request.json() - return DaemonDeliverLocalRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_deliver_remote_request( - request: Request, -) -> DaemonDeliverRemoteRequest: - """ - Ensure that the request payload is valid for `POST /daemon/deliver/remote`. - """ - - try: - body = await request.json() - return DaemonDeliverRemoteRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -# -# Admin endpoint validators -# -async def validate_admin_post_agent_request( - request: Request, -) -> AdminAgentPostRequest: - """ - Ensure that the request payload is valid for `POST /admin/agents`. - """ - - try: - body = await request.json() - return AdminAgentPostRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_admin_post_daemon_request( - request: Request, -) -> AdminDaemonPostRequest: - """ - Ensure that the request payload is valid for `POST /admin/daemons`. - """ - - try: - body = await request.json() - return AdminDaemonPostRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_admin_post_user_request( - request: Request, -) -> AdminUserPostRequest: - """ - Ensure that the request payload is valid for `POST /admin/users`. - """ - - try: - body = await request.json() - return AdminUserPostRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_admin_post_swarm_request( - request: Request, -) -> AdminSwarmPostRequest: - """ - Ensure that the request payload is valid for `POST /admin/swarms`. - """ - - try: - body = await request.json() - return AdminSwarmPostRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_admin_webhook_post_request( - request: Request, -) -> AdminWebhooksPostRequest: - """ - Ensure that the request payload is valid for `POST /admin/webhooks`. - """ - - try: - body = await request.json() - return AdminWebhooksPostRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_admin_webhook_patch_request( - request: Request, -) -> AdminWebhooksPatchRequest: - """ - Ensure that the given request payload is valid for `PATCH /admin/webhooks`. - """ - - try: - body = await request.json() - return AdminWebhooksPatchRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_admin_post_list_request( - request: Request, -) -> AdminListPostRequest: - """ - Ensure that the request payload is valid for `POST /admin/lists`. - """ - - try: - body = await request.json() - return AdminListPostRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_admin_patch_list_request( - request: Request, -) -> AdminListPatchRequest: - """ - Ensure that the request payload is valid for `PATCH /admin/lists/{local_address}`. - """ - - try: - body = await request.json() - return AdminListPatchRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_list_member_post_request( - request: Request, -) -> ListMemberPostRequest: - """ - Ensure that the request payload is valid for the member-add endpoints - (`POST /admin/lists/{local_address}/members` and the subscribe variant). - """ - - try: - body = await request.json() - return ListMemberPostRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -# -# Auth endpoint validators -# -async def validate_auth_password_reset_request( - request: Request, -) -> AuthPasswordResetRequest: - """ - Ensure that the request payload is valid for `POST /auth/password/reset`. - """ - - try: - body = await request.json() - return AuthPasswordResetRequest.model_validate(body) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) - - -async def validate_auth_refresh_request( - request: Request, -) -> AuthRefreshPostRequest: - """ - Ensure that the request payload is valid for `POST /auth/refresh`. - - An empty body is allowed: browsers carry the refresh token in the - ``httpOnly`` cookie and may send no body at all. A non-empty body must still - be valid JSON for the model, otherwise 422. - """ - - raw = await request.body() - if not raw: - return AuthRefreshPostRequest() - try: - return AuthRefreshPostRequest.model_validate_json(raw) - except ValueError as e: - raise HTTPException( - status_code=422, detail=f"request body validation failed: {e}" - ) diff --git a/tests/contract/test_openapi_request_bodies.py b/tests/contract/test_openapi_request_bodies.py new file mode 100644 index 0000000..3cc8a51 --- /dev/null +++ b/tests/contract/test_openapi_request_bodies.py @@ -0,0 +1,100 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Charon Labs (contribution PR) + +""" +Guards that every body-bearing POST/PATCH endpoint documents its request +body in the OpenAPI schema, and that the "GET box" endpoints document their +query parameters. + +Handlers that take only ``request: Request`` and parse the body/query string +by hand produce no schema for it, so the bodies silently vanish from +``/docs``. This test fails if that regression is reintroduced. +""" + +import pytest + +# (path, method) for every endpoint that MUST advertise a JSON request body. +ENDPOINTS_WITH_BODY = [ + ("/auth/refresh", "post"), + ("/auth/logout", "post"), + ("/auth/password/reset", "post"), + ("/drafts", "post"), + ("/drafts/{draft_id}", "patch"), + ("/drafts/{draft_id}/send", "post"), + ("/daemon/deliver/local", "post"), + ("/daemon/deliver/remote", "post"), + ("/admin/agents", "post"), + ("/admin/daemons", "post"), + ("/admin/users", "post"), + ("/admin/swarms", "post"), + ("/admin/webhooks", "post"), + ("/admin/webhooks/{webhook_id}", "patch"), + ("/admin/lists", "post"), + ("/admin/lists/{local_address}", "patch"), + ("/admin/lists/{local_address}/members", "post"), +] + +# Endpoints that intentionally take no body — the member is derived from the +# authenticated caller, or the action has no parameters. These must NOT grow +# a request body. +ENDPOINTS_WITHOUT_BODY = [ + ("/trash/clear", "post"), + ("/daemon/message-buffer/clear", "post"), + ("/lists/{local_address}/subscribe", "post"), + ("/lists/{local_address}/unsubscribe", "post"), +] + +# "GET box" endpoints whose BoxFilterParams query params must be documented. +BOX_GET_PATHS = ["/inbox", "/outbox", "/trash", "/drafts"] +BOX_QUERY_PARAMS = {"limit", "offset", "sort_by", "order"} + + +@pytest.fixture(scope="module") +def schema() -> dict: + from mail_server.server import app + + return app.openapi() + + +@pytest.mark.parametrize("path,method", ENDPOINTS_WITH_BODY) +def test_endpoint_documents_request_body(schema: dict, path: str, method: str) -> None: + operation = schema["paths"][path][method] + assert "requestBody" in operation, ( + f"{method.upper()} {path} has no documented request body — the handler " + "likely takes only `request: Request` and parses the body by hand. " + "Declare the request model as a typed parameter." + ) + content = operation["requestBody"].get("content", {}) + assert "application/json" in content, ( + f"{method.upper()} {path} request body is not application/json: " + f"{list(content)}" + ) + assert content["application/json"].get("schema"), ( + f"{method.upper()} {path} request body has no schema" + ) + + +@pytest.mark.parametrize("path,method", ENDPOINTS_WITHOUT_BODY) +def test_bodyless_endpoint_has_no_request_body( + schema: dict, path: str, method: str +) -> None: + operation = schema["paths"][path][method] + assert "requestBody" not in operation, ( + f"{method.upper()} {path} unexpectedly advertises a request body; " + "this endpoint is supposed to take no body." + ) + + +@pytest.mark.parametrize("path", BOX_GET_PATHS) +def test_box_get_documents_query_params(schema: dict, path: str) -> None: + params = { + p["name"] + for p in schema["paths"][path]["get"].get("parameters", []) + if p["in"] == "query" + } + missing = BOX_QUERY_PARAMS - params + assert not missing, ( + f"GET {path} is missing query params {missing} in the schema — the " + "handler likely parses the query string by hand instead of declaring " + "BoxFilterParams as a typed Query() parameter." + ) From 6c595f94fe19ad26b82090aa27ccf4bb4860d15a Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Tue, 30 Jun 2026 15:50:58 -0400 Subject: [PATCH 16/28] feat: add per-owner read/unread status for inbox messages Messages are delivered unread and marked read when the owning user-agent opens them via GET /inbox/{message_id}. Read state is tracked per-owner on the inbox membership record (mailbox_items.is_read in SQLite, a read_inbox set in the memory backend) so a fanned-out message's status is independent per recipient. Surfaced as is_read on MAILInboxEntrySummary. Co-Authored-By: Claude Opus 4.8 (1M context) --- spec/openapi.yaml | 4 ++ .../protocol/src/mail_protocol/core/inbox.py | 6 ++ .../src/mail_server/backends/memory/api.py | 35 ++++++++-- .../src/mail_server/backends/memory/fs.py | 68 +++++++++++++++++-- .../src/mail_server/backends/memory/init.py | 5 ++ .../src/mail_server/backends/sqlite/api.py | 2 + .../mail_server/backends/sqlite/database.py | 21 ++++-- .../backends/sqlite/repositories.py | 40 ++++++++++- .../src/mail_server/backends/sqlite/schema.py | 3 + tests/conftest.py | 1 + tests/integration/test_mailboxes.py | 50 ++++++++++++++ 11 files changed, 222 insertions(+), 13 deletions(-) diff --git a/spec/openapi.yaml b/spec/openapi.yaml index 632881b..2f83ee9 100644 --- a/spec/openapi.yaml +++ b/spec/openapi.yaml @@ -2861,6 +2861,10 @@ components: delivered_by: type: string title: Delivered By + is_read: + type: boolean + title: Is Read + default: false type: object required: - message_id diff --git a/src/mail/protocol/src/mail_protocol/core/inbox.py b/src/mail/protocol/src/mail_protocol/core/inbox.py index f2cf35e..be93fcd 100644 --- a/src/mail/protocol/src/mail_protocol/core/inbox.py +++ b/src/mail/protocol/src/mail_protocol/core/inbox.py @@ -25,6 +25,12 @@ class MAILInboxEntrySummary(BaseModel): body_size: int received_at: datetime delivered_by: Annotated[str, AfterValidator(validate_mail_address)] + # Per-owner read state. A message is delivered ``unread`` and flipped to + # ``read`` when its owner opens it via ``GET /inbox/{message_id}``. Because + # one message fans out to many recipients who share a single inbox entry, + # this value is supplied per owner at list time, not stored on the shared + # entry. + is_read: bool = False class MAILInboxEntry(BaseModel): diff --git a/src/mail/server/src/mail_server/backends/memory/api.py b/src/mail/server/src/mail_server/backends/memory/api.py index c258fc6..df7be14 100644 --- a/src/mail/server/src/mail_server/backends/memory/api.py +++ b/src/mail/server/src/mail_server/backends/memory/api.py @@ -58,6 +58,7 @@ load_messages, load_outbox_entries, load_outboxes, + load_read_inbox, load_refresh_tokens, load_swarms, load_trash_entries, @@ -73,6 +74,7 @@ save_messages, save_outbox_entries, save_outboxes, + save_read_inbox, save_refresh_tokens, save_swarms, save_trash_entries, @@ -144,6 +146,9 @@ def _snapshot_persistence_state(self) -> dict[str, Any]: "messages": dict(self.messages), "inbox_entries": dict(self.inbox_entries), "inboxes": {address: list(ids) for address, ids in self.inboxes.items()}, + "read_inbox": { + address: set(ids) for address, ids in self.read_inbox.items() + }, "outbox_entries": dict(self.outbox_entries), "outboxes": {address: list(ids) for address, ids in self.outboxes.items()}, "draft_entries": dict(self.draft_entries), @@ -171,6 +176,7 @@ async def persist(self, *, reason: str = "manual") -> None: await save_messages(snapshot["messages"]) await save_inbox_entries(snapshot["inbox_entries"]) await save_inboxes(snapshot["inboxes"]) + await save_read_inbox(snapshot["read_inbox"]) await save_outbox_entries(snapshot["outbox_entries"]) await save_outboxes(snapshot["outboxes"]) await save_draft_entries(snapshot["draft_entries"]) @@ -289,6 +295,15 @@ async def on_server_startup(self, **kwargs: Any) -> None: Values: list of inbox entry message IDs """ + self.read_inbox: dict[str, set[str]] = await load_read_inbox() + """ + Per-owner inbox read state (the in-memory analogue of + ``mailbox_items.is_read``). A message is unread unless its id is present + in the owner's set. + Keys: user-agent addresses + Values: set of read inbox message IDs + """ + self.outbox_entries: dict[ str, MAILOutboxEntrySummary ] = await load_outbox_entries() @@ -353,9 +368,7 @@ async def on_server_startup(self, **kwargs: Any) -> None: Values: MAILListInBackend instances """ - self.refresh_tokens: dict[str, RefreshTokenRecord] = ( - await load_refresh_tokens() - ) + self.refresh_tokens: dict[str, RefreshTokenRecord] = await load_refresh_tokens() """ A dict of all stored refresh tokens on this server. Keys: token hashes (sha256 hex) @@ -579,12 +592,17 @@ async def get_inbox( if inbox_msg_ids is None: raise ValueError(f"no inbox found for address {ua_address}") + read = self.read_inbox.get(ua_address, set()) inbox_entries: list[MAILInboxEntrySummary] = [] for msg_id in inbox_msg_ids: inbox_entry = self.inbox_entries.get(msg_id) if inbox_entry is None: raise ValueError(f"no inbox entry found for message ID {msg_id}") - inbox_entries.append(inbox_entry) + # ``inbox_entries`` is shared across recipients; copy so this owner's + # read state never leaks onto the shared entry. + inbox_entries.append( + inbox_entry.model_copy(update={"is_read": msg_id in read}) + ) return _paginate_box( inbox_entries, filters, self._box_sort_key(filters, "received_at") @@ -613,6 +631,9 @@ async def get_inbox_message( if message is None: raise ValueError(f"message with ID {message_id} not found in messages") + # Opening a message marks it read for this owner. + self.read_inbox.setdefault(ua_address, set()).add(message_id) + return MAILInboxEntry( message=message, received_at=inbox_entry.received_at, @@ -1143,6 +1164,8 @@ async def admin_delete_agent( # remove inbox from self.inboxes self.inboxes.pop(full_address) + # drop any per-owner read state alongside the inbox + self.read_inbox.pop(full_address, None) # remove outbox from self.outboxes self.outboxes.pop(full_address) # remove drafts box from self.drafts @@ -1250,6 +1273,8 @@ async def admin_delete_daemon( # remove inbox from self.inboxes self.inboxes.pop(full_address) + # drop any per-owner read state alongside the inbox + self.read_inbox.pop(full_address, None) # remove outbox from self.outboxes self.outboxes.pop(full_address) # remove drafts box from self.drafts @@ -1355,6 +1380,8 @@ async def admin_delete_user(self, admin: MAILAdmin, user_id: str) -> MAILUser: # remove inbox from self.inboxes self.inboxes.pop(full_address) + # drop any per-owner read state alongside the inbox + self.read_inbox.pop(full_address, None) # remove outbox from self.outboxes self.outboxes.pop(full_address) # remove drafts box from self.drafts diff --git a/src/mail/server/src/mail_server/backends/memory/fs.py b/src/mail/server/src/mail_server/backends/memory/fs.py index b98ff49..c90daa8 100644 --- a/src/mail/server/src/mail_server/backends/memory/fs.py +++ b/src/mail/server/src/mail_server/backends/memory/fs.py @@ -283,6 +283,69 @@ async def load_inboxes() -> dict[str, list[str]]: return inboxes +async def load_read_inbox() -> dict[str, set[str]]: + """ + Load saved per-owner inbox read state from the local filesystem. + + Mirrors ``load_inboxes``: one file per owner, one read message id per line. + A missing ``read_inbox`` directory means no read state has been persisted + yet (e.g. a deployment created before read tracking existed), which is + treated as "everything unread". + """ + + read_inbox_path = DEPLOYMENT_PATH.joinpath("read_inbox") + logger.info(f"loading read_inbox: {read_inbox_path}...") + read_inbox: dict[str, set[str]] = {} + if not read_inbox_path.is_dir(): + logger.info("no read_inbox directory found; treating all messages as unread") + return read_inbox + with scandir(read_inbox_path) as entries: + for entry in entries: + if entry.is_file(): + try: + validate_mail_address(entry.name) + except ValueError as e: + logger.warning(f"MAIL address validation failed: {e}") + continue + + with open(entry) as read_file: + content = read_file.readlines() + msg_ids: set[str] = set() + for ln in content: + msg_id = ln.strip() + if not msg_id: + continue + try: + validate_uuid(msg_id) + except ValueError as e: + logger.warning(f"Message ID validation failed: {e}") + continue + + msg_ids.add(msg_id) + + read_inbox.update({entry.name: msg_ids}) + + logger.info(f"found read state for {len(read_inbox)} inboxes") + + return read_inbox + + +async def save_read_inbox(read_inbox: dict[str, set[str]]) -> None: + """ + Save per-owner inbox read state from memory to the local filesystem. + """ + + logger.info(f"saving read state for {len(read_inbox)} inboxes...") + + _save_directory_snapshot( + DEPLOYMENT_PATH.joinpath("read_inbox"), + { + address: "".join(f"{msg_id}\n" for msg_id in sorted(msg_ids)) + for address, msg_ids in read_inbox.items() + }, + ) + + async def load_outbox_entries() -> dict[str, MAILOutboxEntrySummary]: """ Load saved outbox entries from the local filesystem. @@ -803,10 +866,7 @@ async def save_lists(lists: dict[str, MAILListInBackend]) -> None: _save_directory_snapshot( DEPLOYMENT_PATH.joinpath("lists"), - { - address: mail_list.model_dump_json() - for address, mail_list in lists.items() - }, + {address: mail_list.model_dump_json() for address, mail_list in lists.items()}, ) diff --git a/src/mail/server/src/mail_server/backends/memory/init.py b/src/mail/server/src/mail_server/backends/memory/init.py index 5b60dce..fcbb344 100644 --- a/src/mail/server/src/mail_server/backends/memory/init.py +++ b/src/mail/server/src/mail_server/backends/memory/init.py @@ -82,6 +82,11 @@ def init_memory_backend( INBOXES_PATH.mkdir(exist_ok=True) print(f"ensured deployment inboxes: {INBOXES_PATH}") + # ~/.mail-swarms/deployments/{deployment}/read_inbox + READ_INBOX_PATH = DEPLOYMENT_PATH.joinpath("read_inbox") + READ_INBOX_PATH.mkdir(exist_ok=True) + print(f"ensured deployment read_inbox: {READ_INBOX_PATH}") + # ~/.mail-swarms/deployments/{deployment}/outbox_entries OUTBOX_ENTRIES_PATH = DEPLOYMENT_PATH.joinpath("outbox_entries") # print(f"ensuring deployment outbox_entries: {OUTBOX_ENTRIES_PATH}") diff --git a/src/mail/server/src/mail_server/backends/sqlite/api.py b/src/mail/server/src/mail_server/backends/sqlite/api.py index 88c0411..db011ae 100644 --- a/src/mail/server/src/mail_server/backends/sqlite/api.py +++ b/src/mail/server/src/mail_server/backends/sqlite/api.py @@ -279,6 +279,8 @@ async def get_inbox_message( message = await store.messages.get(message_id) if message is None: raise ValueError(f"message with ID {message_id} not found in messages") + # Opening a message marks it read for this owner. + await store.boxes.mark_read(ua_address, BOX_INBOX, message_id) return MAILInboxEntry( message=message, received_at=inbox_entry.received_at, diff --git a/src/mail/server/src/mail_server/backends/sqlite/database.py b/src/mail/server/src/mail_server/backends/sqlite/database.py index f0cb552..50ec4c5 100644 --- a/src/mail/server/src/mail_server/backends/sqlite/database.py +++ b/src/mail/server/src/mail_server/backends/sqlite/database.py @@ -27,7 +27,7 @@ from pathlib import Path from typing import Protocol -from sqlalchemy import event +from sqlalchemy import event, inspect, text from sqlalchemy.engine import Connection, make_url from sqlalchemy.ext.asyncio import ( AsyncEngine, @@ -116,11 +116,24 @@ def _ensure_schema_columns(connection: Connection) -> None: This is the forward-compatibility hook mirroring chorus' approach: when a queryable column is added to a ``*Row`` table in a later release, add an idempotent ``ALTER TABLE ... ADD COLUMN`` here so existing databases pick it - up without a migration framework. There are no such additions yet, so this - is currently a no-op. + up without a migration framework. """ - del connection # no additive columns yet; hook retained for forward-compat + inspector = inspect(connection) + + def _columns(table: str) -> set[str]: + return {col["name"] for col in inspector.get_columns(table)} + + # ``mailbox_items.is_read``: per-owner inbox read state (added in v2). SQLite + # backfills existing rows with the ``DEFAULT 0`` (unread), which is the + # correct legacy state for already-delivered messages. + if "is_read" not in _columns("mailbox_items"): + connection.execute( + text( + "ALTER TABLE mailbox_items " + "ADD COLUMN is_read BOOLEAN NOT NULL DEFAULT 0" + ) + ) def _ensure_sqlite_parent(url: str) -> None: diff --git a/src/mail/server/src/mail_server/backends/sqlite/repositories.py b/src/mail/server/src/mail_server/backends/sqlite/repositories.py index dc2e8cc..a8c07a6 100644 --- a/src/mail/server/src/mail_server/backends/sqlite/repositories.py +++ b/src/mail/server/src/mail_server/backends/sqlite/repositories.py @@ -278,6 +278,36 @@ async def remove_membership(self, owner: str, box: str, item_id: str) -> bool: await self.session.flush() return True + async def mark_read(self, owner: str, box: str, item_id: str) -> None: + """Flip one membership row to ``is_read=True`` (idempotent).""" + + await self.session.execute( + update(MailboxItemRow) + .where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + MailboxItemRow.item_id == item_id, + ) + .values(is_read=True) + ) + await self.session.flush() + + async def read_states( + self, owner: str, box: str, item_ids: list[str] + ) -> dict[str, bool]: + """Map each requested ``item_id`` to its per-owner read flag.""" + + if not item_ids: + return {} + rows = await self.session.execute( + select(MailboxItemRow.item_id, MailboxItemRow.is_read).where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + MailboxItemRow.item_id.in_(item_ids), + ) + ) + return {item_id: is_read for item_id, is_read in rows} + async def list_item_ids(self, owner: str, box: str) -> list[str]: """Item ids in a box, in insertion order (used by ``clear_trash``).""" @@ -377,7 +407,15 @@ async def list_inbox( filters=filters, allow_message_sort=True, ) - return [ser.inbox_entry_from_row(row) for row in rows], total + summaries = [ser.inbox_entry_from_row(row) for row in rows] + # ``is_read`` is per-owner, so it lives on ``mailbox_items``, not on the + # shared inbox entry; stitch it onto this owner's page. + read = await self.read_states( + owner, BOX_INBOX, [s.message_id for s in summaries] + ) + for summary in summaries: + summary.is_read = read.get(summary.message_id, False) + return summaries, total async def list_outbox( self, owner: str, filters: BoxFilterParams diff --git a/src/mail/server/src/mail_server/backends/sqlite/schema.py b/src/mail/server/src/mail_server/backends/sqlite/schema.py index 5d7212f..604eead 100644 --- a/src/mail/server/src/mail_server/backends/sqlite/schema.py +++ b/src/mail/server/src/mail_server/backends/sqlite/schema.py @@ -202,6 +202,9 @@ class MailboxItemRow(Base): box: Mapped[str] = mapped_column(String(8)) item_id: Mapped[str] = mapped_column(String(64)) entered_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + # Per-owner read state. Only meaningful for ``box == "inbox"`` (other boxes + # leave it at the default). Set ``True`` when the owner opens the message. + is_read: Mapped[bool] = mapped_column(default=False) class MessageBufferRow(Base): diff --git a/tests/conftest.py b/tests/conftest.py index c445925..7b3318d 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -58,6 +58,7 @@ def deployment_dir( "messages", "inbox_entries", "inboxes", + "read_inbox", "outbox_entries", "outboxes", "draft_entries", diff --git a/tests/integration/test_mailboxes.py b/tests/integration/test_mailboxes.py index 301cf68..15eaca5 100644 --- a/tests/integration/test_mailboxes.py +++ b/tests/integration/test_mailboxes.py @@ -11,6 +11,7 @@ USER = "user:alice@localhost" OTHER_USER = "user:bob@localhost" +AGENT = "sage@chorus@localhost" DAEMON = "daemon:dummy@localhost" @@ -65,6 +66,55 @@ def test_inbox_open_isolated_between_users( assert response.status_code == 404 +def test_inbox_delivered_message_starts_unread( + app_client: TestClient, headers_for, deliver_message +) -> None: + deliver_message(USER, [OTHER_USER]) + response = app_client.get("/inbox", headers=headers_for(OTHER_USER)) + assert response.status_code == 200 + entries = response.json()["entries"] + assert len(entries) == 1 + assert entries[0]["is_read"] is False + + +def test_inbox_open_marks_message_read( + app_client: TestClient, headers_for, deliver_message +) -> None: + message_id = deliver_message(USER, [OTHER_USER]) + + # Opening the message flips it to read. + open_response = app_client.get( + f"/inbox/{message_id}", headers=headers_for(OTHER_USER) + ) + assert open_response.status_code == 200 + + list_response = app_client.get("/inbox", headers=headers_for(OTHER_USER)) + entries = list_response.json()["entries"] + assert len(entries) == 1 + assert entries[0]["message_id"] == message_id + assert entries[0]["is_read"] is True + + +def test_inbox_read_status_is_per_owner( + app_client: TestClient, headers_for, deliver_message +) -> None: + """One recipient opening a message must not mark it read for another.""" + + message_id = deliver_message(USER, [OTHER_USER, AGENT]) + + # bob opens the message; sage never does. + app_client.get(f"/inbox/{message_id}", headers=headers_for(OTHER_USER)) + + bob_entries = app_client.get("/inbox", headers=headers_for(OTHER_USER)).json()[ + "entries" + ] + sage_entries = app_client.get("/inbox", headers=headers_for(AGENT)).json()[ + "entries" + ] + assert bob_entries[0]["is_read"] is True + assert sage_entries[0]["is_read"] is False + + # ─── Outbox ──────────────────────────────────────────────────────── From 9da55cd1432b460182938d72d40ca584384f2b0e Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Tue, 16 Jun 2026 14:39:30 -0400 Subject: [PATCH 17/28] docs: rebased v2 docs branch with main --- docs/howtos/authenticate-user-agent.md | 16 +++++++++++++++ docs/howtos/initialize-memory-backend.md | 15 ++++++++++++++ docs/howtos/manage-mailing-lists.md | 17 ++++++++++++++++ docs/howtos/manage-swarms.md | 16 +++++++++++++++ docs/tutorials/build-minimal-http-client.md | 17 ++++++++++++++++ docs/tutorials/run-local-mail.md | 22 +++++++++++++++++++++ docs/tutorials/send-first-message.md | 19 ++++++++++++++++++ 7 files changed, 122 insertions(+) diff --git a/docs/howtos/authenticate-user-agent.md b/docs/howtos/authenticate-user-agent.md index 22f9c82..54fcb2a 100644 --- a/docs/howtos/authenticate-user-agent.md +++ b/docs/howtos/authenticate-user-agent.md @@ -69,3 +69,19 @@ To obtain a fresh access token with your credentials, repeat steps 1-2. - `src/mail/client/src/mail_client/commands/whoami.py` - `src/mail/server/src/mail_server/routers/auth.py` - `spec/openapi.yaml` +<<<<<<< HEAD +======= + +## Steps to Cover + +1. Set `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. +2. Run `mail login`. +3. Store the returned token in `MAIL_TOKEN`. +4. Run `mail whoami`. +5. Use the token in an HTTP `Authorization: Bearer ...` header. +6. Refresh or replace expired tokens. + +## Validation + +`mail whoami` returns the expected user-agent type and address. +>>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/howtos/initialize-memory-backend.md b/docs/howtos/initialize-memory-backend.md index 8adc8f2..c233475 100644 --- a/docs/howtos/initialize-memory-backend.md +++ b/docs/howtos/initialize-memory-backend.md @@ -120,3 +120,18 @@ TODO - `src/mail/server/src/mail_server/backend_init.py` - `src/mail/server/src/mail_server/backends/memory/init.py` - `src/mail/server/docs/tutorials/quickstart.md` +<<<<<<< HEAD +======= + +## Steps to Cover + +1. Run `uv run backend-init`. +2. Customize deployment, swarm, host, agents, daemons, users, or admins. +3. Locate generated credential files. +4. Remove or protect plaintext password files after capture. +5. Reinitialize clean state when needed. + +## Validation + +The server starts successfully and the generated user-agents can authenticate. +>>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/howtos/manage-mailing-lists.md b/docs/howtos/manage-mailing-lists.md index 260be13..2dada11 100644 --- a/docs/howtos/manage-mailing-lists.md +++ b/docs/howtos/manage-mailing-lists.md @@ -193,3 +193,20 @@ the field placement on the wire. - `src/mail/client/src/mail_client/commands/list_member_post.py` - `src/mail/server/src/mail_server/routers/lists.py` - `src/mail/protocol/src/mail_protocol/core/lists.py` +<<<<<<< HEAD +======= + +## Steps to Cover + +1. List available mailing lists. +2. Inspect one list by address. +3. Create a list as an admin. +4. Subscribe and unsubscribe as a user-agent. +5. Add or remove members as an admin. +6. Send a message to a list address. + +## Validation + +List membership changes are reflected in list lookup and list-address sends +deliver to expected recipients. +>>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/howtos/manage-swarms.md b/docs/howtos/manage-swarms.md index b3d1205..1a2ce72 100644 --- a/docs/howtos/manage-swarms.md +++ b/docs/howtos/manage-swarms.md @@ -78,3 +78,19 @@ If this operation was successful, information on the newly-deleted swarm will be - `src/mail/client/src/mail_client/commands/swarm_delete.py` - `src/mail/server/src/mail_server/routers/swarms.py` - `src/mail/server/src/mail_server/routers/admin.py` +<<<<<<< HEAD +======= + +## Steps to Cover + +1. List swarms. +2. Inspect a swarm by name. +3. Create a swarm as an admin. +4. Add initial description, keywords, and agents. +5. Delete a test swarm. + +## Validation + +The created swarm is visible through public swarm lookup and removable through +admin commands. +>>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/tutorials/build-minimal-http-client.md b/docs/tutorials/build-minimal-http-client.md index bae6ed9..c0a498e 100644 --- a/docs/tutorials/build-minimal-http-client.md +++ b/docs/tutorials/build-minimal-http-client.md @@ -296,3 +296,20 @@ the CLI. - `src/mail/protocol/src/mail_protocol/network/responses.py` - `src/mail/server/src/mail_server/routers/auth.py` - `src/mail/server/src/mail_server/routers/drafts.py` +<<<<<<< HEAD +======= + +## Draft Outline + +1. Start from a known server URL and credentials. +2. Obtain a bearer token from `POST /auth/token`. +3. Validate identity with `GET /auth/whoami`. +4. Create a draft with `POST /drafts/`. +5. Send the draft with `POST /drafts/{draft_id}/send`. +6. Parse success and validation errors. + +## Not Here + +- Full endpoint listings belong in [HTTP API](../references/http-api.md). +- Protocol motivation belongs in [MAIL v2 Overview](../explanations/mail-v2-overview.md). +>>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/tutorials/run-local-mail.md b/docs/tutorials/run-local-mail.md index 4f42d14..2484fcd 100644 --- a/docs/tutorials/run-local-mail.md +++ b/docs/tutorials/run-local-mail.md @@ -13,6 +13,7 @@ the message was delivered. New contributors and first-time users who have cloned the repository and want a working local loop before reading deeper docs. +<<<<<<< HEAD ## Not Here - Exhaustive command flags belong in reference pages. @@ -163,6 +164,8 @@ uv run mail outbox-open {message_id} Like in the `agent`'s inbox, you should see the message contents as you composed them, with a subject of "Test subject" and a body of "This is a message body". +======= +>>>>>>> 4bed686 (docs: rebased v2 docs branch with main) ## Source Material - `README.md` @@ -171,3 +174,22 @@ Like in the `agent`'s inbox, you should see the message contents as you composed - `src/mail/server/.env.example` - `src/mail/server/src/mail_server/backend_init.py` - `src/mail/daemon/src/mail_daemon/maild/api.py` +<<<<<<< HEAD +======= + +## Draft Outline + +1. Install workspace dependencies with `uv sync`. +2. Configure the server environment. +3. Initialize the memory backend with `backend-init`. +4. Start `mail-server`. +5. Log in as a sender and recipient with `mail login`. +6. Start `mail-daemon` with daemon credentials. +7. Compose and send a message. +8. Open inbox and outbox entries to confirm delivery. + +## Not Here + +- Exhaustive command flags belong in reference pages. +- Deployment hardening belongs in how-to guides and explanations. +>>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/tutorials/send-first-message.md b/docs/tutorials/send-first-message.md index 188969b..2d22757 100644 --- a/docs/tutorials/send-first-message.md +++ b/docs/tutorials/send-first-message.md @@ -145,3 +145,22 @@ Delivered By: daemon:{daemon_name}@example.com - `src/mail/client/src/mail_client/commands/send.py` - `src/mail/client/src/mail_client/commands/inbox_open.py` - `src/mail/client/src/mail_client/commands/outbox_open.py` +<<<<<<< HEAD +======= + +## Draft Outline + +1. Verify `mail --help` works. +2. Log in with `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. +3. Store the returned token in `MAIL_TOKEN`. +4. Run `mail whoami`. +5. Compose a draft. +6. Send the draft to one recipient. +7. Open the outbox message. +8. If testing with a second account, open the recipient inbox. + +## Not Here + +- Admin account creation belongs in [Manage User-Agents](../howtos/manage-user-agents.md). +- Complete CLI option tables belong in [Client CLI](../references/client-cli.md). +>>>>>>> 4bed686 (docs: rebased v2 docs branch with main) From 6b28141e38ae5001cf6a99a10e72f4b0c90e69c2 Mon Sep 17 00:00:00 2001 From: minichorus-pm Date: Tue, 23 Jun 2026 16:18:18 -0400 Subject: [PATCH 18/28] docs: webhook delivery contract, manage-webhooks how-to, and build-webhook-receiver tutorial MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three new pages plus three index updates, addressing the webhook documentation gap in the v2 docs branch (the first of the two docs offers from the dev-list thread on Final Prep for v2). The conceptual contract. What webhooks are for, the payload shape (with v2's reply_to and tags from PR #74), the HMAC-SHA256 scheme (timestamp.body, not body.timestamp — bug-shape worth flagging explicitly), the headers MAIL sends, the receiver verification checklist, the retry ladder (6 attempts: immediate, +1s, +30s, +5min, +1h, +6h — total window ~7h31m), the retry conditions (timeout / 5xx / 429 retry; 2xx / other 4xx don't), and the 'inbox is source of truth' contract that shapes how receivers should handle internal failures. Sources verified against src/mail/server/src/mail_server/backends/base.py (_handle_webhook_delivered and _webhook_delivered_post on origin main; the v2-docs branch is currently behind main on the schema changes from PR #74). Wrote against the canonical main-branch shape so the docs match the v2 release contract; the docs branch needs the rebase before merge. Operator's guide. Secret generation, POST /admin/webhooks (with events and url), GET /admin/webhooks for listing, GET /admin/webhooks/{id} for inspection, PATCH /admin/webhooks/{id} for URL / secret rotation (event types are immutable in v2), DELETE /admin/webhooks/{id}. Includes the rotation coordination note (both sides must update at the same moment to avoid signature failures in flight). Implementer's walkthrough. A single-file FastAPI receiver with: - verify_signature using HMAC over raw bytes (calls out the two most common bugs: re-encoded JSON breaking signature; missing the timestamp.body prefix). - is_duplicate / mark_processed for event_id-based dedup with a 24-hour garbage-collection window. - is_timestamp_in_window for 5-min skew rejection. - The full endpoint composing them with the right error codes (503 if secret not configured, 408 for skew, 403 for bad signature, 200 with status=duplicate for retries). - Registration command for end-to-end test. - Diagnostic checklist for the 'nothing arrives' case. docs/{explanations,howtos,tutorials}/README.md each gain a row linking to the new page. - docs/howtos/manage-mailing-lists.md exists as a stub today; drafting that one is the second piece of the offer and will land as a separate commit on the same branch. - docs/references/http-api.md is a stub overall (not just for webhooks). The webhook-specific endpoints could be sketched there in a later pass; deferred so this commit stays focused on the conceptual + tutorial layer. The four facts that are easiest to get wrong (and that I had ground truth on from the chorus-side webhook receiver): - HMAC inputs: 'timestamp.raw_body' not 'raw_body.timestamp'. - X-MAIL-Timestamp value: Unix seconds as a STRING, used in both the HMAC and the header so receivers can recompute from the header alone. - Signature header format: 'sha256='. - Body bytes: payload.model_dump_json() (Pydantic's canonical JSON), posted as-is — re-encoded JSON has different bytes and breaks verification. All four match what _webhook_delivered_post does in src/mail/server/src/mail_server/backends/base.py:653. --- docs/explanations/README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/explanations/README.md b/docs/explanations/README.md index 7b08513..86c2de7 100644 --- a/docs/explanations/README.md +++ b/docs/explanations/README.md @@ -12,8 +12,15 @@ They are for understanding, not for step-by-step tasks or exhaustive lookup. | [Addressing Model](addressing-model.md) | Why does MAIL use host-scoped and swarm-scoped addresses? | | [Delivery Model](delivery-model.md) | Why are daemons responsible for message delivery? | | [Security Model](security-model.md) | What are the main trust boundaries and risks? | +<<<<<<< HEAD | [Mailing Lists](mailing-lists.md) | What is a list? How does it expand, what does its policy mean, and how do admin and user-agent permissions split? | +<<<<<<< HEAD | [Webhook Delivery](webhook-delivery.md) | What is the webhook contract — payload shape, HMAC signing, retry behavior, and the inbox-is-source-of-truth assumption? | +======= +======= +| [Webhook Delivery](webhook-delivery.md) | What is the webhook contract — payload shape, HMAC signing, retry behavior, and the inbox-is-source-of-truth assumption? | +>>>>>>> 54e649b (docs: webhook delivery contract, manage-webhooks how-to, and build-webhook-receiver tutorial) +>>>>>>> 5e5b5c5 (docs: webhook delivery contract, manage-webhooks how-to, and build-webhook-receiver tutorial) | [MAIL v1 Legacy Runtime](mail-v1-legacy.md) | How should readers interpret the archived v1 runtime and docs? | | [Documentation System](documentation-system.md) | How should maintainers decide where a new page belongs? | From 6e8b09c2d8c5b09451199aaf15e41ea306e7c496 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 15:27:26 -0400 Subject: [PATCH 19/28] fix: resolve merge-conflict markers left in docs after v2 rebase The v2-docs rebase (9da55cd) committed unresolved conflict markers into eight doc files. Resolve each by keeping the finished (HEAD) content and dropping the leftover pre-writing scaffolding (Draft Outline / Steps to Cover / Validation) and duplicated Not Here blocks. No written content is lost; the run-local-mail tutorial body was wrapped inside a HEAD block. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanations/README.md | 7 ------- docs/howtos/authenticate-user-agent.md | 16 --------------- docs/howtos/initialize-memory-backend.md | 15 -------------- docs/howtos/manage-mailing-lists.md | 17 ---------------- docs/howtos/manage-swarms.md | 16 --------------- docs/tutorials/build-minimal-http-client.md | 17 ---------------- docs/tutorials/run-local-mail.md | 22 --------------------- docs/tutorials/send-first-message.md | 19 ------------------ 8 files changed, 129 deletions(-) diff --git a/docs/explanations/README.md b/docs/explanations/README.md index 86c2de7..7b08513 100644 --- a/docs/explanations/README.md +++ b/docs/explanations/README.md @@ -12,15 +12,8 @@ They are for understanding, not for step-by-step tasks or exhaustive lookup. | [Addressing Model](addressing-model.md) | Why does MAIL use host-scoped and swarm-scoped addresses? | | [Delivery Model](delivery-model.md) | Why are daemons responsible for message delivery? | | [Security Model](security-model.md) | What are the main trust boundaries and risks? | -<<<<<<< HEAD | [Mailing Lists](mailing-lists.md) | What is a list? How does it expand, what does its policy mean, and how do admin and user-agent permissions split? | -<<<<<<< HEAD | [Webhook Delivery](webhook-delivery.md) | What is the webhook contract — payload shape, HMAC signing, retry behavior, and the inbox-is-source-of-truth assumption? | -======= -======= -| [Webhook Delivery](webhook-delivery.md) | What is the webhook contract — payload shape, HMAC signing, retry behavior, and the inbox-is-source-of-truth assumption? | ->>>>>>> 54e649b (docs: webhook delivery contract, manage-webhooks how-to, and build-webhook-receiver tutorial) ->>>>>>> 5e5b5c5 (docs: webhook delivery contract, manage-webhooks how-to, and build-webhook-receiver tutorial) | [MAIL v1 Legacy Runtime](mail-v1-legacy.md) | How should readers interpret the archived v1 runtime and docs? | | [Documentation System](documentation-system.md) | How should maintainers decide where a new page belongs? | diff --git a/docs/howtos/authenticate-user-agent.md b/docs/howtos/authenticate-user-agent.md index 54fcb2a..22f9c82 100644 --- a/docs/howtos/authenticate-user-agent.md +++ b/docs/howtos/authenticate-user-agent.md @@ -69,19 +69,3 @@ To obtain a fresh access token with your credentials, repeat steps 1-2. - `src/mail/client/src/mail_client/commands/whoami.py` - `src/mail/server/src/mail_server/routers/auth.py` - `spec/openapi.yaml` -<<<<<<< HEAD -======= - -## Steps to Cover - -1. Set `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. -2. Run `mail login`. -3. Store the returned token in `MAIL_TOKEN`. -4. Run `mail whoami`. -5. Use the token in an HTTP `Authorization: Bearer ...` header. -6. Refresh or replace expired tokens. - -## Validation - -`mail whoami` returns the expected user-agent type and address. ->>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/howtos/initialize-memory-backend.md b/docs/howtos/initialize-memory-backend.md index c233475..8adc8f2 100644 --- a/docs/howtos/initialize-memory-backend.md +++ b/docs/howtos/initialize-memory-backend.md @@ -120,18 +120,3 @@ TODO - `src/mail/server/src/mail_server/backend_init.py` - `src/mail/server/src/mail_server/backends/memory/init.py` - `src/mail/server/docs/tutorials/quickstart.md` -<<<<<<< HEAD -======= - -## Steps to Cover - -1. Run `uv run backend-init`. -2. Customize deployment, swarm, host, agents, daemons, users, or admins. -3. Locate generated credential files. -4. Remove or protect plaintext password files after capture. -5. Reinitialize clean state when needed. - -## Validation - -The server starts successfully and the generated user-agents can authenticate. ->>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/howtos/manage-mailing-lists.md b/docs/howtos/manage-mailing-lists.md index 2dada11..260be13 100644 --- a/docs/howtos/manage-mailing-lists.md +++ b/docs/howtos/manage-mailing-lists.md @@ -193,20 +193,3 @@ the field placement on the wire. - `src/mail/client/src/mail_client/commands/list_member_post.py` - `src/mail/server/src/mail_server/routers/lists.py` - `src/mail/protocol/src/mail_protocol/core/lists.py` -<<<<<<< HEAD -======= - -## Steps to Cover - -1. List available mailing lists. -2. Inspect one list by address. -3. Create a list as an admin. -4. Subscribe and unsubscribe as a user-agent. -5. Add or remove members as an admin. -6. Send a message to a list address. - -## Validation - -List membership changes are reflected in list lookup and list-address sends -deliver to expected recipients. ->>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/howtos/manage-swarms.md b/docs/howtos/manage-swarms.md index 1a2ce72..b3d1205 100644 --- a/docs/howtos/manage-swarms.md +++ b/docs/howtos/manage-swarms.md @@ -78,19 +78,3 @@ If this operation was successful, information on the newly-deleted swarm will be - `src/mail/client/src/mail_client/commands/swarm_delete.py` - `src/mail/server/src/mail_server/routers/swarms.py` - `src/mail/server/src/mail_server/routers/admin.py` -<<<<<<< HEAD -======= - -## Steps to Cover - -1. List swarms. -2. Inspect a swarm by name. -3. Create a swarm as an admin. -4. Add initial description, keywords, and agents. -5. Delete a test swarm. - -## Validation - -The created swarm is visible through public swarm lookup and removable through -admin commands. ->>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/tutorials/build-minimal-http-client.md b/docs/tutorials/build-minimal-http-client.md index c0a498e..bae6ed9 100644 --- a/docs/tutorials/build-minimal-http-client.md +++ b/docs/tutorials/build-minimal-http-client.md @@ -296,20 +296,3 @@ the CLI. - `src/mail/protocol/src/mail_protocol/network/responses.py` - `src/mail/server/src/mail_server/routers/auth.py` - `src/mail/server/src/mail_server/routers/drafts.py` -<<<<<<< HEAD -======= - -## Draft Outline - -1. Start from a known server URL and credentials. -2. Obtain a bearer token from `POST /auth/token`. -3. Validate identity with `GET /auth/whoami`. -4. Create a draft with `POST /drafts/`. -5. Send the draft with `POST /drafts/{draft_id}/send`. -6. Parse success and validation errors. - -## Not Here - -- Full endpoint listings belong in [HTTP API](../references/http-api.md). -- Protocol motivation belongs in [MAIL v2 Overview](../explanations/mail-v2-overview.md). ->>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/tutorials/run-local-mail.md b/docs/tutorials/run-local-mail.md index 2484fcd..4f42d14 100644 --- a/docs/tutorials/run-local-mail.md +++ b/docs/tutorials/run-local-mail.md @@ -13,7 +13,6 @@ the message was delivered. New contributors and first-time users who have cloned the repository and want a working local loop before reading deeper docs. -<<<<<<< HEAD ## Not Here - Exhaustive command flags belong in reference pages. @@ -164,8 +163,6 @@ uv run mail outbox-open {message_id} Like in the `agent`'s inbox, you should see the message contents as you composed them, with a subject of "Test subject" and a body of "This is a message body". -======= ->>>>>>> 4bed686 (docs: rebased v2 docs branch with main) ## Source Material - `README.md` @@ -174,22 +171,3 @@ Like in the `agent`'s inbox, you should see the message contents as you composed - `src/mail/server/.env.example` - `src/mail/server/src/mail_server/backend_init.py` - `src/mail/daemon/src/mail_daemon/maild/api.py` -<<<<<<< HEAD -======= - -## Draft Outline - -1. Install workspace dependencies with `uv sync`. -2. Configure the server environment. -3. Initialize the memory backend with `backend-init`. -4. Start `mail-server`. -5. Log in as a sender and recipient with `mail login`. -6. Start `mail-daemon` with daemon credentials. -7. Compose and send a message. -8. Open inbox and outbox entries to confirm delivery. - -## Not Here - -- Exhaustive command flags belong in reference pages. -- Deployment hardening belongs in how-to guides and explanations. ->>>>>>> 4bed686 (docs: rebased v2 docs branch with main) diff --git a/docs/tutorials/send-first-message.md b/docs/tutorials/send-first-message.md index 2d22757..188969b 100644 --- a/docs/tutorials/send-first-message.md +++ b/docs/tutorials/send-first-message.md @@ -145,22 +145,3 @@ Delivered By: daemon:{daemon_name}@example.com - `src/mail/client/src/mail_client/commands/send.py` - `src/mail/client/src/mail_client/commands/inbox_open.py` - `src/mail/client/src/mail_client/commands/outbox_open.py` -<<<<<<< HEAD -======= - -## Draft Outline - -1. Verify `mail --help` works. -2. Log in with `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. -3. Store the returned token in `MAIL_TOKEN`. -4. Run `mail whoami`. -5. Compose a draft. -6. Send the draft to one recipient. -7. Open the outbox message. -8. If testing with a second account, open the recipient inbox. - -## Not Here - -- Admin account creation belongs in [Manage User-Agents](../howtos/manage-user-agents.md). -- Complete CLI option tables belong in [Client CLI](../references/client-cli.md). ->>>>>>> 4bed686 (docs: rebased v2 docs branch with main) From 9d357ddaab0a2a639c33a8b19c00d0b558992c8f Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 15:31:01 -0400 Subject: [PATCH 20/28] docs: correct drift in written v2 docs against current code Fix seven concrete mismatches found cross-checking docs against source: - manage-swarms: command is 'swarm-list', not 'swarms-list' - manage-mailing-lists: 'list-patch' CLI verb is a stub that raises NotImplementedError; document the PATCH /admin/lists endpoint instead - manage-mailing-lists: management commands take the local name@swarm form, only sends use the full list:name@swarm@host recipient form - authenticate-user-agent: document the mail refresh / MAIL_REFRESH_TOKEN flow for interactive principals instead of re-login only - addressing-model: identifier hard cap is 31, not 32 - delivery-model: /daemon/deliver/remote route is wired but its handler raises NotImplementedError; say 'not yet functional', not 'not implemented' - build-minimal-http-client: token response includes refresh_token and expires_in; send response includes mail_version, reply_to, and tags Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanations/addressing-model.md | 2 +- docs/explanations/delivery-model.md | 5 +- docs/howtos/authenticate-user-agent.md | 14 ++++- docs/howtos/manage-mailing-lists.md | 57 ++++++++++++++------- docs/howtos/manage-swarms.md | 4 +- docs/tutorials/build-minimal-http-client.md | 11 +++- 6 files changed, 67 insertions(+), 26 deletions(-) diff --git a/docs/explanations/addressing-model.md b/docs/explanations/addressing-model.md index ddbb3bf..1f6f7d4 100644 --- a/docs/explanations/addressing-model.md +++ b/docs/explanations/addressing-model.md @@ -112,7 +112,7 @@ Identifiers in MAIL — every `ua_id`, `agent`, `swarm`, and `list_id` — must So `welfare-discourse` is valid; `Welfare_Discourse`, `-leading`, `trailing-`, and `double--hyphen` are not. Each identifier must be at least one character and -SHOULD be at most 32 (the reference implementation enforces 32 as a hard cap). +SHOULD be at most 31 (the reference implementation enforces 31 as a hard cap). The `host` segment must be a valid domain name — or, in the reference implementation, an IP address. diff --git a/docs/explanations/delivery-model.md b/docs/explanations/delivery-model.md index b2fbc67..cbac982 100644 --- a/docs/explanations/delivery-model.md +++ b/docs/explanations/delivery-model.md @@ -104,8 +104,9 @@ notified of new mail rather than having to poll; see [HTTP API](../references/ht The implemented delivery path is **local**: `POST /daemon/deliver/local` carries messages between user-agents on the *same* server. A second endpoint, `POST /daemon/deliver/remote`, is reserved for future delivery of messages that -arrive from other MAIL servers; it is not yet implemented. For now, treat -delivery as a within-host operation. +arrive from other MAIL servers. The route is wired, but its backend handler +currently raises `NotImplementedError`, so remote delivery is not yet +functional. For now, treat delivery as a within-host operation. ## Pre-send versus post-send errors diff --git a/docs/howtos/authenticate-user-agent.md b/docs/howtos/authenticate-user-agent.md index 22f9c82..ecca026 100644 --- a/docs/howtos/authenticate-user-agent.md +++ b/docs/howtos/authenticate-user-agent.md @@ -38,6 +38,8 @@ Store this token as an environment variable called `MAIL_TOKEN`: MAIL_TOKEN={token} ``` +If you logged in as a **user** or **admin** (an *interactive principal*), `login` also prints a **refresh token**. Store it as `MAIL_REFRESH_TOKEN` so you can renew your access token later without re-entering your password (see step 6). Agents and daemons are not issued refresh tokens and re-authenticate with their credentials instead. + ### 4. Run `mail whoami` With your `MAIL_SERVER` and `MAIL_TOKEN` environment variables set, you can now view your own user-agent information using the `whoami` command: @@ -61,7 +63,17 @@ curl {server_url}/auth/whoami \ Server-issued access tokens will expire after a predetermined length of time (e.g. 15, 30, or 60 minutes). If you attempt to hit a MAIL server endpoint with a previously-valid token and get a `401` response, that likely means your token has expired. -To obtain a fresh access token with your credentials, repeat steps 1-2. +If you saved a refresh token in step 3 (users and admins only), renew your access token without re-entering credentials by running `mail refresh` with `MAIL_SERVER` and `MAIL_REFRESH_TOKEN` set: + +```bash +MAIL_SERVER={server_url} +MAIL_REFRESH_TOKEN={refresh_token} +uv run mail refresh +``` + +Refresh tokens are **rotated**: each `mail refresh` invalidates the token you sent and prints a replacement, so update `MAIL_REFRESH_TOKEN` with the new value every time. Store the new access token in `MAIL_TOKEN` as in step 3. + +Agents and daemons are not issued refresh tokens; they obtain a fresh access token by logging in again (repeat steps 1-2). ## Source Material diff --git a/docs/howtos/manage-mailing-lists.md b/docs/howtos/manage-mailing-lists.md index 260be13..dc8a70c 100644 --- a/docs/howtos/manage-mailing-lists.md +++ b/docs/howtos/manage-mailing-lists.md @@ -17,11 +17,22 @@ action. Admin credentials are required for create / patch / member add-remove / delete; user-agent credentials are sufficient for read / subscribe / unsubscribe / send. -The list address shape is `list:@@` — see -[Addressing Model](../explanations/addressing-model.md). Anywhere -this document writes `{list_address}`, it expects the full -`list:`-prefixed form (e.g., -`list:announcements@chorus@example.com`). +Two address forms appear below, and the CLI is strict about which it +wants — see [Addressing Model](../explanations/addressing-model.md): + +- **List-management commands** (`list-get`, `list-subscribe`, + `list-unsubscribe`, `list-member-post`, `list-member-delete`, + `list-delete`) take the **local** `{name}@{swarm}` form (e.g. + `announcements@chorus`). The server resolves it to the list's + canonical `list:{name}@{swarm}@{host}` address using its own host. +- **Sending to a list** (step 8) names the list as a message + recipient, so it uses the **full** routable form + `list:{name}@{swarm}@{host}` (e.g. + `list:announcements@chorus@example.com`), just like any other + recipient. + +This how-to writes `{list_address}` for the local management form and +`{list_recipient}` for the full send form. ## Steps @@ -114,18 +125,24 @@ If successful, details on the mailing list will be printed to the console. ### 6. Update list policy as an admin -Admins can update the policy on an existing list with the -`mail-admin` command `list-patch`. The list's canonical address -(`name`, `swarm`, `host`) is immutable; only the policy fields can -change. +Policy is the only mutable part of a list — the canonical address +(`name`, `swarm`, `host`) is immutable. The server exposes +`PATCH /admin/lists/{name}@{swarm}` accepting an `AdminListPatchRequest` +body (a single optional `policy` object). + +> **Not yet available from the CLI.** The `mail-admin list-patch` +> command is registered but currently a stub: it declares no +> arguments and its handler raises `NotImplementedError` +> (`src/mail/client/src/mail_client/commands/list_patch.py`). Until it +> is implemented, patch a list's policy by calling the endpoint +> directly — for example with `curl` (the path takes the **local** +> `{name}@{swarm}` form): ```bash -MAIL_SERVER={server_url} -MAIL_TOKEN={admin_jwt} -uv run mail-admin list-patch {list_address} \ - --visibility public \ - --join-policy open \ - --send-policy open +curl -s -X PATCH "$MAIL_SERVER/admin/lists/announcements@chorus" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"policy":{"visibility":"public","join_policy":"open","send_policy":"open"}}' ``` For v1, only `public` / `open` / `open` are honored; the other @@ -154,14 +171,16 @@ flow through a list](../explanations/mailing-lists.md#how-messages-flow-through- ### 8. Send a message to a list address -If they are authorized to do so, user-agents can send a message to -a list address by specifying the list address in the `mail` command -`send`: +If they are authorized to do so, user-agents can send a message to a +list by naming the list as a recipient of the `mail` command `send`. +Because this is a message recipient (not a management target), use the +**full** routable form `list:{name}@{swarm}@{host}` here — written +`{list_recipient}` below: ```bash MAIL_SERVER={server_url} MAIL_TOKEN={ua_jwt} -uv run mail send {draft_id} {list_address} +uv run mail send {draft_id} {list_recipient} ``` The receiving server expands the list and delivers one copy to diff --git a/docs/howtos/manage-swarms.md b/docs/howtos/manage-swarms.md index b3d1205..dd1ef2e 100644 --- a/docs/howtos/manage-swarms.md +++ b/docs/howtos/manage-swarms.md @@ -15,12 +15,12 @@ read-only swarm inspection. ### 1. List swarms -Authorized user-agents can list all swarms on a MAIL server via the `mail` CLI command `swarms-list`: +Authorized user-agents can list all swarms on a MAIL server via the `mail` CLI command `swarm-list` (aliases: `swarms`, `sl`): ```bash MAIL_SERVER={server_url} MAIL_TOKEN={ua_jwt} -uv run mail swarms-list +uv run mail swarm-list ``` This will print the name, keywords, and number of agents for each swarm on the server. diff --git a/docs/tutorials/build-minimal-http-client.md b/docs/tutorials/build-minimal-http-client.md index bae6ed9..30e897c 100644 --- a/docs/tutorials/build-minimal-http-client.md +++ b/docs/tutorials/build-minimal-http-client.md @@ -69,9 +69,15 @@ curl -s -X POST "$MAIL_SERVER/auth/token" \ The response carries a JSON Web Token: ```json -{"access_token":"eyJhbGciOiJIUzI1NiIs...","token_type":"bearer","metadata":{}} +{"access_token":"eyJhbGciOiJIUzI1NiIs...","token_type":"bearer","refresh_token":"...","expires_in":1800,"metadata":{}} ``` +`expires_in` is the access-token lifetime in seconds. For an *interactive +principal* — a user or admin, as with the `admin` account used here — +`refresh_token` is populated (agents and daemons receive `refresh_token: null` +and re-authenticate with their credentials). This tutorial only needs +`access_token`; renewing via the refresh token is out of scope here. + Capture the token so the authenticated calls can reuse it: ```bash @@ -173,11 +179,14 @@ The response is the assembled message. It has a `message_id` distinct from the ```json { "message": { + "mail_version": "2.0", "message_id": "5a2c1b9d-7e3f-4c8a-9d21-0b4e6f8a1c2d", + "reply_to": null, "sender": "admin:dummy@localhost", "recipients": ["supervisor@default@localhost"], "subject": "Hello over HTTP", "body": "My first MAIL message, sent with curl.", + "tags": [], "sent_at": "2026-06-16T23:11:05Z", "metadata": {} }, From d0a7fd3f6278aff29108f11162cfca052dd228e5 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 15:34:08 -0400 Subject: [PATCH 21/28] docs: fix index omissions and finish memory-backend how-to - docs/README.md 'Proposed Layout' now lists the pages that already exist: Build a Webhook Receiver (tutorials), Manage Webhooks (how-tos), and Mailing Lists + Webhook Delivery (explanations). - initialize-memory-backend: write the previously-TODO 'reinitialize a clean slate' step (delete the deployment dir under ~/.mail-swarms and re-run backend-init) and flip its stale 'Status: stub' to 'draft'. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/README.md | 4 ++++ docs/howtos/initialize-memory-backend.md | 23 ++++++++++++++++++++--- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/docs/README.md b/docs/README.md index 4167169..9c58a6d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -29,6 +29,7 @@ to exactly one of four categories. - [Run MAIL Locally](tutorials/run-local-mail.md) - [Send Your First MAIL Message](tutorials/send-first-message.md) - [Build a Minimal HTTP Client](tutorials/build-minimal-http-client.md) +- [Build a Webhook Receiver](tutorials/build-webhook-receiver.md) ### How-To Guides @@ -40,6 +41,7 @@ to exactly one of four categories. - [Manage User-Agents](howtos/manage-user-agents.md) - [Manage Swarms](howtos/manage-swarms.md) - [Manage Mailing Lists](howtos/manage-mailing-lists.md) +- [Manage Webhooks](howtos/manage-webhooks.md) - [Regenerate API Artifacts](howtos/regenerate-api-artifacts.md) - [Run the Test Suite](howtos/run-tests.md) @@ -63,6 +65,8 @@ to exactly one of four categories. - [Addressing Model](explanations/addressing-model.md) - [Delivery Model](explanations/delivery-model.md) - [Security Model](explanations/security-model.md) +- [Mailing Lists](explanations/mailing-lists.md) +- [Webhook Delivery](explanations/webhook-delivery.md) - [MAIL v1 Legacy Runtime](explanations/mail-v1-legacy.md) - [Documentation System](explanations/documentation-system.md) diff --git a/docs/howtos/initialize-memory-backend.md b/docs/howtos/initialize-memory-backend.md index 8adc8f2..7a82a0a 100644 --- a/docs/howtos/initialize-memory-backend.md +++ b/docs/howtos/initialize-memory-backend.md @@ -1,6 +1,6 @@ # Initialize the Memory Backend -Status: stub +Status: draft ## Goal @@ -111,9 +111,26 @@ rm -rf ~/.mail-swarms/deployments/{deployment}/.secrets where `deployment` is the name chosen for your deployment. -### 5. Reinitialize clean slate when needed +### 5. Reinitialize a clean slate when needed -TODO +All state for a deployment lives under a single directory, +`~/.mail-swarms/deployments/{deployment}/`, which holds the `swarms/`, +`user_agents/`, and `messages/` stores plus the `message_buffer.lock` file and +the `.secrets/` folder from step 3. Re-running `backend-init` against an existing +deployment reuses that directory rather than clearing it, so for a guaranteed +clean slate — a fresh swarm, user-agents, and credentials — stop any running +`mail-server`, remove the deployment directory, and initialize again: + +```bash +# stop mail-server first, then: +rm -rf ~/.mail-swarms/deployments/{deployment} +uv run backend-init --deployment "{deployment}" +``` + +where `deployment` is the name chosen for your deployment. This regenerates the +swarm, user-agents, and fresh plaintext credentials — capture and protect them +again as in steps 3-4. To reset *every* deployment at once, remove the whole +`~/.mail-swarms/deployments` directory instead. ## Source Material From 63738c1b303233c6522692dc29459a51a4e2158f Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 15:43:19 -0400 Subject: [PATCH 22/28] docs: write hand-authored reference pages (P2, references category) Fill in six reference stubs against verified source: - repository-layout: workspace/package map, scripts, tests, legacy - protocol-specification: SPEC.md + openapi roles, section map, contract tests, and the known spec-vs-impl identifier-length divergence (31 vs 32) - http-api: full route inventory with auth levels + status codes - data-models: protocol model field tables, constants, validators - storage-backends: backend contract, memory vs sqlite, backend-init - configuration: exhaustive env var + CLI flag inventory Also refine addressing-model to distinguish the spec's SHOULD-32 from the reference implementation's hard cap of 31. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanations/addressing-model.md | 6 +- docs/references/configuration.md | 118 ++++++++-- docs/references/data-models.md | 271 ++++++++++++++++++++-- docs/references/http-api.md | 180 ++++++++++++-- docs/references/protocol-specification.md | 95 ++++++-- docs/references/repository-layout.md | 150 ++++++++++-- docs/references/storage-backends.md | 153 ++++++++++-- 7 files changed, 842 insertions(+), 131 deletions(-) diff --git a/docs/explanations/addressing-model.md b/docs/explanations/addressing-model.md index 1f6f7d4..13019aa 100644 --- a/docs/explanations/addressing-model.md +++ b/docs/explanations/addressing-model.md @@ -111,8 +111,10 @@ Identifiers in MAIL — every `ua_id`, `agent`, `swarm`, and `list_id` — must ``` So `welfare-discourse` is valid; `Welfare_Discourse`, `-leading`, `trailing-`, -and `double--hyphen` are not. Each identifier must be at least one character and -SHOULD be at most 31 (the reference implementation enforces 31 as a hard cap). +and `double--hyphen` are not. Each identifier must be at least one character. The +[protocol spec](../references/protocol-specification.md) recommends a maximum of +32 characters, but the reference implementation enforces a hard cap of 31 — +longer identifiers are rejected with a validation error. The `host` segment must be a valid domain name — or, in the reference implementation, an IP address. diff --git a/docs/references/configuration.md b/docs/references/configuration.md index 614c875..3c9df0d 100644 --- a/docs/references/configuration.md +++ b/docs/references/configuration.md @@ -1,30 +1,108 @@ # Configuration -Status: stub +Status: draft -## Scope +This page lists every environment variable and CLI flag across the four MAIL v2 +packages, with defaults and whether each is required. Default host/port come from +[`mail_protocol.constants`](../../src/mail/protocol/src/mail_protocol/constants.py): +`MAIL_DEFAULT_HOST = 127.0.0.1`, `MAIL_DEFAULT_PORT = 8865`. -List environment variables, CLI defaults, and runtime settings for active MAIL -v2 packages. +## Server (`mail-server`) -## Source of Truth +### Required environment variables -- `src/mail/server/.env.example` -- `src/mail/server/src/mail_server/cli.py` -- `src/mail/server/src/mail_server/server.py` -- `src/mail/daemon/src/mail_daemon/maild/api.py` -- `src/mail/client/src/mail_client/commands/` +These are read at import/startup — the server process fails to boot (raising +`RuntimeError`) if any is unset. -## Entries to Cover +| Variable | Effect | +| --- | --- | +| `MAIL_HOST` | Canonical host identity for the deployment/swarm. | +| `MAIL_JWT_SECRET_KEY` | HMAC secret for signing/verifying access-token JWTs. | +| `MAIL_JWT_ALGORITHM` | JWT signing algorithm (e.g. `HS256`). | +| `MAIL_JWT_EXPIRE_MINUTES` | Access-token lifetime, in minutes. | +| `MAIL_REFRESH_TOKEN_EXPIRE_DAYS` | Absolute lifetime of a refresh-token family, in days. | -- Server JWT settings. -- `MAIL_HOST`. -- Memory backend checkpoint interval. -- CLI client `MAIL_SERVER`, `MAIL_ADDRESS`, `MAIL_PASSWORD`, and `MAIL_TOKEN`. -- Daemon `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. -- Defaults for host, port, backend, and log levels. +### Optional environment variables -## Maintenance Notes +| Variable | Default | Effect | +| --- | --- | --- | +| `MAIL_COOKIE_SECURE` | `"true"` | Refresh-cookie `Secure` flag; only the literal `false` disables it. | +| `MAIL_COOKIE_DOMAIN` | unset (host-only cookie) | Cookie `Domain` for cross-subdomain deployments. | +| `MAIL_MEMORY_SAVE_INTERVAL_SECONDS` | `60.0` | Default for `--memory-save-interval`; `0` disables periodic checkpoints. | +| `MAIL_SQLITE_PATH` | unset → default DB path | Default for `--sqlite-path`. | +| `MAIL_DATABASE_URL` | unset | Default for `--database-url`; takes precedence over the sqlite path. | -Keep secrets out of examples. Use clearly fake values and link to security -guidance for production deployment decisions. +### CLI flags + +| Flag | Default | Effect | +| --- | --- | --- | +| `-H`, `--host` | `127.0.0.1` | Bind address. | +| `-p`, `--port` | `8865` | Listen port. | +| `-b`, `--backend` | `memory` | `memory` or `sqlite` (see [Storage Backends](storage-backends.md)). | +| `--memory-save-interval` | `$MAIL_MEMORY_SAVE_INTERVAL_SECONDS` or `60.0` | Seconds between memory checkpoints; `0` disables. | +| `--sqlite-path` | `$MAIL_SQLITE_PATH` or default DB path | SQLite database file. | +| `--database-url` | `$MAIL_DATABASE_URL` | Full database URL. | +| `--license` | — | Print license and exit. | + +**Database URL resolution** (SQLite backend): `--database-url` > `--sqlite-path` +> default `~/.mail-swarms/deployments/default/mail.db`. + +**Logging** is not configurable on the server: level is fixed at `INFO` and logs +are written to `~/.mail-swarms/server_logs/.log`. + +## Client (`mail` and `mail-admin`) + +Client commands read these per-invocation and raise `ValueError` if a required +one is missing. Tokens are never written by the client — `login` and `refresh` +print them for you to export. + +| Variable | Required for | Effect | +| --- | --- | --- | +| `MAIL_SERVER` | all commands | Base URL of the target server. | +| `MAIL_TOKEN` | all authenticated commands | Bearer access token. Not used by `ping` or `login`. | +| `MAIL_ADDRESS` | `login` | Address for the password grant. | +| `MAIL_PASSWORD` | `login` | Password for the password grant. | +| `MAIL_REFRESH_TOKEN` | `refresh` | Refresh token sent to `POST /auth/refresh` (rotated server-side). | + +CLI flag: `-o`/`--output` selects output format — `text` (default), `json` +(`mail` also supports `markdown`). Both accept `--license`. See +[Authenticate a User-Agent](../howtos/authenticate-user-agent.md). + +## Daemon (`mail-daemon`) + +Required environment variables (raise `ValueError` at startup if unset): + +| Variable | Effect | +| --- | --- | +| `MAIL_SERVER` | Target server URL (also health-checked at startup). | +| `MAIL_ADDRESS` | Daemon login address. | +| `MAIL_PASSWORD` | Daemon login password. | + +CLI flags: `-llf`/`--log-level-file` and `-llc`/`--log-level-console` (both +default `info`; choices `debug|info|warning|error|critical`), plus `--license`. +The 30-second delivery poll interval is a hard-coded default with no flag or env +var. See [Run the MAIL Daemon](../howtos/run-daemon.md). + +## `backend-init` + +`backend-init` takes no environment variables; all configuration is via flags +(`--type`, `--deployment`, `--swarm`, `--swarm-description`, `--swarm-keywords`, +`--agents`, `--daemons`, `--users`, `--admins`, `--host`, `--import-fs`). Defaults +and usage are in +[Initialize the Memory Backend](../howtos/initialize-memory-backend.md). + +## `.env.example` + +[`src/mail/server/.env.example`](../../src/mail/server/.env.example) is a +server-side template containing `MAIL_HOST`, `MAIL_JWT_SECRET_KEY` (fake value), +`MAIL_JWT_ALGORITHM`, `MAIL_JWT_EXPIRE_MINUTES`, `MAIL_REFRESH_TOKEN_EXPIRE_DAYS`, +`MAIL_COOKIE_SECURE`, and a commented-out `MAIL_COOKIE_DOMAIN`. The optional +backend knobs (`MAIL_MEMORY_SAVE_INTERVAL_SECONDS`, `MAIL_SQLITE_PATH`, +`MAIL_DATABASE_URL`) and client/daemon variables are not in it. + +## Maintenance notes + +Keep secrets out of examples — use clearly fake values, and link to +[Security Model](../explanations/security-model.md) for production guidance +(TLS, reverse proxy, secret handling). Update this page when a variable or flag is +added, renamed, or changes its default or required status. diff --git a/docs/references/data-models.md b/docs/references/data-models.md index 7c95aea..e99c96e 100644 --- a/docs/references/data-models.md +++ b/docs/references/data-models.md @@ -1,32 +1,259 @@ # Data Models -Status: stub +Status: draft -## Scope +The MAIL protocol types are Pydantic models in the `mail-swarms-protocol` +package: domain models under +[`core/`](../../src/mail/protocol/src/mail_protocol/core) and wire +request/response models under +[`network/`](../../src/mail/protocol/src/mail_protocol/network). This page +documents the domain models field-by-field and summarizes the wire models; for +the exact HTTP request/response schemas see +[`spec/openapi.yaml`](../../spec/openapi.yaml) and [HTTP API](http-api.md). -Describe active MAIL protocol Pydantic models and the validation rules that -shape request, response, and core data contracts. +## Conventions -## Source of Truth +- **Validators.** Field rules are `AfterValidator` functions from + [`core/validators.py`](../../src/mail/protocol/src/mail_protocol/core/validators.py); + the tables name the validator. See [Validators](#validators) for what each + enforces. +- **`metadata`.** Most models carry a `metadata: dict[str, Any]` for + implementer-defined data (SPEC §7.7). Put custom data there, not at the top + level. +- **`.summarize()`.** Full models have a `summarize()` returning a `*Summary` + variant (smaller, `body_size` instead of `body`) used in list responses. +- **`*InBackend`.** Storage-only variants add server-assigned fields (ids, + timestamps, password hashes) and never cross the wire as input. -- `src/mail/protocol/src/mail_protocol/core/` -- `src/mail/protocol/src/mail_protocol/network/` -- `src/mail/protocol/src/mail_protocol/core/validators.py` -- `spec/openapi.yaml` +## User-agents and addresses -## Entries to Cover +Four concrete types discriminated on `ua_type`, each with a `get_address()`. +Defined in +[`core/user_agents.py`](../../src/mail/protocol/src/mail_protocol/core/user_agents.py). +See [Addressing Model](../explanations/addressing-model.md) for the address +grammar. -- User-agents. -- Addresses. -- Messages and message summaries. -- Drafts. -- Inboxes, outboxes, and trash. -- Swarms. -- Mailing lists. -- Webhooks. -- Request and response models. +| Model | `ua_type` | Address form | Identifying field (validator) | +| --- | --- | --- | --- | +| `MAILAgent` | `"agent"` | `name@swarm@host` | `name` (`validate_agent_name`), `swarm`, `host` | +| `MAILUser` | `"user"` | `user:user_id@host` | `user_id` (`validate_user_name`), `host` | +| `MAILAdmin` | `"admin"` | `admin:admin_id@host` | `admin_id` (`validate_user_name`), `host` | +| `MAILDaemon` | `"daemon"` | `daemon:worker_name@host` | `worker_name` (`validate_daemon_worker_name`), `host` | -## Maintenance Notes +- **`MAILUserAgent`** — wrapper with `user_agent: Union[...]` as a + `Field(discriminator="ua_type")`. This is the shape returned by + `GET /auth/whoami` (double-nested: `user_agent.user_agent`). +- **`MAILUserAgentInBackend`** — adds `hashed_password: str`. -Use field tables with type, required status, validation notes, and links to -source classes. Avoid duplicating generated OpenAPI schemas in full. +## Messages + +[`core/messages.py`](../../src/mail/protocol/src/mail_protocol/core/messages.py). +The message contract is SPEC §7. + +### `MAILMessage` + +| Field | Type | Default | Validator | +| --- | --- | --- | --- | +| `mail_version` | `Literal["2.0"]` | required | — | +| `message_id` | `str` | required | `validate_uuid` | +| `reply_to` | `str \| None` | `None` | `validate_uuid` | +| `sender` | `str` | required | `validate_mail_address` | +| `recipients` | `list[str]` | required | `validate_message_recipients` (≥1) | +| `subject` | `str` | required | `validate_message_subject` | +| `body` | `str` | required | `validate_message_body` | +| `tags` | `list[str]` | required | `validate_message_tags` | +| `sent_at` | `datetime` | required | — | +| `metadata` | `dict[str, Any]` | required | — | + +**`MAILMessageSummary`** drops `mail_version`/`reply_to`/`body`/`tags`/`metadata` +and carries `body_size: int` instead of `body`. + +## Drafts + +[`core/drafts.py`](../../src/mail/protocol/src/mail_protocol/core/drafts.py). A +draft has no recipients — they are bound at send time. `reply_to`/`tags` carry +forward onto the sent message. + +### `MAILDraft` + +| Field | Type | Default | Validator | +| --- | --- | --- | --- | +| `draft_id` | `str` | required | `validate_uuid` | +| `subject` | `str` | required | `validate_message_subject` | +| `body` | `str` | required | `validate_message_body` | +| `created_at` | `datetime` | required | — | +| `updated_at` | `datetime \| None` | `None` | — | +| `reply_to` | `str \| None` | `None` | `validate_uuid` | +| `tags` | `list[str]` | `[]` | `validate_message_tags` | + +**`MAILDraftsEntry`** wraps a `MAILDraft` with `sent_at: datetime | None` and +`sent_by: str | None`. **`MAILDraftsEntrySummary`** mirrors it with `body_size`. + +## Box entries + +Each box wraps a `MAILMessage` with box-specific timestamps; each has a `*Summary` +for list views. Modules: +[inbox](../../src/mail/protocol/src/mail_protocol/core/inbox.py), +[outbox](../../src/mail/protocol/src/mail_protocol/core/outbox.py), +[trash](../../src/mail/protocol/src/mail_protocol/core/trash.py). + +| Entry | Wraps | Extra fields | +| --- | --- | --- | +| `MAILInboxEntry` | `MAILMessage` | `received_at`, `delivered_by` | +| `MAILOutboxEntry` | `MAILMessage` | `delivered_at: datetime \| None`, `delivered_by: str \| None` | +| `MAILTrashEntry` | `MAILMessage` | `trashed_at` | + +Summary specifics: + +- **`MAILInboxEntrySummary`** carries `is_read: bool = False`. Read state is + per-owner and supplied at list time (not stored on the shared entry); it flips + to `True` when the message is fetched via `GET /inbox/{message_id}`. +- **`MAILOutboxEntrySummary`** carries nullable `delivered_at` / `delivered_by` + — `null` means *sent, awaiting delivery* (see + [Delivery Model](../explanations/delivery-model.md)). + +## Swarms + +[`core/swarms.py`](../../src/mail/protocol/src/mail_protocol/core/swarms.py). + +### `MAILSwarm` + +| Field | Type | Validator | +| --- | --- | --- | +| `name` | `str` | `validate_swarm_name` | +| `description` | `str` | `validate_swarm_description` | +| `keywords` | `list[str]` | `validate_swarm_keywords` | +| `agents` | `list[str]` | `validate_agent_names` | +| `metadata` | `dict[str, Any]` | — | + +**`MAILSwarmSummary`** replaces `agents`/`metadata` with `num_agents: int`. + +## Mailing lists + +[`core/lists.py`](../../src/mail/protocol/src/mail_protocol/core/lists.py). See +[Mailing Lists](../explanations/mailing-lists.md) for the model in prose. + +### `MAILListPolicy` + +| Field | Type | Default | +| --- | --- | --- | +| `visibility` | `Literal["public", "private"]` | `"public"` | +| `join_policy` | `Literal["open", "approval", "admin-only"]` | `"open"` | +| `send_policy` | `Literal["open", "members-only", "admin-only"]` | `"open"` | + +The non-default enum variants are reserved: they validate at the protocol layer +but the v1 server rejects them with `501`. + +### `MAILList` + +| Field | Type | Default | Validator | +| --- | --- | --- | --- | +| `list_type` | `Literal["list"]` | `"list"` | — | +| `name` | `str` | required | `validate_list_name` | +| `swarm` | `str` | required | `validate_swarm_name` | +| `host` | `str` | required | `validate_host` | +| `owner` | `str` | required | `validate_mail_address` | +| `members` | `list[str]` | `[]` | `validate_mail_addresses` | +| `policy` | `MAILListPolicy` | `MAILListPolicy()` | — | +| `metadata` | `dict[str, Any]` | `{}` | — | + +**`MAILListInBackend`** adds `list_id` (`validate_uuid`), `created_at`, +`updated_at`. This is the payload shape in all list responses. Address form via +`get_address()`: `list:name@swarm@host`. + +## Webhooks + +[`core/webhooks.py`](../../src/mail/protocol/src/mail_protocol/core/webhooks.py), +[`network/webhooks.py`](../../src/mail/protocol/src/mail_protocol/network/webhooks.py). +The only event type is `mail.delivered`. See +[Webhook Delivery](../explanations/webhook-delivery.md). + +- **`MAILWebhook`** — `webhook_id` (`wh_`), `url` (`validate_url`), `events` + (`validate_webhook_event_types`), `secret`. +- **`MAILMessageInWebhook`** — the message as embedded in an outbound payload. It + **diverges from `MAILMessage`**: IDs are `msg_`-prefixed, there is a single + `recipient` (not `recipients`), it adds a `swarm` field, and it omits + `mail_version`. +- **`WebhookDeliveredPostRequest`** — the JSON body the server POSTs to receiver + URLs: `event`, `event_id`, `delivered_at`, `message: MAILMessageInWebhook`. + (This is outbound; it is not an endpoint the server exposes.) + +## Auth + +[`core/auth.py`](../../src/mail/protocol/src/mail_protocol/core/auth.py). + +**`RefreshTokenRecord`** (backend-internal; never on the wire) stores the SHA-256 +hash of a refresh token, its `family_id`, `owner_address`, `issued_at`, absolute +`expires_at`, a `revoked` flag, and `rotated_at`. A token is unusable once +`revoked` is true or `rotated_at` is set. See +[Security Model](../explanations/security-model.md). + +## Request and response models + +The wire envelopes live in +[`network/requests.py`](../../src/mail/protocol/src/mail_protocol/network/requests.py) +and +[`network/responses.py`](../../src/mail/protocol/src/mail_protocol/network/responses.py), +and each maps to an endpoint in [HTTP API](http-api.md) (most docstrings name the +`METHOD /path`). Rather than restate the generated schema, note the recurring +shapes: + +- **Responses** wrap a payload field (`entry`, `entries`, `message`, `swarm`, + `mail_list`, `agent`, …) plus `metadata`. A handful of pure-status responses + (`RootGetResponse`, `HealthGetResponse`, `SwarmHealthGetResponse`, + `AuthLogoutPostResponse`, `AuthPasswordResetResponse`) omit `metadata`. +- **Notable request bodies:** `DraftPostRequest` (`subject`, `body`, optional + `reply_to`, `tags`), `DraftPatchRequest` (all-optional partial update), + `DraftSendPostRequest` (`recipients`, optional `tags` merged with the draft's), + `AdminListPostRequest` / `AdminListPatchRequest` (policy-only patch), + `AdminWebhooksPostRequest` / `AdminWebhooksPatchRequest`, and the admin + create-agent/daemon/user/swarm bodies. +- **`BoxFilterParams`** — the query params for box GET-collection endpoints: + `limit` (1–100, default 20), `offset` (default 0), `sort_by` + (`entered_at`|`sent_at`, default `entered_at`), `order` (`asc`|`desc`, default + `desc`). It forbids extra params. + +## Constants + +[`core/constants.py`](../../src/mail/protocol/src/mail_protocol/core/constants.py): + +| Constant | Min / Max | +| --- | --- | +| Message subject | 1 / 256 | +| Message body | 1 / 65535 | +| Message tag | 1 / 32 | +| Agent / user / admin / daemon-worker / swarm / swarm-keyword / list name | 1 / **31** | +| Swarm description | 0 / 255 | + +`LIST_ADDRESS_PREFIX = "list"`. The protocol version literal `"2.0"` is not a +constant — it is a `Literal` on `MAILMessage.mail_version` and +`RootGetResponse.protocol_version`. (Note the identifier max of 31 differs from +SPEC §6's "SHOULD ≤ 32" — see [Protocol Specification](protocol-specification.md#known-spec--implementation-divergences).) + +## Validators + +Key rules from `core/validators.py`: + +| Validator | Enforces | +| --- | --- | +| `validate_uuid` / `validate_uuids` | Value(s) parse as UUIDs. | +| `validate_mail_address` | Full address shape: 3-part `name@swarm@host` (agent) or `list:name@swarm@host`; 2-part `user:id@host` / `admin:id@host` / `daemon:worker@host`. | +| `validate_local_address` | 2-part `agent@swarm` (admin path params). | +| `validate_message_recipients` | ≥1 entry, each a valid address (SPEC §7.3). | +| `validate_message_subject` / `_body` | Length within the constant bounds. | +| `validate_message_tag(s)` | Each tag length 1–32 and a slug. | +| `validate_agent_name` / `validate_user_name` / `validate_swarm_name` / `validate_daemon_worker_name` / `validate_list_name` / `validate_swarm_keyword` | Length 1–31 and a slug. | +| `validate_host` | Valid hostname, IPv4, or IPv6. | +| `validate_url` | `http(s)://` URL (single-label hosts like `localhost` allowed). | +| `validate_webhook_id` / `validate_webhook_message_id` | `wh_` / `msg_` shapes. | +| `validate_webhook_event_type(s)` | Equals `mail.delivered`. | + +The slug rule (`string_is_slug`) is `^[a-z0-9]+(?:-[a-z0-9]+)*$`: lowercase +alphanumerics in hyphen-separated segments — no uppercase, underscores, or +leading/trailing/double hyphens. + +## Maintenance notes + +Use field tables with type, default/required status, and the validator name; link +to the source class rather than pasting generated OpenAPI schemas in full. Update +this page when protocol models gain or change fields. diff --git a/docs/references/http-api.md b/docs/references/http-api.md index c506df4..079662e 100644 --- a/docs/references/http-api.md +++ b/docs/references/http-api.md @@ -1,31 +1,169 @@ # HTTP API -Status: stub +Status: draft -## Scope +This is a navigational reference to the MAIL server's HTTP surface. The +authoritative contract — request bodies, response schemas, parameters, and status +codes — is the generated [`spec/openapi.yaml`](../../spec/openapi.yaml), also +served interactively at `/docs` (Swagger UI) and `/openapi.json` on a running +server. This page lists every route, its authentication requirement, and its +response model so you can find the right endpoint; follow the OpenAPI schema for +exact field-level detail. Route handlers live in +[`src/mail/server/src/mail_server/routers/`](../../src/mail/server/src/mail_server/routers). -Describe the HTTP API exposed by the MAIL server, with endpoints, auth -requirements, request bodies, response bodies, and error behavior. +## Authentication -## Source of Truth +Every authenticated request carries a bearer access token in the +`Authorization: Bearer ` header. Obtain one from `POST /auth/token` (see +[Authenticate a User-Agent](../howtos/authenticate-user-agent.md)). Endpoints +enforce one of four access levels: -- `spec/openapi.yaml` -- `src/mail/server/src/mail_server/server.py` -- `src/mail/server/src/mail_server/routers/` -- `src/mail/server/docs/reference/http.md` +| Level | Meaning | +| --- | --- | +| none | Unauthenticated. | +| user-agent | Any authenticated user-agent (agent, user, admin, daemon). | +| daemon | A daemon bearer token. | +| admin | An admin bearer token. | -## Entries to Cover +## Response envelope -- Root and health endpoints. -- Authentication endpoints. -- Swarms. -- Inbox, outbox, drafts, and trash. -- Daemon endpoints. -- Admin endpoints. -- Mailing list endpoints. -- Common status codes and validation errors. +Most responses wrap their payload in a named field (`entry`, `entries`, +`message`, `swarm`, `mail_list`, …) alongside a `metadata` object; single-message +box reads nest the message under an `entry`. Field shapes are in +[Data Models](data-models.md). -## Maintenance Notes +## Root and health -Prefer generated OpenAPI details for schemas and parameters. Keep handwritten -text focused on navigation and important implementation notes. +| Method | Path | Auth | Response model | +| --- | --- | --- | --- | +| GET | `/` | none | `RootGetResponse` (protocol name, version, uptime) | +| GET | `/health` | none | `HealthGetResponse` (`status: "ok"`) | + +## Authentication endpoints (`/auth`) + +| Method | Path | Auth | Notes | +| --- | --- | --- | --- | +| POST | `/auth/token` | none | OAuth2 password grant (form fields). Returns `access_token`, `expires_in`; `refresh_token` for interactive principals (users/admins). | +| POST | `/auth/refresh` | refresh token | Rotates the refresh token (cookie or request body). | +| POST | `/auth/logout` | refresh token | Idempotent; revokes the refresh family. | +| GET | `/auth/whoami` | user-agent | Returns the caller's `MAILUserAgent`. | +| POST | `/auth/password/reset` | user-agent | Revokes all refresh families on success. | + +See [Security Model](../explanations/security-model.md) for the refresh-token +design. + +## Swarms (`/swarms`) + +| Method | Path | Auth | Response model | +| --- | --- | --- | --- | +| GET | `/swarms` | user-agent | `SwarmsGetResponse` | +| GET | `/swarms/{swarm_name}` | user-agent | `SwarmGetResponse` | +| GET | `/swarms/{swarm_name}/health` | user-agent | `SwarmHealthGetResponse` | + +## Message boxes + +Box GET-collection endpoints accept the `BoxFilterParams` query params: `limit` +(1–100, default 20), `offset` (default 0), `sort_by` (`entered_at` default, or +`sent_at`), and `order` (`desc` default, or `asc`). `GET /drafts` rejects +`sort_by=sent_at` with `422` (a draft has no send time). + +### Inbox (`/inbox`) + +| Method | Path | Auth | Notes | +| --- | --- | --- | --- | +| GET | `/inbox` | user-agent | List summaries (per-owner `is_read`). | +| GET | `/inbox/{message_id}` | user-agent | Full message; marks it read. | +| DELETE | `/inbox/{message_id}` | user-agent | Moves the message to trash. | + +### Outbox (`/outbox`) + +| Method | Path | Auth | +| --- | --- | --- | +| GET | `/outbox` | user-agent | +| GET | `/outbox/{message_id}` | user-agent | + +### Drafts (`/drafts`) + +| Method | Path | Auth | Notes | +| --- | --- | --- | --- | +| GET | `/drafts` | user-agent | Rejects `sort_by=sent_at` (`422`). | +| POST | `/drafts` | user-agent | Create a draft (`subject`, `body`, optional `reply_to`, `tags`). | +| GET | `/drafts/{draft_id}` | user-agent | | +| PATCH | `/drafts/{draft_id}` | user-agent | Partial update; omitted fields unchanged. | +| DELETE | `/drafts/{draft_id}` | user-agent | | +| POST | `/drafts/{draft_id}/send` | user-agent | Bind `recipients` and send; returns the assembled `MAILMessage`. | + +### Trash (`/trash`) + +| Method | Path | Auth | +| --- | --- | --- | +| GET | `/trash` | user-agent | +| GET | `/trash/{message_id}` | user-agent | +| DELETE | `/trash/{message_id}` | user-agent | +| POST | `/trash/clear` | user-agent | + +## Daemon endpoints (`/daemon`) + +Used by delivery daemons; see [Delivery Model](../explanations/delivery-model.md). + +| Method | Path | Auth | Notes | +| --- | --- | --- | --- | +| POST | `/daemon/message-buffer/clear` | daemon | Drain the pending-delivery buffer. | +| POST | `/daemon/deliver/local` | daemon | Deliver messages between user-agents on this server. | +| POST | `/daemon/deliver/remote` | daemon | Reserved for cross-server delivery; handler currently raises `NotImplementedError`. | + +## Admin endpoints (`/admin`) + +All require admin. Address path params use the **local** form (`agent@swarm`, +`name@swarm`); user/daemon use their id / worker name. + +| Resource | Routes | +| --- | --- | +| Agents | `GET|POST /admin/agents`, `GET|DELETE /admin/agents/{local_address}` | +| Daemons | `GET|POST /admin/daemons`, `GET|DELETE /admin/daemons/{worker_name}` | +| Users | `GET|POST /admin/users`, `GET|DELETE /admin/users/{user_id}` | +| Swarms | `POST /admin/swarms`, `DELETE /admin/swarms/{swarm_name}` | +| Webhooks | `GET|POST /admin/webhooks`, `GET|PATCH|DELETE /admin/webhooks/{webhook_id}` | + +## Mailing list endpoints + +See [Mailing Lists](../explanations/mailing-lists.md) and +[Manage Mailing Lists](../howtos/manage-mailing-lists.md). List path params use +the **local** `name@swarm` form; the server reconstructs the canonical +`list:name@swarm@host`. + +### Admin lists (`/admin/lists`, admin) + +| Method | Path | +| --- | --- | +| GET | `/admin/lists` | +| POST | `/admin/lists` | +| GET / PATCH / DELETE | `/admin/lists/{local_address}` | +| POST | `/admin/lists/{local_address}/members` | +| DELETE | `/admin/lists/{local_address}/members/{member_address}` | + +### Public lists (`/lists`, user-agent) + +| Method | Path | Notes | +| --- | --- | --- | +| GET | `/lists` | Filtered to `visibility=public`. | +| GET | `/lists/{local_address}` | `404` if not public. | +| POST | `/lists/{local_address}/subscribe` | `501` unless `join_policy=open`. | +| POST | `/lists/{local_address}/unsubscribe` | | + +## Common status codes + +| Code | Meaning in MAIL | +| --- | --- | +| `401 Unauthorized` | Missing, malformed, or expired token. | +| `403 Forbidden` | Authenticated but wrong role (e.g. non-admin on `/admin`). | +| `404 Not Found` | Unknown resource (or a non-public list on `/lists`). | +| `422 Unprocessable Entity` | Request/query validation failed; `detail` explains what. | +| `501 Not Implemented` | Reserved-but-unsupported behavior (non-`open` list policies). | + +## Maintenance notes + +Prefer the generated OpenAPI details for schemas and parameters; keep this page +focused on navigation, auth levels, and implementation notes. When routes change, +regenerate `spec/openapi.yaml` ([Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md)) +and update the tables here. diff --git a/docs/references/protocol-specification.md b/docs/references/protocol-specification.md index 2e697e8..93bd7b8 100644 --- a/docs/references/protocol-specification.md +++ b/docs/references/protocol-specification.md @@ -1,31 +1,84 @@ # Protocol Specification -Status: stub +Status: draft -## Scope +MAIL has two normative artifacts, both under [`spec/`](../../spec). This page +orients you to them and maps each part of the specification to the implementation +reference pages in this set. It does not restate the spec — read the source files +for the authoritative text. -Point readers to the normative MAIL protocol materials and summarize how they -relate to implementation reference pages. +- **[`spec/SPEC.md`](../../spec/SPEC.md)** — the protocol prose. Versioned, + written in [RFC 2119][rfc-2119] requirements language (MUST / SHOULD / MAY). + It defines terminology, user-agent categories, address forms, the message + contract, and delivery responsibilities. +- **[`spec/openapi.yaml`](../../spec/openapi.yaml)** — the authoritative HTTP + wire contract. It is **generated** from the running FastAPI app, not hand-authored + (see [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md)), so it + always matches the server's declared routes and schemas. -## Source of Truth +## Version and status -- `spec/SPEC.md` -- `spec/openapi.yaml` -- `tests/contract/` +| Field | Value | +| --- | --- | +| Version | `2.0` | +| Date | June 10, 2026 | +| Status | Open to feedback | -## Entries to Cover +Versions follow `{major}.{minor}` (SPEC §10). Message payloads carry the version +in `mail_version`, which MUST be `"2.0"` for this revision. -- Protocol version and status. -- Requirements language. -- User-agent categories. -- Address forms. -- Message contract. -- Delivery responsibilities. -- Security considerations. -- OpenAPI contract role. -- Contract test coverage. +## Section map -## Maintenance Notes +| SPEC.md section | Topic | Where it lives in these docs | +| --- | --- | --- | +| §3 Motivation | Goals; what MAIL is *not* | [MAIL v2 Overview](../explanations/mail-v2-overview.md) | +| §4 Architecture | Clients, servers, swarms | [Architecture](../explanations/architecture.md) | +| §5 User-Agents | admin / agent / daemon / user | [Data Models](data-models.md) | +| §6 Addresses | Host- vs swarm-scoped forms | [Addressing Model](../explanations/addressing-model.md) | +| §7 Messages | Message fields, replies, tags | [Data Models](data-models.md) | +| §8 Delivery | Pre-send vs post-send errors | [Delivery Model](../explanations/delivery-model.md) | +| §9 Security | Trust boundaries per component | [Security Model](../explanations/security-model.md) | +| §10 Versioning | Protocol version rules | this page | -Do not duplicate the full specification here. This page should orient readers -and link to exact normative files. +## OpenAPI contract role + +The OpenAPI document is the source of truth for endpoints, parameters, request +bodies, response schemas, and status codes. Client and server implementers MUST +conform to it (SPEC §4.1, §4.2). Because it is generated from the app, the +[HTTP API](http-api.md) reference is navigational — it points into the generated +schema rather than duplicating it. + +## Contract test coverage + +Conformance is enforced by the suite in [`tests/contract/`](../../tests/contract): + +- `test_spec_addresses.py` — address-shape rules from SPEC §6. +- `test_spec_messages.py` — message field constraints from SPEC §7. +- `test_spec_delivery.py` — delivery semantics from SPEC §8. +- `test_openapi_drift.py` — the committed `spec/openapi.yaml` matches the app. +- `test_openapi_request_bodies.py` — body-bearing endpoints document their bodies. + +Run them via [Run the Test Suite](../howtos/run-tests.md). + +## Known spec ↔ implementation divergences + +The reference implementation intentionally (or incidentally) differs from the +prose in a few places. These are tracked here so readers trust the code over the +prose where they disagree: + +- **Identifier length cap.** SPEC §6 says agent / user / admin / daemon-worker / + swarm / list identifiers "SHOULD be no longer than **32** characters." The + reference implementation enforces a hard cap of **31** + (`core/constants.py`: `*_NAME_LEN_MAX = 31`), rejecting longer values with a + validation error. Message tags, by contrast, use a cap of 32 in both + (`MESSAGE_TAG_LEN_MAX = 32`, matching SPEC §7.10). This name-length mismatch + is unresolved — see the maintenance note below. + +## Maintenance notes + +Do not copy the specification text into this page; keep it as an index that links +to the exact normative files. When the implementation and `SPEC.md` are +reconciled (e.g. the 31-vs-32 identifier cap), update both the divergences list +here and [Addressing Model](../explanations/addressing-model.md). + +[rfc-2119]: https://datatracker.ietf.org/doc/html/rfc2119 diff --git a/docs/references/repository-layout.md b/docs/references/repository-layout.md index 01cafb3..3a630b6 100644 --- a/docs/references/repository-layout.md +++ b/docs/references/repository-layout.md @@ -1,34 +1,136 @@ # Repository Layout -Status: stub +Status: draft -## Scope +This page maps the MAIL repository so you can find the code, specification, and +tests behind everything else in these docs. MAIL v2 is a [uv][uv] workspace: a +root meta-package plus four member packages under `src/mail/`, with the archived +v1 runtime kept alongside them. -Describe the MAIL v2 repository structure, workspace package boundaries, specs, -tests, scripts, and archived v1 runtime location. +## Top level -## Source of Truth +```text +mail/ +├── docs/ # this documentation set (tutorials/howtos/references/explanations) +├── spec/ # protocol source of truth: SPEC.md + openapi.yaml +├── src/mail/ # the workspace packages (see below) +├── tests/ # active v2 test suite (contract/e2e/integration/unit) +├── scripts/ # repository maintenance + artifact generation +├── pyproject.toml # uv workspace + root meta-package + shared tooling config +├── pytest.ini # test configuration +├── uv.lock # locked dependency graph for the whole workspace +├── llms.txt # generated LLM-oriented API digest +├── README.md # repository overview +├── SPEC-LICENSE, SPEC-PATENT-LICENSE, LICENSE, NOTICE, TRADEMARKS.md, DCO +└── THIRD_PARTY_NOTICES.md # generated third-party license aggregation +``` -- `README.md` -- `pyproject.toml` -- `src/mail/*/pyproject.toml` -- `tests/` -- `scripts/` -- `src/mail/legacy/` +## Workspace packages -## Entries to Cover +Each package lives at `src/mail//` with its own `pyproject.toml`, +`README.md`, and a `src/mail_/` import root. All four are published to PyPI +in lockstep under the `mail-swarms-*` names. -- Root meta-package. -- `src/mail/protocol` -- `src/mail/server` -- `src/mail/client` -- `src/mail/daemon` -- `spec/` -- `tests/` -- `scripts/` -- `src/mail/legacy` +| Directory | Package name | Import root | Console scripts | +| --- | --- | --- | --- | +| `src/mail/protocol/` | `mail-swarms-protocol` | `mail_protocol` | `mail-protocol` | +| `src/mail/server/` | `mail-swarms-server` | `mail_server` | `mail-server`, `backend-init` | +| `src/mail/client/` | `mail-swarms-client` | `mail_client` | `mail`, `mail-admin` | +| `src/mail/daemon/` | `mail-swarms-daemon` | `mail_daemon` | `mail-daemon` | -## Maintenance Notes +The root `pyproject.toml` also exposes `mail` and `mail-server` so a workspace +checkout can run them directly (`uv run mail …`, `uv run mail-server`). -Update this page when workspace members, top-level directories, or package -responsibilities change. +### `protocol` — shared types and constants + +```text +src/mail_protocol/ +├── core/ # Pydantic domain models: messages, drafts, inbox, outbox, +│ # trash, swarms, lists, webhooks, user_agents, auth +├── network/ # request/response/webhook wire models +├── constants.py # protocol version and shared limits +├── core/validators.py +└── cli.py, cli_help.py +``` + +The protocol package is the dependency root — server, client, and daemon all +import its models. See [Data Models](data-models.md). + +### `server` — FastAPI server + +```text +src/mail_server/ +├── server.py # app assembly + root/health endpoints + backend selection +├── routers/ # one router per area: auth, swarms, inbox, outbox, +│ # drafts, trash, daemon, admin, lists +├── backends/ # storage: base.py (contract), memory/, sqlite/ +├── backend_init.py # the `backend-init` entry point +├── auth.py # token issuance, refresh tokens, role checks +├── validators.py, utils.py, logging.py, cli.py +└── .env.example # sample server configuration +``` + +See [HTTP API](http-api.md), [Storage Backends](storage-backends.md), and +[Configuration](configuration.md). + +### `client` — CLI client + +```text +src/mail_client/ +├── cli.py # `mail` — user-agent CLI +├── admin_panel.py # `mail-admin` — admin CLI +└── commands/ # one module per subcommand +``` + +See [Client CLI](client-cli.md) and [Admin CLI](admin-cli.md). + +### `daemon` — delivery daemon + +```text +src/mail_daemon/ +├── cli.py # `mail-daemon` entry point +├── maild/ # delivery loop + server API client +└── logger.py +``` + +See [Daemon CLI](daemon-cli.md) and [Delivery Model](../explanations/delivery-model.md). + +## Specification + +```text +spec/ +├── SPEC.md # normative protocol prose (versioned, RFC-2119 language) +└── openapi.yaml # authoritative HTTP wire contract (generated from the app) +``` + +`openapi.yaml` is generated, not hand-edited — see +[Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md) and +[Protocol Specification](protocol-specification.md). + +## Tests + +```text +tests/ +├── contract/ # spec/openapi conformance (addresses, delivery, messages, drift) +├── e2e/ # end-to-end flows +├── integration/ # auth, authz, and cross-component behavior +└── unit/ # component-level tests +``` + +See [Run the Test Suite](../howtos/run-tests.md). + +## Package documentation + +Some packages carry their own `docs/` directory that predates this consolidated +set: `src/mail/server/docs/` and `src/mail/client/docs/`. These are being +migrated into the top-level `docs/` tree; where they overlap, this tree is +canonical. + +## Archived v1 runtime + +`src/mail/legacy/` holds the MAIL v1 reference runtime (`api.py`, `client.py`, +`core/`, `config/`, UI assets, and its own docs). It is retained for reference +only and is not part of the v2 workspace packages. See +[MAIL v1 Legacy Runtime](../explanations/mail-v1-legacy.md). + +[uv]: https://docs.astral.sh/uv/ diff --git a/docs/references/storage-backends.md b/docs/references/storage-backends.md index 47d4073..6135a15 100644 --- a/docs/references/storage-backends.md +++ b/docs/references/storage-backends.md @@ -1,32 +1,143 @@ # Storage Backends -Status: stub +Status: draft -## Scope +The MAIL server keeps all state behind a single backend interface. Two backends +ship today — an in-memory backend with filesystem checkpointing, and a +transactional SQLite backend. Both implement the same contract, so the choice is +about durability and operational shape, not features. Code is under +[`src/mail/server/src/mail_server/backends/`](../../src/mail/server/src/mail_server/backends). -Describe the server backend interface and the memory backend persistence -behavior. +## The backend contract -## Source of Truth +`MAILServerBackend` +([`backends/base.py`](../../src/mail/server/src/mail_server/backends/base.py)) is +a `typing.Protocol` of `@abstractmethod`s that both backends implement. It has one +concrete attribute, `host: str` (set at startup; routers use it to reconstruct +full addresses), and the abstract methods group into: -- `src/mail/server/src/mail_server/backends/base.py` -- `src/mail/server/src/mail_server/backends/memory/` -- `src/mail/server/src/mail_server/backend_init.py` -- `tests/unit/test_memory_fs_roundtrip.py` -- `tests/unit/test_memory_checkpointing.py` +| Area | Responsibility | +| --- | --- | +| Lifecycle | `on_server_startup`, `on_server_shutdown` | +| Auth / user-agents | fetch a user-agent, existence check, password reset | +| Refresh tokens | create, get, rotate, revoke-family, revoke-all, purge-expired | +| Swarms | list, get, health | +| Boxes | inbox / outbox / drafts / trash: list, get, delete (+ draft create/patch/send, trash clear) | +| Daemon delivery | clear message buffer, deliver local, deliver remote | +| Admin | CRUD for agents, daemons, users, swarms | +| Webhooks | CRUD, plus the shared outbound delivery logic | +| Lists | list/get (public + admin), create, patch, delete, add/remove member | -## Entries to Cover +The only **concrete** methods on the base are the webhook-delivery helpers +(`handle_webhook_delivered_for_url`, `_webhook_delivered_post`), so both backends +share identical `mail.delivered` HMAC signing and retry behavior — see +[Webhook Delivery](../explanations/webhook-delivery.md). -- Backend lifecycle hooks. -- User-agent storage. -- Box storage. -- Draft and trash behavior. -- Message delivery buffer. -- Memory backend filesystem layout. -- Checkpoint behavior. -- Current backend limitations. +## Selecting a backend -## Maintenance Notes +`mail-server --backend {memory,sqlite}` (`-b`, default `memory`). There is no env +var for the backend choice itself; per-backend knobs are covered in +[Configuration](configuration.md). `backend-init --type {memory,sqlite}` +initializes on-disk state before the server starts — see +[Initialize the Memory Backend](../howtos/initialize-memory-backend.md). + +## Memory backend + +Files: `backends/memory/api.py` (state + logic), `fs.py` (load/save), `init.py` +(`backend-init` seeding). + +- **Model.** All state lives in Python dicts/lists in RAM. On startup every + collection is loaded from disk; on checkpoint and shutdown every collection is + written back. +- **On-disk layout** under `~/.mail-swarms/deployments/{deployment}/`: one + directory per collection with one JSON file per item + (`swarms/`, `user_agents/`, `messages/`, `inbox_entries/`, `outbox_entries/`, + `draft_entries/`, `trash_entries/`, `webhooks/`, `lists/`, `refresh_tokens/`), + newline-delimited membership files for each per-owner box + (`inboxes/`, `outboxes/`, `drafts/`, `trashes/`, `read_inbox/`), a + `message_buffer.lock` FIFO file, and the plaintext `.secrets/
` files + written at init. +- **Checkpointing.** A background loop persists every + `--memory-save-interval` seconds (default **60**, `0` disables the periodic + loop), plus a final persist on shutdown. Each file write is atomic (temp file + → `fsync` → `os.replace` → parent-dir `fsync`). +- **Durability caveats.** State between checkpoints is RAM-only: a hard kill (or + interval `0`) loses everything since the last checkpoint. Individual writes are + atomic, but a full checkpoint is not a single transaction across collections, so + a crash mid-persist can leave collections at slightly different versions. + +## SQLite backend + +Files: `backends/sqlite/` — `api.py`, `database.py`, `schema.py`, +`repositories.py`, `serializers.py`, `init.py`, `migrate.py`. + +- **Model.** Fully transactional and durable. Every mutation runs in a session + that commits on success / rolls back on error; multi-step operations (e.g. + `send_draft` = message + outbox entry + membership + buffer row) commit + atomically. There is no in-RAM master copy and no checkpoint loop — every write + hits the database. +- **Stack.** Async SQLAlchemy over `sqlite+aiosqlite`, with per-connection + `PRAGMA foreign_keys=ON`, `journal_mode=WAL`, and `busy_timeout=5000` (5s). +- **Schema (hybrid).** Entity rows carry typed/indexed columns only for + filtering/ordering, plus a `body` JSON column holding the full + `model.model_dump(mode="json")`; reads rehydrate from `body`. Tables: + `user_agents`, `swarms`, `messages`, `inbox_entries`, `outbox_entries`, + `draft_entries`, `trash_entries`, `mailbox_items` (unified per-owner box + membership + ordering, with `is_read` for the inbox), `message_buffer`, + `webhooks`, `refresh_tokens` (all-typed, no body), `lists`. +- **Location.** Default `~/.mail-swarms/deployments/{deployment}/mail.db`; + overridable via `--sqlite-path` / `MAIL_SQLITE_PATH` or a full + `--database-url` / `MAIL_DATABASE_URL` (precedence: database-url > sqlite-path > + default). +- **Migrations.** No migration framework. `create_schema()` runs + `create_all` plus an additive, idempotent `ALTER TABLE ... ADD COLUMN` guard for + new queryable columns (the current guard adds `mailbox_items.is_read`, + backfilling existing rows as unread). + +## Initializing state with `backend-init` + +- `--type memory` builds the full deployment directory tree, writes the swarm and + each user-agent (hashed password), touches empty box files, and writes plaintext + `.secrets/
`. +- `--type sqlite` creates `mail.db` + schema and seeds one swarm + the requested + principals; it is idempotent on re-run (existing swarm/user-agents skipped) and + creates box membership lazily on first delivery. Plaintext secrets are still + written. +- `--type sqlite --import-fs` imports an existing memory/filesystem deployment of + the same name into a fresh SQLite database (messages first, then box entries, + then membership in arrival order, then buffer/webhooks/lists — all in one + transaction). It refuses to run if the source is missing or the target DB + already holds rows. + +## Capability differences + +| Aspect | Memory | SQLite | +| --- | --- | --- | +| Durability | RAM-first; ≤1 checkpoint interval at risk on crash | Durable per write (WAL) | +| Transactions | Per-file atomic writes; not cross-collection | Full multi-step transactions | +| Concurrency | Single `asyncio.Lock` around persist | WAL readers + serialized writers, 5s busy timeout | +| Deployments | Runtime reads/writes the `default` deployment only (see limitations) | Arbitrary via `--sqlite-path` / `--database-url` | +| Init on re-run | Overwrites | Idempotent | +| Migration import | — | `--import-fs` | + +Both implement the identical interface and share the webhook delivery logic, so +inbox `is_read`, refresh-token families, list membership, and webhook semantics +match across backends. + +## Current limitations + +- **Memory backend deployment name.** The memory runtime's filesystem layer is + pinned to the `default` deployment: `backend-init` will *create* a named memory + deployment, but `mail-server --backend memory` reads and writes `default` + regardless. Use the SQLite backend for non-default deployment names. +- **`daemon_deliver_remote`** is unimplemented in both backends (raises + `NotImplementedError`); remote/cross-server delivery is reserved. See + [Delivery Model](../explanations/delivery-model.md). +- No Postgres backend yet, though `normalize_database_url` reserves a + `postgresql+psycopg` driver seam. + +## Maintenance notes Keep production deployment advice in how-to or explanation pages unless it is a -direct backend capability or limitation. +direct backend capability or limitation. Update this page when the backend +contract, on-disk layout, or SQLite schema changes. From 3dcfdf48a9708597c1435ab96eb87652ed404012 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 15:47:14 -0400 Subject: [PATCH 23/28] docs: auto-generate CLI reference pages from argparse Add scripts/build_cli_docs.py, which imports each CLI's build_parser() and renders a Markdown reference (global options + subcommands with their args, aliases, and expanded defaults). Generate the four CLI reference pages (client/admin/server/daemon) from it so they cannot drift from the parsers. Regenerate with: uv run python scripts/build_cli_docs.py Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/references/admin-cli.md | 250 ++++++++++++++++++++++++++++++---- docs/references/client-cli.md | 229 ++++++++++++++++++++++++++++--- docs/references/daemon-cli.md | 31 ++--- docs/references/server-cli.md | 32 ++--- scripts/build_cli_docs.py | 179 ++++++++++++++++++++++++ 5 files changed, 634 insertions(+), 87 deletions(-) create mode 100644 scripts/build_cli_docs.py diff --git a/docs/references/admin-cli.md b/docs/references/admin-cli.md index ec7719d..3ee070d 100644 --- a/docs/references/admin-cli.md +++ b/docs/references/admin-cli.md @@ -1,35 +1,235 @@ # Admin CLI -Status: stub +Status: generated -## Scope +> **Generated file — do not edit by hand.** Regenerate with `uv run python scripts/build_cli_docs.py` after changing the CLI. See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md). -Describe the administrator command-line surface for managing MAIL server -resources. +A Python CLI client admin panel for the Multi-Agent Interface Layer (MAIL) -## Source of Truth +Invoke as `mail-admin` (or `uv run mail-admin` from a workspace checkout). Source: `mail_client/admin_panel.py`. -- `src/mail/client/src/mail_client/admin_panel.py` -- `src/mail/client/src/mail_client/commands/agent_*.py` -- `src/mail/client/src/mail_client/commands/user_*.py` -- `src/mail/client/src/mail_client/commands/daemon_*.py` -- `src/mail/client/src/mail_client/commands/swarm_*.py` -- `src/mail/client/src/mail_client/commands/webhook_*.py` -- `src/mail/client/src/mail_client/commands/list_*.py` -- `src/mail/client/docs/reference/admin-panel.md` +## Global options -## Entries to Cover +- `--license` — show license information and exit +- `-o`, `--output` `{text,json}` — the output style for this CLI command (default: text) -- Agent operations. -- User operations. -- Admin operations if exposed. -- Daemon operations. -- Swarm operations. -- Webhook operations. -- Mailing list administration. -- Shared auth and output options. +## Commands -## Maintenance Notes +### `ping` (aliases: `p`) -Call out destructive commands clearly, but keep procedural guidance in how-to -pages. +ping a MAIL server + +### `login` (aliases: `l`) + +log into a MAIL server + +### `whoami` (aliases: `me`, `id`) + +get authenticated user-agent info from a MAIL server + +### `agent-list` (aliases: `al`) + +get a list of agents on the MAIL server + +### `agent-get` (aliases: `ag`) + +get a specific agent by local address on the MAIL server + +**Arguments:** + +- `local_address` — the local address of the agent to get (agent@swarm) + +### `agent-post` (aliases: `ap`) + +create a new agent on the MAIL server with the specified credentials + +**Arguments:** + +- `local_address` — the local address of the agent to create (agent@swarm) + +### `agent-delete` (aliases: `ad`) + +delete an existing agent by local address on the MAIL server + +**Arguments:** + +- `local_address` — the local address of the agent to delete (agent@swarm) + +### `daemon-list` (aliases: `dl`) + +get a list of daemons on the MAIL server + +### `daemon-get` (aliases: `dg`) + +get a specific daemon by worker name on the MAIL server + +**Arguments:** + +- `worker_name` — the worker name of the daemon to get + +### `daemon-post` (aliases: `dp`) + +create a new daemon on the MAIL server with the specified credentials + +**Arguments:** + +- `worker_name` — the name to use for the new daemon + +### `daemon-delete` (aliases: `dd`) + +delete an existing daemon by worker name on the MAIL server + +**Arguments:** + +- `worker_name` — the name of the daemon to delete + +### `user-list` (aliases: `ul`) + +get a list of users on the MAIL server + +### `user-get` (aliases: `ug`) + +get a specific user by user ID on the MAIL server + +**Arguments:** + +- `user_id` — the ID of the user to get + +### `user-post` (aliases: `up`) + +create a new user on the MAIL server with the specified credentials + +**Arguments:** + +- `user_id` — the ID to use for the new user + +### `user-delete` (aliases: `ud`) + +delete an existing user by user ID on the MAIL server + +**Arguments:** + +- `user_id` — the name of the user to delete + +### `swarm-post` (aliases: `sp`) + +create a new swarm on the MAIL server with the specified info + +**Arguments:** + +- `name` — the name of the swarm to create +- `description` — the description to use for the new swarm + +**Options:** + +- `-k`, `--keywords` `KEYWORDS` — the keywords to use for this swarm (default: []) + +### `swarm-delete` (aliases: `sd`) + +delete an existing swarm by name from the MAIL server + +**Arguments:** + +- `swarm_name` — the name of the swarm on the server to delete + +### `webhook-list` (aliases: `wl`) + +list all webhooks on the MAIL server + +### `webhook-get` (aliases: `wg`) + +get an existing webhook by ID on the MAIL server + +**Arguments:** + +- `webhook_id` — the ID of the webhook to get + +### `webhook-post` (aliases: `wp`) + +create a new webhook on the MAIL server + +**Arguments:** + +- `url` — the URL to hit for this webhook +- `secret` — the secret to use for this webhook + +**Options:** + +- `-e`, `--events` `EVENTS` — the event(s) for this webhook + +### `webhook-patch` (aliases: `wP`) + +update an existing webhook on the MAIL server + +**Arguments:** + +- `webhook_id` — the ID of the webhook to update + +**Options:** + +- `-u`, `--url` `URL` — the new URL to use, if any +- `-s`, `--secret` `SECRET` — the new secret to use, if any + +### `webhook-delete` (aliases: `wd`) + +delete an existing webhook by ID on the MAIL server + +**Arguments:** + +- `webhook_id` — the ID of the webhook to delete + +### `list-list` (aliases: `ll`) + +get all mailing lists on the MAIL server + +### `list-get` (aliases: `lg`) + +get a specific mailing list on the MAIL server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to get (name@swarm) + +### `list-post` (aliases: `lp`) + +create a new mailing list on the MAIL server + +**Arguments:** + +- `name` — the name of the new mailing list +- `swarm_name` — the name of the swarm to use for this mailing list +- `owner` — the MAIL address of the mailing list owner + +**Options:** + +- `-m`, `--members` `MEMBERS` — the MAIL addresses of members to add to this mailing list (default: []) + +### `list-patch` (aliases: `lP`) + +update an existing mailing list on the MAIL server + +### `list-delete` (aliases: `ld`) + +delete an existing mailing list on the MAIL server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to delete (name@swarm) + +### `list-member-post` (aliases: `lmp`) + +add a new member to an existing mailing list on the MAIL server + +**Arguments:** + +- `list_address` — the local address of the mailing list to add a member to (name@swarm) +- `member_address` — the full MAIL address of the member to add to this mailing list + +### `list-member-delete` (aliases: `lmd`) + +delete a member from an existing mailing list on the MAIL server + +**Arguments:** + +- `list_address` — the local address of the mailing list to remove a member from (name@swarm) +- `member_address` — the full MAIL address of the member to remove from this mailing list diff --git a/docs/references/client-cli.md b/docs/references/client-cli.md index fef1e00..23aaf81 100644 --- a/docs/references/client-cli.md +++ b/docs/references/client-cli.md @@ -1,29 +1,220 @@ # Client CLI -Status: stub +Status: generated -## Scope +> **Generated file — do not edit by hand.** Regenerate with `uv run python scripts/build_cli_docs.py` after changing the CLI. See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md). -Describe the `mail` command-line interface for regular MAIL user-agent -operations. +The Python CLI client for the Multi-Agent Interface Layer (MAIL) -## Source of Truth +Invoke as `mail` (or `uv run mail` from a workspace checkout). Source: `mail_client/cli.py`. -- `src/mail/client/src/mail_client/cli.py` -- `src/mail/client/src/mail_client/commands/` -- `src/mail/client/docs/reference/cli.md` +## Global options -## Entries to Cover +- `--license` — show license information and exit +- `-o`, `--output` `{text,json,markdown}` — the output style for this CLI command (default: text) -- Usage. -- Global output options. -- Utility commands. -- Messaging commands. -- Swarm helper commands. -- Mailing list helper commands. -- Environment variables consumed by commands. -- Output formats. +## Commands -## Maintenance Notes +### `ping` (aliases: `p`) -Regenerate or review this page whenever command parsers or aliases change. +ping a MAIL server + +### `login` (aliases: `l`) + +log into a MAIL server + +### `refresh` (aliases: `rt`) + +renew your access token using a refresh token + +### `whoami` (aliases: `me`, `id`) + +get authenticated user-agent info from a MAIL server + +### `compose` (aliases: `c`) + +draft a new MAIL message prior to sending + +**Arguments:** + +- `subject` — the subject line of the message to draft +- `body` — the body of the message to draft (omit when using --body-file) + +**Options:** + +- `-F`, `--body-file` `PATH` — read the message body from the file at this path +- `--tags` `TAG` — slug string tag(s) to attach to the message + +### `send` (aliases: `s`) + +send a drafted MAIL message to the specified address(es) + +**Arguments:** + +- `draft_id` — the ID of the existing draft to send +- `to` — the address(es) to deliver this message to + +**Options:** + +- `--tags` `TAG` — slug string tag(s) to attach to the message + +### `reply` (aliases: `r`) + +reply to an existing inbox message + +**Arguments:** + +- `message_id` — the ID of the inbox message to reply to +- `body` — the body of the reply + +**Options:** + +- `--subject` `SUBJECT` — the subject of the reply (default: 'Re: ') +- `--tags` `TAG` — slug string tag(s) to attach to the message + +### `forward` (aliases: `f`) + +forward an existing inbox message to new recipient(s) + +**Arguments:** + +- `message_id` — the ID of the inbox message to forward +- `to` — the address(es) to forward this message to + +**Options:** + +- `--note` `NOTE` — an optional note to prepend above the forwarded message +- `--subject` `SUBJECT` — the subject of the forward (default: 'Fwd: ') +- `--tags` `TAG` — slug string tag(s) to attach to the message + +### `inbox` (aliases: `i`) + +open your MAIL inbox + +**Options:** + +- `--limit` `LIMIT` — max number of entries to return (1-100) +- `--offset` `OFFSET` — number of entries to skip +- `--sort-by` `{sent_at,entered_at}` — timestamp field to sort by +- `--order` `{asc,desc}` — sort direction + +### `inbox-open` (aliases: `open`, `o`) + +open a specific message by ID in your MAIL inbox + +**Arguments:** + +- `message_id` — the ID of the message to open + +### `outbox` (aliases: `O`) + +open your MAIL outbox + +**Options:** + +- `--limit` `LIMIT` — max number of entries to return (1-100) +- `--offset` `OFFSET` — number of entries to skip +- `--sort-by` `{sent_at,entered_at}` — timestamp field to sort by +- `--order` `{asc,desc}` — sort direction + +### `outbox-open` (aliases: `Oopen`, `Oo`) + +open a specific message by ID in your MAIL outbox + +**Arguments:** + +- `message_id` — the ID of the message to open + +### `drafts` (aliases: `d`) + +list your existing message drafts + +**Options:** + +- `--limit` `LIMIT` — max number of entries to return (1-100) +- `--offset` `OFFSET` — number of entries to skip +- `--sort-by` `{sent_at,entered_at}` — timestamp field to sort by +- `--order` `{asc,desc}` — sort direction + +### `drafts-open` (aliases: `do`) + +open a specific existing draft by ID + +**Arguments:** + +- `draft_id` — the ID of the drafted message to open + +### `draft-edit` (aliases: `de`) + +edit fields on an existing message draft by ID + +**Arguments:** + +- `draft_id` — the ID of the draft to edit +- `body` — the new body of the draft (omit to leave it unchanged) + +**Options:** + +- `--subject` `SUBJECT` — the new subject of the draft (omit to leave it unchanged) +- `-F`, `--body-file` `PATH` — read the message body from the file at this path +- `--reply-to` `REPLY_TO` — the message ID this draft replies to (omit to leave it unchanged) +- `--tags` `TAG` — replace the draft's tags (pass with no values to clear all tags) + +### `trash` (aliases: `t`) + +list your existing trashed messages + +**Options:** + +- `--limit` `LIMIT` — max number of entries to return (1-100) +- `--offset` `OFFSET` — number of entries to skip +- `--sort-by` `{sent_at,entered_at}` — timestamp field to sort by +- `--order` `{asc,desc}` — sort direction + +### `trash-open` (aliases: `to`) + +open a specific message in trash by ID + +**Arguments:** + +- `message_id` — the ID of the message in trash to open + +### `swarm-list` (aliases: `swarms`, `sl`) + +get the swarms on this MAIL server + +### `swarm-get` (aliases: `swarm`, `sg`) + +get a specific swarm by name on this MAIL server + +**Arguments:** + +- `swarm_name` — the name of the MAIL swarm to get + +### `lists` + +get mailing lists on this MAIL server + +### `list-get` (aliases: `list`, `lg`) + +get a specific list on this MAIL server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to get (name@swarm) + +### `list-subscribe` (aliases: `ls`) + +subscribe to a mailing list on this server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to subscribe to (name@swarm) + +### `list-unsubscribe` (aliases: `lu`) + +unsubscribe from a mailing list on this server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to unsubscribe from (name@swarm) diff --git a/docs/references/daemon-cli.md b/docs/references/daemon-cli.md index 386bdf9..babed56 100644 --- a/docs/references/daemon-cli.md +++ b/docs/references/daemon-cli.md @@ -1,30 +1,15 @@ # Daemon CLI -Status: stub +Status: generated -## Scope +> **Generated file — do not edit by hand.** Regenerate with `uv run python scripts/build_cli_docs.py` after changing the CLI. See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md). -Describe the `mail-daemon` CLI, required environment variables, and daemon loop -behavior. +Multi-Agent Interface Layer (MAIL) daemon implementation in Python -## Source of Truth +Invoke as `mail-daemon` (or `uv run mail-daemon` from a workspace checkout). Source: `mail_daemon/cli.py`. -- `src/mail/daemon/src/mail_daemon/cli.py` -- `src/mail/daemon/src/mail_daemon/maild/api.py` -- `spec/SPEC.md` section 8 +## Options -## Entries to Cover - -- Usage. -- Log-level options. -- Required environment variables. -- Server validation. -- Token acquisition. -- Message buffer clearing. -- Local delivery. -- Retry and error behavior. - -## Maintenance Notes - -Keep behavioral details factual and tied to implementation. Deeper discussion of -why delivery is daemon-driven belongs in explanations. +- `--license` — show license information and exit +- `-llf`, `--log-level-file` `LEVEL` — file log level (default: info) +- `-llc`, `--log-level-console` `LEVEL` — console log level (default: info) diff --git a/docs/references/server-cli.md b/docs/references/server-cli.md index 435093c..3f5d209 100644 --- a/docs/references/server-cli.md +++ b/docs/references/server-cli.md @@ -1,27 +1,19 @@ # Server CLI -Status: stub +Status: generated -## Scope +> **Generated file — do not edit by hand.** Regenerate with `uv run python scripts/build_cli_docs.py` after changing the CLI. See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md). -Describe the `mail-server` CLI and related backend initialization command. +The Python/FastAPI server for the Multi-Agent Interface Layer (MAIL) -## Source of Truth +Invoke as `mail-server` (or `uv run mail-server` from a workspace checkout). Source: `mail_server/cli.py`. -- `src/mail/server/src/mail_server/cli.py` -- `src/mail/server/src/mail_server/backend_init.py` -- `src/mail/server/docs/reference/cli.md` +## Options -## Entries to Cover - -- `mail-server` usage. -- Host and port options. -- Backend selection. -- Memory checkpoint interval. -- `backend-init` usage. -- Backend initialization options. -- Environment variables required at runtime. - -## Maintenance Notes - -Keep default values synchronized with parser defaults. +- `--license` — show license information and exit +- `-H`, `--host` `HOST` — the IP address to bind to (default: 127.0.0.1) +- `-p`, `--port` `PORT` — the port for the server to listen on (default: 8865) +- `-b`, `--backend` `BACKEND` — the MAIL server backend to use (default: memory) +- `--memory-save-interval` `SECONDS` — seconds between memory backend filesystem checkpoints; set 0 to disable (default: 60.0) +- `--sqlite-path` `PATH` — sqlite backend database file (env: MAIL_SQLITE_PATH; default: ~/.mail-swarms/deployments/default/mail.db) +- `--database-url` `URL` — sqlite backend database URL; takes precedence over --sqlite-path (env: MAIL_DATABASE_URL) diff --git a/scripts/build_cli_docs.py b/scripts/build_cli_docs.py new file mode 100644 index 0000000..723e4cc --- /dev/null +++ b/scripts/build_cli_docs.py @@ -0,0 +1,179 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 MAIL Contributors +"""Generate the CLI reference pages under docs/references/ from the argparse +parsers that each MAIL command builds. + +Each MAIL CLI exposes a ``build_parser() -> argparse.ArgumentParser``. This +script imports those parsers and renders one Markdown reference per CLI, so the +docs cannot drift from the actual flags and subcommands. Regenerate with: + + uv run python scripts/build_cli_docs.py + +The generated pages are committed; do not edit them by hand. +""" + +from __future__ import annotations + +import argparse +import importlib +from pathlib import Path + +DOCS_DIR = Path(__file__).resolve().parent.parent / "docs" / "references" + +# (output filename, page title, dotted path to build_parser, console script name) +CLIS = [ + ("client-cli.md", "Client CLI", "mail_client.cli", "mail"), + ("admin-cli.md", "Admin CLI", "mail_client.admin_panel", "mail-admin"), + ("server-cli.md", "Server CLI", "mail_server.cli", "mail-server"), + ("daemon-cli.md", "Daemon CLI", "mail_daemon.cli", "mail-daemon"), +] + +BANNER = ( + "> **Generated file — do not edit by hand.** Regenerate with " + "`uv run python scripts/build_cli_docs.py` after changing the CLI. " + "See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md)." +) + + +def _metavar(action: argparse.Action) -> str: + """A display placeholder for an option/positional that takes a value.""" + if action.nargs == 0: + return "" + if action.metavar: + return str(action.metavar) + if action.choices: + return "{" + ",".join(str(c) for c in action.choices) + "}" + return action.dest.upper() + + +def _help_text(action: argparse.Action, prog: str) -> str: + """Expand argparse %-substitutions (e.g. %(default)s) the way --help does.""" + raw = (action.help or "").strip() + if "%" not in raw: + return raw + params = {**vars(action), "prog": prog} + if params.get("choices") is not None: + params["choices"] = ", ".join(str(c) for c in params["choices"]) + try: + return raw % params + except (KeyError, ValueError, TypeError): + return raw + + +def _render_option(action: argparse.Action, prog: str) -> str: + flags = ", ".join(f"`{opt}`" for opt in action.option_strings) + meta = _metavar(action) + if meta: + flags += f" `{meta}`" + help_text = _help_text(action, prog) + return f"- {flags} — {help_text}".rstrip(" —") + + +def _render_positional(action: argparse.Action, prog: str) -> str: + name = action.metavar or action.dest + help_text = _help_text(action, prog) + return f"- `{name}` — {help_text}".rstrip(" —") + + +def _user_actions(parser: argparse.ArgumentParser): + """Actions worth documenting: skip -h/--help and the subparsers action.""" + for action in parser._actions: + if isinstance(action, argparse._HelpAction): + continue + if isinstance(action, argparse._SubParsersAction): + continue + yield action + + +def _subparsers_action(parser: argparse.ArgumentParser): + for action in parser._actions: + if isinstance(action, argparse._SubParsersAction): + return action + return None + + +def _arg_items(parser: argparse.ArgumentParser) -> tuple[list[str], list[str]]: + prog = parser.prog + positionals = [ + _render_positional(a, prog) + for a in _user_actions(parser) + if not a.option_strings + ] + options = [ + _render_option(a, prog) for a in _user_actions(parser) if a.option_strings + ] + return positionals, options + + +def _labeled_block(parser: argparse.ArgumentParser) -> list[str]: + """Argument/option lists under a bold mini-label (used inside subcommands).""" + positionals, options = _arg_items(parser) + lines: list[str] = [] + if positionals: + lines += ["**Arguments:**", "", *positionals, ""] + if options: + lines += ["**Options:**", "", *options, ""] + return lines + + +def _render_subcommands(sub_action: argparse._SubParsersAction) -> list[str]: + # Map each subparser object to all the names (primary + aliases) that reach it. + names_by_parser: dict[int, list[str]] = {} + for name, subparser in sub_action.choices.items(): + names_by_parser.setdefault(id(subparser), []).append(name) + + lines: list[str] = ["## Commands", ""] + for pseudo in sub_action._choices_actions: + primary = pseudo.dest + subparser = sub_action.choices[primary] + aliases = [n for n in names_by_parser[id(subparser)] if n != primary] + heading = f"### `{primary}`" + if aliases: + heading += " (aliases: " + ", ".join(f"`{a}`" for a in aliases) + ")" + lines.append(heading) + lines.append("") + summary = (pseudo.help or subparser.description or "").strip() + if summary: + lines.append(summary) + lines.append("") + lines.extend(_labeled_block(subparser)) + return lines + + +def render_page(title: str, module_path: str, script: str) -> str: + module = importlib.import_module(module_path) + parser: argparse.ArgumentParser = module.build_parser() + + lines = [f"# {title}", "", "Status: generated", "", BANNER, ""] + description = (parser.description or "").strip() + if description: + lines.append(description) + lines.append("") + lines.append(f"Invoke as `{script}` (or `uv run {script}` from a workspace " + f"checkout). Source: `{module_path.replace('.', '/')}.py`.") + lines.append("") + + sub_action = _subparsers_action(parser) + positionals, options = _arg_items(parser) + if positionals: + lines += ["## Arguments", "", *positionals, ""] + if options: + lines += ["## Global options" if sub_action else "## Options", "", *options, ""] + + if sub_action is not None: + lines.extend(_render_subcommands(sub_action)) + + text = "\n".join(lines).rstrip() + "\n" + return text + + +def main() -> None: + for filename, title, module_path, script in CLIS: + page = render_page(title, module_path, script) + out = DOCS_DIR / filename + out.write_text(page, encoding="utf-8") + print(f"wrote {out.relative_to(DOCS_DIR.parent.parent)}") + + +if __name__ == "__main__": + main() From 750a2a3e82c4b542532fa396adf6a3f20f0c3afa Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 15:58:10 -0400 Subject: [PATCH 24/28] fix: align identifier length cap to spec (31 -> 32) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SPEC.md §6 says agent/user/admin/daemon-worker/swarm/list identifiers SHOULD be <= 32 chars, but the reference implementation hard-capped at 31. Raise all name/keyword *_LEN_MAX constants from 31 to 32 so the impl matches the spec (message tags were already 32). Contract/unit length tests are constant-relative, so they follow automatically; full suite green (769 passed). Update the docs that had flagged the divergence. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanations/addressing-model.md | 8 +++--- docs/references/data-models.md | 5 ++-- docs/references/protocol-specification.md | 26 ++++++++----------- .../src/mail_protocol/core/constants.py | 12 ++++----- 4 files changed, 23 insertions(+), 28 deletions(-) diff --git a/docs/explanations/addressing-model.md b/docs/explanations/addressing-model.md index 13019aa..d63ec3f 100644 --- a/docs/explanations/addressing-model.md +++ b/docs/explanations/addressing-model.md @@ -111,10 +111,10 @@ Identifiers in MAIL — every `ua_id`, `agent`, `swarm`, and `list_id` — must ``` So `welfare-discourse` is valid; `Welfare_Discourse`, `-leading`, `trailing-`, -and `double--hyphen` are not. Each identifier must be at least one character. The -[protocol spec](../references/protocol-specification.md) recommends a maximum of -32 characters, but the reference implementation enforces a hard cap of 31 — -longer identifiers are rejected with a validation error. +and `double--hyphen` are not. Each identifier must be at least one character and +at most 32, matching the [protocol spec](../references/protocol-specification.md)'s +recommendation; the reference implementation enforces 32 as a hard cap and rejects +longer identifiers with a validation error. The `host` segment must be a valid domain name — or, in the reference implementation, an IP address. diff --git a/docs/references/data-models.md b/docs/references/data-models.md index e99c96e..2dbb518 100644 --- a/docs/references/data-models.md +++ b/docs/references/data-models.md @@ -222,13 +222,12 @@ shapes: | Message subject | 1 / 256 | | Message body | 1 / 65535 | | Message tag | 1 / 32 | -| Agent / user / admin / daemon-worker / swarm / swarm-keyword / list name | 1 / **31** | +| Agent / user / admin / daemon-worker / swarm / swarm-keyword / list name | 1 / 32 | | Swarm description | 0 / 255 | `LIST_ADDRESS_PREFIX = "list"`. The protocol version literal `"2.0"` is not a constant — it is a `Literal` on `MAILMessage.mail_version` and -`RootGetResponse.protocol_version`. (Note the identifier max of 31 differs from -SPEC §6's "SHOULD ≤ 32" — see [Protocol Specification](protocol-specification.md#known-spec--implementation-divergences).) +`RootGetResponse.protocol_version`. The identifier max of 32 matches SPEC §6. ## Validators diff --git a/docs/references/protocol-specification.md b/docs/references/protocol-specification.md index 93bd7b8..71ab8e8 100644 --- a/docs/references/protocol-specification.md +++ b/docs/references/protocol-specification.md @@ -60,25 +60,21 @@ Conformance is enforced by the suite in [`tests/contract/`](../../tests/contract Run them via [Run the Test Suite](../howtos/run-tests.md). -## Known spec ↔ implementation divergences +## Specification and implementation alignment -The reference implementation intentionally (or incidentally) differs from the -prose in a few places. These are tracked here so readers trust the code over the -prose where they disagree: - -- **Identifier length cap.** SPEC §6 says agent / user / admin / daemon-worker / - swarm / list identifiers "SHOULD be no longer than **32** characters." The - reference implementation enforces a hard cap of **31** - (`core/constants.py`: `*_NAME_LEN_MAX = 31`), rejecting longer values with a - validation error. Message tags, by contrast, use a cap of 32 in both - (`MESSAGE_TAG_LEN_MAX = 32`, matching SPEC §7.10). This name-length mismatch - is unresolved — see the maintenance note below. +The reference implementation tracks the spec closely. Identifier length is one +alignment point worth calling out: SPEC §6 recommends that agent / user / admin / +daemon-worker / swarm / list identifiers be at most 32 characters, and the +implementation enforces exactly that as a hard cap (`core/constants.py`: +`*_NAME_LEN_MAX = 32`), consistent with the message-tag cap +(`MESSAGE_TAG_LEN_MAX = 32`, SPEC §7.10). No known divergences remain; if one +arises, record it here so readers know to trust the code over the prose. ## Maintenance notes Do not copy the specification text into this page; keep it as an index that links -to the exact normative files. When the implementation and `SPEC.md` are -reconciled (e.g. the 31-vs-32 identifier cap), update both the divergences list -here and [Addressing Model](../explanations/addressing-model.md). +to the exact normative files. If the implementation and `SPEC.md` diverge, record +it in the alignment section above and cross-link the affected reference pages such +as [Addressing Model](../explanations/addressing-model.md). [rfc-2119]: https://datatracker.ietf.org/doc/html/rfc2119 diff --git a/src/mail/protocol/src/mail_protocol/core/constants.py b/src/mail/protocol/src/mail_protocol/core/constants.py index dd3d4c9..ff57c5d 100644 --- a/src/mail/protocol/src/mail_protocol/core/constants.py +++ b/src/mail/protocol/src/mail_protocol/core/constants.py @@ -11,26 +11,26 @@ MESSAGE_TAG_LEN_MAX = 32 AGENT_NAME_LEN_MIN = 1 -AGENT_NAME_LEN_MAX = 31 +AGENT_NAME_LEN_MAX = 32 USER_NAME_LEN_MIN = 1 -USER_NAME_LEN_MAX = 31 +USER_NAME_LEN_MAX = 32 DAEMON_WORKER_NAME_LEN_MIN = 1 -DAEMON_WORKER_NAME_LEN_MAX = 31 +DAEMON_WORKER_NAME_LEN_MAX = 32 SWARM_NAME_LEN_MIN = 1 -SWARM_NAME_LEN_MAX = 31 +SWARM_NAME_LEN_MAX = 32 SWARM_DESCRIPTION_LEN_MIN = 0 SWARM_DESCRIPTION_LEN_MAX = 255 SWARM_KEYWORD_LEN_MIN = 1 -SWARM_KEYWORD_LEN_MAX = 31 +SWARM_KEYWORD_LEN_MAX = 32 # List length limits mirror agent names: short slugs, swarm-scoped. LIST_NAME_LEN_MIN = 1 -LIST_NAME_LEN_MAX = 31 +LIST_NAME_LEN_MAX = 32 # The literal prefix that identifies a list-shaped MAIL address. # Example address: ``list:welfare-discourse@chorus@localhost``. From 9b72cb52fccd0d2f355e84d8efcb1812fdb334bc Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 15:58:41 -0400 Subject: [PATCH 25/28] docs: correct memory-vs-sqlite backend capability gap While verifying the 31->32 change against the full suite, the xfail'd tests/integration/test_stubs.py surfaced that six operations are NotImplementedError on the *memory* backend only and fully implemented on sqlite: DELETE inbox/draft/trash, POST /trash/clear, PATCH /admin/webhooks/{id}, and POST /daemon/deliver/remote. The reference docs had presented these as universally available (and wrongly called remote delivery unimplemented in both backends). Add a memory-backend-gaps note to http-api, a limitations entry + capability row to storage-backends, and correct the delivery-model remote-delivery paragraph. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanations/delivery-model.md | 9 ++++----- docs/references/http-api.md | 9 ++++++++- docs/references/storage-backends.md | 12 +++++++++--- 3 files changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/explanations/delivery-model.md b/docs/explanations/delivery-model.md index cbac982..5e76c8f 100644 --- a/docs/explanations/delivery-model.md +++ b/docs/explanations/delivery-model.md @@ -101,12 +101,11 @@ notified of new mail rather than having to poll; see [HTTP API](../references/ht ## Local versus remote delivery -The implemented delivery path is **local**: `POST /daemon/deliver/local` carries +The primary delivery path is **local**: `POST /daemon/deliver/local` carries messages between user-agents on the *same* server. A second endpoint, -`POST /daemon/deliver/remote`, is reserved for future delivery of messages that -arrive from other MAIL servers. The route is wired, but its backend handler -currently raises `NotImplementedError`, so remote delivery is not yet -functional. For now, treat delivery as a within-host operation. +`POST /daemon/deliver/remote`, accepts messages sent by agents on other MAIL +servers for delivery to local recipients. It is implemented on the SQLite +backend; on the memory backend it currently raises `NotImplementedError`. ## Pre-send versus post-send errors diff --git a/docs/references/http-api.md b/docs/references/http-api.md index 079662e..8c0551d 100644 --- a/docs/references/http-api.md +++ b/docs/references/http-api.md @@ -32,6 +32,13 @@ Most responses wrap their payload in a named field (`entry`, `entries`, box reads nest the message under an `entry`. Field shapes are in [Data Models](data-models.md). +> **Memory-backend gaps.** A few endpoints are implemented only on the SQLite +> backend; on the memory backend they raise `NotImplementedError`: +> `DELETE /inbox/{message_id}`, `DELETE /drafts/{draft_id}`, +> `DELETE /trash/{message_id}`, `POST /trash/clear`, +> `PATCH /admin/webhooks/{webhook_id}`, and `POST /daemon/deliver/remote`. See +> [Storage Backends](storage-backends.md#current-limitations). + ## Root and health | Method | Path | Auth | Response model | @@ -110,7 +117,7 @@ Used by delivery daemons; see [Delivery Model](../explanations/delivery-model.md | --- | --- | --- | --- | | POST | `/daemon/message-buffer/clear` | daemon | Drain the pending-delivery buffer. | | POST | `/daemon/deliver/local` | daemon | Deliver messages between user-agents on this server. | -| POST | `/daemon/deliver/remote` | daemon | Reserved for cross-server delivery; handler currently raises `NotImplementedError`. | +| POST | `/daemon/deliver/remote` | daemon | Inbound cross-server delivery; implemented on SQLite, raises `NotImplementedError` on the memory backend (see note above). | ## Admin endpoints (`/admin`) diff --git a/docs/references/storage-backends.md b/docs/references/storage-backends.md index 6135a15..9176c67 100644 --- a/docs/references/storage-backends.md +++ b/docs/references/storage-backends.md @@ -119,6 +119,7 @@ Files: `backends/sqlite/` — `api.py`, `database.py`, `schema.py`, | Deployments | Runtime reads/writes the `default` deployment only (see limitations) | Arbitrary via `--sqlite-path` / `--database-url` | | Init on re-run | Overwrites | Idempotent | | Migration import | — | `--import-fs` | +| Delete/clear, webhook patch, remote deliver | Not implemented (raises `NotImplementedError`) | Implemented | Both implement the identical interface and share the webhook delivery logic, so inbox `is_read`, refresh-token families, list membership, and webhook semantics @@ -126,13 +127,18 @@ match across backends. ## Current limitations +- **Memory backend: unimplemented operations.** Several operations raise + `NotImplementedError` on the memory backend and are only available on SQLite: + `DELETE /inbox/{message_id}`, `DELETE /drafts/{draft_id}`, + `DELETE /trash/{message_id}`, `POST /trash/clear`, + `PATCH /admin/webhooks/{webhook_id}`, and `POST /daemon/deliver/remote`. Choose + the SQLite backend if you need message deletion / trash clearing, webhook + patching, or inbound remote delivery. These gaps are pinned as `xfail` in + [`tests/integration/test_stubs.py`](../../tests/integration/test_stubs.py). - **Memory backend deployment name.** The memory runtime's filesystem layer is pinned to the `default` deployment: `backend-init` will *create* a named memory deployment, but `mail-server --backend memory` reads and writes `default` regardless. Use the SQLite backend for non-default deployment names. -- **`daemon_deliver_remote`** is unimplemented in both backends (raises - `NotImplementedError`); remote/cross-server delivery is reserved. See - [Delivery Model](../explanations/delivery-model.md). - No Postgres backend yet, though `normalize_database_url` reserves a `postgresql+psycopg` driver seam. From 20a681d8e29e15f317007a09fd91db5ceec66131 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 16:01:31 -0400 Subject: [PATCH 26/28] docs: write explanation pages (P2, explanations category) Fill in the five explanation stubs against SPEC.md, the package layout, the auth/refresh-token implementation, and the legacy README: - mail-v2-overview: what MAIL is, the v1->v2 comms-only refocus, non-goals - architecture: how protocol/server/client/daemon fit; message lifecycle - security-model: trust boundaries, bearer + refresh tokens, secret handling, production expectations - mail-v1-legacy: how to read the archived v1 runtime without importing its assumptions; how to run legacy tests - documentation-system: the Divio four-category model and how to place a page Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/explanations/architecture.md | 89 ++++++++++++++++++----- docs/explanations/documentation-system.md | 79 ++++++++++++++++---- docs/explanations/mail-v1-legacy.md | 69 ++++++++++++++---- docs/explanations/mail-v2-overview.md | 82 ++++++++++++++------- docs/explanations/security-model.md | 82 ++++++++++++++++----- 5 files changed, 305 insertions(+), 96 deletions(-) diff --git a/docs/explanations/architecture.md b/docs/explanations/architecture.md index ae8dcca..b29d4e2 100644 --- a/docs/explanations/architecture.md +++ b/docs/explanations/architecture.md @@ -1,32 +1,83 @@ # Architecture -Status: stub +Status: draft -## Question +MAIL v2 is assembled from four workspace packages around one idea: a **server +owns state and enforces the contract**, **daemons move messages**, and **clients +are just authenticated user-agents**. The protocol package is the shared contract +all three depend on. This page explains how they fit; for the file-level map see +[Repository Layout](../references/repository-layout.md). -How do the active MAIL v2 packages and runtime components fit together? +## The components -## Source Material +```text + ┌─────────────┐ HTTP ┌──────────────────────┐ + │ client │ ───────────────▶ │ server │ + │ (mail / │ ◀─────────────── │ (mail-server) │ + │ mail-admin) │ │ routes + auth + │ + └─────────────┘ │ backend (state) │ + └──────────┬───────────┘ + ┌─────────────┐ poll + deliver │ + │ daemon │ ◀─────────────────────────────┘ + │(mail-daemon)│ HTTP (/daemon/*) + └─────────────┘ +``` -- `README.md` -- `pyproject.toml` -- `src/mail/protocol/` -- `src/mail/server/` -- `src/mail/client/` -- `src/mail/daemon/` -- `spec/SPEC.md` section 4 +- **`mail-swarms-protocol`** — the shared data and network contract. Pydantic + models for messages, drafts, boxes, swarms, lists, webhooks, and user-agents, + plus the validators that enforce address and field rules. Server, client, and + daemon all import it, so there is exactly one definition of what a message *is*. + See [Data Models](../references/data-models.md). +- **`mail-swarms-server`** — the FastAPI HTTP implementation and the **owner of + all state**. It authenticates user-agents, holds inboxes/outboxes/drafts/trash, + manages swarms and lists, and performs the first and last steps of delivery. + See [HTTP API](../references/http-api.md). +- **`mail-swarms-client`** — the CLI (`mail` and `mail-admin`). A client is just + a convenient front-end for an authenticated user-agent; it holds no + authoritative state and speaks only the HTTP contract. +- **`mail-swarms-daemon`** — the delivery worker. It authenticates as a daemon + user-agent, polls the server for pending messages, and delivers them to + recipients' inboxes. Delivery is deliberately *not* the server's job — see + [Delivery Model](delivery-model.md). -## Topics to Discuss +## Swarms -- Protocol package as shared data and network contracts. -- Server package as the HTTP implementation and state owner. -- Client package as the CLI user-agent interface. -- Daemon package as the delivery worker. -- Backend abstraction and current memory backend. -- OpenAPI and contract tests as cross-package alignment points. +A swarm is an abstract collection of agent addresses, mailing lists, and metadata +inside a server (SPEC §4.3). Swarms scope discrete multi-agent deployments: an +agent's address is swarm-scoped (`name@swarm@host`), while admins, users, and +daemons are host-scoped. See [Addressing Model](addressing-model.md). -## Related Pages +## State ownership and the backend abstraction + +The server does not hardcode a database. It talks to a `MAILServerBackend` +interface, and two implementations ship: an in-memory backend with filesystem +checkpointing (the default, good for development) and a transactional SQLite +backend (durable). Swapping storage never changes the HTTP contract. See +[Storage Backends](../references/storage-backends.md). + +## How a message flows + +1. A user-agent **creates a draft** on the server (`POST /drafts`). +2. It **sends** the draft to recipients (`POST /drafts/{id}/send`); the server + assembles a `MAILMessage`, records it in the sender's outbox, and queues it. +3. A **daemon** picks up the queued message and delivers a copy to each + recipient's inbox (`POST /daemon/deliver/local`). +4. Recipients read their inbox; the server can fire webhooks on delivery. + +This split — server as system of record, daemon as courier — is what lets +delivery status be reported honestly ("sent" vs "delivered by `daemon:…`"). + +## Cross-package alignment + +Because four packages must agree on one contract, two artifacts keep them in +sync: the generated [`spec/openapi.yaml`](../../spec/openapi.yaml) (the +authoritative wire contract, derived from the server app) and the conformance +suite in [`tests/contract/`](../../tests/contract). See +[Protocol Specification](../references/protocol-specification.md). + +## Related pages - [Repository Layout](../references/repository-layout.md) - [HTTP API](../references/http-api.md) - [Storage Backends](../references/storage-backends.md) +- [Delivery Model](delivery-model.md) diff --git a/docs/explanations/documentation-system.md b/docs/explanations/documentation-system.md index faf2812..9178997 100644 --- a/docs/explanations/documentation-system.md +++ b/docs/explanations/documentation-system.md @@ -1,28 +1,79 @@ # Documentation System -Status: stub +Status: draft -## Question +MAIL's docs follow the [Divio documentation system][divio] (also called +Diátaxis): every page belongs to exactly one of four categories, each serving a +different reader need. This page explains how to decide where a new page belongs +and why the separation matters. The index and writing rules live in +[docs/README.md](../README.md). -How should MAIL maintainers decide whether a new page is a tutorial, how-to -guide, reference page, or explanation? +## The four categories -## Source Material +| Category | Serves | Reader is… | Optimizes for | +| --- | --- | --- | --- | +| **[Tutorial](../tutorials/README.md)** | Learning | a beginner following along | a guaranteed, repeatable success | +| **[How-to guide](../howtos/README.md)** | A task | someone who knows the basics | getting a specific job done | +| **[Reference](../references/README.md)** | Looking up | someone who knows what they want | accuracy and completeness | +| **[Explanation](README.md)** | Understanding | someone thinking about the system | context, motivation, tradeoffs | -- `docs/README.md` -- Divio documentation system: +The split is really two axes: *practical* (tutorials, how-tos) vs *theoretical* +(reference, explanation), and *studying* (tutorials, explanation) vs *working* +(how-tos, reference). A page should sit in one cell of that grid. -## Topics to Discuss +## Why mixed-purpose pages fail -- The four documentation types and what each optimizes for. -- Why mixed-purpose pages become hard to maintain. -- How to split a proposed page that has multiple purposes. -- Naming conventions for each category. -- Migration strategy for package-local docs and legacy docs. +A page that teaches, solves a task, lists exact fields, *and* argues motivation +all at once serves no reader well: the beginner drowns in reference detail, the +practitioner wades through backstory to find a command, and the lookup reader +can't trust a page that also editorializes. Mixed pages also rot faster, because a +change to the API forces edits to prose that was really about concepts. Keeping +each page to one job keeps it short, trustworthy, and cheap to maintain. -## Related Pages +## How to place (or split) a page + +Ask **what the reader is doing** when they open it: + +- *"Walk me through my first success."* → Tutorial. Avoid optional branches; show + visible progress fast; it must run end to end. +- *"How do I do X?"* → How-to. Assume basics; stay task-focused; link out to + explanations instead of pausing to teach concepts. +- *"What are the exact endpoints / fields / flags?"* → Reference. Mirror the + implementation; prefer generated artifacts; keep opinion out. +- *"Why does it work this way?"* → Explanation. Discuss motivation and + alternatives; link to tutorials, how-tos, and reference for action and lookup. + +If a proposed page answers more than one of these, split it and cross-link the +parts. A common shape in this repo: an explanation (e.g. +[Mailing Lists](mailing-lists.md)) paired with a how-to +([Manage Mailing Lists](../howtos/manage-mailing-lists.md)) and a reference +([HTTP API](../references/http-api.md)). + +## Naming conventions + +- **Tutorials** read as an outcome or a journey: *Run MAIL Locally*, *Send Your + First MAIL Message*. +- **How-tos** are imperative tasks: *Manage Swarms*, *Authenticate a User-Agent*. +- **Reference** pages are noun topics: *HTTP API*, *Data Models*, *Configuration*. +- **Explanations** are concept nouns: *Delivery Model*, *Security Model*. + +Files are kebab-cased within the category directory +(`howtos/manage-swarms.md`); reference pages that mirror generated artifacts note +that they are generated (e.g. the CLI references). + +## Migrating package-local and legacy docs + +Some packages still carry their own `docs/` (`src/mail/server/docs/`, +`src/mail/client/docs/`) that predate this set, and the v1 archive has its own +docs under `src/mail/legacy/docs/`. This top-level tree is canonical; migrate +package-local material into the right category here and treat legacy docs as +historical reference (see [MAIL v1 Legacy Runtime](mail-v1-legacy.md)). + +## Related pages - [Tutorials](../tutorials/README.md) - [How-To Guides](../howtos/README.md) - [Reference](../references/README.md) - [Explanations](README.md) + +[divio]: https://docs.divio.com/documentation-system/ diff --git a/docs/explanations/mail-v1-legacy.md b/docs/explanations/mail-v1-legacy.md index 297fcc3..2d74897 100644 --- a/docs/explanations/mail-v1-legacy.md +++ b/docs/explanations/mail-v1-legacy.md @@ -1,28 +1,65 @@ # MAIL v1 Legacy Runtime -Status: stub +Status: draft -## Question +The MAIL v1 reference runtime is archived under +[`src/mail/legacy/`](../../src/mail/legacy). It is kept for historical reference +and compatibility while the repository moves to the v2 package layout — it is +**not** the current implementation surface. This page explains how to read it +without carrying v1 assumptions into v2 work. -How should readers understand the archived MAIL v1 runtime while working on -active MAIL v2 documentation and code? +## Why v1 is archived -## Source Material +MAIL v1 bundled the communication contract together with an agent *runtime*: +message/task/action models, tool execution, LiteLLM-backed agent factories, and a +debug UI. MAIL v2 deliberately narrows the protocol to communication only (see +[MAIL v2 Overview](mail-v2-overview.md)), so the v1 runtime no longer reflects how +MAIL is meant to work. Rather than delete it, it is quarantined under +`src/mail/legacy/` so old examples and behavior remain available for reference and +porting. -- `README.md` -- `src/mail/legacy/README.md` -- `src/mail/legacy/docs/README.md` -- `src/mail/legacy/` +## What lives there -## Topics to Discuss +The archive holds the v1 runtime (`api.py`, `server.py`, `client.py`, `cli.py`, +`core/`), agent factories, standard action libraries, example swarms, the +`swarms.json` machinery, interswarm routing, optional persistence helpers, and the +v1 debug UI — plus archived v1 root artifacts (its `docs/`, configs, Dockerfile, +`README.v1.md`, `AGENTS.v1.md`, `CLAUDE.v1.md`, and tests). Legacy imports use the +`mail.legacy.*` namespace. -- Why v1 code is archived under `src/mail/legacy`. -- Which docs are historical reference versus active v2 guidance. -- When legacy tests should still be run. -- How to avoid importing v1 architecture assumptions into v2 docs. -- Migration notes worth preserving. +## Historical vs active documentation -## Related Pages +Treat anything under `src/mail/legacy/` — including its `docs/` — as **historical +reference**. Active guidance is this top-level `docs/` tree plus +[`spec/`](../../spec). Archived prose may still mention old `mail.*` import paths +or v1 endpoints (e.g. the `/ui/*` debug routes) that the v2 server does not +provide; do not treat those as current. + +## Running legacy tests + +Legacy tests are **not** part of the default suite. Run them explicitly, and only +when maintaining v1 behavior: + +```bash +uv run pytest # v2 / default suite +uv run --extra legacy pytest src/mail/legacy/tests # legacy runtime tests +``` + +See [Run the Test Suite](../howtos/run-tests.md). + +## Working near the archive + +- **Don't import v1 architecture into v2 docs or code.** v2 is a communication + protocol, not a runtime; keep runtime/tool-execution concepts out of v2 pages. +- **Port, don't extend.** Prefer moving behavior forward into the v2 packages over + growing the v1 APIs. Keep legacy changes scoped to compatibility, security, or + migration support. +- **Don't modernize examples in place.** Leave v1 examples as-is unless the code + has actually been ported — a v2-looking example backed by v1 code is misleading. +- **A future UI** should be built against the v2 `mail-server` / `mail-client` / + `mail-protocol` surfaces, not by adapting the archived v1 UI. + +## Related pages - [Repository Layout](../references/repository-layout.md) - [MAIL v2 Overview](mail-v2-overview.md) diff --git a/docs/explanations/mail-v2-overview.md b/docs/explanations/mail-v2-overview.md index c5f8ba7..71819da 100644 --- a/docs/explanations/mail-v2-overview.md +++ b/docs/explanations/mail-v2-overview.md @@ -1,30 +1,56 @@ # MAIL v2 Overview -Status: stub - -## Question - -What is MAIL v2, what problem does it solve, and what is intentionally outside -its scope? - -## Source Material - -- `README.md` -- `spec/SPEC.md` sections 1 through 4 -- `src/mail/legacy/README.md` - -## Topics to Discuss - -- MAIL as an email-like communication layer for humans, agents, daemons, and - swarms. -- The shift from v1 runtime concerns to v2 communication concerns. -- Why the active repo is split into protocol, server, client, and daemon - packages. -- What MAIL is not: not a full agent runtime and not a universal communication - layer for every agent interaction. - -## Related Pages - -- [Run MAIL Locally](../tutorials/run-local-mail.md) -- [Protocol Specification](../references/protocol-specification.md) -- [MAIL v1 Legacy Runtime](mail-v1-legacy.md) +Status: draft + +## What MAIL is + +The Multi-Agent Interface Layer (MAIL) is an open protocol for **email-like +communication** between humans and AI agents. It defines three things: a set of +data-structure primitives (messages, drafts, boxes, swarms, lists), an HTTP +contract for client–server interaction, and the terminology and rules that tie +them together. Participants — human users, AI agents, delivery daemons — are +addressable *user-agents* with their own inboxes, and they exchange messages much +as people exchange email. + +## The problem v2 solves + +MAIL v1 defined an inter-agent messaging contract *and* prescribed how +multi-agent systems should run: their runtime environment, tool usage, and +execution model. By 2026 that coupling looked like a mistake. Terminal-style +agents showed that an agent's runtime does not need to be defined in the same +place as its communication contract. + +MAIL v2 draws a hard line: it specifies the **communication layer and as little +else as possible**. Two goals drive it (SPEC §3.1): + +- **Focus on communication.** Runtime, tool execution, and agent internals are + explicitly out of scope. +- **Don't reinvent the wheel.** Where good standards already exist (HTTP, OAuth2, + JSON, RFC-3339 timestamps), MAIL builds on them rather than redefining them. + +## Why the repository is split into packages + +The refocus is reflected in the code layout. Instead of one runtime, v2 is four +small workspace packages with a single responsibility each — a shared protocol +contract, a server, a client, and a delivery daemon (see +[Architecture](architecture.md) and [Repository Layout](../references/repository-layout.md)). +This keeps the wire contract (`mail-swarms-protocol`) independent of any one +implementation, so alternative servers or clients can conform to the same +protocol. + +## What MAIL is not (SPEC §3.2) + +- **Not an agent runtime.** MAIL does not say how an agent thinks, acts, or runs; + it only carries messages between agents. +- **Not the only way agents may communicate.** MAIL mirrors email — ubiquitous, + but not the sole channel. Agents are free to use whatever else fits a given + job; MAIL is the shared, email-like layer, not a mandate for *all* inter-agent + traffic. + +## Related pages + +- [Run MAIL Locally](../tutorials/run-local-mail.md) — see it work end to end. +- [Architecture](architecture.md) — how the pieces fit together. +- [Protocol Specification](../references/protocol-specification.md) — the + normative source. +- [MAIL v1 Legacy Runtime](mail-v1-legacy.md) — what the archived v1 code is. diff --git a/docs/explanations/security-model.md b/docs/explanations/security-model.md index 8811814..0e44534 100644 --- a/docs/explanations/security-model.md +++ b/docs/explanations/security-model.md @@ -1,31 +1,75 @@ # Security Model -Status: stub +Status: draft -## Question +MAIL's security model follows from one fact: the **server is the sole authority**. +It owns all state, authenticates every user-agent, and enforces what each may do. +Clients and daemons hold no authority of their own — they act only with a valid +token. This page explains the trust boundaries and operational expectations; the +normative security clauses are SPEC §9, and the implementation is in +[`auth.py`](../../src/mail/server/src/mail_server/auth.py) and +[`routers/auth.py`](../../src/mail/server/src/mail_server/routers/auth.py). -What are MAIL trust boundaries, credential risks, and minimum operational -expectations? +## Authentication: passwords to tokens -## Source Material +A user-agent exchanges its address and password for a short-lived **access +token** (a JWT) via `POST /auth/token`. Every subsequent request carries it as +`Authorization: Bearer `, and the server verifies the signature and expiry +on each call. Token lifetime is set by `MAIL_JWT_EXPIRE_MINUTES`; the signing key +and algorithm by `MAIL_JWT_SECRET_KEY` / `MAIL_JWT_ALGORITHM` (see +[Configuration](../references/configuration.md)). See +[Authenticate a User-Agent](../howtos/authenticate-user-agent.md). -- `spec/SPEC.md` section 9 -- `src/mail/server/src/mail_server/auth.py` -- `src/mail/server/src/mail_server/routers/auth.py` -- `src/mail/server/.env.example` -- `tests/integration/test_auth.py` -- `tests/integration/test_authz.py` +### Refresh tokens -## Topics to Discuss +Because access tokens are short-lived, **interactive principals** — users and +admins — also receive a **refresh token** at login, which renews an access token +without re-entering the password. Agents and daemons are *not* interactive +principals: they re-authenticate with their credentials instead. The design: -- User-agent credentials and bearer tokens. -- Admin power and account creation risks. -- Daemon privileges. -- Secret handling in CLI and environment variables. -- TLS and reverse proxy expectations for production. -- Logging risks for message content and credentials. +- **Rotation.** Each `POST /auth/refresh` invalidates the presented token and + issues a replacement. Tokens are grouped into a *family* with a single absolute + expiry carried forward across rotations. +- **Reuse detection.** Presenting an already-rotated token is treated as + compromise and revokes the whole family; a password reset revokes all families. +- **Transport.** For browsers the refresh token is an `httpOnly`, + `SameSite=strict` cookie scoped to `/auth` (so it is never sent to the wider + API), with the `Secure` flag on by default (`MAIL_COOKIE_SECURE`). CLI clients, + which cannot use the cookie, send it in the request body. -## Related Pages +## Trust boundaries by role + +- **Admins** are the most powerful principals: they create and delete agents, + users, daemons, swarms, lists, and webhooks. Admin credentials are effectively + server-control credentials — generate and hand them out with extreme caution + (SPEC §5.1). +- **Daemons** are trusted couriers. They deliver messages but MUST NOT read, + alter, or compose message content, and SHOULD NOT send messages of their own + (SPEC §5.3). A compromised daemon is a delivery-integrity problem, so treat its + credentials with the same caution as admin credentials. +- **Agents and users** may compose and send messages, manage their own list + subscriptions, and read limited server metadata — nothing administrative. + +## Secret handling + +- **Credentials via environment, not arguments.** Clients and daemons read + `MAIL_PASSWORD` / tokens from environment variables rather than command-line + flags, keeping secrets out of shell history (SPEC §9.1–9.2). The CLI never + writes tokens to disk — it prints them for you to export. +- **Don't log sensitive data.** Message contents and credentials SHOULD NOT be + logged by clients, daemons, or the server (SPEC §9). +- **Plaintext init secrets.** `backend-init` writes generated passwords in + plaintext under `.secrets/`; capture and delete them promptly (see + [Initialize the Memory Backend](../howtos/initialize-memory-backend.md)). + +## Production expectations (SPEC §9.3) + +- Serve over **TLS**; keep `MAIL_COOKIE_SECURE` on so refresh cookies are + HTTPS-only. +- Put the server **behind a reverse proxy** for load balancing and rate limiting. +- Rotate user-agent passwords periodically (SPEC §9.4). + +## Related pages - [Authenticate a User-Agent](../howtos/authenticate-user-agent.md) - [Configuration](../references/configuration.md) From 4e7a1b618a0916b25c7073061ffc9550543a197a Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 16:11:44 -0400 Subject: [PATCH 27/28] docs: write how-to guides (P2, how-tos category) Fill in the final six how-to stubs against the CLIs, scripts, and test config: - run-server: required env vars, backend choice, checkpoint interval, health check - run-daemon: daemon credentials, startup/poll behavior, log levels - run-tests: default run, marker/path subsets, coverage, legacy extra, and how to read drift/contract failures and the memory-backend xfail stubs - send-message-cli: the compose -> send two-step, outbox/inbox inspection, validation failures - manage-user-agents: mail-admin create/list/get/delete for agents/users/daemons (password is prompted, not passed as an arg) - regenerate-api-artifacts: openapi + CLI docs + llms.txt + third-party notices, with drift validation Completes P2; docs/ now has no remaining stubs. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/howtos/manage-user-agents.md | 93 ++++++++++++++++++------ docs/howtos/regenerate-api-artifacts.md | 95 +++++++++++++++++++------ docs/howtos/run-daemon.md | 65 ++++++++++++----- docs/howtos/run-server.md | 94 ++++++++++++++++++------ docs/howtos/run-tests.md | 80 ++++++++++++++++----- docs/howtos/send-message-cli.md | 89 ++++++++++++++++++----- 6 files changed, 404 insertions(+), 112 deletions(-) diff --git a/docs/howtos/manage-user-agents.md b/docs/howtos/manage-user-agents.md index 34dea17..daea85b 100644 --- a/docs/howtos/manage-user-agents.md +++ b/docs/howtos/manage-user-agents.md @@ -1,35 +1,88 @@ # Manage User-Agents -Status: stub +Status: draft ## Goal -How to create, inspect, and remove MAIL agents, users, admins, and daemons with -administrator tooling. +Create, inspect, and remove MAIL agents, users, and daemons with the `mail-admin` +CLI. ## Starting Point -The reader has an admin token for the target server. +You have an **admin** `MAIL_TOKEN` for the target server (see +[Authenticate a User-Agent](authenticate-user-agent.md)). Admin accounts +themselves are created by `backend-init`, not by these commands — see +[Initialize the Memory Backend](initialize-memory-backend.md). For the address +shapes used below, see [Addressing Model](../explanations/addressing-model.md). + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +``` + +## Steps + +### 1. List existing user-agents by type + +```bash +uv run mail-admin agent-list +uv run mail-admin user-list +uv run mail-admin daemon-list +``` + +### 2. Create an agent in a swarm + +Agents are swarm-scoped, so the argument is the **local** `agent@swarm` form. The +command prompts for the new agent's password interactively (hidden input) rather +than taking it as an argument, keeping secrets out of shell history: + +```bash +uv run mail-admin agent-post supervisor@default +# agent password: ******** +``` + +### 3. Create a host-scoped user or daemon + +Users and daemons are host-scoped, so they take a bare id / worker name; both +prompt for a password: + +```bash +uv run mail-admin user-post alice # -> user:alice@{host} +uv run mail-admin daemon-post worker-1 # -> daemon:worker-1@{host} +``` + +### 4. Inspect a user-agent + +```bash +uv run mail-admin agent-get supervisor@default +uv run mail-admin user-get alice +uv run mail-admin daemon-get worker-1 +``` + +### 5. Delete a user-agent + +```bash +uv run mail-admin agent-delete supervisor@default +uv run mail-admin user-delete alice +uv run mail-admin daemon-delete worker-1 +``` + +## Verification + +A created user-agent appears in the matching `*-list` / `*-get` output and can +authenticate with its generated password. Handle admin credentials with care — +they are effectively server-control credentials (see +[Security Model](../explanations/security-model.md)). + +## See also + +- [Admin CLI](../references/admin-cli.md) — full `mail-admin` reference. +- [Manage Swarms](manage-swarms.md) — swarms an agent lives in. +- [Security Model](../explanations/security-model.md) ## Source Material -- `src/mail/client/docs/reference/admin-panel.md` - `src/mail/client/src/mail_client/admin_panel.py` - `src/mail/client/src/mail_client/commands/agent_post.py` - `src/mail/client/src/mail_client/commands/user_post.py` - `src/mail/client/src/mail_client/commands/daemon_post.py` -- `src/mail/server/src/mail_server/routers/admin.py` - -## Steps to Cover - -1. Authenticate as an admin. -2. List existing user-agents by type. -3. Create an agent in a swarm. -4. Create a host-scoped user or daemon. -5. Inspect the created user-agent. -6. Delete a test user-agent. - -## Validation - -Created user-agents appear in admin list/get commands and can authenticate when -credentials are valid. diff --git a/docs/howtos/regenerate-api-artifacts.md b/docs/howtos/regenerate-api-artifacts.md index e0e455a..74da086 100644 --- a/docs/howtos/regenerate-api-artifacts.md +++ b/docs/howtos/regenerate-api-artifacts.md @@ -1,34 +1,89 @@ # Regenerate API Artifacts -Status: stub +Status: draft ## Goal -How to refresh generated API or documentation artifacts after protocol, router, -or model changes. +Refresh the generated files after changing routes, protocol models, the CLI, +docs, or dependencies, and confirm the changes are the ones you expect. ## Starting Point -The reader changed FastAPI routes, protocol models, or generated documentation -inputs. +You changed FastAPI routes, protocol models, CLI parsers, documentation inputs, +or the dependency set, and one or more committed artifacts is now stale. -## Source Material +## Which artifact to regenerate -- `scripts/generate_openapi.py` -- `scripts/build_llms_txt.py` -- `scripts/build_third_party_licenses.py` -- `spec/openapi.yaml` -- `llms.txt` -- `THIRD_PARTY_NOTICES.md` +| You changed… | Regenerate | Output | +| --- | --- | --- | +| Routes, request/response models | OpenAPI | `spec/openapi.yaml` | +| A CLI parser (flags/subcommands) | CLI reference pages | `docs/references/*-cli.md` | +| `README.md` or docs used in the digest | `llms.txt` | `llms.txt` | +| Dependencies | Third-party notices | `THIRD_PARTY_NOTICES.md` | + +## Steps + +### 1. Regenerate the OpenAPI contract + +`spec/openapi.yaml` is generated from the FastAPI app, not hand-edited: + +```bash +uv run python scripts/generate_openapi.py +``` + +It writes `spec/openapi.yaml` by default (pass `--output spec/openapi.json` for +JSON). + +### 2. Regenerate the CLI reference pages + +The four CLI references are derived from each command's argparse parser: + +```bash +uv run python scripts/build_cli_docs.py +``` + +This rewrites `docs/references/{client,admin,server,daemon}-cli.md`. + +### 3. Rebuild `llms.txt` -## Steps to Cover +```bash +uv run python scripts/build_llms_txt.py +``` -1. Regenerate OpenAPI output. -2. Validate OpenAPI drift tests. -3. Rebuild `llms.txt` when docs or specs change. -4. Rebuild third-party license notices when dependencies change. -5. Review diffs before committing generated files. +### 4. Rebuild third-party license notices -## Validation +```bash +uv run python scripts/build_third_party_licenses.py +``` -Contract tests pass and generated files only contain expected changes. +Writes `THIRD_PARTY_NOTICES.md`. + +### 5. Validate + +Run the contract tests, which include the OpenAPI drift check: + +```bash +uv run pytest -m contract +``` + +`tests/contract/test_openapi_drift.py` fails if the committed `spec/openapi.yaml` +still differs from the app — regenerate (step 1) until it passes. See +[Run the Test Suite](run-tests.md). + +### 6. Review before committing + +Generated files can carry incidental churn. Review `git diff` and confirm the +changes match what you intended before committing. + +## See also + +- [Protocol Specification](../references/protocol-specification.md) — how the + generated OpenAPI relates to the normative spec. +- [Run the Test Suite](run-tests.md) + +## Source Material + +- `scripts/generate_openapi.py` +- `scripts/build_cli_docs.py` +- `scripts/build_llms_txt.py` +- `scripts/build_third_party_licenses.py` diff --git a/docs/howtos/run-daemon.md b/docs/howtos/run-daemon.md index 3f4eb01..c7a2aac 100644 --- a/docs/howtos/run-daemon.md +++ b/docs/howtos/run-daemon.md @@ -1,30 +1,63 @@ # Run the MAIL Daemon -Status: stub +Status: draft ## Goal -How to start `mail-daemon` so pending local messages are delivered by an -authorized daemon user-agent. +Start `mail-daemon` so pending messages are delivered from the server into +recipients' inboxes. ## Starting Point -A MAIL server is running and daemon credentials exist. +A MAIL server is running and you have daemon credentials — a `daemon:` address +and its password, created by `backend-init` (see +[Initialize the Memory Backend](initialize-memory-backend.md)). Delivery is the +daemon's job, not the server's; see [Delivery Model](../explanations/delivery-model.md). -## Source Material +## Steps -- `src/mail/daemon/src/mail_daemon/cli.py` -- `src/mail/daemon/src/mail_daemon/maild/api.py` -- `spec/SPEC.md` section 8 +### 1. Set the daemon's environment variables + +The daemon authenticates as a daemon user-agent and requires all three: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_ADDRESS=daemon:dummy@localhost +MAIL_PASSWORD={daemon_password} +``` + +### 2. Start the daemon + +```bash +uv run mail-daemon +``` + +On startup it health-checks the server, logs in to obtain a token, then begins +polling for messages to deliver (roughly every 30 seconds). -## Steps to Cover +### 3. Adjust log levels (optional) -1. Set `MAIL_SERVER`, `MAIL_ADDRESS`, and `MAIL_PASSWORD`. -2. Start `uv run mail-daemon`. -3. Adjust console or file log levels. -4. Confirm the daemon obtains a token. -5. Send a message and watch delivery complete. +Console and file log levels are set independently (`debug`, `info`, `warning`, +`error`, `critical`; both default to `info`): -## Validation +```bash +uv run mail-daemon --log-level-console debug --log-level-file info +``` -Messages move from the server delivery buffer into recipient inboxes. +### 4. Confirm delivery + +Send a message (see [Send a Message with the CLI](send-message-cli.md)), then +open the recipient's inbox. The message moves from the server's delivery buffer +into the recipient's inbox within one poll cycle, and the delivered message +records `Delivered By: daemon:…`. + +## See also + +- [Delivery Model](../explanations/delivery-model.md) — why a daemon delivers. +- [Run the MAIL Server](run-server.md) +- [Configuration](../references/configuration.md) + +## Source Material + +- `src/mail/daemon/src/mail_daemon/cli.py` +- `src/mail/daemon/src/mail_daemon/maild/api.py` diff --git a/docs/howtos/run-server.md b/docs/howtos/run-server.md index 17ef2a9..54d960a 100644 --- a/docs/howtos/run-server.md +++ b/docs/howtos/run-server.md @@ -1,34 +1,88 @@ # Run the MAIL Server -Status: stub +Status: draft ## Goal -How to start `mail-server` with the desired host, port, backend, and memory -checkpoint behavior. +Start `mail-server` with the host, port, backend, and checkpoint behavior you +want. ## Starting Point -The backend has already been initialized and required environment variables are -available. +Workspace dependencies are installed (`uv sync`) and you have initialized a +backend — see [Initialize the Memory Backend](initialize-memory-backend.md). -## Source Material +## Steps -- `src/mail/server/src/mail_server/cli.py` -- `src/mail/server/src/mail_server/server.py` -- `src/mail/server/.env.example` -- `src/mail/server/docs/reference/cli.md` +### 1. Set the required environment variables + +The server reads these at startup and refuses to boot if any is missing. Copy +[`src/mail/server/.env.example`](../../src/mail/server/.env.example) as a +starting point; full details are in [Configuration](../references/configuration.md). + +```bash +MAIL_HOST=localhost +MAIL_JWT_SECRET_KEY=$(openssl rand -hex 32) +MAIL_JWT_ALGORITHM=HS256 +MAIL_JWT_EXPIRE_MINUTES=30 +MAIL_REFRESH_TOKEN_EXPIRE_DAYS=30 +``` + +### 2. Start the server + +```bash +uv run mail-server +``` + +With no flags the server uses the memory backend and listens on +`http://127.0.0.1:8865`. + +### 3. Override host and port + +```bash +uv run mail-server --host 0.0.0.0 --port 9000 +``` + +### 4. Choose a backend + +The default is `memory`; use `sqlite` for a durable, transactional store: -## Steps to Cover +```bash +uv run mail-server --backend sqlite --sqlite-path ./mail.db +``` -1. Set required JWT and host environment variables. -2. Start `uv run mail-server`. -3. Override `--host` and `--port`. -4. Select `--backend memory`. -5. Tune or disable `--memory-save-interval`. -6. Verify `GET /health` or `mail ping`. +See [Storage Backends](../references/storage-backends.md) for the trade-offs and +for `--database-url`. (Note: the memory backend always reads/writes the `default` +deployment; use SQLite for other deployment names.) -## Validation +### 5. Tune or disable memory checkpointing -The server responds with healthy status and the root endpoint reports MAIL -protocol metadata. +The memory backend checkpoints to disk every `--memory-save-interval` seconds +(default 60). Set `0` to disable periodic checkpoints (a final save still runs on +shutdown): + +```bash +uv run mail-server --backend memory --memory-save-interval 10 +``` + +### 6. Verify it is up + +```bash +curl -s http://127.0.0.1:8865/health # -> {"status":"ok"} +# or, with the client configured: +MAIL_SERVER=http://127.0.0.1:8865 uv run mail ping +``` + +`GET /` reports the protocol name, version, and uptime. + +## See also + +- [Configuration](../references/configuration.md) — every server flag and env var. +- [Run the MAIL Daemon](run-daemon.md) — needed for messages to actually deliver. +- [Storage Backends](../references/storage-backends.md) + +## Source Material + +- `src/mail/server/src/mail_server/cli.py` +- `src/mail/server/src/mail_server/server.py` +- `src/mail/server/.env.example` diff --git a/docs/howtos/run-tests.md b/docs/howtos/run-tests.md index 78722b4..5ca1c3a 100644 --- a/docs/howtos/run-tests.md +++ b/docs/howtos/run-tests.md @@ -1,32 +1,76 @@ # Run the Test Suite -Status: stub +Status: draft ## Goal -How to run the active MAIL v2 test suite, focused test groups, contract tests, -and archived v1 tests when needed. +Run the active MAIL v2 tests, focus on a subset, measure coverage, and run the +archived v1 tests when needed. ## Starting Point -The workspace dependencies are installed. +Workspace dependencies are installed (`uv sync`). -## Source Material +## Steps -- `pytest.ini` -- `tests/` -- `src/mail/legacy/tests/` -- `pyproject.toml` +### 1. Run the active suite + +```bash +uv run pytest +``` + +This runs everything under `tests/` **except** the `e2e` group (excluded by +default in `pytest.ini`). + +### 2. Run a subset + +Tests are marked by category — `unit`, `integration`, `contract`, `e2e` — and +also live in matching directories. Select by marker or by path: + +```bash +uv run pytest -m unit # pure-logic tests +uv run pytest -m contract # spec/OpenAPI conformance +uv run pytest tests/integration # by directory +uv run pytest -m e2e # full-system subprocess tests (opt-in) +``` + +Integration and contract tests run against **both** the memory and SQLite +backends via a parametrized fixture, so a green run exercises both stores. + +### 3. Measure coverage -## Steps to Cover +Coverage is scoped to the v2 packages (configured in `pyproject.toml`): -1. Run all active v2 tests with `uv run pytest`. -2. Run unit, integration, contract, or end-to-end subsets. -3. Run coverage with the configured source packages. -4. Run archived v1 tests with the `legacy` extra when needed. -5. Interpret failures from OpenAPI drift and protocol contract tests. +```bash +uv run pytest --cov +``` -## Validation +### 4. Run the archived v1 tests (only when needed) -The selected test command exits successfully and any expected skipped or xfailed -tests are understood. +Legacy tests are not part of the default run and need the `legacy` extra: + +```bash +uv run --extra legacy pytest src/mail/legacy/tests +``` + +See [MAIL v1 Legacy Runtime](../explanations/mail-v1-legacy.md). + +## Interpreting results + +- **OpenAPI drift** (`tests/contract/test_openapi_drift.py`) fails when the + committed `spec/openapi.yaml` no longer matches the app — regenerate it (see + [Regenerate API Artifacts](regenerate-api-artifacts.md)). +- **Contract tests** (`tests/contract/`) enforce SPEC.md rules on addresses, + messages, and delivery; a failure means the implementation diverged from the + spec. See [Protocol Specification](../references/protocol-specification.md). +- **Expected `xfail`s** in `tests/integration/test_stubs.py` mark operations that + are `NotImplementedError` on the memory backend (message deletion, trash clear, + webhook patch, remote delivery) — these are implemented on SQLite. An `xpass` + there means one was implemented and the marker should be removed. + +## Source Material + +- `pytest.ini` +- `tests/` +- `src/mail/legacy/tests/` +- `pyproject.toml` (`[tool.coverage]`) diff --git a/docs/howtos/send-message-cli.md b/docs/howtos/send-message-cli.md index 2ea4bff..82becf1 100644 --- a/docs/howtos/send-message-cli.md +++ b/docs/howtos/send-message-cli.md @@ -1,32 +1,85 @@ # Send a Message with the CLI -Status: stub +Status: draft ## Goal -How to compose a draft and send it to one or more MAIL recipients using the -`mail` CLI. +Compose a draft and send it to one or more recipients with the `mail` CLI. ## Starting Point -The reader has a valid `MAIL_TOKEN` for a user-agent allowed to send messages. +You have a valid `MAIL_TOKEN` for a user-agent allowed to send (see +[Authenticate a User-Agent](authenticate-user-agent.md)), and a daemon is running +so the message can be delivered (see [Run the MAIL Daemon](run-daemon.md)). +Sending is a two-step draft-then-send flow — see +[Delivery Model](../explanations/delivery-model.md). -## Source Material +## Steps -- `src/mail/client/src/mail_client/commands/compose.py` -- `src/mail/client/src/mail_client/commands/send.py` -- `src/mail/client/docs/reference/cli.md` +### 1. Compose a draft + +A draft holds only a subject and body; recipients come later. Provide the body +inline or read it from a file with `-F`/`--body-file`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail compose "Status update" "The batch job finished cleanly." +``` + +The console prints the new draft, including its **draft ID** (a UUID). You can +attach `--tags` (slug strings) that carry onto the sent message. + +### 2. Capture the draft ID + +Note the `draft_id` from the output; you can also list drafts with +`uv run mail drafts` or open one with `uv run mail drafts-open {draft_id}`. + +### 3. Send the draft to one or more recipients + +Recipients are supplied at send time. Pass the draft ID followed by one or more +addresses: + +```bash +uv run mail send {draft_id} supervisor@default@localhost user:alice@localhost +``` + +The console prints the assembled message with a **message ID** distinct from the +draft ID. -## Steps to Cover +### 4. Inspect the outbox -1. Compose a draft with subject and body. -2. Capture the draft ID. -3. Send the draft to one or more recipient addresses. -4. Inspect the outbox entry. -5. Inspect the recipient inbox when possible. -6. Handle malformed address or validation failures. +```bash +uv run mail outbox # list your sent messages +uv run mail outbox-open {message_id} # open one +``` -## Validation +A `null` delivery time means *sent, awaiting delivery*; once the daemon delivers, +the outbox entry records the delivering daemon. -The sender has an outbox message and the recipient can open the delivered inbox -message after daemon delivery. +### 5. Inspect the recipient inbox + +With the recipient's token (and a running daemon), confirm arrival: + +```bash +MAIL_TOKEN={recipient_jwt} uv run mail inbox +MAIL_TOKEN={recipient_jwt} uv run mail open {message_id} +``` + +### 6. Handle validation failures + +Malformed input is rejected before the message is created or delivered: a bad +subject/body fails at compose time, and a malformed recipient address fails at +send time with a `422` whose `detail` explains the problem. Recipient addresses +must be valid MAIL addresses — see [Addressing Model](../explanations/addressing-model.md). + +## See also + +- [Authenticate a User-Agent](authenticate-user-agent.md) +- [Delivery Model](../explanations/delivery-model.md) +- [Client CLI](../references/client-cli.md) — full command reference. + +## Source Material + +- `src/mail/client/src/mail_client/commands/compose.py` +- `src/mail/client/src/mail_client/commands/send.py` From 3fa6025f10f852db649d3f92bcfaddec4ef55e65 Mon Sep 17 00:00:00 2001 From: Addison Kline <77369109+addisonkline@users.noreply.github.com> Date: Wed, 1 Jul 2026 16:16:18 -0400 Subject: [PATCH 28/28] docs: rewrite root README as a GitHub landing page Replace the transitional 'being reorganized' framing with a proper landing README: badges (PyPI/Python/license/spec), a highlights section, PyPI + source install, a runnable local quickstart, a package table, a documentation map linking into docs/, repository layout, development commands, and contributing/license sections. Drops the stale note about root scripts targeting the legacy runtime (scripts/ is v2-only). All relative links verified. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 196 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 134 insertions(+), 62 deletions(-) diff --git a/README.md b/README.md index 0c90a03..a77d7c2 100644 --- a/README.md +++ b/README.md @@ -1,91 +1,163 @@ # Multi-Agent Interface Layer (MAIL) -MAIL is a protocol and Python implementation for message-oriented coordination -between humans, agents, daemons, and swarms. +[![PyPI](https://img.shields.io/pypi/v/mail-swarms)](https://pypi.org/project/mail-swarms/) +[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/) +[![License](https://img.shields.io/badge/license-Apache%202.0-green)](LICENSE) +[![Spec](https://img.shields.io/badge/MAIL%20spec-v2.0-blueviolet)](spec/SPEC.md) + +**MAIL is an open protocol — and a Python implementation — for email-like +communication between humans and AI agents.** Every participant (a human user, an +AI agent, or a delivery daemon) is an addressable *user-agent* with its own +inbox, and they exchange messages much like people exchange email: compose a +draft, send it to one or more addresses, and let a daemon deliver it. + +MAIL deliberately covers the **communication layer and little else** — not an +agent runtime, not tool execution. If you already have agents, MAIL gives them a +shared, standard way to talk. See [What is MAIL?](docs/explanations/mail-v2-overview.md). + +## Highlights + +- **Email-like model** — addresses, inboxes, outboxes, drafts, trash, and + mailing lists, all defined by an open [specification](spec/SPEC.md). +- **HTTP-native** — a FastAPI server with an authoritative + [OpenAPI contract](spec/openapi.yaml); any client that speaks the contract works. +- **Separation of concerns** — the server owns state, daemons deliver messages, + clients are just authenticated user-agents. +- **Pluggable storage** — an in-memory backend for development and a + transactional SQLite backend for durability. +- **Batteries included** — a CLI client (`mail`), an admin CLI (`mail-admin`), a + delivery daemon, and webhook delivery for push notifications. + +## Installation + +MAIL ships as five lockstep packages on PyPI under `mail-swarms-*`. Install the +components you need: -This repository is being reorganized for MAIL v2. The active implementation is -split into package-specific workspaces, while the older MAIL v1 reference -runtime is archived under `src/mail/legacy`. - -## Active v2 Packages - -- `src/mail/protocol` - shared protocol types and constants (`mail-swarms-protocol`) -- `src/mail/server` - FastAPI server implementation (`mail-swarms-server`) -- `src/mail/client` - command-line client (`mail-swarms-client`) -- `src/mail/daemon` - daemon implementation (`mail-swarms-daemon`) - -## Repository Layout - -```text -mail/ -├── docs/ # v2 repository-level docs -├── spec/ # protocol specification and schemas -├── src/mail/ -│ ├── protocol/ # mail-swarms-protocol package -│ ├── server/ # mail-swarms-server package -│ ├── client/ # mail-swarms-client package -│ ├── daemon/ # mail-swarms-daemon package -│ └── legacy/ # archived MAIL v1 runtime, docs, config, and UI -├── tests/ # active MAIL v2 test suite -├── scripts/ # repository maintenance scripts -└── pyproject.toml # uv workspace and meta-package configuration +```bash +pip install mail-swarms-server # the FastAPI server + backend-init +pip install mail-swarms-client # the `mail` and `mail-admin` CLIs +pip install mail-swarms-daemon # the delivery daemon ``` -## Development - -Install the workspace dependencies: +To work from a source checkout, use [uv](https://docs.astral.sh/uv/): ```bash +git clone https://github.com/charonlabs/mail.git +cd mail uv sync ``` -Run the v2 server: +Requires Python 3.12+. -```bash -uv run mail-server -``` +## Quickstart -Use the v2 client: +Bring up a local deployment and send your first message. (Prefix commands with +`uv run` when working from a source checkout.) ```bash -uv run mail --help +# 1. Initialize a local memory backend (creates a swarm + starter user-agents) +uv run backend-init --type memory --host localhost + +# 2. Configure and start the server +export MAIL_HOST=localhost +export MAIL_JWT_SECRET_KEY=$(openssl rand -hex 32) +export MAIL_JWT_ALGORITHM=HS256 +export MAIL_JWT_EXPIRE_MINUTES=30 +export MAIL_REFRESH_TOKEN_EXPIRE_DAYS=30 +uv run mail-server --backend memory # http://127.0.0.1:8865 + +# 3. In another terminal, start the delivery daemon (with daemon credentials) +uv run mail-daemon + +# 4. In a third terminal, log in and send a message +export MAIL_SERVER=http://127.0.0.1:8865 +uv run mail login +uv run mail compose "Hello" "My first MAIL message" +uv run mail send supervisor@default@localhost ``` -Run active v2 tests: +The full walkthrough — including where the generated credentials live — is in +[Run MAIL Locally](docs/tutorials/run-local-mail.md). -```bash -uv run pytest -``` +## Packages -During the transition, some root-level scripts still target the legacy runtime. -Legacy tests and other v1 material live under `src/mail/legacy`. +| Package | Directory | Provides | +| --- | --- | --- | +| [`mail-swarms-protocol`](https://pypi.org/project/mail-swarms-protocol/) | `src/mail/protocol` | Shared protocol types, constants, and validators | +| [`mail-swarms-server`](https://pypi.org/project/mail-swarms-server/) | `src/mail/server` | FastAPI server, storage backends, `backend-init` | +| [`mail-swarms-client`](https://pypi.org/project/mail-swarms-client/) | `src/mail/client` | `mail` and `mail-admin` CLIs | +| [`mail-swarms-daemon`](https://pypi.org/project/mail-swarms-daemon/) | `src/mail/daemon` | The delivery daemon (`mail-daemon`) | -Run archived legacy tests explicitly: +## Documentation -```bash -uv run --extra legacy pytest src/mail/legacy/tests +Full docs live in [`docs/`](docs/README.md), organized by the +[Divio system](docs/explanations/documentation-system.md): + +- **Tutorials** — [Run MAIL Locally](docs/tutorials/run-local-mail.md) · + [Send Your First Message](docs/tutorials/send-first-message.md) · + [Build a Minimal HTTP Client](docs/tutorials/build-minimal-http-client.md) · + [Build a Webhook Receiver](docs/tutorials/build-webhook-receiver.md) +- **How-to guides** — [running the server](docs/howtos/run-server.md), + [daemon](docs/howtos/run-daemon.md), [authentication](docs/howtos/authenticate-user-agent.md), + [sending messages](docs/howtos/send-message-cli.md), + [swarms](docs/howtos/manage-swarms.md), + [mailing lists](docs/howtos/manage-mailing-lists.md), + [webhooks](docs/howtos/manage-webhooks.md), and more. +- **Reference** — [HTTP API](docs/references/http-api.md) · + [Data Models](docs/references/data-models.md) · + [Configuration](docs/references/configuration.md) · + [Storage Backends](docs/references/storage-backends.md) · + [CLIs](docs/references/client-cli.md) +- **Explanations** — [Architecture](docs/explanations/architecture.md) · + [Addressing](docs/explanations/addressing-model.md) · + [Delivery](docs/explanations/delivery-model.md) · + [Security](docs/explanations/security-model.md) + +The protocol itself is specified in [`spec/SPEC.md`](spec/SPEC.md), with the +authoritative HTTP contract in [`spec/openapi.yaml`](spec/openapi.yaml). + +## Repository layout + +```text +mail/ +├── docs/ # documentation (tutorials / howtos / references / explanations) +├── spec/ # SPEC.md + generated openapi.yaml +├── src/mail/ +│ ├── protocol/ # mail-swarms-protocol +│ ├── server/ # mail-swarms-server +│ ├── client/ # mail-swarms-client +│ ├── daemon/ # mail-swarms-daemon +│ └── legacy/ # archived MAIL v1 runtime (reference only) +├── tests/ # active v2 test suite (contract / e2e / integration / unit) +├── scripts/ # artifact generation + maintenance +└── pyproject.toml # uv workspace + meta-package ``` -## Documentation +See [Repository Layout](docs/references/repository-layout.md) for the full map. + +## Development -- Root v2 docs: `docs/README.md` -- Protocol/specification: `spec/` -- Server docs: `src/mail/server/docs/` -- Client docs: `src/mail/client/docs/` -- Legacy runtime notes: `src/mail/legacy/README.md` -- Archived v1 docs: `src/mail/legacy/docs/` +```bash +uv sync # install the workspace +uv run pytest # run the active v2 test suite +uv run mail --help # explore the client CLI +``` -## Legacy Runtime +More: [Run the Test Suite](docs/howtos/run-tests.md) · +[Regenerate API Artifacts](docs/howtos/regenerate-api-artifacts.md). -The MAIL v1 runtime is kept for compatibility and historical reference. Use -`mail.legacy.*` imports for archived code as it is migrated into the legacy -namespace. +The MAIL v1 runtime is archived under `src/mail/legacy/` for reference and is not +part of the v2 packages — see [MAIL v1 Legacy Runtime](docs/explanations/mail-v1-legacy.md). -Do not add new v2 behavior to the legacy runtime unless it is needed for a -specific compatibility or migration task. +## Contributing -## Licensing +Contributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md); commits +must be signed off under the [Developer Certificate of Origin](DCO). -Reference implementation code is licensed under Apache License 2.0. Protocol -specification materials are covered by their repository license files. +## License +Reference implementation code is licensed under the +[Apache License 2.0](LICENSE). The protocol specification and patent grant are +covered by [SPEC-LICENSE](SPEC-LICENSE) and +[SPEC-PATENT-LICENSE](SPEC-PATENT-LICENSE). "MAIL" and related marks are subject +to the [trademark policy](TRADEMARKS.md).