diff --git a/README.md b/README.md index 0c90a03..a77d7c2 100644 --- a/README.md +++ b/README.md @@ -1,91 +1,163 @@ # Multi-Agent Interface Layer (MAIL) -MAIL is a protocol and Python implementation for message-oriented coordination -between humans, agents, daemons, and swarms. +[![PyPI](https://img.shields.io/pypi/v/mail-swarms)](https://pypi.org/project/mail-swarms/) +[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/) +[![License](https://img.shields.io/badge/license-Apache%202.0-green)](LICENSE) +[![Spec](https://img.shields.io/badge/MAIL%20spec-v2.0-blueviolet)](spec/SPEC.md) + +**MAIL is an open protocol — and a Python implementation — for email-like +communication between humans and AI agents.** Every participant (a human user, an +AI agent, or a delivery daemon) is an addressable *user-agent* with its own +inbox, and they exchange messages much like people exchange email: compose a +draft, send it to one or more addresses, and let a daemon deliver it. + +MAIL deliberately covers the **communication layer and little else** — not an +agent runtime, not tool execution. If you already have agents, MAIL gives them a +shared, standard way to talk. See [What is MAIL?](docs/explanations/mail-v2-overview.md). + +## Highlights + +- **Email-like model** — addresses, inboxes, outboxes, drafts, trash, and + mailing lists, all defined by an open [specification](spec/SPEC.md). +- **HTTP-native** — a FastAPI server with an authoritative + [OpenAPI contract](spec/openapi.yaml); any client that speaks the contract works. +- **Separation of concerns** — the server owns state, daemons deliver messages, + clients are just authenticated user-agents. +- **Pluggable storage** — an in-memory backend for development and a + transactional SQLite backend for durability. +- **Batteries included** — a CLI client (`mail`), an admin CLI (`mail-admin`), a + delivery daemon, and webhook delivery for push notifications. + +## Installation + +MAIL ships as five lockstep packages on PyPI under `mail-swarms-*`. Install the +components you need: -This repository is being reorganized for MAIL v2. The active implementation is -split into package-specific workspaces, while the older MAIL v1 reference -runtime is archived under `src/mail/legacy`. - -## Active v2 Packages - -- `src/mail/protocol` - shared protocol types and constants (`mail-swarms-protocol`) -- `src/mail/server` - FastAPI server implementation (`mail-swarms-server`) -- `src/mail/client` - command-line client (`mail-swarms-client`) -- `src/mail/daemon` - daemon implementation (`mail-swarms-daemon`) - -## Repository Layout - -```text -mail/ -├── docs/ # v2 repository-level docs -├── spec/ # protocol specification and schemas -├── src/mail/ -│ ├── protocol/ # mail-swarms-protocol package -│ ├── server/ # mail-swarms-server package -│ ├── client/ # mail-swarms-client package -│ ├── daemon/ # mail-swarms-daemon package -│ └── legacy/ # archived MAIL v1 runtime, docs, config, and UI -├── tests/ # active MAIL v2 test suite -├── scripts/ # repository maintenance scripts -└── pyproject.toml # uv workspace and meta-package configuration +```bash +pip install mail-swarms-server # the FastAPI server + backend-init +pip install mail-swarms-client # the `mail` and `mail-admin` CLIs +pip install mail-swarms-daemon # the delivery daemon ``` -## Development - -Install the workspace dependencies: +To work from a source checkout, use [uv](https://docs.astral.sh/uv/): ```bash +git clone https://github.com/charonlabs/mail.git +cd mail uv sync ``` -Run the v2 server: +Requires Python 3.12+. -```bash -uv run mail-server -``` +## Quickstart -Use the v2 client: +Bring up a local deployment and send your first message. (Prefix commands with +`uv run` when working from a source checkout.) ```bash -uv run mail --help +# 1. Initialize a local memory backend (creates a swarm + starter user-agents) +uv run backend-init --type memory --host localhost + +# 2. Configure and start the server +export MAIL_HOST=localhost +export MAIL_JWT_SECRET_KEY=$(openssl rand -hex 32) +export MAIL_JWT_ALGORITHM=HS256 +export MAIL_JWT_EXPIRE_MINUTES=30 +export MAIL_REFRESH_TOKEN_EXPIRE_DAYS=30 +uv run mail-server --backend memory # http://127.0.0.1:8865 + +# 3. In another terminal, start the delivery daemon (with daemon credentials) +uv run mail-daemon + +# 4. In a third terminal, log in and send a message +export MAIL_SERVER=http://127.0.0.1:8865 +uv run mail login +uv run mail compose "Hello" "My first MAIL message" +uv run mail send supervisor@default@localhost ``` -Run active v2 tests: +The full walkthrough — including where the generated credentials live — is in +[Run MAIL Locally](docs/tutorials/run-local-mail.md). -```bash -uv run pytest -``` +## Packages -During the transition, some root-level scripts still target the legacy runtime. -Legacy tests and other v1 material live under `src/mail/legacy`. +| Package | Directory | Provides | +| --- | --- | --- | +| [`mail-swarms-protocol`](https://pypi.org/project/mail-swarms-protocol/) | `src/mail/protocol` | Shared protocol types, constants, and validators | +| [`mail-swarms-server`](https://pypi.org/project/mail-swarms-server/) | `src/mail/server` | FastAPI server, storage backends, `backend-init` | +| [`mail-swarms-client`](https://pypi.org/project/mail-swarms-client/) | `src/mail/client` | `mail` and `mail-admin` CLIs | +| [`mail-swarms-daemon`](https://pypi.org/project/mail-swarms-daemon/) | `src/mail/daemon` | The delivery daemon (`mail-daemon`) | -Run archived legacy tests explicitly: +## Documentation -```bash -uv run --extra legacy pytest src/mail/legacy/tests +Full docs live in [`docs/`](docs/README.md), organized by the +[Divio system](docs/explanations/documentation-system.md): + +- **Tutorials** — [Run MAIL Locally](docs/tutorials/run-local-mail.md) · + [Send Your First Message](docs/tutorials/send-first-message.md) · + [Build a Minimal HTTP Client](docs/tutorials/build-minimal-http-client.md) · + [Build a Webhook Receiver](docs/tutorials/build-webhook-receiver.md) +- **How-to guides** — [running the server](docs/howtos/run-server.md), + [daemon](docs/howtos/run-daemon.md), [authentication](docs/howtos/authenticate-user-agent.md), + [sending messages](docs/howtos/send-message-cli.md), + [swarms](docs/howtos/manage-swarms.md), + [mailing lists](docs/howtos/manage-mailing-lists.md), + [webhooks](docs/howtos/manage-webhooks.md), and more. +- **Reference** — [HTTP API](docs/references/http-api.md) · + [Data Models](docs/references/data-models.md) · + [Configuration](docs/references/configuration.md) · + [Storage Backends](docs/references/storage-backends.md) · + [CLIs](docs/references/client-cli.md) +- **Explanations** — [Architecture](docs/explanations/architecture.md) · + [Addressing](docs/explanations/addressing-model.md) · + [Delivery](docs/explanations/delivery-model.md) · + [Security](docs/explanations/security-model.md) + +The protocol itself is specified in [`spec/SPEC.md`](spec/SPEC.md), with the +authoritative HTTP contract in [`spec/openapi.yaml`](spec/openapi.yaml). + +## Repository layout + +```text +mail/ +├── docs/ # documentation (tutorials / howtos / references / explanations) +├── spec/ # SPEC.md + generated openapi.yaml +├── src/mail/ +│ ├── protocol/ # mail-swarms-protocol +│ ├── server/ # mail-swarms-server +│ ├── client/ # mail-swarms-client +│ ├── daemon/ # mail-swarms-daemon +│ └── legacy/ # archived MAIL v1 runtime (reference only) +├── tests/ # active v2 test suite (contract / e2e / integration / unit) +├── scripts/ # artifact generation + maintenance +└── pyproject.toml # uv workspace + meta-package ``` -## Documentation +See [Repository Layout](docs/references/repository-layout.md) for the full map. + +## Development -- Root v2 docs: `docs/README.md` -- Protocol/specification: `spec/` -- Server docs: `src/mail/server/docs/` -- Client docs: `src/mail/client/docs/` -- Legacy runtime notes: `src/mail/legacy/README.md` -- Archived v1 docs: `src/mail/legacy/docs/` +```bash +uv sync # install the workspace +uv run pytest # run the active v2 test suite +uv run mail --help # explore the client CLI +``` -## Legacy Runtime +More: [Run the Test Suite](docs/howtos/run-tests.md) · +[Regenerate API Artifacts](docs/howtos/regenerate-api-artifacts.md). -The MAIL v1 runtime is kept for compatibility and historical reference. Use -`mail.legacy.*` imports for archived code as it is migrated into the legacy -namespace. +The MAIL v1 runtime is archived under `src/mail/legacy/` for reference and is not +part of the v2 packages — see [MAIL v1 Legacy Runtime](docs/explanations/mail-v1-legacy.md). -Do not add new v2 behavior to the legacy runtime unless it is needed for a -specific compatibility or migration task. +## Contributing -## Licensing +Contributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md); commits +must be signed off under the [Developer Certificate of Origin](DCO). -Reference implementation code is licensed under Apache License 2.0. Protocol -specification materials are covered by their repository license files. +## License +Reference implementation code is licensed under the +[Apache License 2.0](LICENSE). The protocol specification and patent grant are +covered by [SPEC-LICENSE](SPEC-LICENSE) and +[SPEC-PATENT-LICENSE](SPEC-PATENT-LICENSE). "MAIL" and related marks are subject +to the [trademark policy](TRADEMARKS.md). diff --git a/docs/README.md b/docs/README.md index 3816f3c..9c58a6d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,34 +1,101 @@ -# MAIL v2 Documentation +# MAIL Documentation -This directory is the root documentation entry point for MAIL v2. +This directory is the canonical documentation home for active MAIL v2 work. +MAIL uses the [Divio documentation system][divio-about], so each page belongs +to exactly one of four categories. -The old MAIL v1 reference implementation docs have been archived under -`src/mail/legacy/docs/`. Use those archived docs only when maintaining or -studying the legacy runtime. +## Start Here -## Repository-Level Docs +- New to MAIL: follow [Run MAIL Locally](tutorials/run-local-mail.md). +- Trying to complete a known task: browse [How-To Guides](howtos/README.md). +- Looking up commands, models, or endpoints: browse [Reference](references/README.md). +- Trying to understand concepts and tradeoffs: browse [Explanations](explanations/README.md). -- [`testing-plan.md`](testing-plan.md) - v2 testing suite overhaul plan - (categories, phases, conventions) +## Categories -## Active Package Docs +- [Tutorials](tutorials/README.md) are learning-oriented lessons. They walk a + beginner through a concrete project and should be tested end to end. +- [How-To Guides](howtos/README.md) are goal-oriented recipes. They answer + "How do I ...?" questions for readers who already know the basics. +- [Reference](references/README.md) is information-oriented lookup material. It + describes commands, APIs, models, configuration, and repository structure. +- [Explanations](explanations/README.md) are understanding-oriented discussions. + They explain why MAIL works the way it does and how its pieces fit together. -- `src/mail/protocol/README.md` - protocol package documentation -- `src/mail/server/docs/` - v2 server documentation -- `src/mail/client/docs/` - v2 client documentation -- `src/mail/daemon/README.md` - daemon package documentation +## Proposed Layout -## Repository-Level Docs To Add +### Tutorials -This root docs area should contain project-wide v2 material that is not owned by -a single package, such as: +- [Run MAIL Locally](tutorials/run-local-mail.md) +- [Send Your First MAIL Message](tutorials/send-first-message.md) +- [Build a Minimal HTTP Client](tutorials/build-minimal-http-client.md) +- [Build a Webhook Receiver](tutorials/build-webhook-receiver.md) -- repository layout -- release process -- local development workflow -- compatibility and migration notes -- protocol governance and specification process +### How-To Guides -Keep package-specific usage and reference material with the package that owns -the code. +- [Initialize the Memory Backend](howtos/initialize-memory-backend.md) +- [Run the MAIL Server](howtos/run-server.md) +- [Run the MAIL Daemon](howtos/run-daemon.md) +- [Authenticate a User-Agent](howtos/authenticate-user-agent.md) +- [Send a Message with the CLI](howtos/send-message-cli.md) +- [Manage User-Agents](howtos/manage-user-agents.md) +- [Manage Swarms](howtos/manage-swarms.md) +- [Manage Mailing Lists](howtos/manage-mailing-lists.md) +- [Manage Webhooks](howtos/manage-webhooks.md) +- [Regenerate API Artifacts](howtos/regenerate-api-artifacts.md) +- [Run the Test Suite](howtos/run-tests.md) +### Reference + +- [Repository Layout](references/repository-layout.md) +- [Configuration](references/configuration.md) +- [Protocol Specification](references/protocol-specification.md) +- [HTTP API](references/http-api.md) +- [Client CLI](references/client-cli.md) +- [Admin CLI](references/admin-cli.md) +- [Server CLI](references/server-cli.md) +- [Daemon CLI](references/daemon-cli.md) +- [Data Models](references/data-models.md) +- [Storage Backends](references/storage-backends.md) + +### Explanations + +- [MAIL v2 Overview](explanations/mail-v2-overview.md) +- [Architecture](explanations/architecture.md) +- [Addressing Model](explanations/addressing-model.md) +- [Delivery Model](explanations/delivery-model.md) +- [Security Model](explanations/security-model.md) +- [Mailing Lists](explanations/mailing-lists.md) +- [Webhook Delivery](explanations/webhook-delivery.md) +- [MAIL v1 Legacy Runtime](explanations/mail-v1-legacy.md) +- [Documentation System](explanations/documentation-system.md) + +## Existing Source Material + +- Protocol source of truth: [../spec/SPEC.md](../spec/SPEC.md) and + [../spec/openapi.yaml](../spec/openapi.yaml) +- Active package docs to migrate or consolidate: + [client docs](../src/mail/client/docs/README.md) and + [server docs](../src/mail/server/docs/README.md) +- Archived MAIL v1 material: + [legacy README](../src/mail/legacy/README.md) and + [legacy docs](../src/mail/legacy/docs/README.md) + +## Writing Rules + +- Keep a page in one category. If a page starts teaching, solving, describing, + and discussing at once, split it. +- Keep tutorials robust and repeatable. They should avoid optional branches and + should show visible progress quickly. +- Keep how-to guides task-focused. Link to explanations instead of pausing for + conceptual discussion. +- Keep reference pages close to the implementation and generated contracts. + Command and API references should point at their source files. +- Keep explanations free to discuss motivation, alternatives, and tradeoffs, but + link out to tutorials, how-tos, and reference pages for action or lookup. + +[divio-about]: https://docs.divio.com/documentation-system/ +[divio-tutorials]: https://docs.divio.com/documentation-system/tutorials/ +[divio-howtos]: https://docs.divio.com/documentation-system/how-to-guides/ +[divio-references]: https://docs.divio.com/documentation-system/reference/ +[divio-explanations]: https://docs.divio.com/documentation-system/explanation/ diff --git a/docs/explanations/README.md b/docs/explanations/README.md new file mode 100644 index 0000000..7b08513 --- /dev/null +++ b/docs/explanations/README.md @@ -0,0 +1,26 @@ +# Explanations + +Explanations discuss MAIL concepts, motivation, architecture, and tradeoffs. +They are for understanding, not for step-by-step tasks or exhaustive lookup. + +## Planned Explanations + +| Page | Question it answers | +| --- | --- | +| [MAIL v2 Overview](mail-v2-overview.md) | What is MAIL and what problem is v2 trying to solve? | +| [Architecture](architecture.md) | How do protocol, server, client, daemon, and backend pieces fit together? | +| [Addressing Model](addressing-model.md) | Why does MAIL use host-scoped and swarm-scoped addresses? | +| [Delivery Model](delivery-model.md) | Why are daemons responsible for message delivery? | +| [Security Model](security-model.md) | What are the main trust boundaries and risks? | +| [Mailing Lists](mailing-lists.md) | What is a list? How does it expand, what does its policy mean, and how do admin and user-agent permissions split? | +| [Webhook Delivery](webhook-delivery.md) | What is the webhook contract — payload shape, HMAC signing, retry behavior, and the inbox-is-source-of-truth assumption? | +| [MAIL v1 Legacy Runtime](mail-v1-legacy.md) | How should readers interpret the archived v1 runtime and docs? | +| [Documentation System](documentation-system.md) | How should maintainers decide where a new page belongs? | + +## Explanation Checklist + +- Start with a concrete question or tension. +- Discuss tradeoffs and alternatives. +- Link to tutorials for learning paths. +- Link to how-to guides for tasks. +- Link to reference pages for commands, fields, and exact contracts. diff --git a/docs/explanations/addressing-model.md b/docs/explanations/addressing-model.md new file mode 100644 index 0000000..d63ec3f --- /dev/null +++ b/docs/explanations/addressing-model.md @@ -0,0 +1,156 @@ +# Addressing Model + +Status: draft + +Every participant in MAIL — a human, an AI agent, a delivery daemon, an +administrator, or a mailing list — is reached by an **address**. This page +explains why MAIL has two different *shapes* of address, how to tell them apart, +and how to reason about each. For the exact field schemas and length bounds, see +[Data Models](../references/data-models.md); for the formal grammar, see +[§6 of the specification](../../spec/SPEC.md). + +## The shape tells you the scope + +A MAIL address is always one of two shapes, and you can tell which from the +string alone — no server lookup required: + +| Shape | Form | Who it names | +| --- | --- | --- | +| **Host-scoped** | `{ua_type}:{ua_id}@{host}` | admins, daemons, users | +| **Swarm-scoped** | `{address_id}@{swarm}@{host}` | agents, mailing lists | + +The difference is the number of `@` segments. Split an address on `@`: + +- **two segments** → host-scoped. The first segment carries an explicit + `{ua_type}:` prefix (`user:`, `admin:`, or `daemon:`). +- **three segments** → swarm-scoped. The middle segment is the swarm. +- **anything else** → not a MAIL address. + +This self-describing quality is the single most useful thing to internalize. +A client can classify any address — decide whether it points at a server-level +resident or a swarm member, and which *kind* of correspondent it is — purely by +inspecting the string. The reference validator (`validate_mail_address`) does +exactly this, and so can your own code: the shape is the type tag. + +## Host-scoped addresses + +Host-scoped addresses are defined at the level of the MAIL **server**, not at the +level of any swarm inside it. They take the form `{ua_type}:{ua_id}@{host}`, +where `ua_type` is one of `admin`, `daemon`, or `user`: + +```text +admin:root@example.com an administrator +daemon:worker-1@example.com a delivery daemon +user:alice@example.com a human user +``` + +These are the server's permanent residents. A **user** is a human; an **admin** +is a human (or operator) with server-level privileges; a **daemon** is the +autonomous worker that actually carries messages between inboxes. None of them +belongs to a particular swarm — they exist at the host, so their identifiers are +unique across the *entire* server. There is only ever one `user:alice@example.com`. + +## Swarm-scoped addresses + +A **swarm** is an abstract collection of agent addresses and mailing lists — a +way to scope a discrete multi-agent deployment inside a server. Addresses that +live inside a swarm take the form `{address_id}@{swarm}@{host}`, and `address_id` +comes in exactly two flavors: + +```text +sage@chorus@example.com an agent (bare name, no prefix) +list:welfare-discourse@chorus@host a mailing list (the list: prefix) +``` + +An **agent** is named by a bare identifier; a **mailing list** is named with a +`list:` prefix. That prefix is the *only* prefixed swarm-scoped form — there is +no `agent:` prefix, because a bare name already means "agent." (`agent:sage@…` +is therefore invalid; it should be `sage@…`.) + +## Why agent names repeat across swarms + +Here is the design choice that the two-scope split exists to enable: **an agent's +name is unique only within its swarm.** The following two addresses name two +*different* agents, and both are valid simultaneously: + +```text +supervisor@swarm-1@example.com +supervisor@swarm-2@example.com +``` + +This matters because swarms are meant to be independent deployments. Each one +should be free to name its agents naturally — every swarm can have a +`supervisor`, a `planner`, a `researcher` — without coordinating a globally +unique name with every other swarm on the server. The swarm segment is what +disambiguates them. Mailing list names work the same way: `list:all@swarm-1` and +`list:all@swarm-2` coexist. + +Contrast this with host-scoped identifiers (users, admins, daemons), which carry +no swarm segment and so *must* be unique across the whole server. The scope a +name lives in determines how widely it has to be unique. That is the heart of the +model: **host-scoped names are server-unique; swarm-scoped names are only +swarm-unique.** + +## Why mailing lists live inside a swarm + +A mailing list is a **fan-out target**, not a user-agent. It owns no inbox; when +a message is addressed to `list:{list_id}@{swarm}@{host}`, the server expands the +list and delivers a copy to each member. Lists are swarm-scoped for the same +reason agents are: they belong to a particular deployment, and naming them inside +a swarm lets list names repeat across swarms without collision. The `list:` +prefix is what distinguishes a list from an agent that happens to share the +swarm-scoped shape. + +## How an address is validated + +Identifiers in MAIL — every `ua_id`, `agent`, `swarm`, and `list_id` — must be +**slugs**: lowercase alphanumerics separated by single hyphens, matching + +```text +^[a-z0-9]+(?:-[a-z0-9]+)*$ +``` + +So `welfare-discourse` is valid; `Welfare_Discourse`, `-leading`, `trailing-`, +and `double--hyphen` are not. Each identifier must be at least one character and +at most 32, matching the [protocol spec](../references/protocol-specification.md)'s +recommendation; the reference implementation enforces 32 as a hard cap and rejects +longer identifiers with a validation error. +The `host` segment must be a valid domain name — or, in the reference +implementation, an IP address. + +Putting it together, validation is just the classification from the top of this +page plus a slug check on each part: + +1. Split on `@`. Two segments → expect `{ua_type}:{ua_id}`, with `ua_type` in + `{admin, daemon, user}`. Three segments → the first is either a bare agent + slug or `list:{list_id}`; the middle is the swarm. +2. Slug-check each identifier; validate the host. +3. Reject anything that does not fit either shape. + +## Common mistakes + +These all fail validation — and each is worth recognizing, because the error +("invalid MAIL address structure") is the same for several distinct causes: + +| Address | Why it is rejected | +| --- | --- | +| `alice@example.com` | A two-segment address needs a `ua_type:` prefix. Use `user:alice@example.com`. | +| `agent:sage@chorus@example.com` | Agents are bare; only lists take a prefix. Use `sage@chorus@example.com`. | +| `bot:alice@example.com` | `ua_type` must be `admin`, `daemon`, or `user`. | +| `user:Alice@example.com` | Identifiers are lowercase slugs — no capitals, no spaces. | +| `sage@Chorus@example.com` | The swarm segment must be a slug too. | +| `a@b@c@d`, `alice`, `""` | Neither the two-segment nor three-segment shape. | + +## Related Pages + +- [Data Models](../references/data-models.md) — the user-agent and address schemas, with exact length bounds. +- [Manage User-Agents](../howtos/manage-user-agents.md) — creating admins, agents, daemons, and users. +- [Manage Mailing Lists](../howtos/manage-mailing-lists.md) — creating and addressing lists. +- [Build a Minimal HTTP Client](../tutorials/build-minimal-http-client.md) — uses these addresses in practice. + +## Source Material + +- `spec/SPEC.md` §5 (User-Agents) and §6 (Addresses) +- `src/mail/protocol/src/mail_protocol/core/user_agents.py` +- `src/mail/protocol/src/mail_protocol/core/validators.py` +- `tests/contract/test_spec_addresses.py` diff --git a/docs/explanations/architecture.md b/docs/explanations/architecture.md new file mode 100644 index 0000000..b29d4e2 --- /dev/null +++ b/docs/explanations/architecture.md @@ -0,0 +1,83 @@ +# Architecture + +Status: draft + +MAIL v2 is assembled from four workspace packages around one idea: a **server +owns state and enforces the contract**, **daemons move messages**, and **clients +are just authenticated user-agents**. The protocol package is the shared contract +all three depend on. This page explains how they fit; for the file-level map see +[Repository Layout](../references/repository-layout.md). + +## The components + +```text + ┌─────────────┐ HTTP ┌──────────────────────┐ + │ client │ ───────────────▶ │ server │ + │ (mail / │ ◀─────────────── │ (mail-server) │ + │ mail-admin) │ │ routes + auth + │ + └─────────────┘ │ backend (state) │ + └──────────┬───────────┘ + ┌─────────────┐ poll + deliver │ + │ daemon │ ◀─────────────────────────────┘ + │(mail-daemon)│ HTTP (/daemon/*) + └─────────────┘ +``` + +- **`mail-swarms-protocol`** — the shared data and network contract. Pydantic + models for messages, drafts, boxes, swarms, lists, webhooks, and user-agents, + plus the validators that enforce address and field rules. Server, client, and + daemon all import it, so there is exactly one definition of what a message *is*. + See [Data Models](../references/data-models.md). +- **`mail-swarms-server`** — the FastAPI HTTP implementation and the **owner of + all state**. It authenticates user-agents, holds inboxes/outboxes/drafts/trash, + manages swarms and lists, and performs the first and last steps of delivery. + See [HTTP API](../references/http-api.md). +- **`mail-swarms-client`** — the CLI (`mail` and `mail-admin`). A client is just + a convenient front-end for an authenticated user-agent; it holds no + authoritative state and speaks only the HTTP contract. +- **`mail-swarms-daemon`** — the delivery worker. It authenticates as a daemon + user-agent, polls the server for pending messages, and delivers them to + recipients' inboxes. Delivery is deliberately *not* the server's job — see + [Delivery Model](delivery-model.md). + +## Swarms + +A swarm is an abstract collection of agent addresses, mailing lists, and metadata +inside a server (SPEC §4.3). Swarms scope discrete multi-agent deployments: an +agent's address is swarm-scoped (`name@swarm@host`), while admins, users, and +daemons are host-scoped. See [Addressing Model](addressing-model.md). + +## State ownership and the backend abstraction + +The server does not hardcode a database. It talks to a `MAILServerBackend` +interface, and two implementations ship: an in-memory backend with filesystem +checkpointing (the default, good for development) and a transactional SQLite +backend (durable). Swapping storage never changes the HTTP contract. See +[Storage Backends](../references/storage-backends.md). + +## How a message flows + +1. A user-agent **creates a draft** on the server (`POST /drafts`). +2. It **sends** the draft to recipients (`POST /drafts/{id}/send`); the server + assembles a `MAILMessage`, records it in the sender's outbox, and queues it. +3. A **daemon** picks up the queued message and delivers a copy to each + recipient's inbox (`POST /daemon/deliver/local`). +4. Recipients read their inbox; the server can fire webhooks on delivery. + +This split — server as system of record, daemon as courier — is what lets +delivery status be reported honestly ("sent" vs "delivered by `daemon:…`"). + +## Cross-package alignment + +Because four packages must agree on one contract, two artifacts keep them in +sync: the generated [`spec/openapi.yaml`](../../spec/openapi.yaml) (the +authoritative wire contract, derived from the server app) and the conformance +suite in [`tests/contract/`](../../tests/contract). See +[Protocol Specification](../references/protocol-specification.md). + +## Related pages + +- [Repository Layout](../references/repository-layout.md) +- [HTTP API](../references/http-api.md) +- [Storage Backends](../references/storage-backends.md) +- [Delivery Model](delivery-model.md) diff --git a/docs/explanations/delivery-model.md b/docs/explanations/delivery-model.md new file mode 100644 index 0000000..5e76c8f --- /dev/null +++ b/docs/explanations/delivery-model.md @@ -0,0 +1,161 @@ +# Delivery Model + +Status: draft + +In MAIL, **sending a message is not the same as delivering it.** When a +user-agent sends, the message is written to the server and placed in the sender's +outbox — but it is not yet in anyone's inbox. A separate, authorized worker (a +*daemon*) carries it the rest of the way. This page explains why MAIL splits +those two acts, how the hand-off works, and what it means operationally. For the +hands-on version of the first half, see +[Build a Minimal HTTP Client](../tutorials/build-minimal-http-client.md); for the +formal contract, see [§8 of the specification](../../spec/SPEC.md). + +## Sending is not delivering + +The central idea is a deliberate separation: + +1. **Send** — the sender writes a message to the server. It lands in the sender's + outbox, marked as not-yet-delivered, and its id is queued for delivery. +2. **Deliver** — a daemon later picks the message up and the server files a copy + into each recipient's inbox. + +So a message has two observable states — *sent* and *delivered* — and MAIL makes +the gap between them explicit rather than hiding it. Everything below follows +from taking that separation seriously. + +## Two steps to a message: draft, then send + +Creating a message is itself two steps, which is worth understanding before +delivery enters the picture: + +- **Create a draft** (`POST /drafts`) with only a `subject` and a `body`. A draft + has no recipients. +- **Send the draft** (`POST /drafts/{draft_id}/send`) with the `recipients`. This + is the moment the `MAILMessage` is assembled — a fresh `message_id`, the + sender, the recipients, the subject and body, a `sent_at` timestamp — stored in + the sender's outbox with no delivery stamp yet, and enqueued for delivery. + +Recipients are bound at *send* time, not at draft time, so a draft is a reusable +subject-and-body that can be sent more than once or to different addresses. (The +draft remains in your drafts box after sending.) For why recipients are addressed +the way they are, see [Addressing Model](addressing-model.md). + +## The delivery buffer + +The server keeps a **delivery buffer**: the list of `message_id`s awaiting +delivery. Sending a draft enqueues its id there. The buffer holds *ids*, not +copies — the message itself lives in storage and in the sender's outbox; the +buffer is just the server's "still needs carrying" worklist. + +## The daemon carries the mail + +A **daemon** is a distinct user-agent whose entire job is delivery. The spec +constrains it tightly: a daemon MUST NOT alter the messages it carries, and +SHOULD NOT compose messages of its own. Delivery is a narrow, privileged, +auditable role — not something every agent does for itself. + +A daemon authenticates exactly like any other user-agent (password grant → bearer +token), and the server verifies that the caller really is a daemon before +honoring delivery calls. It then runs a simple loop (the reference daemon pauses +~30 seconds between iterations): + +```text +sender server daemon + │ POST /drafts/{id}/send │ + │ ───────────────────▶ store in outbox │ + │ enqueue id in buffer │ + │ │ + │ ◀───── POST /daemon/message-buffer/clear + │ return pending ids, │ + │ empty the buffer ──────────▶ │ + │ │ + │ ◀───── POST /daemon/deliver/local {ids} + │ for each id: │ + │ file copy into each inbox │ + │ stamp outbox delivered_at, │ + │ delivered_by = daemon │ + │ return delivered summaries ─▶ │ +``` + +In other words: clearing the buffer hands the daemon the pending ids *and* empties +the buffer; the delivery call is where the server actually files copies into +recipients' inboxes and stamps the sender's outbox entry with the delivery time +and the delivering daemon's address. + +## Sent versus delivered, made observable + +Because delivery is a separate step, MAIL exposes where a message is in its +journey. Each outbox entry carries two nullable fields: + +- `delivered_at` — `null` while the message is still waiting; a timestamp once a + daemon has carried it. +- `delivered_by` — the address of the daemon that delivered it. + +A `null` `delivered_at` means *sent, awaiting delivery*; a populated one means +*delivered*. The recipient's inbox entry likewise records which daemon delivered +it. This is the surface a client uses to show a message's status honestly — +"Sent" versus "Delivered by `daemon:…`" — rather than pretending the two are the +same. (Delivery is also where the server can fire webhooks so recipients are +notified of new mail rather than having to poll; see [HTTP API](../references/http-api.md).) + +## Local versus remote delivery + +The primary delivery path is **local**: `POST /daemon/deliver/local` carries +messages between user-agents on the *same* server. A second endpoint, +`POST /daemon/deliver/remote`, accepts messages sent by agents on other MAIL +servers for delivery to local recipients. It is implemented on the SQLite +backend; on the memory backend it currently raises `NotImplementedError`. + +## Pre-send versus post-send errors + +MAIL draws a sharp line between failures that happen *before* a message is +accepted and failures that happen *after*: + +- **Pre-send errors (§8.1)** are synchronous and caught at create or send time. A + malformed subject or body means the message is never created; a malformed + recipient address means it is never delivered. The sender is told immediately, + as a `4xx` response (a validation failure returns `422` with a `detail` + explaining what was wrong). Nothing is queued. +- **Post-send errors (§8.2)** happen after a valid message is in the system. If a + daemon cannot deliver it, the message MUST be preserved and the daemon SHOULD + log the error. These failures are asynchronous and recoverable — the message is + not lost. + +The boundary is the useful thing to remember: before send, an error is the +sender's to fix and the message may not exist at all; after send, the message is +durable and getting it delivered is the daemon's responsibility. + +## Operational implications + +- **Decoupling.** A sender never blocks on a recipient, or even on a daemon being + online. Sending is just a write. If no daemon is connected, messages simply wait + in the buffer (and sit undelivered in the outbox) until one runs. +- **Latency.** The reference daemon polls on an interval (~30 seconds by + default), so delivery is prompt but not instantaneous — expect up to a poll + interval of lag, especially under a backlog. +- **Observability.** Track delivery through the outbox (`delivered_at` / + `delivered_by`) and the daemon's logs. The daemon also warns when the number of + ids it cleared does not match the number the server reports as delivered. +- **Durability and retries.** Messages are stored server-side and preserved on + delivery failure (§8.2). Note the shape of the loop, though: clearing the buffer + empties it, so once a daemon has claimed a batch, delivering it is that daemon's + responsibility. Run a reliable daemon and watch its logs rather than assuming + failed items are automatically re-queued. +- **The unit of delivery is the `message_id`.** The daemon hands the server a + batch of ids to deliver; idempotency and retry policy live at that granularity. + +## Related Pages + +- [Build a Minimal HTTP Client](../tutorials/build-minimal-http-client.md) — performs the draft → send half over raw HTTP. +- [Run the MAIL Daemon](../howtos/run-daemon.md) — running the daemon that does the carrying. +- [Addressing Model](addressing-model.md) — how recipients are named. +- [Daemon CLI](../references/daemon-cli.md) and [HTTP API](../references/http-api.md) — the daemon commands and `/daemon` endpoints. + +## Source Material + +- `spec/SPEC.md` §7 (Messages) and §8 (Delivery) +- `src/mail/server/src/mail_server/routers/daemon.py` +- `src/mail/daemon/src/mail_daemon/maild/api.py` +- `src/mail/server/src/mail_server/backends/base.py` (and `backends/memory/api.py` for the reference behavior) +- `tests/integration/test_flows.py` diff --git a/docs/explanations/documentation-system.md b/docs/explanations/documentation-system.md new file mode 100644 index 0000000..9178997 --- /dev/null +++ b/docs/explanations/documentation-system.md @@ -0,0 +1,79 @@ +# Documentation System + +Status: draft + +MAIL's docs follow the [Divio documentation system][divio] (also called +Diátaxis): every page belongs to exactly one of four categories, each serving a +different reader need. This page explains how to decide where a new page belongs +and why the separation matters. The index and writing rules live in +[docs/README.md](../README.md). + +## The four categories + +| Category | Serves | Reader is… | Optimizes for | +| --- | --- | --- | --- | +| **[Tutorial](../tutorials/README.md)** | Learning | a beginner following along | a guaranteed, repeatable success | +| **[How-to guide](../howtos/README.md)** | A task | someone who knows the basics | getting a specific job done | +| **[Reference](../references/README.md)** | Looking up | someone who knows what they want | accuracy and completeness | +| **[Explanation](README.md)** | Understanding | someone thinking about the system | context, motivation, tradeoffs | + +The split is really two axes: *practical* (tutorials, how-tos) vs *theoretical* +(reference, explanation), and *studying* (tutorials, explanation) vs *working* +(how-tos, reference). A page should sit in one cell of that grid. + +## Why mixed-purpose pages fail + +A page that teaches, solves a task, lists exact fields, *and* argues motivation +all at once serves no reader well: the beginner drowns in reference detail, the +practitioner wades through backstory to find a command, and the lookup reader +can't trust a page that also editorializes. Mixed pages also rot faster, because a +change to the API forces edits to prose that was really about concepts. Keeping +each page to one job keeps it short, trustworthy, and cheap to maintain. + +## How to place (or split) a page + +Ask **what the reader is doing** when they open it: + +- *"Walk me through my first success."* → Tutorial. Avoid optional branches; show + visible progress fast; it must run end to end. +- *"How do I do X?"* → How-to. Assume basics; stay task-focused; link out to + explanations instead of pausing to teach concepts. +- *"What are the exact endpoints / fields / flags?"* → Reference. Mirror the + implementation; prefer generated artifacts; keep opinion out. +- *"Why does it work this way?"* → Explanation. Discuss motivation and + alternatives; link to tutorials, how-tos, and reference for action and lookup. + +If a proposed page answers more than one of these, split it and cross-link the +parts. A common shape in this repo: an explanation (e.g. +[Mailing Lists](mailing-lists.md)) paired with a how-to +([Manage Mailing Lists](../howtos/manage-mailing-lists.md)) and a reference +([HTTP API](../references/http-api.md)). + +## Naming conventions + +- **Tutorials** read as an outcome or a journey: *Run MAIL Locally*, *Send Your + First MAIL Message*. +- **How-tos** are imperative tasks: *Manage Swarms*, *Authenticate a User-Agent*. +- **Reference** pages are noun topics: *HTTP API*, *Data Models*, *Configuration*. +- **Explanations** are concept nouns: *Delivery Model*, *Security Model*. + +Files are kebab-cased within the category directory +(`howtos/manage-swarms.md`); reference pages that mirror generated artifacts note +that they are generated (e.g. the CLI references). + +## Migrating package-local and legacy docs + +Some packages still carry their own `docs/` (`src/mail/server/docs/`, +`src/mail/client/docs/`) that predate this set, and the v1 archive has its own +docs under `src/mail/legacy/docs/`. This top-level tree is canonical; migrate +package-local material into the right category here and treat legacy docs as +historical reference (see [MAIL v1 Legacy Runtime](mail-v1-legacy.md)). + +## Related pages + +- [Tutorials](../tutorials/README.md) +- [How-To Guides](../howtos/README.md) +- [Reference](../references/README.md) +- [Explanations](README.md) + +[divio]: https://docs.divio.com/documentation-system/ diff --git a/docs/explanations/mail-v1-legacy.md b/docs/explanations/mail-v1-legacy.md new file mode 100644 index 0000000..2d74897 --- /dev/null +++ b/docs/explanations/mail-v1-legacy.md @@ -0,0 +1,66 @@ +# MAIL v1 Legacy Runtime + +Status: draft + +The MAIL v1 reference runtime is archived under +[`src/mail/legacy/`](../../src/mail/legacy). It is kept for historical reference +and compatibility while the repository moves to the v2 package layout — it is +**not** the current implementation surface. This page explains how to read it +without carrying v1 assumptions into v2 work. + +## Why v1 is archived + +MAIL v1 bundled the communication contract together with an agent *runtime*: +message/task/action models, tool execution, LiteLLM-backed agent factories, and a +debug UI. MAIL v2 deliberately narrows the protocol to communication only (see +[MAIL v2 Overview](mail-v2-overview.md)), so the v1 runtime no longer reflects how +MAIL is meant to work. Rather than delete it, it is quarantined under +`src/mail/legacy/` so old examples and behavior remain available for reference and +porting. + +## What lives there + +The archive holds the v1 runtime (`api.py`, `server.py`, `client.py`, `cli.py`, +`core/`), agent factories, standard action libraries, example swarms, the +`swarms.json` machinery, interswarm routing, optional persistence helpers, and the +v1 debug UI — plus archived v1 root artifacts (its `docs/`, configs, Dockerfile, +`README.v1.md`, `AGENTS.v1.md`, `CLAUDE.v1.md`, and tests). Legacy imports use the +`mail.legacy.*` namespace. + +## Historical vs active documentation + +Treat anything under `src/mail/legacy/` — including its `docs/` — as **historical +reference**. Active guidance is this top-level `docs/` tree plus +[`spec/`](../../spec). Archived prose may still mention old `mail.*` import paths +or v1 endpoints (e.g. the `/ui/*` debug routes) that the v2 server does not +provide; do not treat those as current. + +## Running legacy tests + +Legacy tests are **not** part of the default suite. Run them explicitly, and only +when maintaining v1 behavior: + +```bash +uv run pytest # v2 / default suite +uv run --extra legacy pytest src/mail/legacy/tests # legacy runtime tests +``` + +See [Run the Test Suite](../howtos/run-tests.md). + +## Working near the archive + +- **Don't import v1 architecture into v2 docs or code.** v2 is a communication + protocol, not a runtime; keep runtime/tool-execution concepts out of v2 pages. +- **Port, don't extend.** Prefer moving behavior forward into the v2 packages over + growing the v1 APIs. Keep legacy changes scoped to compatibility, security, or + migration support. +- **Don't modernize examples in place.** Leave v1 examples as-is unless the code + has actually been ported — a v2-looking example backed by v1 code is misleading. +- **A future UI** should be built against the v2 `mail-server` / `mail-client` / + `mail-protocol` surfaces, not by adapting the archived v1 UI. + +## Related pages + +- [Repository Layout](../references/repository-layout.md) +- [MAIL v2 Overview](mail-v2-overview.md) +- [Run the Test Suite](../howtos/run-tests.md) diff --git a/docs/explanations/mail-v2-overview.md b/docs/explanations/mail-v2-overview.md new file mode 100644 index 0000000..71819da --- /dev/null +++ b/docs/explanations/mail-v2-overview.md @@ -0,0 +1,56 @@ +# MAIL v2 Overview + +Status: draft + +## What MAIL is + +The Multi-Agent Interface Layer (MAIL) is an open protocol for **email-like +communication** between humans and AI agents. It defines three things: a set of +data-structure primitives (messages, drafts, boxes, swarms, lists), an HTTP +contract for client–server interaction, and the terminology and rules that tie +them together. Participants — human users, AI agents, delivery daemons — are +addressable *user-agents* with their own inboxes, and they exchange messages much +as people exchange email. + +## The problem v2 solves + +MAIL v1 defined an inter-agent messaging contract *and* prescribed how +multi-agent systems should run: their runtime environment, tool usage, and +execution model. By 2026 that coupling looked like a mistake. Terminal-style +agents showed that an agent's runtime does not need to be defined in the same +place as its communication contract. + +MAIL v2 draws a hard line: it specifies the **communication layer and as little +else as possible**. Two goals drive it (SPEC §3.1): + +- **Focus on communication.** Runtime, tool execution, and agent internals are + explicitly out of scope. +- **Don't reinvent the wheel.** Where good standards already exist (HTTP, OAuth2, + JSON, RFC-3339 timestamps), MAIL builds on them rather than redefining them. + +## Why the repository is split into packages + +The refocus is reflected in the code layout. Instead of one runtime, v2 is four +small workspace packages with a single responsibility each — a shared protocol +contract, a server, a client, and a delivery daemon (see +[Architecture](architecture.md) and [Repository Layout](../references/repository-layout.md)). +This keeps the wire contract (`mail-swarms-protocol`) independent of any one +implementation, so alternative servers or clients can conform to the same +protocol. + +## What MAIL is not (SPEC §3.2) + +- **Not an agent runtime.** MAIL does not say how an agent thinks, acts, or runs; + it only carries messages between agents. +- **Not the only way agents may communicate.** MAIL mirrors email — ubiquitous, + but not the sole channel. Agents are free to use whatever else fits a given + job; MAIL is the shared, email-like layer, not a mandate for *all* inter-agent + traffic. + +## Related pages + +- [Run MAIL Locally](../tutorials/run-local-mail.md) — see it work end to end. +- [Architecture](architecture.md) — how the pieces fit together. +- [Protocol Specification](../references/protocol-specification.md) — the + normative source. +- [MAIL v1 Legacy Runtime](mail-v1-legacy.md) — what the archived v1 code is. diff --git a/docs/explanations/mailing-lists.md b/docs/explanations/mailing-lists.md new file mode 100644 index 0000000..cc46b8f --- /dev/null +++ b/docs/explanations/mailing-lists.md @@ -0,0 +1,235 @@ +# Mailing Lists + +Status: draft + +## Scope + +The conceptual model behind MAIL's mailing lists: what they are, how +they're addressed, how messages flow through them, how the policy +shape works, and how admin and user-agent permissions split. The +matching how-to for *operating* lists via the CLI is [Manage +Mailing Lists](../howtos/manage-mailing-lists.md); the formal route +reference is in [HTTP API](../references/http-api.md). + +## What a list is + +A MAIL list is a **swarm-scoped, addressable fan-out target**. It is +not a user-agent and it does not own an inbox. When a sender +addresses a message to a list, MAIL's local delivery path expands +the list and delivers one copy of the message to each member +(see [Local versus remote delivery in the Delivery +Model](delivery-model.md#local-versus-remote-delivery) for the +broader pipeline). + +The address shape is: + +``` +list:@@ +``` + +The `list:` prefix is what distinguishes a list address from the +other three address shapes (`agent`, `user`, `admin`); see +[Addressing Model](addressing-model.md) for the full address +taxonomy. Subscribers (members) are themselves addressable +user-agents on the same host — typically the same swarm, though +cross-swarm membership is possible. + +## Why lists exist + +Three concrete problems lists solve cleanly: + +- **One sender, many recipients with a single send.** Without + lists, a sender broadcasting to N recipients has to issue N + send requests (or send to one address and have the receiver + re-broadcast). Lists let the fan-out happen server-side in + one atomic dispatch. +- **Stable address for a changing audience.** Members can be + added or removed without the sender needing to know. A + `list:announcements@chorus@chrn.ai` address persists; the + set of recipients behind it can change daily. +- **Policy control on send / join / visibility separated from + membership.** Who can join, who can post, and who can see + the list are three different questions; the list's `policy` + object addresses each independently. + +## Anatomy of a list + +A MAIL list has the following fields (see +[`MAILList`](../references/data-models.md) for the formal +schema): + +| Field | Meaning | +| --- | --- | +| `name` | The swarm-scoped identifier (e.g., `announcements`). | +| `swarm` | The swarm this list belongs to. | +| `host` | The MAIL host the list lives on. | +| `owner` | The MAIL address of the user-agent that created or owns the list. | +| `members` | The current list of subscribed user-agent addresses. | +| `policy` | The visibility / join / send policy (see below). | +| `metadata` | Free-form key/value pairs for downstream consumers. | + +Once created, the canonical address (`name`, `swarm`, `host`) is +**immutable for the life of the list**. The `policy` and `members` +fields can change; admin-side patches in v1 are limited to policy +edits. + +## The policy shape + +The `policy` object has three fields, each an enumeration with the +v1 variant the server actually honors flagged below: + +```python +class MAILListPolicy: + visibility: "public" | "private" # v1 honors: public + join_policy: "open" | "approval" | "admin-only" # v1 honors: open + send_policy: "open" | "members-only" | "admin-only" # v1 honors: open +``` + +The wire format reserves all enumerations now so future +contributions can extend the server without changing the protocol +shape. Other variants pass protocol-layer validation but are +rejected at the endpoint layer in v1 with `501 Not Implemented`. + +What "open" means in each field: + +- `visibility: public` — the list address appears in `GET /lists` + and `GET /lists/{addr}` for any authenticated user-agent. +- `join_policy: open` — any user-agent can `POST + /lists/{addr}/subscribe` to add themselves as a member without + admin intervention. +- `send_policy: open` — any user-agent can address a message to + the list and have it expanded. + +A v1 list is therefore effectively a **public open-open** list: +anyone can see it, anyone can join, anyone can post. + +The deferred variants (`approval`, `admin-only`, `members-only`, +`private`) define the structure that v1.1+ can fill in. Designing +a list with `join_policy: admin-only` today means the policy is +recorded faithfully but the server returns `501` on any +self-subscribe attempt; readers can use that signal to know "this +list will become admin-managed when the server honors it." + +## How messages flow through a list + +When a sender addresses a message to `list:@@`, +the following happens on the receiving MAIL server: + +1. **Local delivery picks up the list address.** The recipient + prefix `list:` triggers the list-expansion path rather than + the normal user-agent delivery. +2. **The list is looked up by address.** If the list does not + exist, the message is dropped with a log line (no error to + the sender; lists are an opportunistic fan-out, not a + reliable RPC). +3. **For each member of the list,** MAIL's local delivery is + invoked again with the member's address. Each member receives + the message in their inbox as if the sender had addressed + them directly, except that the message's `metadata` carries + a `list_address` field pointing back at the originating list. +4. **Nested list members are rejected.** A list address inside + another list's member set logs a warning and is skipped; + v1 does not support recursive expansion. +5. **Webhook firing happens per-member, not per-list.** Each + recipient's webhook fires individually; the originating list + surfaces via the per-event `metadata.list_address`. + +The `metadata.list_address` field is what lets downstream +consumers (a webhook receiver, an inbox UI) distinguish "I was +sent this directly" from "I was sent this because I'm on a list." +See the [Webhook Delivery](webhook-delivery.md) explainer for +how the field appears on the wire. + +## Admin and user-agent permission split + +Lists have a clean two-layer permission model: + +### Admin-only operations + +- **Create a list** (`POST /admin/lists`). The list address must + be unique on the server. +- **Patch policy** (`PATCH /admin/lists/{addr}`). Only `policy` + is mutable; the address is fixed for the life of the list. +- **Add or remove members** (`POST /admin/lists/{addr}/members`, + `DELETE /admin/lists/{addr}/members/{member}`). Forcible + membership change without the member's consent. +- **Delete a list** (`DELETE /admin/lists/{addr}`). +- **Read everything** (`GET /admin/lists`, `GET + /admin/lists/{addr}`). Admin reads are not gated by + `visibility`. + +### User-agent operations + +- **Read public lists** (`GET /lists`, `GET /lists/{addr}`). Lists + with `visibility: public` appear; private lists do not. +- **Self-subscribe** (`POST /lists/{addr}/subscribe`). Honored + when `join_policy: open`; returns `501` for deferred variants. + Membership is permission-blind at storage — the router gates + on policy, not the storage layer. +- **Self-unsubscribe** (`POST /lists/{addr}/unsubscribe`). + Symmetric: members can always leave. +- **Send to a list** (compose + send with a list address as + recipient). Honored when `send_policy: open`; deferred variants + similarly return `501`. + +### Why this split + +The split reflects the broader MAIL trust model (see [Security +Model](security-model.md)). Admins have the authority to shape +the list as an object: who exists, who's on it, what its policy +is. User-agents have the authority to participate within the +policy bounds the admin has set. + +This means a deployment can have a `join_policy: open` list that +any user-agent can join, but the *list itself* — its existence, +its members at create time, its policy — is an admin's +responsibility. Conversely, a `join_policy: admin-only` list (when +v1.1+ honors it) means even discoverable lists can't be joined +without going through the admin. + +## Addressability examples + +A few address-shape examples to make the model concrete: + +| Address | Means | +| --- | --- | +| `bob@chorus@example.com` | The user-agent `bob` in the `chorus` swarm. | +| `list:announcements@chorus@example.com` | The list `announcements` in the `chorus` swarm. | +| `admin:ops@example.com` | The admin `ops` on the host (not swarm-scoped). | +| `user:alice@example.com` | The end-user `alice` on the host. | + +Sending to a list looks identical to sending to a user-agent +from the sender's side — the difference is what the receiving +server does with the address. + +## Things lists are not + +A few clarifying negatives: + +- **Lists are not a queue or buffer.** Messages are expanded + synchronously into per-member deliveries; there is no + list-level inbox or pending state. +- **Lists do not retain a "sent through the list" history.** The + history lives in the senders' outboxes and the recipients' + inboxes. The list as an object has no message log. +- **Lists are not a privacy boundary.** Members of a list whose + `visibility: public` is honored can be enumerated by any + authenticated user-agent via `GET /lists/{addr}`. Treat the + member list as discoverable in v1. + +## See also + +- [Manage Mailing Lists](../howtos/manage-mailing-lists.md) — the + task-oriented CLI walkthrough for operating lists. +- [Addressing Model](addressing-model.md) — the broader address + taxonomy lists sit within. +- [Delivery Model](delivery-model.md) — the local-vs-remote + delivery pipeline that handles list expansion. +- [Webhook Delivery](webhook-delivery.md) — how list deliveries + carry the `metadata.list_address` field for downstream + consumers. +- [Data Models](../references/data-models.md) — formal + field-by-field schema for `MAILList`, `MAILListInBackend`, and + `MAILListPolicy`. +- [HTTP API](../references/http-api.md) — the formal route + reference, including admin and user-agent list endpoints. diff --git a/docs/explanations/security-model.md b/docs/explanations/security-model.md new file mode 100644 index 0000000..0e44534 --- /dev/null +++ b/docs/explanations/security-model.md @@ -0,0 +1,76 @@ +# Security Model + +Status: draft + +MAIL's security model follows from one fact: the **server is the sole authority**. +It owns all state, authenticates every user-agent, and enforces what each may do. +Clients and daemons hold no authority of their own — they act only with a valid +token. This page explains the trust boundaries and operational expectations; the +normative security clauses are SPEC §9, and the implementation is in +[`auth.py`](../../src/mail/server/src/mail_server/auth.py) and +[`routers/auth.py`](../../src/mail/server/src/mail_server/routers/auth.py). + +## Authentication: passwords to tokens + +A user-agent exchanges its address and password for a short-lived **access +token** (a JWT) via `POST /auth/token`. Every subsequent request carries it as +`Authorization: Bearer `, and the server verifies the signature and expiry +on each call. Token lifetime is set by `MAIL_JWT_EXPIRE_MINUTES`; the signing key +and algorithm by `MAIL_JWT_SECRET_KEY` / `MAIL_JWT_ALGORITHM` (see +[Configuration](../references/configuration.md)). See +[Authenticate a User-Agent](../howtos/authenticate-user-agent.md). + +### Refresh tokens + +Because access tokens are short-lived, **interactive principals** — users and +admins — also receive a **refresh token** at login, which renews an access token +without re-entering the password. Agents and daemons are *not* interactive +principals: they re-authenticate with their credentials instead. The design: + +- **Rotation.** Each `POST /auth/refresh` invalidates the presented token and + issues a replacement. Tokens are grouped into a *family* with a single absolute + expiry carried forward across rotations. +- **Reuse detection.** Presenting an already-rotated token is treated as + compromise and revokes the whole family; a password reset revokes all families. +- **Transport.** For browsers the refresh token is an `httpOnly`, + `SameSite=strict` cookie scoped to `/auth` (so it is never sent to the wider + API), with the `Secure` flag on by default (`MAIL_COOKIE_SECURE`). CLI clients, + which cannot use the cookie, send it in the request body. + +## Trust boundaries by role + +- **Admins** are the most powerful principals: they create and delete agents, + users, daemons, swarms, lists, and webhooks. Admin credentials are effectively + server-control credentials — generate and hand them out with extreme caution + (SPEC §5.1). +- **Daemons** are trusted couriers. They deliver messages but MUST NOT read, + alter, or compose message content, and SHOULD NOT send messages of their own + (SPEC §5.3). A compromised daemon is a delivery-integrity problem, so treat its + credentials with the same caution as admin credentials. +- **Agents and users** may compose and send messages, manage their own list + subscriptions, and read limited server metadata — nothing administrative. + +## Secret handling + +- **Credentials via environment, not arguments.** Clients and daemons read + `MAIL_PASSWORD` / tokens from environment variables rather than command-line + flags, keeping secrets out of shell history (SPEC §9.1–9.2). The CLI never + writes tokens to disk — it prints them for you to export. +- **Don't log sensitive data.** Message contents and credentials SHOULD NOT be + logged by clients, daemons, or the server (SPEC §9). +- **Plaintext init secrets.** `backend-init` writes generated passwords in + plaintext under `.secrets/`; capture and delete them promptly (see + [Initialize the Memory Backend](../howtos/initialize-memory-backend.md)). + +## Production expectations (SPEC §9.3) + +- Serve over **TLS**; keep `MAIL_COOKIE_SECURE` on so refresh cookies are + HTTPS-only. +- Put the server **behind a reverse proxy** for load balancing and rate limiting. +- Rotate user-agent passwords periodically (SPEC §9.4). + +## Related pages + +- [Authenticate a User-Agent](../howtos/authenticate-user-agent.md) +- [Configuration](../references/configuration.md) +- [Protocol Specification](../references/protocol-specification.md) diff --git a/docs/explanations/webhook-delivery.md b/docs/explanations/webhook-delivery.md new file mode 100644 index 0000000..fec2591 --- /dev/null +++ b/docs/explanations/webhook-delivery.md @@ -0,0 +1,263 @@ +# Webhook Delivery + +Status: draft + +## Scope + +How MAIL notifies external consumers when mail is delivered: the event +shape, the security model, the retry behavior, and the assumptions the +contract places on receivers. + +This document is for implementers building a webhook consumer (a +service that receives MAIL events and routes them somewhere else). +The matching how-to for *registering* webhooks via the admin API is +[Manage Webhooks](../howtos/manage-webhooks.md); the matching tutorial +for *building* a receiver end-to-end is [Build a Webhook +Receiver](../tutorials/build-webhook-receiver.md). + +## What webhooks are for + +The MAIL inbox is the durable surface: mail lives there, indexed by +recipient, and any authenticated user-agent can poll it via the +inbox endpoints. Webhooks are a *push* alternative: when a message +is delivered to a recipient's inbox, the MAIL server fires an HTTP +`POST` to one or more registered URLs with a structured payload, so +downstream services can react without polling. + +Webhooks do not replace the inbox. The inbox is the source of truth. +A webhook that fails to deliver is a notification missed; the message +itself is still readable by the recipient via the normal inbox API. +This shapes the security and retry contract below. + +## Event types + +The only `event` value in v2 is `mail.delivered`. A future release +may add other event types; receivers should reject events whose +`event` field they do not recognize, but should not fail registration +on the presence of an unknown event in the `events` array (the +`/admin/webhooks` validator already gates that). + +## Payload shape + +Every `mail.delivered` event is delivered as a JSON request body +shaped like: + +```json +{ + "event": "mail.delivered", + "event_id": "evt_", + "delivered_at": "2026-06-24T19:31:00.000000+00:00", + "message": { + "message_id": "msg_", + "reply_to": null, + "sender": "alice@chorus@example.com", + "recipient": "bob@chorus@example.com", + "subject": "Daily briefing", + "body": "…", + "tags": [], + "sent_at": "2026-06-24T19:30:55.123456+00:00", + "swarm": "chorus", + "metadata": {} + } +} +``` + +Field notes: + +- `event_id` is unique per delivery attempt SET but is reused across + retries (see [Retries](#retries)). Receivers MUST treat + `event_id` as the dedup key — a webhook receiver that processes + the same `event_id` more than once is a bug. +- `message_id` is prefixed with `msg_`. The bare UUID is stored on + the canonical `MAILMessage`; the prefix is added at webhook + payload construction time. Use the prefixed form when fetching the + full message via the inbox API. +- `reply_to`, when set, is the prefixed `message_id` of the original + message this is replying to. +- `tags` is a list of slug-shaped strings the sender attached. +- `metadata.list_address`, when present, indicates the delivery + originated from a list expansion. Use it to surface the originating + list to the end-recipient. + +Refer to [Data Models](../references/data-models.md) for the full +field-by-field schema of the inner `MAILMessageInWebhook`. + +## Security model + +### Why HMAC + +MAIL emits webhooks to URLs configured by an administrator. The +receiver needs to verify that an incoming request actually came from +MAIL (not from a third party who guessed or scanned the URL). The +shared mechanism is an HMAC signature over the request body, computed +with a secret known only to MAIL and the receiver. + +### What gets signed + +MAIL computes the signature as: + +``` +signature = HMAC-SHA256(secret, f"{timestamp}.{raw_body}") +``` + +Where: + +- `timestamp` is the value of the `X-MAIL-Timestamp` header (Unix + seconds since the epoch, as a string). +- `raw_body` is the *exact byte sequence* of the request body. MAIL + signs `payload.model_dump_json()` (Pydantic's canonical JSON + serialization) and posts those same bytes as the request body. + Re-encoding via `json=...` would produce different bytes (different + key order, whitespace, type coercion) and break verification. + +The `secret` is the value supplied when the webhook was registered. + +### Headers sent on every webhook POST + +| Header | Value | +| ------------------ | ------------------------------------------------- | +| `Content-Type` | `application/json` | +| `X-MAIL-Event-Id` | The `event_id` from the payload. | +| `X-MAIL-Timestamp` | Unix seconds since epoch, as a string. | +| `X-MAIL-Signature` | `sha256=` where `` is the HMAC digest. | +| `User-Agent` | `Multi-Agent-Interface-Layer-Server/2.0.0 (...)` | + +### Receiver verification + +A correct receiver does the following on every request: + +1. Read `X-MAIL-Timestamp` and reject the request (`408` or `400`) + if it is more than 5 minutes from the receiver's clock. This + bounds the replay window. +2. Read `X-MAIL-Signature` and strip the `sha256=` prefix. +3. Recompute `HMAC-SHA256(secret, f"{timestamp}.{raw_body}")` over + the raw request body bytes (NOT the parsed JSON). +4. Compare to the received digest using a constant-time comparison. + Reject (`403`) if they differ. +5. Read `X-MAIL-Event-Id` and check it against a recent-events store. + If it has been processed in the last ~24 hours, return `200` with + a no-op response (the request is a retry; the original processing + stands). +6. Process the event. Return `200` (or `202`) on success. + +Step 5 is where the dedup contract lives. MAIL retries on transient +failure (see below) and reuses the same `event_id` across retries. +A receiver that does not dedup will process the same delivery +multiple times under load or after any transient outage. + +### What if the secret isn't configured + +A receiver that has registered a webhook but does not yet have the +secret in its environment SHOULD reject incoming requests with +`503 Service Unavailable` (not `403`). `403` would suggest a real +authentication failure; `503` correctly signals "I'm not ready, +please retry." + +## Retries + +MAIL fires up to **six attempts** per event, with the following +delays between attempts: + +| Attempt | Delay before this attempt | Cumulative wall-clock | +| ------- | ------------------------- | ---------------------- | +| 1 | (immediate) | 0 | +| 2 | 1 second | ~1 s | +| 3 | 30 seconds | ~31 s | +| 4 | 5 minutes | ~5 min | +| 5 | 1 hour | ~1 h | +| 6 | 6 hours | ~7 h | + +After the sixth attempt, MAIL gives up. The total retry window is +roughly **7 hours and 31 seconds** from the first attempt. + +A retry is triggered when `_webhook_delivered_post` returns `True`, +which happens for any of: + +- `httpx.TimeoutException` on the request. +- A `5xx` status code from the receiver. +- A `429 Too Many Requests` status code. + +A retry is NOT triggered (and the event is considered delivered or +abandoned) for: + +- A `2xx` status code (success). +- A `4xx` status code other than `429` (the receiver explicitly + rejected the request; retries won't change that). + +### Implications for receivers + +- A receiver that needs to throttle MAIL's webhook firing should + return `429` rather than starve. MAIL backs off cleanly. +- A receiver that detects a permanently malformed payload should + return `4xx` (not `5xx`). MAIL will not retry, which is the + correct behavior — the next event will succeed. +- A receiver should NOT return `5xx` for "I couldn't route this + internally but I have the message stored." That makes MAIL retry + unnecessarily. Instead, return `200` — MAIL's inbox is the source + of truth; the routing failure does not need MAIL's help to + recover. + +## The "inbox is source of truth" contract + +This is the single most important assumption a receiver makes: + +> If a webhook delivery fails, the message is not lost. The +> recipient can still poll their MAIL inbox via the regular HTTP +> API. The webhook is a notification — its failure shapes UX, not +> correctness. + +In practice this means: + +- A receiver that successfully verifies the signature, accepts the + event_id as new, but then fails internally while processing the + event SHOULD STILL RETURN `200`. The event is recorded as + processed; the internal failure is the receiver's problem to + recover from (it can read the message from the MAIL inbox on its + own schedule). +- A receiver MUST NOT return `5xx` to "force MAIL to retry." MAIL's + retries are for transport failures, not for receiver-internal + bugs. The retry schedule above is short enough that downstream + systems can fail and recover quickly without webhook help. + +## Reliability and ordering + +The webhook contract guarantees: + +- **At-least-once delivery** within the 7-hour retry window. After + retries exhaust, the message is still in the recipient's inbox + and can be fetched there. +- **Per-event idempotency via `event_id`.** Receivers MUST dedup on + `event_id` to handle retries correctly. + +The contract does NOT guarantee: + +- **Ordering.** Webhooks for related events (e.g., several mails to + the same recipient in rapid succession) may arrive out of order + due to retry interleavings or concurrent firing. Receivers MUST + treat each event independently. The `sent_at` and `delivered_at` + timestamps can be used to reconstruct ordering if needed. +- **Exactly-once delivery.** Dedup by `event_id` collapses the + at-least-once delivery to at-most-once *processing* in the + receiver's domain, but MAIL itself can fire the same event_id up + to six times. +- **Synchronous delivery.** Webhook firing happens asynchronously + on the MAIL server. A successful `POST /drafts/{id}/send` (or + similar) does not block on webhook delivery. + +## See also + +- [Manage Webhooks](../howtos/manage-webhooks.md) — registering, + inspecting, and deleting webhooks via the admin API. +- [Build a Webhook Receiver](../tutorials/build-webhook-receiver.md) + — step-by-step tutorial walking through the signature + verification, dedup, and processing of a real receiver. +- [Delivery Model](delivery-model.md) — broader context on how MAIL + routes a message from sender to recipient inbox. +- [Security Model](security-model.md) — the broader auth and + authorization model the webhook contract sits within. +- [Data Models](../references/data-models.md) — formal field-by-field + schemas for `MAILWebhook`, `MAILMessageInWebhook`, and the + envelope. +- [HTTP API](../references/http-api.md) — the formal route list, + including the webhook firing target shape and the admin + registration endpoints. diff --git a/docs/howtos/README.md b/docs/howtos/README.md new file mode 100644 index 0000000..67587c2 --- /dev/null +++ b/docs/howtos/README.md @@ -0,0 +1,29 @@ +# How-To Guides + +How-to guides solve specific MAIL tasks for readers who already know the basic +shape of the system. Each guide should start from a realistic state, provide +ordered steps, and stop when the task is complete. + +## Planned Guides + +| Page | Task | +| --- | --- | +| [Initialize the Memory Backend](initialize-memory-backend.md) | Create local server state for development or testing. | +| [Run the MAIL Server](run-server.md) | Start and configure `mail-server`. | +| [Run the MAIL Daemon](run-daemon.md) | Start `mail-daemon` against an existing server. | +| [Authenticate a User-Agent](authenticate-user-agent.md) | Obtain and use a MAIL bearer token. | +| [Send a Message with the CLI](send-message-cli.md) | Compose and send a message from the command line. | +| [Manage User-Agents](manage-user-agents.md) | Create, inspect, and remove agents, users, admins, and daemons. | +| [Manage Swarms](manage-swarms.md) | Create, inspect, and delete swarms. | +| [Manage Mailing Lists](manage-mailing-lists.md) | Create lists and manage subscriptions or members. | +| [Manage Webhooks](manage-webhooks.md) | Register, inspect, update, and delete webhook subscriptions. | +| [Regenerate API Artifacts](regenerate-api-artifacts.md) | Refresh generated OpenAPI or documentation artifacts. | +| [Run the Test Suite](run-tests.md) | Run focused or full repository tests. | + +## How-To Checklist + +- Title the page as "How to ...". +- Assume the reader knows what outcome they want. +- Provide commands and expected success checks. +- Link to reference material for exhaustive options. +- Link to explanations for background instead of embedding long discussion. diff --git a/docs/howtos/authenticate-user-agent.md b/docs/howtos/authenticate-user-agent.md new file mode 100644 index 0000000..ecca026 --- /dev/null +++ b/docs/howtos/authenticate-user-agent.md @@ -0,0 +1,83 @@ +# Authenticate a User-Agent + +Status: draft + +## Goal + +How to exchange MAIL address credentials for a bearer token and use that token +with CLI or HTTP requests. + +## Starting Point + +The reader has a server URL, a MAIL address, and a password. + +## Steps + +### 1. Set environment variables + +In order to log into a MAIL server using the MAIL client CLI, you must set the following environment variables: +- `MAIL_SERVER`: The URL of the MAIL server to log into, e.g. `https://mail-swarms.example.com`. +- `MAIL_ADDRESS`: The address of the MAIL user-agent to log in as, e.g. `user:example@example.com`. +- `MAIL_PASSWORD`: The password for the MAIL user-agent to log in as. + +### 2. Run `mail login` + +With the environment variables set as described in step 1, log into the MAIL server using the CLI client command `login`: + +```bash +uv run mail login +``` + +### 3. Store the returned token in `MAIL_TOKEN` + +Running the `login` command above should print a temporary access token to the console. +This can be used in subsequent operations with the `mail` client CLI, rather than `MAIL_ADDRESS` and `MAIL_PASSWORD`. +Store this token as an environment variable called `MAIL_TOKEN`: + +```env +MAIL_TOKEN={token} +``` + +If you logged in as a **user** or **admin** (an *interactive principal*), `login` also prints a **refresh token**. Store it as `MAIL_REFRESH_TOKEN` so you can renew your access token later without re-entering your password (see step 6). Agents and daemons are not issued refresh tokens and re-authenticate with their credentials instead. + +### 4. Run `mail whoami` + +With your `MAIL_SERVER` and `MAIL_TOKEN` environment variables set, you can now view your own user-agent information using the `whoami` command: + +```bash +uv run mail whoami +``` + +This will print the authenticated user-agent's MAIL address and user-agent type. Ensure these are both the expected values. + +### 5. Use the token in an HTTP `Authorization: Bearer ...` header + +Since the MAIL server accepts access tokens in the `Authorization` header, you can attempt to hit the `whoami` endpoint with a raw HTTP request rather than through the `mail` client CLI: + +```bash +curl {server_url}/auth/whoami \ +-H "Authorization: Bearer {token}" +``` + +### 6. Refresh or replace expired tokens + +Server-issued access tokens will expire after a predetermined length of time (e.g. 15, 30, or 60 minutes). If you attempt to hit a MAIL server endpoint with a previously-valid token and get a `401` response, that likely means your token has expired. + +If you saved a refresh token in step 3 (users and admins only), renew your access token without re-entering credentials by running `mail refresh` with `MAIL_SERVER` and `MAIL_REFRESH_TOKEN` set: + +```bash +MAIL_SERVER={server_url} +MAIL_REFRESH_TOKEN={refresh_token} +uv run mail refresh +``` + +Refresh tokens are **rotated**: each `mail refresh` invalidates the token you sent and prints a replacement, so update `MAIL_REFRESH_TOKEN` with the new value every time. Store the new access token in `MAIL_TOKEN` as in step 3. + +Agents and daemons are not issued refresh tokens; they obtain a fresh access token by logging in again (repeat steps 1-2). + +## Source Material + +- `src/mail/client/src/mail_client/commands/login.py` +- `src/mail/client/src/mail_client/commands/whoami.py` +- `src/mail/server/src/mail_server/routers/auth.py` +- `spec/openapi.yaml` diff --git a/docs/howtos/initialize-memory-backend.md b/docs/howtos/initialize-memory-backend.md new file mode 100644 index 0000000..7a82a0a --- /dev/null +++ b/docs/howtos/initialize-memory-backend.md @@ -0,0 +1,139 @@ +# Initialize the Memory Backend + +Status: draft + +## Goal + +How to create a local memory backend with initial swarms, user-agents, and +credentials for development. + +## Starting Point + +The repository is cloned, dependencies are installed, and the reader wants local +server state for `mail-server --backend memory`. + +## Steps + +### 1. Run `backend-init` + +You can initialize an in-memory backend for a MAIL server by running the `backend-init` script: + +```bash +uv run backend-init +``` + +By default (i.e., with no specified arguments), this script will generate a new MAIL server backend with the following attributes: +- **Backend Type**: `memory` +- **Deployment**: `default` +- **Swarm Name**: `default` +- **Swarm Description**: `A MAIL swarm` +- **Swarm Keywords**: `[]` +- **Agents**: `['supervisor']` +- **Daemons**: `['dummy']` +- **Users**: `['dummy']` +- **Admins**: `['dummy']` +- **Host**: `example.com` + +### 2. Customize deployment, swarm, host, agents, daemons, users, or admins + +To initialize a new backend with a different deployment name (e.g. `example`), specify the `-d`/`--deployment` argument for the `backend-init` script: + +```bash +uv run backend-init --deployment "example" +``` + +To initialize a new backend with a different swarm name (e.g. `example`), specify the `-s`/`--swarm` argument: + +```bash +uv run backend-init --swarm "example" +``` + +To initialize a new backend with a different swarm description (e.g. `My custom description`), specify the `-sd`/`--swarm-description` argument: + +```bash +uv run backend-init --swarm-description "My custom description" +``` + +To initialize a new backend with a different list of swarm keywords (e.g. `['dev', 'internal']`), specify the `-sk`/`--swarm-keywords` argument: + +```bash +uv run backend-init --swarm-keywords "dev" "internal" +``` + +To initialize a new backend with a different list of agent names (e.g. `['meta', 'scribe']`), specify the `--agents` argument: + +```bash +uv run backend-init --agents "meta" "scribe" +``` + +For a different list of daemon names (e.g. `['worker-1', 'worker-2']`), specify the `--daemons` argument: + +```bash +uv run backend-init --daemons "worker-1" "worker-2" +``` + +For a different list of user names (e.g. `['user-1', 'user-2', 'user-3']`), specify the `--users` argument: + +```bash +uv run backend-init --users "user-1" "user-2" "user-3" +``` + +For a different list of admin names (e.g. `['a1', 'a2', 'a3', 'a4']`), specify the `--admins` argument: + +```bash +uv run backend-init --admins "a1" "a2" "a3" "a4" +``` + +To initialize a new backend with a different host (e.g. `my-site.com`), specify the `-H`/`--host` argument: + +```bash +uv run backend-init --host "my-site.com" +``` + +### 3. Locate generated credential files + +Upon memory backend initialization, a password for each created user-agent is randomly-generated. These passwords will be stored in plaintext inside `~/.mail-swarms/deployments/{deployment}/.secrets`, where `deployment` is the name of the deployment created with `backend-init`. + +For example, if you have a deployment named `default` and a user-agent address `supervisor@default@example.com`, you can view its plaintext password: + +```bash +cat ~/.mail-swarms/deployments/default/.secrets/supervisor@default@example.com +``` + +### 4. Remove or protect plaintext password files after capture + +Copy the plaintext passwords for the user-agents you intend to keep into a safe place. +Once you have these copied, you can simply delete the `.secrets` folder in your deployment: + +```bash +rm -rf ~/.mail-swarms/deployments/{deployment}/.secrets +``` + +where `deployment` is the name chosen for your deployment. + +### 5. Reinitialize a clean slate when needed + +All state for a deployment lives under a single directory, +`~/.mail-swarms/deployments/{deployment}/`, which holds the `swarms/`, +`user_agents/`, and `messages/` stores plus the `message_buffer.lock` file and +the `.secrets/` folder from step 3. Re-running `backend-init` against an existing +deployment reuses that directory rather than clearing it, so for a guaranteed +clean slate — a fresh swarm, user-agents, and credentials — stop any running +`mail-server`, remove the deployment directory, and initialize again: + +```bash +# stop mail-server first, then: +rm -rf ~/.mail-swarms/deployments/{deployment} +uv run backend-init --deployment "{deployment}" +``` + +where `deployment` is the name chosen for your deployment. This regenerates the +swarm, user-agents, and fresh plaintext credentials — capture and protect them +again as in steps 3-4. To reset *every* deployment at once, remove the whole +`~/.mail-swarms/deployments` directory instead. + +## Source Material + +- `src/mail/server/src/mail_server/backend_init.py` +- `src/mail/server/src/mail_server/backends/memory/init.py` +- `src/mail/server/docs/tutorials/quickstart.md` diff --git a/docs/howtos/manage-mailing-lists.md b/docs/howtos/manage-mailing-lists.md new file mode 100644 index 0000000..dc8a70c --- /dev/null +++ b/docs/howtos/manage-mailing-lists.md @@ -0,0 +1,214 @@ +# Manage Mailing Lists + +Status: draft + +## Goal + +How to create mailing lists, inspect them, manage subscriptions or +members, edit policy, and delete them. The conceptual model — the +address shape, the policy structure, the admin/user permission +split — is in [Mailing Lists](../explanations/mailing-lists.md); +this how-to assumes you've at least skimmed it. + +## Starting Point + +The reader has credentials with permissions appropriate for the list +action. Admin credentials are required for create / patch / member +add-remove / delete; user-agent credentials are sufficient for +read / subscribe / unsubscribe / send. + +Two address forms appear below, and the CLI is strict about which it +wants — see [Addressing Model](../explanations/addressing-model.md): + +- **List-management commands** (`list-get`, `list-subscribe`, + `list-unsubscribe`, `list-member-post`, `list-member-delete`, + `list-delete`) take the **local** `{name}@{swarm}` form (e.g. + `announcements@chorus`). The server resolves it to the list's + canonical `list:{name}@{swarm}@{host}` address using its own host. +- **Sending to a list** (step 8) names the list as a message + recipient, so it uses the **full** routable form + `list:{name}@{swarm}@{host}` (e.g. + `list:announcements@chorus@example.com`), just like any other + recipient. + +This how-to writes `{list_address}` for the local management form and +`{list_recipient}` for the full send form. + +## Steps + +### 1. List available mailing lists + +MAIL user-agents can view all mailing lists visible to them by using the `mail` CLI command `lists` with valid credentials: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail lists +``` + +This will print all list addresses visible to the authenticated user-agent to the console. + +### 2. Inspect a list by address + +User-agents can inspect a specific list (visible to them) by address through the `mail` command `list-get`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail list-get {list_address} +``` + +If the specified list by address exists, its ID, owner, member user-agents, policies, and metadata will be printed to the console. + +### 3. Create a list as an admin + +To create a new mailing list on a MAIL server, use the `mail-admin` CLI client with valid admin credentials: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-post {list_name} {swarm_name} {list_owner} +``` + +Note that `list_name`, `swarm_name`, and `list_owner` are required arguments. You can optionally specify a list of MAIL user-agent addresses to add as members upon list creation: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-post {list_name} {swarm_name} {list_owner} \ +--members "user:dummy@example.com" "supervisor@default@example.com" +``` + +### 4. Subscribe and unsubscribe as a user-agent + +Non-`admin` user-agents may subscribe to or unsubscribe from mailing lists. +To subscribe to an existing mailing list by a given address, use the `mail` CLI client with authorized credentials: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail list-subscribe {list_address} +``` + +If the operation was successful, details on the list subscribed to will be printed to the console. +To unsubscribe from an existing mailing list by a given address, use the `mail` CLI client with authorized credentials: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail list-unsubscribe {list_address} +``` + +If the operation was successful, details on the list unsubscribed from will be printed to the console. + +### 5. Add or remove members as an admin + +Admins can add a MAIL user-agent by address to an existing mailing list with the `mail-admin` command `list-member-post`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-member-post {list_address} {member_address} +``` + +If successful, details on the mailing list will be printed to the console. + +Similarly, admins can remove existing members by address from a mailing list with `list-member-delete`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-member-delete {list_address} {member_address} +``` + +If successful, details on the mailing list will be printed to the console. + +### 6. Update list policy as an admin + +Policy is the only mutable part of a list — the canonical address +(`name`, `swarm`, `host`) is immutable. The server exposes +`PATCH /admin/lists/{name}@{swarm}` accepting an `AdminListPatchRequest` +body (a single optional `policy` object). + +> **Not yet available from the CLI.** The `mail-admin list-patch` +> command is registered but currently a stub: it declares no +> arguments and its handler raises `NotImplementedError` +> (`src/mail/client/src/mail_client/commands/list_patch.py`). Until it +> is implemented, patch a list's policy by calling the endpoint +> directly — for example with `curl` (the path takes the **local** +> `{name}@{swarm}` form): + +```bash +curl -s -X PATCH "$MAIL_SERVER/admin/lists/announcements@chorus" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"policy":{"visibility":"public","join_policy":"open","send_policy":"open"}}' +``` + +For v1, only `public` / `open` / `open` are honored; the other +variants are reserved in the wire format and rejected at the +endpoint layer with `501`. See [Mailing +Lists](../explanations/mailing-lists.md#the-policy-shape) for +context on the deferred variants. + +### 7. Delete a list as an admin + +To remove an existing list entirely, use `list-delete`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin list-delete {list_address} +``` + +The list is removed from the server and the canonical address +becomes available for re-creation. In-flight messages already +expanded into per-member deliveries before the delete are +unaffected (they live in the recipients' inboxes); messages +addressed to the list after the delete are dropped per the +unknown-list path described in [Mailing Lists → How messages +flow through a list](../explanations/mailing-lists.md#how-messages-flow-through-a-list). + +### 8. Send a message to a list address + +If they are authorized to do so, user-agents can send a message to a +list by naming the list as a recipient of the `mail` command `send`. +Because this is a message recipient (not a management target), use the +**full** routable form `list:{name}@{swarm}@{host}` here — written +`{list_recipient}` below: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail send {draft_id} {list_recipient} +``` + +The receiving server expands the list and delivers one copy to +each member's inbox. Each member's webhook (if any) fires with a +`metadata.list_address` field naming the originating list so +downstream consumers can distinguish list deliveries from direct +ones — see [Webhook +Delivery](../explanations/webhook-delivery.md#payload-shape) for +the field placement on the wire. + +## See also + +- [Mailing Lists](../explanations/mailing-lists.md) — the + conceptual model: address shape, policy fields, expansion + semantics, permission split. +- [Webhook Delivery](../explanations/webhook-delivery.md) — how + list deliveries surface to webhook receivers via + `metadata.list_address`. +- [Addressing Model](../explanations/addressing-model.md) — the + full address taxonomy lists sit within. +- [HTTP API](../references/http-api.md) — the formal route + reference for admin and user-agent list endpoints. + +## Source Material + +- `src/mail/client/src/mail_client/commands/lists.py` +- `src/mail/client/src/mail_client/commands/list_post.py` +- `src/mail/client/src/mail_client/commands/list_subscribe.py` +- `src/mail/client/src/mail_client/commands/list_member_post.py` +- `src/mail/server/src/mail_server/routers/lists.py` +- `src/mail/protocol/src/mail_protocol/core/lists.py` diff --git a/docs/howtos/manage-swarms.md b/docs/howtos/manage-swarms.md new file mode 100644 index 0000000..dd1ef2e --- /dev/null +++ b/docs/howtos/manage-swarms.md @@ -0,0 +1,80 @@ +# Manage Swarms + +Status: draft + +## Goal + +How to create, inspect, and delete MAIL swarms. + +## Starting Point + +The reader has an admin token for changes or a regular user-agent token for +read-only swarm inspection. + +## Steps + +### 1. List swarms + +Authorized user-agents can list all swarms on a MAIL server via the `mail` CLI command `swarm-list` (aliases: `swarms`, `sl`): + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail swarm-list +``` + +This will print the name, keywords, and number of agents for each swarm on the server. + +### 2. Inspect a swarm by name + +Authorized user-agents can inspect a specific, existing swarm by name on a MAIL server via the `mail` command `swarm-get`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail swarm-get {swarm_name} +``` + +If a swarm with the specified name exists, its description, keywords, and full list of agents will be printed to the console. + +### 3. Create a swarm as an admin + +Authorized admins can create a new swarm on a MAIL server via the `mail-admin` CLI command `swarm-post`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin swarm-post {swarm_name} {swarm_description} +``` + +The arguments `swarm_name` and `swarm_description` are required. Optionally, the swarm's `keywords` can be specified as well: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin swarm-post {swarm_name} {swarm_description} \ +--keywords "kw-1" "kw-2" +``` + +If this operation was successful, information on the new swarm will be printed to the console. + +### 4. Delete a swarm as an admin + +Authorized admins can delete an existing swarm by name on a MAIL server via the `mail-admin` command `swarm-delete`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +uv run mail-admin swarm-delete {swarm_name} +``` + +If this operation was successful, information on the newly-deleted swarm will be printed to the console. + +## Source Material + +- `src/mail/client/src/mail_client/commands/swarm_list.py` +- `src/mail/client/src/mail_client/commands/swarm_get.py` +- `src/mail/client/src/mail_client/commands/swarm_post.py` +- `src/mail/client/src/mail_client/commands/swarm_delete.py` +- `src/mail/server/src/mail_server/routers/swarms.py` +- `src/mail/server/src/mail_server/routers/admin.py` diff --git a/docs/howtos/manage-user-agents.md b/docs/howtos/manage-user-agents.md new file mode 100644 index 0000000..daea85b --- /dev/null +++ b/docs/howtos/manage-user-agents.md @@ -0,0 +1,88 @@ +# Manage User-Agents + +Status: draft + +## Goal + +Create, inspect, and remove MAIL agents, users, and daemons with the `mail-admin` +CLI. + +## Starting Point + +You have an **admin** `MAIL_TOKEN` for the target server (see +[Authenticate a User-Agent](authenticate-user-agent.md)). Admin accounts +themselves are created by `backend-init`, not by these commands — see +[Initialize the Memory Backend](initialize-memory-backend.md). For the address +shapes used below, see [Addressing Model](../explanations/addressing-model.md). + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={admin_jwt} +``` + +## Steps + +### 1. List existing user-agents by type + +```bash +uv run mail-admin agent-list +uv run mail-admin user-list +uv run mail-admin daemon-list +``` + +### 2. Create an agent in a swarm + +Agents are swarm-scoped, so the argument is the **local** `agent@swarm` form. The +command prompts for the new agent's password interactively (hidden input) rather +than taking it as an argument, keeping secrets out of shell history: + +```bash +uv run mail-admin agent-post supervisor@default +# agent password: ******** +``` + +### 3. Create a host-scoped user or daemon + +Users and daemons are host-scoped, so they take a bare id / worker name; both +prompt for a password: + +```bash +uv run mail-admin user-post alice # -> user:alice@{host} +uv run mail-admin daemon-post worker-1 # -> daemon:worker-1@{host} +``` + +### 4. Inspect a user-agent + +```bash +uv run mail-admin agent-get supervisor@default +uv run mail-admin user-get alice +uv run mail-admin daemon-get worker-1 +``` + +### 5. Delete a user-agent + +```bash +uv run mail-admin agent-delete supervisor@default +uv run mail-admin user-delete alice +uv run mail-admin daemon-delete worker-1 +``` + +## Verification + +A created user-agent appears in the matching `*-list` / `*-get` output and can +authenticate with its generated password. Handle admin credentials with care — +they are effectively server-control credentials (see +[Security Model](../explanations/security-model.md)). + +## See also + +- [Admin CLI](../references/admin-cli.md) — full `mail-admin` reference. +- [Manage Swarms](manage-swarms.md) — swarms an agent lives in. +- [Security Model](../explanations/security-model.md) + +## Source Material + +- `src/mail/client/src/mail_client/admin_panel.py` +- `src/mail/client/src/mail_client/commands/agent_post.py` +- `src/mail/client/src/mail_client/commands/user_post.py` +- `src/mail/client/src/mail_client/commands/daemon_post.py` diff --git a/docs/howtos/manage-webhooks.md b/docs/howtos/manage-webhooks.md new file mode 100644 index 0000000..573c0e5 --- /dev/null +++ b/docs/howtos/manage-webhooks.md @@ -0,0 +1,148 @@ +# Manage Webhooks + +Status: draft + +## Goal + +How to register, inspect, update, and delete webhook subscriptions on +a MAIL server using the admin API. Webhooks let downstream services +receive `mail.delivered` events without polling — see [Webhook +Delivery](../explanations/webhook-delivery.md) for the conceptual +contract. + +## Starting Point + +You have admin credentials for the MAIL server, and you know the +public URL the webhook should fire against. You have a shared secret +already agreed with the receiver, or you're prepared to generate one. + +## Steps + +### 1. Generate a secret (if you don't already have one) + +Webhook signatures use an HMAC-SHA256 with a shared secret. The +secret must be known to both MAIL and the receiver, and not exposed +elsewhere. A reasonable generator: + +```bash +python -c "import secrets; print(secrets.token_urlsafe(32))" +``` + +Save the resulting string in a secure location accessible to the +receiver process. The receiver loads it from a config file or env +var; MAIL stores it on the registered webhook record. + +### 2. Register the webhook + +`POST /admin/webhooks` with the receiver URL, the events to +subscribe to, and the secret. v2 supports one event type +(`mail.delivered`); future versions may add more. + +```bash +ADMIN_TOKEN="$(cat ~/.mail/admin.token)" +SECRET="…" # from step 1 + +curl -sS -X POST "$MAIL_SERVER/admin/webhooks" \ + -H "Authorization: Bearer $ADMIN_TOKEN" \ + -H "Content-Type: application/json" \ + -d "$(jq -n \ + --arg url "https://my-receiver.example.com/mail/webhook" \ + --arg secret "$SECRET" \ + '{url: $url, events: ["mail.delivered"], secret: $secret}')" +``` + +A successful response returns the new webhook record (including its +generated `webhook_id`): + +```json +{ + "webhook": { + "webhook_id": "wh_abc12345-…", + "url": "https://my-receiver.example.com/mail/webhook", + "events": ["mail.delivered"], + "secret": "…" + }, + "metadata": {} +} +``` + +Save the `webhook_id` — you'll need it to inspect, update, or delete +the registration later. The secret is also stored in MAIL's backend; +the receiver only needs its own copy. + +### 3. List all registered webhooks + +`GET /admin/webhooks` returns the IDs of every webhook on the +server: + +```bash +curl -sS "$MAIL_SERVER/admin/webhooks" \ + -H "Authorization: Bearer $ADMIN_TOKEN" +``` + +```json +{ + "webhook_ids": ["wh_abc12345-…", "wh_def67890-…"], + "metadata": {} +} +``` + +To get the full record for a specific webhook, use +`GET /admin/webhooks/{webhook_id}`: + +```bash +curl -sS "$MAIL_SERVER/admin/webhooks/wh_abc12345-…" \ + -H "Authorization: Bearer $ADMIN_TOKEN" +``` + +### 4. Update an existing webhook + +`PATCH /admin/webhooks/{webhook_id}` can change the receiver URL or +rotate the secret. The webhook_id and the subscribed events are +immutable; to change events you must delete and re-register. + +```bash +curl -sS -X PATCH "$MAIL_SERVER/admin/webhooks/wh_abc12345-…" \ + -H "Authorization: Bearer $ADMIN_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"url": "https://new-receiver.example.com/mail/webhook", "secret": "new-secret"}' +``` + +When rotating a secret, coordinate with the receiver so both sides +update at the same moment; otherwise webhooks delivered between the +two updates will fail signature verification on the receiver side. + +### 5. Delete a webhook + +`DELETE /admin/webhooks/{webhook_id}` removes the registration. MAIL +will stop firing webhooks to that URL immediately. + +```bash +curl -sS -X DELETE "$MAIL_SERVER/admin/webhooks/wh_abc12345-…" \ + -H "Authorization: Bearer $ADMIN_TOKEN" +``` + +In-flight retries for events that were already being delivered when +the webhook was deleted are not interrupted; if a retry attempt +succeeds, the receiver still gets the event. After the retry +schedule exhausts (or succeeds), no further events fire. + +## Validation + +After registering a webhook, you can confirm it works end-to-end by: + +1. Sending a test message to a recipient on the server. +2. Watching the receiver's logs for an incoming `POST` with the + expected event_id, signature, and payload. +3. Confirming the receiver returns `200`. (A non-2xx response will + trigger MAIL's retry ladder; see [Webhook + Delivery](../explanations/webhook-delivery.md#retries).) + +## See also + +- [Webhook Delivery](../explanations/webhook-delivery.md) — the + contract MAIL emits and the receiver must verify. +- [Build a Webhook Receiver](../tutorials/build-webhook-receiver.md) + — implementer's tutorial for writing a receiver from scratch. +- [HTTP API](../references/http-api.md) — full route and response + reference. diff --git a/docs/howtos/regenerate-api-artifacts.md b/docs/howtos/regenerate-api-artifacts.md new file mode 100644 index 0000000..74da086 --- /dev/null +++ b/docs/howtos/regenerate-api-artifacts.md @@ -0,0 +1,89 @@ +# Regenerate API Artifacts + +Status: draft + +## Goal + +Refresh the generated files after changing routes, protocol models, the CLI, +docs, or dependencies, and confirm the changes are the ones you expect. + +## Starting Point + +You changed FastAPI routes, protocol models, CLI parsers, documentation inputs, +or the dependency set, and one or more committed artifacts is now stale. + +## Which artifact to regenerate + +| You changed… | Regenerate | Output | +| --- | --- | --- | +| Routes, request/response models | OpenAPI | `spec/openapi.yaml` | +| A CLI parser (flags/subcommands) | CLI reference pages | `docs/references/*-cli.md` | +| `README.md` or docs used in the digest | `llms.txt` | `llms.txt` | +| Dependencies | Third-party notices | `THIRD_PARTY_NOTICES.md` | + +## Steps + +### 1. Regenerate the OpenAPI contract + +`spec/openapi.yaml` is generated from the FastAPI app, not hand-edited: + +```bash +uv run python scripts/generate_openapi.py +``` + +It writes `spec/openapi.yaml` by default (pass `--output spec/openapi.json` for +JSON). + +### 2. Regenerate the CLI reference pages + +The four CLI references are derived from each command's argparse parser: + +```bash +uv run python scripts/build_cli_docs.py +``` + +This rewrites `docs/references/{client,admin,server,daemon}-cli.md`. + +### 3. Rebuild `llms.txt` + +```bash +uv run python scripts/build_llms_txt.py +``` + +### 4. Rebuild third-party license notices + +```bash +uv run python scripts/build_third_party_licenses.py +``` + +Writes `THIRD_PARTY_NOTICES.md`. + +### 5. Validate + +Run the contract tests, which include the OpenAPI drift check: + +```bash +uv run pytest -m contract +``` + +`tests/contract/test_openapi_drift.py` fails if the committed `spec/openapi.yaml` +still differs from the app — regenerate (step 1) until it passes. See +[Run the Test Suite](run-tests.md). + +### 6. Review before committing + +Generated files can carry incidental churn. Review `git diff` and confirm the +changes match what you intended before committing. + +## See also + +- [Protocol Specification](../references/protocol-specification.md) — how the + generated OpenAPI relates to the normative spec. +- [Run the Test Suite](run-tests.md) + +## Source Material + +- `scripts/generate_openapi.py` +- `scripts/build_cli_docs.py` +- `scripts/build_llms_txt.py` +- `scripts/build_third_party_licenses.py` diff --git a/docs/howtos/run-daemon.md b/docs/howtos/run-daemon.md new file mode 100644 index 0000000..c7a2aac --- /dev/null +++ b/docs/howtos/run-daemon.md @@ -0,0 +1,63 @@ +# Run the MAIL Daemon + +Status: draft + +## Goal + +Start `mail-daemon` so pending messages are delivered from the server into +recipients' inboxes. + +## Starting Point + +A MAIL server is running and you have daemon credentials — a `daemon:` address +and its password, created by `backend-init` (see +[Initialize the Memory Backend](initialize-memory-backend.md)). Delivery is the +daemon's job, not the server's; see [Delivery Model](../explanations/delivery-model.md). + +## Steps + +### 1. Set the daemon's environment variables + +The daemon authenticates as a daemon user-agent and requires all three: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_ADDRESS=daemon:dummy@localhost +MAIL_PASSWORD={daemon_password} +``` + +### 2. Start the daemon + +```bash +uv run mail-daemon +``` + +On startup it health-checks the server, logs in to obtain a token, then begins +polling for messages to deliver (roughly every 30 seconds). + +### 3. Adjust log levels (optional) + +Console and file log levels are set independently (`debug`, `info`, `warning`, +`error`, `critical`; both default to `info`): + +```bash +uv run mail-daemon --log-level-console debug --log-level-file info +``` + +### 4. Confirm delivery + +Send a message (see [Send a Message with the CLI](send-message-cli.md)), then +open the recipient's inbox. The message moves from the server's delivery buffer +into the recipient's inbox within one poll cycle, and the delivered message +records `Delivered By: daemon:…`. + +## See also + +- [Delivery Model](../explanations/delivery-model.md) — why a daemon delivers. +- [Run the MAIL Server](run-server.md) +- [Configuration](../references/configuration.md) + +## Source Material + +- `src/mail/daemon/src/mail_daemon/cli.py` +- `src/mail/daemon/src/mail_daemon/maild/api.py` diff --git a/docs/howtos/run-server.md b/docs/howtos/run-server.md new file mode 100644 index 0000000..54d960a --- /dev/null +++ b/docs/howtos/run-server.md @@ -0,0 +1,88 @@ +# Run the MAIL Server + +Status: draft + +## Goal + +Start `mail-server` with the host, port, backend, and checkpoint behavior you +want. + +## Starting Point + +Workspace dependencies are installed (`uv sync`) and you have initialized a +backend — see [Initialize the Memory Backend](initialize-memory-backend.md). + +## Steps + +### 1. Set the required environment variables + +The server reads these at startup and refuses to boot if any is missing. Copy +[`src/mail/server/.env.example`](../../src/mail/server/.env.example) as a +starting point; full details are in [Configuration](../references/configuration.md). + +```bash +MAIL_HOST=localhost +MAIL_JWT_SECRET_KEY=$(openssl rand -hex 32) +MAIL_JWT_ALGORITHM=HS256 +MAIL_JWT_EXPIRE_MINUTES=30 +MAIL_REFRESH_TOKEN_EXPIRE_DAYS=30 +``` + +### 2. Start the server + +```bash +uv run mail-server +``` + +With no flags the server uses the memory backend and listens on +`http://127.0.0.1:8865`. + +### 3. Override host and port + +```bash +uv run mail-server --host 0.0.0.0 --port 9000 +``` + +### 4. Choose a backend + +The default is `memory`; use `sqlite` for a durable, transactional store: + +```bash +uv run mail-server --backend sqlite --sqlite-path ./mail.db +``` + +See [Storage Backends](../references/storage-backends.md) for the trade-offs and +for `--database-url`. (Note: the memory backend always reads/writes the `default` +deployment; use SQLite for other deployment names.) + +### 5. Tune or disable memory checkpointing + +The memory backend checkpoints to disk every `--memory-save-interval` seconds +(default 60). Set `0` to disable periodic checkpoints (a final save still runs on +shutdown): + +```bash +uv run mail-server --backend memory --memory-save-interval 10 +``` + +### 6. Verify it is up + +```bash +curl -s http://127.0.0.1:8865/health # -> {"status":"ok"} +# or, with the client configured: +MAIL_SERVER=http://127.0.0.1:8865 uv run mail ping +``` + +`GET /` reports the protocol name, version, and uptime. + +## See also + +- [Configuration](../references/configuration.md) — every server flag and env var. +- [Run the MAIL Daemon](run-daemon.md) — needed for messages to actually deliver. +- [Storage Backends](../references/storage-backends.md) + +## Source Material + +- `src/mail/server/src/mail_server/cli.py` +- `src/mail/server/src/mail_server/server.py` +- `src/mail/server/.env.example` diff --git a/docs/howtos/run-tests.md b/docs/howtos/run-tests.md new file mode 100644 index 0000000..5ca1c3a --- /dev/null +++ b/docs/howtos/run-tests.md @@ -0,0 +1,76 @@ +# Run the Test Suite + +Status: draft + +## Goal + +Run the active MAIL v2 tests, focus on a subset, measure coverage, and run the +archived v1 tests when needed. + +## Starting Point + +Workspace dependencies are installed (`uv sync`). + +## Steps + +### 1. Run the active suite + +```bash +uv run pytest +``` + +This runs everything under `tests/` **except** the `e2e` group (excluded by +default in `pytest.ini`). + +### 2. Run a subset + +Tests are marked by category — `unit`, `integration`, `contract`, `e2e` — and +also live in matching directories. Select by marker or by path: + +```bash +uv run pytest -m unit # pure-logic tests +uv run pytest -m contract # spec/OpenAPI conformance +uv run pytest tests/integration # by directory +uv run pytest -m e2e # full-system subprocess tests (opt-in) +``` + +Integration and contract tests run against **both** the memory and SQLite +backends via a parametrized fixture, so a green run exercises both stores. + +### 3. Measure coverage + +Coverage is scoped to the v2 packages (configured in `pyproject.toml`): + +```bash +uv run pytest --cov +``` + +### 4. Run the archived v1 tests (only when needed) + +Legacy tests are not part of the default run and need the `legacy` extra: + +```bash +uv run --extra legacy pytest src/mail/legacy/tests +``` + +See [MAIL v1 Legacy Runtime](../explanations/mail-v1-legacy.md). + +## Interpreting results + +- **OpenAPI drift** (`tests/contract/test_openapi_drift.py`) fails when the + committed `spec/openapi.yaml` no longer matches the app — regenerate it (see + [Regenerate API Artifacts](regenerate-api-artifacts.md)). +- **Contract tests** (`tests/contract/`) enforce SPEC.md rules on addresses, + messages, and delivery; a failure means the implementation diverged from the + spec. See [Protocol Specification](../references/protocol-specification.md). +- **Expected `xfail`s** in `tests/integration/test_stubs.py` mark operations that + are `NotImplementedError` on the memory backend (message deletion, trash clear, + webhook patch, remote delivery) — these are implemented on SQLite. An `xpass` + there means one was implemented and the marker should be removed. + +## Source Material + +- `pytest.ini` +- `tests/` +- `src/mail/legacy/tests/` +- `pyproject.toml` (`[tool.coverage]`) diff --git a/docs/howtos/send-message-cli.md b/docs/howtos/send-message-cli.md new file mode 100644 index 0000000..82becf1 --- /dev/null +++ b/docs/howtos/send-message-cli.md @@ -0,0 +1,85 @@ +# Send a Message with the CLI + +Status: draft + +## Goal + +Compose a draft and send it to one or more recipients with the `mail` CLI. + +## Starting Point + +You have a valid `MAIL_TOKEN` for a user-agent allowed to send (see +[Authenticate a User-Agent](authenticate-user-agent.md)), and a daemon is running +so the message can be delivered (see [Run the MAIL Daemon](run-daemon.md)). +Sending is a two-step draft-then-send flow — see +[Delivery Model](../explanations/delivery-model.md). + +## Steps + +### 1. Compose a draft + +A draft holds only a subject and body; recipients come later. Provide the body +inline or read it from a file with `-F`/`--body-file`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={ua_jwt} +uv run mail compose "Status update" "The batch job finished cleanly." +``` + +The console prints the new draft, including its **draft ID** (a UUID). You can +attach `--tags` (slug strings) that carry onto the sent message. + +### 2. Capture the draft ID + +Note the `draft_id` from the output; you can also list drafts with +`uv run mail drafts` or open one with `uv run mail drafts-open {draft_id}`. + +### 3. Send the draft to one or more recipients + +Recipients are supplied at send time. Pass the draft ID followed by one or more +addresses: + +```bash +uv run mail send {draft_id} supervisor@default@localhost user:alice@localhost +``` + +The console prints the assembled message with a **message ID** distinct from the +draft ID. + +### 4. Inspect the outbox + +```bash +uv run mail outbox # list your sent messages +uv run mail outbox-open {message_id} # open one +``` + +A `null` delivery time means *sent, awaiting delivery*; once the daemon delivers, +the outbox entry records the delivering daemon. + +### 5. Inspect the recipient inbox + +With the recipient's token (and a running daemon), confirm arrival: + +```bash +MAIL_TOKEN={recipient_jwt} uv run mail inbox +MAIL_TOKEN={recipient_jwt} uv run mail open {message_id} +``` + +### 6. Handle validation failures + +Malformed input is rejected before the message is created or delivered: a bad +subject/body fails at compose time, and a malformed recipient address fails at +send time with a `422` whose `detail` explains the problem. Recipient addresses +must be valid MAIL addresses — see [Addressing Model](../explanations/addressing-model.md). + +## See also + +- [Authenticate a User-Agent](authenticate-user-agent.md) +- [Delivery Model](../explanations/delivery-model.md) +- [Client CLI](../references/client-cli.md) — full command reference. + +## Source Material + +- `src/mail/client/src/mail_client/commands/compose.py` +- `src/mail/client/src/mail_client/commands/send.py` diff --git a/docs/references/README.md b/docs/references/README.md new file mode 100644 index 0000000..15d96f2 --- /dev/null +++ b/docs/references/README.md @@ -0,0 +1,29 @@ +# Reference + +Reference pages describe MAIL machinery: commands, endpoints, models, +configuration, repository layout, and implementation-defined behavior. +Reference material should be terse, structured consistently, and tied to source +files or generated contracts. + +## Planned Reference Pages + +| Page | Describes | Source of truth | +| --- | --- | --- | +| [Repository Layout](repository-layout.md) | Active workspace, specs, tests, scripts, and legacy code. | `README.md`, `pyproject.toml` | +| [Configuration](configuration.md) | Environment variables and runtime settings. | CLI modules, `.env.example` | +| [Protocol Specification](protocol-specification.md) | Normative MAIL protocol documents. | `spec/SPEC.md`, `spec/openapi.yaml` | +| [HTTP API](http-api.md) | REST endpoints, auth, request and response bodies. | `spec/openapi.yaml`, FastAPI routers | +| [Client CLI](client-cli.md) | `mail` commands and options. | `src/mail/client/src/mail_client/cli.py` | +| [Admin CLI](admin-cli.md) | Administrator commands and options. | `src/mail/client/src/mail_client/admin_panel.py` | +| [Server CLI](server-cli.md) | `mail-server` options. | `src/mail/server/src/mail_server/cli.py` | +| [Daemon CLI](daemon-cli.md) | `mail-daemon` options and env vars. | `src/mail/daemon/src/mail_daemon/cli.py` | +| [Data Models](data-models.md) | Pydantic models used by protocol and network contracts. | `src/mail/protocol/src/mail_protocol/` | +| [Storage Backends](storage-backends.md) | Backend interfaces and memory backend behavior. | `src/mail/server/src/mail_server/backends/` | + +## Reference Checklist + +- Match structure to code structure where practical. +- Prefer tables for options, fields, and endpoint summaries. +- Include examples only to clarify syntax. +- Link to tutorials and how-tos instead of becoming step-by-step guidance. +- Update reference pages in the same change as command, API, or model changes. diff --git a/docs/references/admin-cli.md b/docs/references/admin-cli.md new file mode 100644 index 0000000..3ee070d --- /dev/null +++ b/docs/references/admin-cli.md @@ -0,0 +1,235 @@ +# Admin CLI + +Status: generated + +> **Generated file — do not edit by hand.** Regenerate with `uv run python scripts/build_cli_docs.py` after changing the CLI. See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md). + +A Python CLI client admin panel for the Multi-Agent Interface Layer (MAIL) + +Invoke as `mail-admin` (or `uv run mail-admin` from a workspace checkout). Source: `mail_client/admin_panel.py`. + +## Global options + +- `--license` — show license information and exit +- `-o`, `--output` `{text,json}` — the output style for this CLI command (default: text) + +## Commands + +### `ping` (aliases: `p`) + +ping a MAIL server + +### `login` (aliases: `l`) + +log into a MAIL server + +### `whoami` (aliases: `me`, `id`) + +get authenticated user-agent info from a MAIL server + +### `agent-list` (aliases: `al`) + +get a list of agents on the MAIL server + +### `agent-get` (aliases: `ag`) + +get a specific agent by local address on the MAIL server + +**Arguments:** + +- `local_address` — the local address of the agent to get (agent@swarm) + +### `agent-post` (aliases: `ap`) + +create a new agent on the MAIL server with the specified credentials + +**Arguments:** + +- `local_address` — the local address of the agent to create (agent@swarm) + +### `agent-delete` (aliases: `ad`) + +delete an existing agent by local address on the MAIL server + +**Arguments:** + +- `local_address` — the local address of the agent to delete (agent@swarm) + +### `daemon-list` (aliases: `dl`) + +get a list of daemons on the MAIL server + +### `daemon-get` (aliases: `dg`) + +get a specific daemon by worker name on the MAIL server + +**Arguments:** + +- `worker_name` — the worker name of the daemon to get + +### `daemon-post` (aliases: `dp`) + +create a new daemon on the MAIL server with the specified credentials + +**Arguments:** + +- `worker_name` — the name to use for the new daemon + +### `daemon-delete` (aliases: `dd`) + +delete an existing daemon by worker name on the MAIL server + +**Arguments:** + +- `worker_name` — the name of the daemon to delete + +### `user-list` (aliases: `ul`) + +get a list of users on the MAIL server + +### `user-get` (aliases: `ug`) + +get a specific user by user ID on the MAIL server + +**Arguments:** + +- `user_id` — the ID of the user to get + +### `user-post` (aliases: `up`) + +create a new user on the MAIL server with the specified credentials + +**Arguments:** + +- `user_id` — the ID to use for the new user + +### `user-delete` (aliases: `ud`) + +delete an existing user by user ID on the MAIL server + +**Arguments:** + +- `user_id` — the name of the user to delete + +### `swarm-post` (aliases: `sp`) + +create a new swarm on the MAIL server with the specified info + +**Arguments:** + +- `name` — the name of the swarm to create +- `description` — the description to use for the new swarm + +**Options:** + +- `-k`, `--keywords` `KEYWORDS` — the keywords to use for this swarm (default: []) + +### `swarm-delete` (aliases: `sd`) + +delete an existing swarm by name from the MAIL server + +**Arguments:** + +- `swarm_name` — the name of the swarm on the server to delete + +### `webhook-list` (aliases: `wl`) + +list all webhooks on the MAIL server + +### `webhook-get` (aliases: `wg`) + +get an existing webhook by ID on the MAIL server + +**Arguments:** + +- `webhook_id` — the ID of the webhook to get + +### `webhook-post` (aliases: `wp`) + +create a new webhook on the MAIL server + +**Arguments:** + +- `url` — the URL to hit for this webhook +- `secret` — the secret to use for this webhook + +**Options:** + +- `-e`, `--events` `EVENTS` — the event(s) for this webhook + +### `webhook-patch` (aliases: `wP`) + +update an existing webhook on the MAIL server + +**Arguments:** + +- `webhook_id` — the ID of the webhook to update + +**Options:** + +- `-u`, `--url` `URL` — the new URL to use, if any +- `-s`, `--secret` `SECRET` — the new secret to use, if any + +### `webhook-delete` (aliases: `wd`) + +delete an existing webhook by ID on the MAIL server + +**Arguments:** + +- `webhook_id` — the ID of the webhook to delete + +### `list-list` (aliases: `ll`) + +get all mailing lists on the MAIL server + +### `list-get` (aliases: `lg`) + +get a specific mailing list on the MAIL server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to get (name@swarm) + +### `list-post` (aliases: `lp`) + +create a new mailing list on the MAIL server + +**Arguments:** + +- `name` — the name of the new mailing list +- `swarm_name` — the name of the swarm to use for this mailing list +- `owner` — the MAIL address of the mailing list owner + +**Options:** + +- `-m`, `--members` `MEMBERS` — the MAIL addresses of members to add to this mailing list (default: []) + +### `list-patch` (aliases: `lP`) + +update an existing mailing list on the MAIL server + +### `list-delete` (aliases: `ld`) + +delete an existing mailing list on the MAIL server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to delete (name@swarm) + +### `list-member-post` (aliases: `lmp`) + +add a new member to an existing mailing list on the MAIL server + +**Arguments:** + +- `list_address` — the local address of the mailing list to add a member to (name@swarm) +- `member_address` — the full MAIL address of the member to add to this mailing list + +### `list-member-delete` (aliases: `lmd`) + +delete a member from an existing mailing list on the MAIL server + +**Arguments:** + +- `list_address` — the local address of the mailing list to remove a member from (name@swarm) +- `member_address` — the full MAIL address of the member to remove from this mailing list diff --git a/docs/references/client-cli.md b/docs/references/client-cli.md new file mode 100644 index 0000000..23aaf81 --- /dev/null +++ b/docs/references/client-cli.md @@ -0,0 +1,220 @@ +# Client CLI + +Status: generated + +> **Generated file — do not edit by hand.** Regenerate with `uv run python scripts/build_cli_docs.py` after changing the CLI. See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md). + +The Python CLI client for the Multi-Agent Interface Layer (MAIL) + +Invoke as `mail` (or `uv run mail` from a workspace checkout). Source: `mail_client/cli.py`. + +## Global options + +- `--license` — show license information and exit +- `-o`, `--output` `{text,json,markdown}` — the output style for this CLI command (default: text) + +## Commands + +### `ping` (aliases: `p`) + +ping a MAIL server + +### `login` (aliases: `l`) + +log into a MAIL server + +### `refresh` (aliases: `rt`) + +renew your access token using a refresh token + +### `whoami` (aliases: `me`, `id`) + +get authenticated user-agent info from a MAIL server + +### `compose` (aliases: `c`) + +draft a new MAIL message prior to sending + +**Arguments:** + +- `subject` — the subject line of the message to draft +- `body` — the body of the message to draft (omit when using --body-file) + +**Options:** + +- `-F`, `--body-file` `PATH` — read the message body from the file at this path +- `--tags` `TAG` — slug string tag(s) to attach to the message + +### `send` (aliases: `s`) + +send a drafted MAIL message to the specified address(es) + +**Arguments:** + +- `draft_id` — the ID of the existing draft to send +- `to` — the address(es) to deliver this message to + +**Options:** + +- `--tags` `TAG` — slug string tag(s) to attach to the message + +### `reply` (aliases: `r`) + +reply to an existing inbox message + +**Arguments:** + +- `message_id` — the ID of the inbox message to reply to +- `body` — the body of the reply + +**Options:** + +- `--subject` `SUBJECT` — the subject of the reply (default: 'Re: ') +- `--tags` `TAG` — slug string tag(s) to attach to the message + +### `forward` (aliases: `f`) + +forward an existing inbox message to new recipient(s) + +**Arguments:** + +- `message_id` — the ID of the inbox message to forward +- `to` — the address(es) to forward this message to + +**Options:** + +- `--note` `NOTE` — an optional note to prepend above the forwarded message +- `--subject` `SUBJECT` — the subject of the forward (default: 'Fwd: ') +- `--tags` `TAG` — slug string tag(s) to attach to the message + +### `inbox` (aliases: `i`) + +open your MAIL inbox + +**Options:** + +- `--limit` `LIMIT` — max number of entries to return (1-100) +- `--offset` `OFFSET` — number of entries to skip +- `--sort-by` `{sent_at,entered_at}` — timestamp field to sort by +- `--order` `{asc,desc}` — sort direction + +### `inbox-open` (aliases: `open`, `o`) + +open a specific message by ID in your MAIL inbox + +**Arguments:** + +- `message_id` — the ID of the message to open + +### `outbox` (aliases: `O`) + +open your MAIL outbox + +**Options:** + +- `--limit` `LIMIT` — max number of entries to return (1-100) +- `--offset` `OFFSET` — number of entries to skip +- `--sort-by` `{sent_at,entered_at}` — timestamp field to sort by +- `--order` `{asc,desc}` — sort direction + +### `outbox-open` (aliases: `Oopen`, `Oo`) + +open a specific message by ID in your MAIL outbox + +**Arguments:** + +- `message_id` — the ID of the message to open + +### `drafts` (aliases: `d`) + +list your existing message drafts + +**Options:** + +- `--limit` `LIMIT` — max number of entries to return (1-100) +- `--offset` `OFFSET` — number of entries to skip +- `--sort-by` `{sent_at,entered_at}` — timestamp field to sort by +- `--order` `{asc,desc}` — sort direction + +### `drafts-open` (aliases: `do`) + +open a specific existing draft by ID + +**Arguments:** + +- `draft_id` — the ID of the drafted message to open + +### `draft-edit` (aliases: `de`) + +edit fields on an existing message draft by ID + +**Arguments:** + +- `draft_id` — the ID of the draft to edit +- `body` — the new body of the draft (omit to leave it unchanged) + +**Options:** + +- `--subject` `SUBJECT` — the new subject of the draft (omit to leave it unchanged) +- `-F`, `--body-file` `PATH` — read the message body from the file at this path +- `--reply-to` `REPLY_TO` — the message ID this draft replies to (omit to leave it unchanged) +- `--tags` `TAG` — replace the draft's tags (pass with no values to clear all tags) + +### `trash` (aliases: `t`) + +list your existing trashed messages + +**Options:** + +- `--limit` `LIMIT` — max number of entries to return (1-100) +- `--offset` `OFFSET` — number of entries to skip +- `--sort-by` `{sent_at,entered_at}` — timestamp field to sort by +- `--order` `{asc,desc}` — sort direction + +### `trash-open` (aliases: `to`) + +open a specific message in trash by ID + +**Arguments:** + +- `message_id` — the ID of the message in trash to open + +### `swarm-list` (aliases: `swarms`, `sl`) + +get the swarms on this MAIL server + +### `swarm-get` (aliases: `swarm`, `sg`) + +get a specific swarm by name on this MAIL server + +**Arguments:** + +- `swarm_name` — the name of the MAIL swarm to get + +### `lists` + +get mailing lists on this MAIL server + +### `list-get` (aliases: `list`, `lg`) + +get a specific list on this MAIL server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to get (name@swarm) + +### `list-subscribe` (aliases: `ls`) + +subscribe to a mailing list on this server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to subscribe to (name@swarm) + +### `list-unsubscribe` (aliases: `lu`) + +unsubscribe from a mailing list on this server by address + +**Arguments:** + +- `list_address` — the local address of the mailing list to unsubscribe from (name@swarm) diff --git a/docs/references/configuration.md b/docs/references/configuration.md new file mode 100644 index 0000000..3c9df0d --- /dev/null +++ b/docs/references/configuration.md @@ -0,0 +1,108 @@ +# Configuration + +Status: draft + +This page lists every environment variable and CLI flag across the four MAIL v2 +packages, with defaults and whether each is required. Default host/port come from +[`mail_protocol.constants`](../../src/mail/protocol/src/mail_protocol/constants.py): +`MAIL_DEFAULT_HOST = 127.0.0.1`, `MAIL_DEFAULT_PORT = 8865`. + +## Server (`mail-server`) + +### Required environment variables + +These are read at import/startup — the server process fails to boot (raising +`RuntimeError`) if any is unset. + +| Variable | Effect | +| --- | --- | +| `MAIL_HOST` | Canonical host identity for the deployment/swarm. | +| `MAIL_JWT_SECRET_KEY` | HMAC secret for signing/verifying access-token JWTs. | +| `MAIL_JWT_ALGORITHM` | JWT signing algorithm (e.g. `HS256`). | +| `MAIL_JWT_EXPIRE_MINUTES` | Access-token lifetime, in minutes. | +| `MAIL_REFRESH_TOKEN_EXPIRE_DAYS` | Absolute lifetime of a refresh-token family, in days. | + +### Optional environment variables + +| Variable | Default | Effect | +| --- | --- | --- | +| `MAIL_COOKIE_SECURE` | `"true"` | Refresh-cookie `Secure` flag; only the literal `false` disables it. | +| `MAIL_COOKIE_DOMAIN` | unset (host-only cookie) | Cookie `Domain` for cross-subdomain deployments. | +| `MAIL_MEMORY_SAVE_INTERVAL_SECONDS` | `60.0` | Default for `--memory-save-interval`; `0` disables periodic checkpoints. | +| `MAIL_SQLITE_PATH` | unset → default DB path | Default for `--sqlite-path`. | +| `MAIL_DATABASE_URL` | unset | Default for `--database-url`; takes precedence over the sqlite path. | + +### CLI flags + +| Flag | Default | Effect | +| --- | --- | --- | +| `-H`, `--host` | `127.0.0.1` | Bind address. | +| `-p`, `--port` | `8865` | Listen port. | +| `-b`, `--backend` | `memory` | `memory` or `sqlite` (see [Storage Backends](storage-backends.md)). | +| `--memory-save-interval` | `$MAIL_MEMORY_SAVE_INTERVAL_SECONDS` or `60.0` | Seconds between memory checkpoints; `0` disables. | +| `--sqlite-path` | `$MAIL_SQLITE_PATH` or default DB path | SQLite database file. | +| `--database-url` | `$MAIL_DATABASE_URL` | Full database URL. | +| `--license` | — | Print license and exit. | + +**Database URL resolution** (SQLite backend): `--database-url` > `--sqlite-path` +> default `~/.mail-swarms/deployments/default/mail.db`. + +**Logging** is not configurable on the server: level is fixed at `INFO` and logs +are written to `~/.mail-swarms/server_logs/.log`. + +## Client (`mail` and `mail-admin`) + +Client commands read these per-invocation and raise `ValueError` if a required +one is missing. Tokens are never written by the client — `login` and `refresh` +print them for you to export. + +| Variable | Required for | Effect | +| --- | --- | --- | +| `MAIL_SERVER` | all commands | Base URL of the target server. | +| `MAIL_TOKEN` | all authenticated commands | Bearer access token. Not used by `ping` or `login`. | +| `MAIL_ADDRESS` | `login` | Address for the password grant. | +| `MAIL_PASSWORD` | `login` | Password for the password grant. | +| `MAIL_REFRESH_TOKEN` | `refresh` | Refresh token sent to `POST /auth/refresh` (rotated server-side). | + +CLI flag: `-o`/`--output` selects output format — `text` (default), `json` +(`mail` also supports `markdown`). Both accept `--license`. See +[Authenticate a User-Agent](../howtos/authenticate-user-agent.md). + +## Daemon (`mail-daemon`) + +Required environment variables (raise `ValueError` at startup if unset): + +| Variable | Effect | +| --- | --- | +| `MAIL_SERVER` | Target server URL (also health-checked at startup). | +| `MAIL_ADDRESS` | Daemon login address. | +| `MAIL_PASSWORD` | Daemon login password. | + +CLI flags: `-llf`/`--log-level-file` and `-llc`/`--log-level-console` (both +default `info`; choices `debug|info|warning|error|critical`), plus `--license`. +The 30-second delivery poll interval is a hard-coded default with no flag or env +var. See [Run the MAIL Daemon](../howtos/run-daemon.md). + +## `backend-init` + +`backend-init` takes no environment variables; all configuration is via flags +(`--type`, `--deployment`, `--swarm`, `--swarm-description`, `--swarm-keywords`, +`--agents`, `--daemons`, `--users`, `--admins`, `--host`, `--import-fs`). Defaults +and usage are in +[Initialize the Memory Backend](../howtos/initialize-memory-backend.md). + +## `.env.example` + +[`src/mail/server/.env.example`](../../src/mail/server/.env.example) is a +server-side template containing `MAIL_HOST`, `MAIL_JWT_SECRET_KEY` (fake value), +`MAIL_JWT_ALGORITHM`, `MAIL_JWT_EXPIRE_MINUTES`, `MAIL_REFRESH_TOKEN_EXPIRE_DAYS`, +`MAIL_COOKIE_SECURE`, and a commented-out `MAIL_COOKIE_DOMAIN`. The optional +backend knobs (`MAIL_MEMORY_SAVE_INTERVAL_SECONDS`, `MAIL_SQLITE_PATH`, +`MAIL_DATABASE_URL`) and client/daemon variables are not in it. + +## Maintenance notes + +Keep secrets out of examples — use clearly fake values, and link to +[Security Model](../explanations/security-model.md) for production guidance +(TLS, reverse proxy, secret handling). Update this page when a variable or flag is +added, renamed, or changes its default or required status. diff --git a/docs/references/daemon-cli.md b/docs/references/daemon-cli.md new file mode 100644 index 0000000..babed56 --- /dev/null +++ b/docs/references/daemon-cli.md @@ -0,0 +1,15 @@ +# Daemon CLI + +Status: generated + +> **Generated file — do not edit by hand.** Regenerate with `uv run python scripts/build_cli_docs.py` after changing the CLI. See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md). + +Multi-Agent Interface Layer (MAIL) daemon implementation in Python + +Invoke as `mail-daemon` (or `uv run mail-daemon` from a workspace checkout). Source: `mail_daemon/cli.py`. + +## Options + +- `--license` — show license information and exit +- `-llf`, `--log-level-file` `LEVEL` — file log level (default: info) +- `-llc`, `--log-level-console` `LEVEL` — console log level (default: info) diff --git a/docs/references/data-models.md b/docs/references/data-models.md new file mode 100644 index 0000000..2dbb518 --- /dev/null +++ b/docs/references/data-models.md @@ -0,0 +1,258 @@ +# Data Models + +Status: draft + +The MAIL protocol types are Pydantic models in the `mail-swarms-protocol` +package: domain models under +[`core/`](../../src/mail/protocol/src/mail_protocol/core) and wire +request/response models under +[`network/`](../../src/mail/protocol/src/mail_protocol/network). This page +documents the domain models field-by-field and summarizes the wire models; for +the exact HTTP request/response schemas see +[`spec/openapi.yaml`](../../spec/openapi.yaml) and [HTTP API](http-api.md). + +## Conventions + +- **Validators.** Field rules are `AfterValidator` functions from + [`core/validators.py`](../../src/mail/protocol/src/mail_protocol/core/validators.py); + the tables name the validator. See [Validators](#validators) for what each + enforces. +- **`metadata`.** Most models carry a `metadata: dict[str, Any]` for + implementer-defined data (SPEC §7.7). Put custom data there, not at the top + level. +- **`.summarize()`.** Full models have a `summarize()` returning a `*Summary` + variant (smaller, `body_size` instead of `body`) used in list responses. +- **`*InBackend`.** Storage-only variants add server-assigned fields (ids, + timestamps, password hashes) and never cross the wire as input. + +## User-agents and addresses + +Four concrete types discriminated on `ua_type`, each with a `get_address()`. +Defined in +[`core/user_agents.py`](../../src/mail/protocol/src/mail_protocol/core/user_agents.py). +See [Addressing Model](../explanations/addressing-model.md) for the address +grammar. + +| Model | `ua_type` | Address form | Identifying field (validator) | +| --- | --- | --- | --- | +| `MAILAgent` | `"agent"` | `name@swarm@host` | `name` (`validate_agent_name`), `swarm`, `host` | +| `MAILUser` | `"user"` | `user:user_id@host` | `user_id` (`validate_user_name`), `host` | +| `MAILAdmin` | `"admin"` | `admin:admin_id@host` | `admin_id` (`validate_user_name`), `host` | +| `MAILDaemon` | `"daemon"` | `daemon:worker_name@host` | `worker_name` (`validate_daemon_worker_name`), `host` | + +- **`MAILUserAgent`** — wrapper with `user_agent: Union[...]` as a + `Field(discriminator="ua_type")`. This is the shape returned by + `GET /auth/whoami` (double-nested: `user_agent.user_agent`). +- **`MAILUserAgentInBackend`** — adds `hashed_password: str`. + +## Messages + +[`core/messages.py`](../../src/mail/protocol/src/mail_protocol/core/messages.py). +The message contract is SPEC §7. + +### `MAILMessage` + +| Field | Type | Default | Validator | +| --- | --- | --- | --- | +| `mail_version` | `Literal["2.0"]` | required | — | +| `message_id` | `str` | required | `validate_uuid` | +| `reply_to` | `str \| None` | `None` | `validate_uuid` | +| `sender` | `str` | required | `validate_mail_address` | +| `recipients` | `list[str]` | required | `validate_message_recipients` (≥1) | +| `subject` | `str` | required | `validate_message_subject` | +| `body` | `str` | required | `validate_message_body` | +| `tags` | `list[str]` | required | `validate_message_tags` | +| `sent_at` | `datetime` | required | — | +| `metadata` | `dict[str, Any]` | required | — | + +**`MAILMessageSummary`** drops `mail_version`/`reply_to`/`body`/`tags`/`metadata` +and carries `body_size: int` instead of `body`. + +## Drafts + +[`core/drafts.py`](../../src/mail/protocol/src/mail_protocol/core/drafts.py). A +draft has no recipients — they are bound at send time. `reply_to`/`tags` carry +forward onto the sent message. + +### `MAILDraft` + +| Field | Type | Default | Validator | +| --- | --- | --- | --- | +| `draft_id` | `str` | required | `validate_uuid` | +| `subject` | `str` | required | `validate_message_subject` | +| `body` | `str` | required | `validate_message_body` | +| `created_at` | `datetime` | required | — | +| `updated_at` | `datetime \| None` | `None` | — | +| `reply_to` | `str \| None` | `None` | `validate_uuid` | +| `tags` | `list[str]` | `[]` | `validate_message_tags` | + +**`MAILDraftsEntry`** wraps a `MAILDraft` with `sent_at: datetime | None` and +`sent_by: str | None`. **`MAILDraftsEntrySummary`** mirrors it with `body_size`. + +## Box entries + +Each box wraps a `MAILMessage` with box-specific timestamps; each has a `*Summary` +for list views. Modules: +[inbox](../../src/mail/protocol/src/mail_protocol/core/inbox.py), +[outbox](../../src/mail/protocol/src/mail_protocol/core/outbox.py), +[trash](../../src/mail/protocol/src/mail_protocol/core/trash.py). + +| Entry | Wraps | Extra fields | +| --- | --- | --- | +| `MAILInboxEntry` | `MAILMessage` | `received_at`, `delivered_by` | +| `MAILOutboxEntry` | `MAILMessage` | `delivered_at: datetime \| None`, `delivered_by: str \| None` | +| `MAILTrashEntry` | `MAILMessage` | `trashed_at` | + +Summary specifics: + +- **`MAILInboxEntrySummary`** carries `is_read: bool = False`. Read state is + per-owner and supplied at list time (not stored on the shared entry); it flips + to `True` when the message is fetched via `GET /inbox/{message_id}`. +- **`MAILOutboxEntrySummary`** carries nullable `delivered_at` / `delivered_by` + — `null` means *sent, awaiting delivery* (see + [Delivery Model](../explanations/delivery-model.md)). + +## Swarms + +[`core/swarms.py`](../../src/mail/protocol/src/mail_protocol/core/swarms.py). + +### `MAILSwarm` + +| Field | Type | Validator | +| --- | --- | --- | +| `name` | `str` | `validate_swarm_name` | +| `description` | `str` | `validate_swarm_description` | +| `keywords` | `list[str]` | `validate_swarm_keywords` | +| `agents` | `list[str]` | `validate_agent_names` | +| `metadata` | `dict[str, Any]` | — | + +**`MAILSwarmSummary`** replaces `agents`/`metadata` with `num_agents: int`. + +## Mailing lists + +[`core/lists.py`](../../src/mail/protocol/src/mail_protocol/core/lists.py). See +[Mailing Lists](../explanations/mailing-lists.md) for the model in prose. + +### `MAILListPolicy` + +| Field | Type | Default | +| --- | --- | --- | +| `visibility` | `Literal["public", "private"]` | `"public"` | +| `join_policy` | `Literal["open", "approval", "admin-only"]` | `"open"` | +| `send_policy` | `Literal["open", "members-only", "admin-only"]` | `"open"` | + +The non-default enum variants are reserved: they validate at the protocol layer +but the v1 server rejects them with `501`. + +### `MAILList` + +| Field | Type | Default | Validator | +| --- | --- | --- | --- | +| `list_type` | `Literal["list"]` | `"list"` | — | +| `name` | `str` | required | `validate_list_name` | +| `swarm` | `str` | required | `validate_swarm_name` | +| `host` | `str` | required | `validate_host` | +| `owner` | `str` | required | `validate_mail_address` | +| `members` | `list[str]` | `[]` | `validate_mail_addresses` | +| `policy` | `MAILListPolicy` | `MAILListPolicy()` | — | +| `metadata` | `dict[str, Any]` | `{}` | — | + +**`MAILListInBackend`** adds `list_id` (`validate_uuid`), `created_at`, +`updated_at`. This is the payload shape in all list responses. Address form via +`get_address()`: `list:name@swarm@host`. + +## Webhooks + +[`core/webhooks.py`](../../src/mail/protocol/src/mail_protocol/core/webhooks.py), +[`network/webhooks.py`](../../src/mail/protocol/src/mail_protocol/network/webhooks.py). +The only event type is `mail.delivered`. See +[Webhook Delivery](../explanations/webhook-delivery.md). + +- **`MAILWebhook`** — `webhook_id` (`wh_`), `url` (`validate_url`), `events` + (`validate_webhook_event_types`), `secret`. +- **`MAILMessageInWebhook`** — the message as embedded in an outbound payload. It + **diverges from `MAILMessage`**: IDs are `msg_`-prefixed, there is a single + `recipient` (not `recipients`), it adds a `swarm` field, and it omits + `mail_version`. +- **`WebhookDeliveredPostRequest`** — the JSON body the server POSTs to receiver + URLs: `event`, `event_id`, `delivered_at`, `message: MAILMessageInWebhook`. + (This is outbound; it is not an endpoint the server exposes.) + +## Auth + +[`core/auth.py`](../../src/mail/protocol/src/mail_protocol/core/auth.py). + +**`RefreshTokenRecord`** (backend-internal; never on the wire) stores the SHA-256 +hash of a refresh token, its `family_id`, `owner_address`, `issued_at`, absolute +`expires_at`, a `revoked` flag, and `rotated_at`. A token is unusable once +`revoked` is true or `rotated_at` is set. See +[Security Model](../explanations/security-model.md). + +## Request and response models + +The wire envelopes live in +[`network/requests.py`](../../src/mail/protocol/src/mail_protocol/network/requests.py) +and +[`network/responses.py`](../../src/mail/protocol/src/mail_protocol/network/responses.py), +and each maps to an endpoint in [HTTP API](http-api.md) (most docstrings name the +`METHOD /path`). Rather than restate the generated schema, note the recurring +shapes: + +- **Responses** wrap a payload field (`entry`, `entries`, `message`, `swarm`, + `mail_list`, `agent`, …) plus `metadata`. A handful of pure-status responses + (`RootGetResponse`, `HealthGetResponse`, `SwarmHealthGetResponse`, + `AuthLogoutPostResponse`, `AuthPasswordResetResponse`) omit `metadata`. +- **Notable request bodies:** `DraftPostRequest` (`subject`, `body`, optional + `reply_to`, `tags`), `DraftPatchRequest` (all-optional partial update), + `DraftSendPostRequest` (`recipients`, optional `tags` merged with the draft's), + `AdminListPostRequest` / `AdminListPatchRequest` (policy-only patch), + `AdminWebhooksPostRequest` / `AdminWebhooksPatchRequest`, and the admin + create-agent/daemon/user/swarm bodies. +- **`BoxFilterParams`** — the query params for box GET-collection endpoints: + `limit` (1–100, default 20), `offset` (default 0), `sort_by` + (`entered_at`|`sent_at`, default `entered_at`), `order` (`asc`|`desc`, default + `desc`). It forbids extra params. + +## Constants + +[`core/constants.py`](../../src/mail/protocol/src/mail_protocol/core/constants.py): + +| Constant | Min / Max | +| --- | --- | +| Message subject | 1 / 256 | +| Message body | 1 / 65535 | +| Message tag | 1 / 32 | +| Agent / user / admin / daemon-worker / swarm / swarm-keyword / list name | 1 / 32 | +| Swarm description | 0 / 255 | + +`LIST_ADDRESS_PREFIX = "list"`. The protocol version literal `"2.0"` is not a +constant — it is a `Literal` on `MAILMessage.mail_version` and +`RootGetResponse.protocol_version`. The identifier max of 32 matches SPEC §6. + +## Validators + +Key rules from `core/validators.py`: + +| Validator | Enforces | +| --- | --- | +| `validate_uuid` / `validate_uuids` | Value(s) parse as UUIDs. | +| `validate_mail_address` | Full address shape: 3-part `name@swarm@host` (agent) or `list:name@swarm@host`; 2-part `user:id@host` / `admin:id@host` / `daemon:worker@host`. | +| `validate_local_address` | 2-part `agent@swarm` (admin path params). | +| `validate_message_recipients` | ≥1 entry, each a valid address (SPEC §7.3). | +| `validate_message_subject` / `_body` | Length within the constant bounds. | +| `validate_message_tag(s)` | Each tag length 1–32 and a slug. | +| `validate_agent_name` / `validate_user_name` / `validate_swarm_name` / `validate_daemon_worker_name` / `validate_list_name` / `validate_swarm_keyword` | Length 1–31 and a slug. | +| `validate_host` | Valid hostname, IPv4, or IPv6. | +| `validate_url` | `http(s)://` URL (single-label hosts like `localhost` allowed). | +| `validate_webhook_id` / `validate_webhook_message_id` | `wh_` / `msg_` shapes. | +| `validate_webhook_event_type(s)` | Equals `mail.delivered`. | + +The slug rule (`string_is_slug`) is `^[a-z0-9]+(?:-[a-z0-9]+)*$`: lowercase +alphanumerics in hyphen-separated segments — no uppercase, underscores, or +leading/trailing/double hyphens. + +## Maintenance notes + +Use field tables with type, default/required status, and the validator name; link +to the source class rather than pasting generated OpenAPI schemas in full. Update +this page when protocol models gain or change fields. diff --git a/docs/references/http-api.md b/docs/references/http-api.md new file mode 100644 index 0000000..8c0551d --- /dev/null +++ b/docs/references/http-api.md @@ -0,0 +1,176 @@ +# HTTP API + +Status: draft + +This is a navigational reference to the MAIL server's HTTP surface. The +authoritative contract — request bodies, response schemas, parameters, and status +codes — is the generated [`spec/openapi.yaml`](../../spec/openapi.yaml), also +served interactively at `/docs` (Swagger UI) and `/openapi.json` on a running +server. This page lists every route, its authentication requirement, and its +response model so you can find the right endpoint; follow the OpenAPI schema for +exact field-level detail. Route handlers live in +[`src/mail/server/src/mail_server/routers/`](../../src/mail/server/src/mail_server/routers). + +## Authentication + +Every authenticated request carries a bearer access token in the +`Authorization: Bearer ` header. Obtain one from `POST /auth/token` (see +[Authenticate a User-Agent](../howtos/authenticate-user-agent.md)). Endpoints +enforce one of four access levels: + +| Level | Meaning | +| --- | --- | +| none | Unauthenticated. | +| user-agent | Any authenticated user-agent (agent, user, admin, daemon). | +| daemon | A daemon bearer token. | +| admin | An admin bearer token. | + +## Response envelope + +Most responses wrap their payload in a named field (`entry`, `entries`, +`message`, `swarm`, `mail_list`, …) alongside a `metadata` object; single-message +box reads nest the message under an `entry`. Field shapes are in +[Data Models](data-models.md). + +> **Memory-backend gaps.** A few endpoints are implemented only on the SQLite +> backend; on the memory backend they raise `NotImplementedError`: +> `DELETE /inbox/{message_id}`, `DELETE /drafts/{draft_id}`, +> `DELETE /trash/{message_id}`, `POST /trash/clear`, +> `PATCH /admin/webhooks/{webhook_id}`, and `POST /daemon/deliver/remote`. See +> [Storage Backends](storage-backends.md#current-limitations). + +## Root and health + +| Method | Path | Auth | Response model | +| --- | --- | --- | --- | +| GET | `/` | none | `RootGetResponse` (protocol name, version, uptime) | +| GET | `/health` | none | `HealthGetResponse` (`status: "ok"`) | + +## Authentication endpoints (`/auth`) + +| Method | Path | Auth | Notes | +| --- | --- | --- | --- | +| POST | `/auth/token` | none | OAuth2 password grant (form fields). Returns `access_token`, `expires_in`; `refresh_token` for interactive principals (users/admins). | +| POST | `/auth/refresh` | refresh token | Rotates the refresh token (cookie or request body). | +| POST | `/auth/logout` | refresh token | Idempotent; revokes the refresh family. | +| GET | `/auth/whoami` | user-agent | Returns the caller's `MAILUserAgent`. | +| POST | `/auth/password/reset` | user-agent | Revokes all refresh families on success. | + +See [Security Model](../explanations/security-model.md) for the refresh-token +design. + +## Swarms (`/swarms`) + +| Method | Path | Auth | Response model | +| --- | --- | --- | --- | +| GET | `/swarms` | user-agent | `SwarmsGetResponse` | +| GET | `/swarms/{swarm_name}` | user-agent | `SwarmGetResponse` | +| GET | `/swarms/{swarm_name}/health` | user-agent | `SwarmHealthGetResponse` | + +## Message boxes + +Box GET-collection endpoints accept the `BoxFilterParams` query params: `limit` +(1–100, default 20), `offset` (default 0), `sort_by` (`entered_at` default, or +`sent_at`), and `order` (`desc` default, or `asc`). `GET /drafts` rejects +`sort_by=sent_at` with `422` (a draft has no send time). + +### Inbox (`/inbox`) + +| Method | Path | Auth | Notes | +| --- | --- | --- | --- | +| GET | `/inbox` | user-agent | List summaries (per-owner `is_read`). | +| GET | `/inbox/{message_id}` | user-agent | Full message; marks it read. | +| DELETE | `/inbox/{message_id}` | user-agent | Moves the message to trash. | + +### Outbox (`/outbox`) + +| Method | Path | Auth | +| --- | --- | --- | +| GET | `/outbox` | user-agent | +| GET | `/outbox/{message_id}` | user-agent | + +### Drafts (`/drafts`) + +| Method | Path | Auth | Notes | +| --- | --- | --- | --- | +| GET | `/drafts` | user-agent | Rejects `sort_by=sent_at` (`422`). | +| POST | `/drafts` | user-agent | Create a draft (`subject`, `body`, optional `reply_to`, `tags`). | +| GET | `/drafts/{draft_id}` | user-agent | | +| PATCH | `/drafts/{draft_id}` | user-agent | Partial update; omitted fields unchanged. | +| DELETE | `/drafts/{draft_id}` | user-agent | | +| POST | `/drafts/{draft_id}/send` | user-agent | Bind `recipients` and send; returns the assembled `MAILMessage`. | + +### Trash (`/trash`) + +| Method | Path | Auth | +| --- | --- | --- | +| GET | `/trash` | user-agent | +| GET | `/trash/{message_id}` | user-agent | +| DELETE | `/trash/{message_id}` | user-agent | +| POST | `/trash/clear` | user-agent | + +## Daemon endpoints (`/daemon`) + +Used by delivery daemons; see [Delivery Model](../explanations/delivery-model.md). + +| Method | Path | Auth | Notes | +| --- | --- | --- | --- | +| POST | `/daemon/message-buffer/clear` | daemon | Drain the pending-delivery buffer. | +| POST | `/daemon/deliver/local` | daemon | Deliver messages between user-agents on this server. | +| POST | `/daemon/deliver/remote` | daemon | Inbound cross-server delivery; implemented on SQLite, raises `NotImplementedError` on the memory backend (see note above). | + +## Admin endpoints (`/admin`) + +All require admin. Address path params use the **local** form (`agent@swarm`, +`name@swarm`); user/daemon use their id / worker name. + +| Resource | Routes | +| --- | --- | +| Agents | `GET|POST /admin/agents`, `GET|DELETE /admin/agents/{local_address}` | +| Daemons | `GET|POST /admin/daemons`, `GET|DELETE /admin/daemons/{worker_name}` | +| Users | `GET|POST /admin/users`, `GET|DELETE /admin/users/{user_id}` | +| Swarms | `POST /admin/swarms`, `DELETE /admin/swarms/{swarm_name}` | +| Webhooks | `GET|POST /admin/webhooks`, `GET|PATCH|DELETE /admin/webhooks/{webhook_id}` | + +## Mailing list endpoints + +See [Mailing Lists](../explanations/mailing-lists.md) and +[Manage Mailing Lists](../howtos/manage-mailing-lists.md). List path params use +the **local** `name@swarm` form; the server reconstructs the canonical +`list:name@swarm@host`. + +### Admin lists (`/admin/lists`, admin) + +| Method | Path | +| --- | --- | +| GET | `/admin/lists` | +| POST | `/admin/lists` | +| GET / PATCH / DELETE | `/admin/lists/{local_address}` | +| POST | `/admin/lists/{local_address}/members` | +| DELETE | `/admin/lists/{local_address}/members/{member_address}` | + +### Public lists (`/lists`, user-agent) + +| Method | Path | Notes | +| --- | --- | --- | +| GET | `/lists` | Filtered to `visibility=public`. | +| GET | `/lists/{local_address}` | `404` if not public. | +| POST | `/lists/{local_address}/subscribe` | `501` unless `join_policy=open`. | +| POST | `/lists/{local_address}/unsubscribe` | | + +## Common status codes + +| Code | Meaning in MAIL | +| --- | --- | +| `401 Unauthorized` | Missing, malformed, or expired token. | +| `403 Forbidden` | Authenticated but wrong role (e.g. non-admin on `/admin`). | +| `404 Not Found` | Unknown resource (or a non-public list on `/lists`). | +| `422 Unprocessable Entity` | Request/query validation failed; `detail` explains what. | +| `501 Not Implemented` | Reserved-but-unsupported behavior (non-`open` list policies). | + +## Maintenance notes + +Prefer the generated OpenAPI details for schemas and parameters; keep this page +focused on navigation, auth levels, and implementation notes. When routes change, +regenerate `spec/openapi.yaml` ([Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md)) +and update the tables here. diff --git a/docs/references/protocol-specification.md b/docs/references/protocol-specification.md new file mode 100644 index 0000000..71ab8e8 --- /dev/null +++ b/docs/references/protocol-specification.md @@ -0,0 +1,80 @@ +# Protocol Specification + +Status: draft + +MAIL has two normative artifacts, both under [`spec/`](../../spec). This page +orients you to them and maps each part of the specification to the implementation +reference pages in this set. It does not restate the spec — read the source files +for the authoritative text. + +- **[`spec/SPEC.md`](../../spec/SPEC.md)** — the protocol prose. Versioned, + written in [RFC 2119][rfc-2119] requirements language (MUST / SHOULD / MAY). + It defines terminology, user-agent categories, address forms, the message + contract, and delivery responsibilities. +- **[`spec/openapi.yaml`](../../spec/openapi.yaml)** — the authoritative HTTP + wire contract. It is **generated** from the running FastAPI app, not hand-authored + (see [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md)), so it + always matches the server's declared routes and schemas. + +## Version and status + +| Field | Value | +| --- | --- | +| Version | `2.0` | +| Date | June 10, 2026 | +| Status | Open to feedback | + +Versions follow `{major}.{minor}` (SPEC §10). Message payloads carry the version +in `mail_version`, which MUST be `"2.0"` for this revision. + +## Section map + +| SPEC.md section | Topic | Where it lives in these docs | +| --- | --- | --- | +| §3 Motivation | Goals; what MAIL is *not* | [MAIL v2 Overview](../explanations/mail-v2-overview.md) | +| §4 Architecture | Clients, servers, swarms | [Architecture](../explanations/architecture.md) | +| §5 User-Agents | admin / agent / daemon / user | [Data Models](data-models.md) | +| §6 Addresses | Host- vs swarm-scoped forms | [Addressing Model](../explanations/addressing-model.md) | +| §7 Messages | Message fields, replies, tags | [Data Models](data-models.md) | +| §8 Delivery | Pre-send vs post-send errors | [Delivery Model](../explanations/delivery-model.md) | +| §9 Security | Trust boundaries per component | [Security Model](../explanations/security-model.md) | +| §10 Versioning | Protocol version rules | this page | + +## OpenAPI contract role + +The OpenAPI document is the source of truth for endpoints, parameters, request +bodies, response schemas, and status codes. Client and server implementers MUST +conform to it (SPEC §4.1, §4.2). Because it is generated from the app, the +[HTTP API](http-api.md) reference is navigational — it points into the generated +schema rather than duplicating it. + +## Contract test coverage + +Conformance is enforced by the suite in [`tests/contract/`](../../tests/contract): + +- `test_spec_addresses.py` — address-shape rules from SPEC §6. +- `test_spec_messages.py` — message field constraints from SPEC §7. +- `test_spec_delivery.py` — delivery semantics from SPEC §8. +- `test_openapi_drift.py` — the committed `spec/openapi.yaml` matches the app. +- `test_openapi_request_bodies.py` — body-bearing endpoints document their bodies. + +Run them via [Run the Test Suite](../howtos/run-tests.md). + +## Specification and implementation alignment + +The reference implementation tracks the spec closely. Identifier length is one +alignment point worth calling out: SPEC §6 recommends that agent / user / admin / +daemon-worker / swarm / list identifiers be at most 32 characters, and the +implementation enforces exactly that as a hard cap (`core/constants.py`: +`*_NAME_LEN_MAX = 32`), consistent with the message-tag cap +(`MESSAGE_TAG_LEN_MAX = 32`, SPEC §7.10). No known divergences remain; if one +arises, record it here so readers know to trust the code over the prose. + +## Maintenance notes + +Do not copy the specification text into this page; keep it as an index that links +to the exact normative files. If the implementation and `SPEC.md` diverge, record +it in the alignment section above and cross-link the affected reference pages such +as [Addressing Model](../explanations/addressing-model.md). + +[rfc-2119]: https://datatracker.ietf.org/doc/html/rfc2119 diff --git a/docs/references/repository-layout.md b/docs/references/repository-layout.md new file mode 100644 index 0000000..3a630b6 --- /dev/null +++ b/docs/references/repository-layout.md @@ -0,0 +1,136 @@ +# Repository Layout + +Status: draft + +This page maps the MAIL repository so you can find the code, specification, and +tests behind everything else in these docs. MAIL v2 is a [uv][uv] workspace: a +root meta-package plus four member packages under `src/mail/`, with the archived +v1 runtime kept alongside them. + +## Top level + +```text +mail/ +├── docs/ # this documentation set (tutorials/howtos/references/explanations) +├── spec/ # protocol source of truth: SPEC.md + openapi.yaml +├── src/mail/ # the workspace packages (see below) +├── tests/ # active v2 test suite (contract/e2e/integration/unit) +├── scripts/ # repository maintenance + artifact generation +├── pyproject.toml # uv workspace + root meta-package + shared tooling config +├── pytest.ini # test configuration +├── uv.lock # locked dependency graph for the whole workspace +├── llms.txt # generated LLM-oriented API digest +├── README.md # repository overview +├── SPEC-LICENSE, SPEC-PATENT-LICENSE, LICENSE, NOTICE, TRADEMARKS.md, DCO +└── THIRD_PARTY_NOTICES.md # generated third-party license aggregation +``` + +## Workspace packages + +Each package lives at `src/mail//` with its own `pyproject.toml`, +`README.md`, and a `src/mail_/` import root. All four are published to PyPI +in lockstep under the `mail-swarms-*` names. + +| Directory | Package name | Import root | Console scripts | +| --- | --- | --- | --- | +| `src/mail/protocol/` | `mail-swarms-protocol` | `mail_protocol` | `mail-protocol` | +| `src/mail/server/` | `mail-swarms-server` | `mail_server` | `mail-server`, `backend-init` | +| `src/mail/client/` | `mail-swarms-client` | `mail_client` | `mail`, `mail-admin` | +| `src/mail/daemon/` | `mail-swarms-daemon` | `mail_daemon` | `mail-daemon` | + +The root `pyproject.toml` also exposes `mail` and `mail-server` so a workspace +checkout can run them directly (`uv run mail …`, `uv run mail-server`). + +### `protocol` — shared types and constants + +```text +src/mail_protocol/ +├── core/ # Pydantic domain models: messages, drafts, inbox, outbox, +│ # trash, swarms, lists, webhooks, user_agents, auth +├── network/ # request/response/webhook wire models +├── constants.py # protocol version and shared limits +├── core/validators.py +└── cli.py, cli_help.py +``` + +The protocol package is the dependency root — server, client, and daemon all +import its models. See [Data Models](data-models.md). + +### `server` — FastAPI server + +```text +src/mail_server/ +├── server.py # app assembly + root/health endpoints + backend selection +├── routers/ # one router per area: auth, swarms, inbox, outbox, +│ # drafts, trash, daemon, admin, lists +├── backends/ # storage: base.py (contract), memory/, sqlite/ +├── backend_init.py # the `backend-init` entry point +├── auth.py # token issuance, refresh tokens, role checks +├── validators.py, utils.py, logging.py, cli.py +└── .env.example # sample server configuration +``` + +See [HTTP API](http-api.md), [Storage Backends](storage-backends.md), and +[Configuration](configuration.md). + +### `client` — CLI client + +```text +src/mail_client/ +├── cli.py # `mail` — user-agent CLI +├── admin_panel.py # `mail-admin` — admin CLI +└── commands/ # one module per subcommand +``` + +See [Client CLI](client-cli.md) and [Admin CLI](admin-cli.md). + +### `daemon` — delivery daemon + +```text +src/mail_daemon/ +├── cli.py # `mail-daemon` entry point +├── maild/ # delivery loop + server API client +└── logger.py +``` + +See [Daemon CLI](daemon-cli.md) and [Delivery Model](../explanations/delivery-model.md). + +## Specification + +```text +spec/ +├── SPEC.md # normative protocol prose (versioned, RFC-2119 language) +└── openapi.yaml # authoritative HTTP wire contract (generated from the app) +``` + +`openapi.yaml` is generated, not hand-edited — see +[Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md) and +[Protocol Specification](protocol-specification.md). + +## Tests + +```text +tests/ +├── contract/ # spec/openapi conformance (addresses, delivery, messages, drift) +├── e2e/ # end-to-end flows +├── integration/ # auth, authz, and cross-component behavior +└── unit/ # component-level tests +``` + +See [Run the Test Suite](../howtos/run-tests.md). + +## Package documentation + +Some packages carry their own `docs/` directory that predates this consolidated +set: `src/mail/server/docs/` and `src/mail/client/docs/`. These are being +migrated into the top-level `docs/` tree; where they overlap, this tree is +canonical. + +## Archived v1 runtime + +`src/mail/legacy/` holds the MAIL v1 reference runtime (`api.py`, `client.py`, +`core/`, `config/`, UI assets, and its own docs). It is retained for reference +only and is not part of the v2 workspace packages. See +[MAIL v1 Legacy Runtime](../explanations/mail-v1-legacy.md). + +[uv]: https://docs.astral.sh/uv/ diff --git a/docs/references/server-cli.md b/docs/references/server-cli.md new file mode 100644 index 0000000..3f5d209 --- /dev/null +++ b/docs/references/server-cli.md @@ -0,0 +1,19 @@ +# Server CLI + +Status: generated + +> **Generated file — do not edit by hand.** Regenerate with `uv run python scripts/build_cli_docs.py` after changing the CLI. See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md). + +The Python/FastAPI server for the Multi-Agent Interface Layer (MAIL) + +Invoke as `mail-server` (or `uv run mail-server` from a workspace checkout). Source: `mail_server/cli.py`. + +## Options + +- `--license` — show license information and exit +- `-H`, `--host` `HOST` — the IP address to bind to (default: 127.0.0.1) +- `-p`, `--port` `PORT` — the port for the server to listen on (default: 8865) +- `-b`, `--backend` `BACKEND` — the MAIL server backend to use (default: memory) +- `--memory-save-interval` `SECONDS` — seconds between memory backend filesystem checkpoints; set 0 to disable (default: 60.0) +- `--sqlite-path` `PATH` — sqlite backend database file (env: MAIL_SQLITE_PATH; default: ~/.mail-swarms/deployments/default/mail.db) +- `--database-url` `URL` — sqlite backend database URL; takes precedence over --sqlite-path (env: MAIL_DATABASE_URL) diff --git a/docs/references/storage-backends.md b/docs/references/storage-backends.md new file mode 100644 index 0000000..9176c67 --- /dev/null +++ b/docs/references/storage-backends.md @@ -0,0 +1,149 @@ +# Storage Backends + +Status: draft + +The MAIL server keeps all state behind a single backend interface. Two backends +ship today — an in-memory backend with filesystem checkpointing, and a +transactional SQLite backend. Both implement the same contract, so the choice is +about durability and operational shape, not features. Code is under +[`src/mail/server/src/mail_server/backends/`](../../src/mail/server/src/mail_server/backends). + +## The backend contract + +`MAILServerBackend` +([`backends/base.py`](../../src/mail/server/src/mail_server/backends/base.py)) is +a `typing.Protocol` of `@abstractmethod`s that both backends implement. It has one +concrete attribute, `host: str` (set at startup; routers use it to reconstruct +full addresses), and the abstract methods group into: + +| Area | Responsibility | +| --- | --- | +| Lifecycle | `on_server_startup`, `on_server_shutdown` | +| Auth / user-agents | fetch a user-agent, existence check, password reset | +| Refresh tokens | create, get, rotate, revoke-family, revoke-all, purge-expired | +| Swarms | list, get, health | +| Boxes | inbox / outbox / drafts / trash: list, get, delete (+ draft create/patch/send, trash clear) | +| Daemon delivery | clear message buffer, deliver local, deliver remote | +| Admin | CRUD for agents, daemons, users, swarms | +| Webhooks | CRUD, plus the shared outbound delivery logic | +| Lists | list/get (public + admin), create, patch, delete, add/remove member | + +The only **concrete** methods on the base are the webhook-delivery helpers +(`handle_webhook_delivered_for_url`, `_webhook_delivered_post`), so both backends +share identical `mail.delivered` HMAC signing and retry behavior — see +[Webhook Delivery](../explanations/webhook-delivery.md). + +## Selecting a backend + +`mail-server --backend {memory,sqlite}` (`-b`, default `memory`). There is no env +var for the backend choice itself; per-backend knobs are covered in +[Configuration](configuration.md). `backend-init --type {memory,sqlite}` +initializes on-disk state before the server starts — see +[Initialize the Memory Backend](../howtos/initialize-memory-backend.md). + +## Memory backend + +Files: `backends/memory/api.py` (state + logic), `fs.py` (load/save), `init.py` +(`backend-init` seeding). + +- **Model.** All state lives in Python dicts/lists in RAM. On startup every + collection is loaded from disk; on checkpoint and shutdown every collection is + written back. +- **On-disk layout** under `~/.mail-swarms/deployments/{deployment}/`: one + directory per collection with one JSON file per item + (`swarms/`, `user_agents/`, `messages/`, `inbox_entries/`, `outbox_entries/`, + `draft_entries/`, `trash_entries/`, `webhooks/`, `lists/`, `refresh_tokens/`), + newline-delimited membership files for each per-owner box + (`inboxes/`, `outboxes/`, `drafts/`, `trashes/`, `read_inbox/`), a + `message_buffer.lock` FIFO file, and the plaintext `.secrets/
` files + written at init. +- **Checkpointing.** A background loop persists every + `--memory-save-interval` seconds (default **60**, `0` disables the periodic + loop), plus a final persist on shutdown. Each file write is atomic (temp file + → `fsync` → `os.replace` → parent-dir `fsync`). +- **Durability caveats.** State between checkpoints is RAM-only: a hard kill (or + interval `0`) loses everything since the last checkpoint. Individual writes are + atomic, but a full checkpoint is not a single transaction across collections, so + a crash mid-persist can leave collections at slightly different versions. + +## SQLite backend + +Files: `backends/sqlite/` — `api.py`, `database.py`, `schema.py`, +`repositories.py`, `serializers.py`, `init.py`, `migrate.py`. + +- **Model.** Fully transactional and durable. Every mutation runs in a session + that commits on success / rolls back on error; multi-step operations (e.g. + `send_draft` = message + outbox entry + membership + buffer row) commit + atomically. There is no in-RAM master copy and no checkpoint loop — every write + hits the database. +- **Stack.** Async SQLAlchemy over `sqlite+aiosqlite`, with per-connection + `PRAGMA foreign_keys=ON`, `journal_mode=WAL`, and `busy_timeout=5000` (5s). +- **Schema (hybrid).** Entity rows carry typed/indexed columns only for + filtering/ordering, plus a `body` JSON column holding the full + `model.model_dump(mode="json")`; reads rehydrate from `body`. Tables: + `user_agents`, `swarms`, `messages`, `inbox_entries`, `outbox_entries`, + `draft_entries`, `trash_entries`, `mailbox_items` (unified per-owner box + membership + ordering, with `is_read` for the inbox), `message_buffer`, + `webhooks`, `refresh_tokens` (all-typed, no body), `lists`. +- **Location.** Default `~/.mail-swarms/deployments/{deployment}/mail.db`; + overridable via `--sqlite-path` / `MAIL_SQLITE_PATH` or a full + `--database-url` / `MAIL_DATABASE_URL` (precedence: database-url > sqlite-path > + default). +- **Migrations.** No migration framework. `create_schema()` runs + `create_all` plus an additive, idempotent `ALTER TABLE ... ADD COLUMN` guard for + new queryable columns (the current guard adds `mailbox_items.is_read`, + backfilling existing rows as unread). + +## Initializing state with `backend-init` + +- `--type memory` builds the full deployment directory tree, writes the swarm and + each user-agent (hashed password), touches empty box files, and writes plaintext + `.secrets/
`. +- `--type sqlite` creates `mail.db` + schema and seeds one swarm + the requested + principals; it is idempotent on re-run (existing swarm/user-agents skipped) and + creates box membership lazily on first delivery. Plaintext secrets are still + written. +- `--type sqlite --import-fs` imports an existing memory/filesystem deployment of + the same name into a fresh SQLite database (messages first, then box entries, + then membership in arrival order, then buffer/webhooks/lists — all in one + transaction). It refuses to run if the source is missing or the target DB + already holds rows. + +## Capability differences + +| Aspect | Memory | SQLite | +| --- | --- | --- | +| Durability | RAM-first; ≤1 checkpoint interval at risk on crash | Durable per write (WAL) | +| Transactions | Per-file atomic writes; not cross-collection | Full multi-step transactions | +| Concurrency | Single `asyncio.Lock` around persist | WAL readers + serialized writers, 5s busy timeout | +| Deployments | Runtime reads/writes the `default` deployment only (see limitations) | Arbitrary via `--sqlite-path` / `--database-url` | +| Init on re-run | Overwrites | Idempotent | +| Migration import | — | `--import-fs` | +| Delete/clear, webhook patch, remote deliver | Not implemented (raises `NotImplementedError`) | Implemented | + +Both implement the identical interface and share the webhook delivery logic, so +inbox `is_read`, refresh-token families, list membership, and webhook semantics +match across backends. + +## Current limitations + +- **Memory backend: unimplemented operations.** Several operations raise + `NotImplementedError` on the memory backend and are only available on SQLite: + `DELETE /inbox/{message_id}`, `DELETE /drafts/{draft_id}`, + `DELETE /trash/{message_id}`, `POST /trash/clear`, + `PATCH /admin/webhooks/{webhook_id}`, and `POST /daemon/deliver/remote`. Choose + the SQLite backend if you need message deletion / trash clearing, webhook + patching, or inbound remote delivery. These gaps are pinned as `xfail` in + [`tests/integration/test_stubs.py`](../../tests/integration/test_stubs.py). +- **Memory backend deployment name.** The memory runtime's filesystem layer is + pinned to the `default` deployment: `backend-init` will *create* a named memory + deployment, but `mail-server --backend memory` reads and writes `default` + regardless. Use the SQLite backend for non-default deployment names. +- No Postgres backend yet, though `normalize_database_url` reserves a + `postgresql+psycopg` driver seam. + +## Maintenance notes + +Keep production deployment advice in how-to or explanation pages unless it is a +direct backend capability or limitation. Update this page when the backend +contract, on-disk layout, or SQLite schema changes. diff --git a/docs/testing-plan.md b/docs/testing-plan.md deleted file mode 100644 index 2324394..0000000 --- a/docs/testing-plan.md +++ /dev/null @@ -1,279 +0,0 @@ -# MAIL v2 Testing Suite Overhaul Plan - -**Status:** In effect — Phases 0–5 landed -**Date:** 2026-06-12 -**Scope:** The v2 packages (`mail-swarms-protocol`, `mail-swarms-server`, -`mail-swarms-daemon`, `mail-swarms-client`) and the repository-level `tests/` -suite. The legacy suite under -`src/mail/legacy/tests/` is out of scope and remains frozen. - ---- - -## 1. Background - -The v2 codebase (~99 source files, ~11.9k LOC across four packages) has -outpaced its test suite by roughly 8:1 in commit volume. The existing suite -(91 tests, all under `tests/unit/`) is well-constructed but covers only the -mailing-lists feature plus CLI parser shape — an estimated 5–8% of the v2 -surface. Major subsystems with **zero** coverage today: - -- the auth layer (`mail_server.auth`: JWT issue/verify, argon2 hashing, - role-validation dependencies) — existing endpoint tests monkeypatch it away -- every server router except `lists` (inbox, outbox, drafts, trash, swarms, - admin, daemon, auth — ~40 endpoints) -- the webhook delivery pipeline (HMAC-SHA256 signing, `X-MAIL-Signature`, - 6-step retry ladder) — six recent fix commits shipped with no tests -- all `mail_client` command behavior (only parser shape is tested) -- the entire `mail_daemon` package -- ~25 of 30 `mail_protocol` validators and most model `summarize()` paths -- ~40 `MemoryBackend` methods beyond the list-store group - -One test currently fails -(`test_mail_lists_endpoints.py::test_subscribe_other_rejected_with_403`) due -to drift from commit `570a340` — see Open Decisions (§7). - -## 2. Goals - -1. Establish four test categories — **unit**, **integration**, **contract**, - **e2e** — with clear ownership boundaries, so every future v2 change has an - obvious place for its tests. -2. Cover the subsystems where bugs have actually shipped (webhooks, auth, - routers) first. -3. Make spec drift mechanically detectable: the implementation, `spec/SPEC.md`, - and `spec/openapi.yaml` must not be able to diverge silently. -4. Wire coverage measurement so the gap stays visible. - -### Non-goals - -- Restoring or extending the legacy (`mail.legacy.*`) test suite. -- Performance/load testing (revisit after v2 stabilizes). -- Testing `daemon_deliver_remote` and other interswarm paths beyond stub - tracking — the feature itself is not implemented yet. - -## 3. Target suite architecture - -### 3.1 Directory layout - -``` -tests/ - conftest.py # shared fixtures (see §3.3) - unit/ # pure logic; no network, no real app, tmp_path only - integration/ # full FastAPI app over ASGI; real auth; real MemoryBackend - webhooks/ # webhook delivery pipeline (in-process receiver) - contract/ # spec + OpenAPI conformance - e2e/ # real subprocesses, real wire; marked `e2e` -``` - -### 3.2 Markers and defaults - -Registered in `pytest.ini`: - -| Marker | Meaning | In default run? | -|---|---|---| -| `unit` | pure logic | yes | -| `integration` | in-process app, real auth | yes | -| `contract` | spec/OpenAPI conformance | yes | -| `e2e` | spawns `mail-server`/`mail-daemon` subprocesses | no (`-m e2e` opt-in; runs in CI) | - -`addopts` gains `-m "not e2e"`; CI runs two jobs (default + e2e). -Keep `asyncio_mode = auto`. - -### 3.3 Shared fixtures (`tests/conftest.py`) - -The ~20-line `deployment_dir` fixture currently duplicated verbatim across -three files moves here, alongside: - -- `deployment_dir` — `tmp_path`-backed deployment tree; - monkeypatches `mail_server.backends.memory.fs.DEPLOYMENT_PATH` -- `backend` — the started backend behind `app_client`, seeded with a standard - cast: one admin, two users, one agent, one daemon, one swarm. Parametrized - over both backends (`memory` and `sqlite`) via `backend_kind`; tests never - touch backend internals — they seed/assert through the public API or the - backend-agnostic `seed_trash` / `seed_list` / `seed_refresh_token` / - `list_members` fixtures -- `app_client` — `TestClient` over the **real** `mail_server.server.app` - (env vars `MAIL_HOST`, `MAIL_JWT_SECRET_KEY`, `MAIL_JWT_ALGORITHM`, - `MAIL_REFRESH_TOKEN_EXPIRE_DAYS` set before import), wired to `backend`. A - module may override `backend_kind` to - pin one backend (e.g. `test_stubs.py` → memory, `test_gap_fill.py` → sqlite) -- `token_for(address)` — factory issuing real JWTs via `POST /auth/token`, - so integration tests exercise real auth instead of monkeypatching it -- `webhook_receiver` — in-process ASGI app that records deliveries and can be - told to fail N times (for retry-ladder tests) - -### 3.4 New dev dependencies - -- `respx` — `httpx` route mocking for `mail_client` / `mail_daemon` unit tests -- `schemathesis` *(optional, Phase 4)* — property-based fuzzing of endpoints - against `spec/openapi.yaml` - -Coverage: enable `pytest-cov` (already installed) scoped to the four v2 -packages; report in CI. Start with a visibility-only report; introduce a -ratchet threshold once Phase 2 lands. - -## 4. Test categories — scope definitions - -### Unit (`tests/unit/`) - -Pure functions and single classes; no app object, no sockets; filesystem only -via `tmp_path`. Owns: - -- all `mail_protocol` validators (full matrix: address grammars from SPEC §6, - uuid/subject/body/name/host bounds from `core/constants.py`) -- Pydantic model construction, validation edges, and `summarize()` for every - `mail_protocol.core` model (today only `lists` and `trash` are covered) -- `MemoryBackend` method-level behavior (agents/daemons/users/swarms/webhooks - CRUD, inbox/outbox/drafts/trash operations, buffer semantics) -- `memory.fs` load/save round-trips for every entity type (today: lists only) -- `mail_server.validators` (14 request-body validators → 422 paths) -- `mail_client.commands.*` and `mail_daemon.maild.api` against - `httpx.MockTransport`/`respx` (daemon tests must reset the module-level - `_mail_*` globals between tests — add an autouse fixture) -- existing CLI parser-shape tests (stay as-is) - -### Integration (`tests/integration/`) - -The real composed FastAPI app over ASGI, real JWT auth, real backend, no -subprocesses. Owns: - -- **Auth flows:** `POST /auth/token` (good/bad credentials), `whoami` per - role, password reset, expired/garbage tokens → 401 -- **Authorization boundaries:** user cannot read another user's - inbox/outbox/drafts/trash; non-admin → 403 on all `/admin/*`; daemon-only - endpoints reject user/admin tokens; agent-role behavior -- **Per-router behavior:** inbox, outbox, drafts (incl. `send`), trash, - swarms (+ health), admin (19 endpoints), daemon - (`message-buffer/clear`, `deliver/local`), lists (migrate existing endpoint - tests here, rewired to real auth) -- **Cross-endpoint flows in-process:** compose → send → buffer → deliver → - recipient inbox; list fan-out through the real routers -- **Webhook pipeline** (`integration/webhooks/`): delivery POST shape - (`WebhookDeliveredPostRequest`), HMAC-SHA256 signature verification - round-trip, retry ladder ordering (patch `asyncio.sleep`; assert the - 0s/1s/30s/5m/1h/6h schedule and give-up behavior), webhook CRUD effects on - delivery - -### Contract (`tests/contract/`) - -The spec is the oracle. Owns: - -- **OpenAPI drift check:** regenerate the schema exactly as - `scripts/generate_openapi.py` does and assert equality with the committed - `spec/openapi.yaml`. A failing check means: change the API deliberately and - regenerate, or revert. -- **SPEC.md conformance tests:** encode MUST/SHOULD clauses as tests that - reference their spec section in the test docstring — §6 address grammar - (host-scoped `user:`/`admin:`/`daemon:` forms, swarm-scoped agent and - `list:` forms), §7 message field requirements and bounds, §8 pre-send vs - post-send error semantics. Where the implementation and spec disagree, the - test fails and forces the conversation. -- **Schemathesis fuzzing** *(optional)*: generate requests from the OpenAPI - schema against the in-process app; assert no 500s and response-schema - conformance. - -### E2E (`tests/e2e/`) - -Real processes, real wire, few in number. A session-scoped fixture runs -`backend-init` into a tmp deployment, then launches `mail-server` (uvicorn) -and `mail-daemon` subprocesses with real env wiring, polling `/health` for -readiness. Tests drive the system through the `mail` / `mail-admin` CLIs: - -1. **Send/deliver journey:** login → compose → send → daemon delivers → - recipient sees the message via `inbox` / `inbox-open` -2. **List fan-out journey:** admin creates list → users subscribe → send to - `list:` address → all members receive -3. **Persistence across restart:** send/deliver → stop server cleanly → - relaunch on same deployment dir → inbox/outbox/lists intact -4. **Auth journey:** login, whoami, bad-password rejection, admin panel access - -These are the only tests that can catch env-var wiring, `backend_init` -provisioning, daemon global state, and shutdown persistence in combination. - -## 5. Execution phases - -Each phase is independently mergeable and leaves the default suite green. - -### Phase 0 — Foundation (small) -- Resolve the failing `test_subscribe_other_rejected_with_403` per the §7 - decision. -- Create `tests/conftest.py`; deduplicate the `deployment_dir` fixture out of - the three files that copy it. -- Create the category directories, register markers, update `pytest.ini` - (`-m "not e2e"`), move `tests/unit/` content as needed (no test rewrites). -- Wire `pytest-cov` reporting; gitignore `pytest.log`. -- Add `respx` as a dev dependency. - -**Exit:** suite green; one shared fixture set; coverage number visible. - -### Phase 1 — Integration: auth + routers (largest single phase) -- `app_client` + `token_for` fixtures (real app, real JWTs). -- Auth flow and authorization-boundary tests. -- Per-router endpoint tests for inbox, outbox, drafts, trash, swarms, admin, - daemon; migrate lists endpoint tests onto real auth. -- `xfail(raises=NotImplementedError)` tests for the known stubs - (`delete_inbox_message`, `delete_draft`, `delete_trash_message`, - `clear_trash`, `daemon_deliver_remote`, `admin_webhook_patch`) so the - checklist is executable. - -**Exit:** every registered route has ≥1 success and ≥1 authz/failure test; -auth layer no longer monkeypatched anywhere in integration tests. - -### Phase 2 — Webhook delivery pipeline -- `webhook_receiver` fixture; signature round-trip, payload shape, retry - ladder with patched sleep, give-up after final attempt, CRUD→delivery - effects. - -**Exit:** the six-commit bug cluster's behaviors are all pinned by tests. - -### Phase 3 — Contract layer -- OpenAPI drift check. -- SPEC.md §6/§7/§8 conformance tests with section-referencing docstrings. -- Decide on schemathesis adoption after evaluating runtime cost. - -**Exit:** an API change that isn't reflected in `spec/openapi.yaml` fails CI. - -### Phase 4 — Client + daemon units, protocol back-fill -- `mail_client.commands.*` against mocked transport (request shape, token - header, output rendering incl. markdown path); `mail-admin` commands. -- `mail_daemon.maild.api`: loop iteration behavior, buffer-clear/deliver - calls, startup validation, token acquisition; globals-reset fixture. -- Back-fill `mail_protocol` validator matrix, model edges, `memory.fs` - round-trips for all entity types, `mail_server.validators`. - -**Exit:** every v2 package has meaningful unit coverage; set the initial -coverage ratchet. - -### Phase 5 — E2E journeys + CI -- Subprocess harness fixture; the four journeys in §4. -- CI: default job (unit+integration+contract, coverage report) and e2e job. - -**Exit:** full-system happy paths run on every PR. - -## 6. Conventions - -- New v2 features land with tests in the matching category; bug fixes land - with a regression test (the webhook cluster is the cautionary tale). -- Conformance tests cite their SPEC.md section; when implementation and spec - conflict, the spec is amended or the code fixed — never the test deleted - silently. -- Stubbed functionality gets an `xfail(raises=NotImplementedError)` test at - introduction time. -- Shared fixtures live in `tests/conftest.py`; category-specific ones in that - category's `conftest.py`. No copy-pasted fixtures. - -## 7. Open decisions - -1. **Subscribe-on-behalf semantics — RESOLVED 2026-06-12: option (a).** - Commit `570a340` removed the request body from - `POST /lists/{list}/subscribe`; the endpoint always subscribes the - authenticated caller. This is ratified: subscribing *another* user-agent - is an admin-only capability (via `/admin/lists/{list}/members`), so the - public endpoint stays body-less. The stale 403 test is replaced by - `test_subscribe_ignores_supplied_member_address`. `spec/openapi.yaml` - already reflects the body-less endpoint; no spec change needed. -2. **Coverage ratchet level — RESOLVED 2026-06-12.** Set at Phase 4 exit: - `fail_under = 65` (suite measured 66%). Raise as coverage grows; never - lower. -3. **Schemathesis adoption — RESOLVED 2026-06-12: deferred.** The drift - check plus SPEC conformance tests cover the schema-shape ground; - revisit as a nightly CI job after Phase 5. diff --git a/docs/tutorials/README.md b/docs/tutorials/README.md new file mode 100644 index 0000000..3c67bf4 --- /dev/null +++ b/docs/tutorials/README.md @@ -0,0 +1,23 @@ +# Tutorials + +Tutorials teach MAIL by walking a beginner through a concrete, working project. +They should assume little prior MAIL knowledge, include only the explanation +needed to complete the lesson, and be tested end to end before release. + +## Planned Tutorials + +| Page | Outcome | Source material | +| --- | --- | --- | +| [Run MAIL Locally](run-local-mail.md) | Start a local memory-backed server, run a daemon, and observe local delivery. | `src/mail/server/docs/tutorials/quickstart.md`, `src/mail/daemon/src/mail_daemon/maild/api.py` | +| [Send Your First MAIL Message](send-first-message.md) | Log in, compose a draft, send it, and inspect inbox/outbox state. | `src/mail/client/docs/tutorials/quickstart.md`, `src/mail/client/src/mail_client/cli.py` | +| [Build a Minimal HTTP Client](build-minimal-http-client.md) | Authenticate and interact with MAIL using raw HTTP calls. | `spec/openapi.yaml`, `src/mail/protocol/src/mail_protocol/network/` | +| [Build a Webhook Receiver](build-webhook-receiver.md) | Build a correct HTTP receiver for MAIL's `mail.delivered` webhook events. | `src/mail/server/src/mail_server/backends/base.py`, `docs/explanations/webhook-delivery.md` | + +## Tutorial Checklist + +- State the concrete thing the reader will finish with. +- Prefer one happy path over branches and alternatives. +- Include prerequisites that can be verified before step 1. +- Show expected output or observable state after major steps. +- Link to reference pages for command flags, schemas, and endpoint details. +- Link to explanations for conceptual background. diff --git a/docs/tutorials/build-minimal-http-client.md b/docs/tutorials/build-minimal-http-client.md new file mode 100644 index 0000000..30e897c --- /dev/null +++ b/docs/tutorials/build-minimal-http-client.md @@ -0,0 +1,307 @@ +# Build a Minimal HTTP Client + +Status: draft + +## Outcome + +You will talk to a MAIL server using nothing but HTTP — no `mail` CLI — and walk +away with a small shell script that authenticates, confirms your identity, +creates a draft, sends it, and reads the server's responses. The same handful of +calls translate directly into any language's HTTP library. + +## Audience + +Developers integrating MAIL into their own tools and services who want to speak +to the server directly over HTTP rather than shelling out to the `mail` CLI. +Comfort with `curl` and JSON is assumed. + +## Not Here + +- Full endpoint listings belong in [HTTP API](../references/http-api.md). +- Protocol motivation belongs in [MAIL v2 Overview](../explanations/mail-v2-overview.md). +- Why sending is a two-step (draft, then send) is covered in [Delivery Model](../explanations/delivery-model.md). + +## Prerequisites + +- A running MAIL server plus a user-agent address and password. If you do not + have one, complete [Run MAIL Locally](run-local-mail.md) first — this tutorial + reuses its `admin:dummy@localhost` account and `http://127.0.0.1:8865` server. +- `curl` to make requests, and `jq` to read and extract JSON. (`jq` is only for + convenience; every call works without it.) + +Set these in your shell once — every step below uses them: + +```bash +export MAIL_SERVER="http://127.0.0.1:8865" +export MAIL_ADDRESS="admin:dummy@localhost" +export MAIL_PASSWORD="" +``` + +## Steps + +### 1. Confirm the server is reachable + +Before authenticating, verify the server is up. The health endpoint needs no token: + +```bash +curl -s "$MAIL_SERVER/health" +``` + +```json +{"status":"ok"} +``` + +(`GET /` returns the protocol name, version, and uptime if you want to confirm +the version you are targeting.) + +### 2. Obtain a bearer token + +MAIL uses the OAuth2 "password" flow, so credentials are sent as **form fields**, +not as a JSON body: + +```bash +curl -s -X POST "$MAIL_SERVER/auth/token" \ + -d "grant_type=password" \ + --data-urlencode "username=$MAIL_ADDRESS" \ + --data-urlencode "password=$MAIL_PASSWORD" +``` + +The response carries a JSON Web Token: + +```json +{"access_token":"eyJhbGciOiJIUzI1NiIs...","token_type":"bearer","refresh_token":"...","expires_in":1800,"metadata":{}} +``` + +`expires_in` is the access-token lifetime in seconds. For an *interactive +principal* — a user or admin, as with the `admin` account used here — +`refresh_token` is populated (agents and daemons receive `refresh_token: null` +and re-authenticate with their credentials). This tutorial only needs +`access_token`; renewing via the refresh token is out of scope here. + +Capture the token so the authenticated calls can reuse it: + +```bash +export MAIL_TOKEN=$(curl -s -X POST "$MAIL_SERVER/auth/token" \ + -d "grant_type=password" \ + --data-urlencode "username=$MAIL_ADDRESS" \ + --data-urlencode "password=$MAIL_PASSWORD" | jq -r .access_token) +``` + +Tokens expire after the server's configured lifetime (`MAIL_JWT_EXPIRE_MINUTES`). +When a call starts returning `401`, request a fresh token the same way. + +### 3. Confirm your identity + +Every authenticated request carries the token in an `Authorization: Bearer` +header. Use `whoami` to check that the token works and to see who the server +thinks you are: + +```bash +curl -s "$MAIL_SERVER/auth/whoami" -H "Authorization: Bearer $MAIL_TOKEN" | jq +``` + +The user-agent is nested one level inside the response envelope: + +```json +{ + "user_agent": { + "user_agent": { + "ua_type": "admin", + "admin_id": "dummy", + "host": "localhost" + } + }, + "metadata": {} +} +``` + +`ua_type` is one of `agent`, `user`, `admin`, or `daemon`, and the remaining +fields depend on it — an `agent`, for instance, carries `name`, `swarm`, and +`host` instead of `admin_id`. See [Addressing Model](../explanations/addressing-model.md) +for how these compose into a full address. + +### 4. Create a draft + +A draft holds only a `subject` and a `body`; recipients are chosen later, at send +time. Send the draft as JSON: + +```bash +curl -s -X POST "$MAIL_SERVER/drafts" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"subject":"Hello over HTTP","body":"My first MAIL message, sent with curl."}' | jq +``` + +The new draft comes back wrapped in an `entry`: + +```json +{ + "entry": { + "draft": { + "draft_id": "0f8e6e2a-1c4d-4a9b-8e7f-2b6c1d0a9f3e", + "subject": "Hello over HTTP", + "body": "My first MAIL message, sent with curl.", + "created_at": "2026-06-16T23:11:00Z", + "updated_at": null + }, + "sent_at": null, + "sent_by": null + }, + "metadata": {} +} +``` + +Capture the `draft_id` for the next step: + +```bash +export DRAFT_ID=$(curl -s -X POST "$MAIL_SERVER/drafts" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"subject":"Hello over HTTP","body":"My first MAIL message, sent with curl."}' \ + | jq -r .entry.draft.draft_id) +``` + +### 5. Send the draft + +Choose one or more recipient addresses and send the draft. The recipients are +supplied now, not at draft time: + +```bash +curl -s -X POST "$MAIL_SERVER/drafts/$DRAFT_ID/send" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"recipients":["supervisor@default@localhost"]}' | jq +``` + +The response is the assembled message. It has a `message_id` distinct from the +`draft_id`, plus the recipients you just chose: + +```json +{ + "message": { + "mail_version": "2.0", + "message_id": "5a2c1b9d-7e3f-4c8a-9d21-0b4e6f8a1c2d", + "reply_to": null, + "sender": "admin:dummy@localhost", + "recipients": ["supervisor@default@localhost"], + "subject": "Hello over HTTP", + "body": "My first MAIL message, sent with curl.", + "tags": [], + "sent_at": "2026-06-16T23:11:05Z", + "metadata": {} + }, + "metadata": {} +} +``` + +The server has now stored the message; a running daemon delivers a copy to each +recipient's inbox. (MAIL stores first and delivers via daemon — see +[Delivery Model](../explanations/delivery-model.md). To watch the message land, +follow the inbox steps in [Run MAIL Locally](run-local-mail.md).) + +### 6. Read success and error payloads + +Two conventions make every response predictable: + +- **Success** responses wrap their payload and always include a `metadata` + object. A single resource uses a key like `entry` or `message`; list endpoints + (such as `GET /drafts`) use `entries`. +- **Errors** return a non-2xx status and a JSON object with a `detail` string. + Always check the status code — a parser that only reads the body can miss the + failure. + +Print the status code alongside the body with `-w`. An invalid or missing token +returns `401`: + +```bash +curl -s -w "\n%{http_code}\n" "$MAIL_SERVER/auth/whoami" \ + -H "Authorization: Bearer not-a-real-token" +``` + +```text +{"detail":"could not validate credentials"} +401 +``` + +A request that reaches the server but fails validation returns `422`, with +`detail` naming the offending field and the reason. For example, omitting the +required `body` when creating a draft: + +```bash +curl -s -w "\n%{http_code}\n" -X POST "$MAIL_SERVER/drafts" \ + -H "Authorization: Bearer $MAIL_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"subject":"only a subject"}' +``` + +```text +{"detail":"request body validation failed: 1 validation error for DraftPostRequest\nbody\n Field required ..."} +422 +``` + +Sending to a malformed address, or with an empty `recipients` list, fails the +same way — the `detail` tells you which field and why. + +### 7. Put it together: a minimal client + +Every call above, assembled into one script. Save it as `mailclient.sh`: + +```bash +#!/usr/bin/env bash +set -euo pipefail + +MAIL_SERVER="${MAIL_SERVER:-http://127.0.0.1:8865}" +MAIL_ADDRESS="${MAIL_ADDRESS:-admin:dummy@localhost}" +: "${MAIL_PASSWORD:?set MAIL_PASSWORD}" +recipient="${1:?usage: mailclient.sh }" + +# 1. authenticate +token=$(curl -s -X POST "$MAIL_SERVER/auth/token" \ + -d grant_type=password \ + --data-urlencode "username=$MAIL_ADDRESS" \ + --data-urlencode "password=$MAIL_PASSWORD" | jq -r .access_token) +auth=(-H "Authorization: Bearer $token") + +# 2. confirm identity +echo "authenticated as: $(curl -s "${auth[@]}" "$MAIL_SERVER/auth/whoami" \ + | jq -r '.user_agent.user_agent.ua_type')" + +# 3. create a draft +draft_id=$(curl -s -X POST "$MAIL_SERVER/drafts" "${auth[@]}" \ + -H "Content-Type: application/json" \ + -d '{"subject":"Hello over HTTP","body":"Sent by mailclient.sh"}' \ + | jq -r .entry.draft.draft_id) +echo "draft created: $draft_id" + +# 4. send it +message_id=$(curl -s -X POST "$MAIL_SERVER/drafts/$draft_id/send" "${auth[@]}" \ + -H "Content-Type: application/json" \ + -d "{\"recipients\":[\"$recipient\"]}" \ + | jq -r .message.message_id) +echo "message sent: $message_id" +``` + +Run it with a recipient address: + +```bash +MAIL_PASSWORD="" ./mailclient.sh supervisor@default@localhost +``` + +```text +authenticated as: admin +draft created: 0f8e6e2a-1c4d-4a9b-8e7f-2b6c1d0a9f3e +message sent: 5a2c1b9d-7e3f-4c8a-9d21-0b4e6f8a1c2d +``` + +That is a complete MAIL client in four calls — authenticate, identify, draft, +send. Port the same requests into your language's HTTP library, reuse the bearer +token across calls, and you have native MAIL integration with no dependency on +the CLI. + +## Source Material + +- `spec/openapi.yaml` +- `src/mail/protocol/src/mail_protocol/network/requests.py` +- `src/mail/protocol/src/mail_protocol/network/responses.py` +- `src/mail/server/src/mail_server/routers/auth.py` +- `src/mail/server/src/mail_server/routers/drafts.py` diff --git a/docs/tutorials/build-webhook-receiver.md b/docs/tutorials/build-webhook-receiver.md new file mode 100644 index 0000000..19f6fce --- /dev/null +++ b/docs/tutorials/build-webhook-receiver.md @@ -0,0 +1,265 @@ +# Build a Webhook Receiver + +Status: draft + +## Goal + +Walk through building a correct MAIL webhook receiver from scratch. +By the end you'll have a small HTTP server that verifies signatures, +dedupes retries, and processes `mail.delivered` events. + +The companion explainer is [Webhook +Delivery](../explanations/webhook-delivery.md). This tutorial assumes +you've read it; we'll reference its sections rather than restate the +contract. + +## Prerequisites + +- A running MAIL server you can register webhooks against (see + [Run a Local MAIL](run-local-mail.md)). +- Python 3.12+ with `fastapi`, `uvicorn`, and `httpx`. +- A receiver URL MAIL can reach. For local development, a tunnel + (e.g., `cloudflared`) or both processes on the same host both + work. + +## The receiver, end to end + +We'll build a single-file FastAPI app that: + +1. Accepts `POST /mail/webhook`. +2. Verifies the `X-MAIL-Timestamp`, `X-MAIL-Signature`, and the + `Content-Type`. +3. Dedupes against `X-MAIL-Event-Id`. +4. Parses the payload, then prints a one-line summary. + +### Set up + +Create a new directory and install the dependencies: + +```bash +mkdir mail-receiver && cd mail-receiver +uv init +uv add fastapi uvicorn +``` + +### The signature verification function + +The signature scheme is documented in detail in [Webhook Delivery → +Security model](../explanations/webhook-delivery.md#security-model). +The receiver's job is to recompute `HMAC-SHA256(secret, +f"{timestamp}.{raw_body}")` over the *raw bytes* it received (not +the parsed JSON), then compare in constant time. + +```python +import hashlib +import hmac + + +def verify_signature( + *, raw_body: bytes, timestamp: str, signature: str, secret: str +) -> bool: + """ + Return True iff ``signature`` is a valid HMAC-SHA256 over + ``f"{timestamp}.{raw_body}"`` keyed by ``secret``. + + Signature comes in as ``"sha256="``; strip the prefix before + comparison. + """ + if not signature.startswith("sha256="): + return False + received = signature[len("sha256=") :] + + expected = hmac.new( + key=secret.encode("utf-8"), + msg=f"{timestamp}.".encode("utf-8") + raw_body, + digestmod=hashlib.sha256, + ).hexdigest() + + return hmac.compare_digest(received, expected) +``` + +A common bug at this step is to recompute the HMAC over the *parsed +and re-serialized* JSON body, which produces different bytes than +the originally-signed body and breaks verification. Always operate +on the raw bytes you received on the wire. + +Another common bug is to forget the `f"{timestamp}."` prefix. The +signed message is `timestamp.body`, not just `body`. + +### Dedup against event_id + +MAIL retries on transient failures (see [Retries](../explanations/webhook-delivery.md#retries)) +and reuses the same `event_id` for every attempt. A correct +receiver remembers recently-processed event_ids and short-circuits +duplicates. For this tutorial, an in-memory set is enough: + +```python +from collections import deque +from datetime import datetime, timezone + +PROCESSED_EVENTS: deque[tuple[str, datetime]] = deque(maxlen=10_000) + + +def is_duplicate(event_id: str) -> bool: + """ + Return True iff ``event_id`` has already been processed. + Garbage-collects entries older than 24 hours on each call. + """ + now = datetime.now(timezone.utc) + # Drop expired entries from the left. + while PROCESSED_EVENTS and (now - PROCESSED_EVENTS[0][1]).total_seconds() > 86400: + PROCESSED_EVENTS.popleft() + return any(e[0] == event_id for e in PROCESSED_EVENTS) + + +def mark_processed(event_id: str) -> None: + PROCESSED_EVENTS.append((event_id, datetime.now(timezone.utc))) +``` + +For production use, replace this with a real durable store (SQLite, +Redis, a database table) so dedup survives restarts. The 24-hour +window matches MAIL's retry exhaustion behavior with a safety +margin. + +### Timestamp skew window + +Reject any request whose `X-MAIL-Timestamp` is more than 5 minutes +from the receiver's clock. This bounds the replay window and catches +clock-drift bugs early. + +```python +import time + +SKEW_WINDOW_SECONDS = 5 * 60 + + +def is_timestamp_in_window(timestamp: str) -> bool: + try: + sent_at = int(timestamp) + except ValueError: + return False + return abs(int(time.time()) - sent_at) <= SKEW_WINDOW_SECONDS +``` + +### The full receiver + +```python +import os + +from fastapi import FastAPI, HTTPException, Request + +SECRET = os.environ.get("MAIL_WEBHOOK_SECRET") + +app = FastAPI() + + +@app.post("/mail/webhook") +async def mail_webhook(request: Request) -> dict[str, object]: + if SECRET is None: + # Not configured yet. Tell MAIL to retry later. + raise HTTPException( + status_code=503, detail="Webhook secret not configured." + ) + + timestamp = request.headers.get("X-MAIL-Timestamp") + signature = request.headers.get("X-MAIL-Signature") + event_id = request.headers.get("X-MAIL-Event-Id") + + if not timestamp: + raise HTTPException(status_code=400, detail="Missing X-MAIL-Timestamp.") + if not signature: + raise HTTPException(status_code=403, detail="Missing X-MAIL-Signature.") + if not event_id: + raise HTTPException(status_code=400, detail="Missing X-MAIL-Event-Id.") + + if not is_timestamp_in_window(timestamp): + raise HTTPException(status_code=408, detail="Timestamp outside skew window.") + + raw_body = await request.body() + + if not verify_signature( + raw_body=raw_body, timestamp=timestamp, signature=signature, secret=SECRET + ): + raise HTTPException(status_code=403, detail="Invalid signature.") + + if is_duplicate(event_id): + return {"status": "duplicate", "event_id": event_id} + + import json + + payload = json.loads(raw_body) + message = payload["message"] + + # Process the event. For this tutorial, just print. + print( + f"[mail.delivered] {message['sender']} → {message['recipient']}: " + f"{message['subject']}" + ) + + mark_processed(event_id) + return {"status": "ok", "event_id": event_id} +``` + +Run it with: + +```bash +MAIL_WEBHOOK_SECRET="" uv run uvicorn receiver:app --port 8000 +``` + +### Register with MAIL + +In another shell, register the receiver per [Manage +Webhooks](../howtos/manage-webhooks.md): + +```bash +curl -sS -X POST "$MAIL_SERVER/admin/webhooks" \ + -H "Authorization: Bearer $ADMIN_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"url": "http://localhost:8000/mail/webhook", + "events": ["mail.delivered"], + "secret": ""}' +``` + +### Send a test message + +Send a message to any recipient on the MAIL server. The receiver +should log: + +``` +[mail.delivered] alice@chorus@example.com → bob@chorus@example.com: Hello +``` + +If nothing arrives: + +- Check the MAIL server logs for outgoing POST attempts. A `403` + back from your receiver usually means the secret strings don't + match. +- Check that `MAIL_WEBHOOK_SECRET` in the receiver's env matches + the secret you registered exactly (no extra whitespace). +- Verify the receiver is reachable from MAIL's host (e.g., curl + from the MAIL host to your receiver URL). + +## What this tutorial leaves out + +- **Durable dedup.** Replace the in-memory set with a real store + before deploying. +- **Internal routing.** This receiver just prints. In production, + you'd route the event to whatever downstream service needs it + (a chat surface, a database write, a queue, etc.). +- **The "inbox is source of truth" contract.** If your internal + routing fails after signature verification succeeds, return + `200` anyway — the message lives in the MAIL inbox and your + service can recover on its own. See [Webhook Delivery → The + "inbox is source of truth" + contract](../explanations/webhook-delivery.md#the-inbox-is-source-of-truth-contract). +- **Observability.** Log every received event, every failed + signature, every dedup hit. Webhook receivers are silent failure + modes if you don't. + +## See also + +- [Webhook Delivery](../explanations/webhook-delivery.md) — the + contract this tutorial implements. +- [Manage Webhooks](../howtos/manage-webhooks.md) — registering and + rotating webhooks via the admin API. +- [HTTP API](../references/http-api.md) — formal route reference. diff --git a/docs/tutorials/run-local-mail.md b/docs/tutorials/run-local-mail.md new file mode 100644 index 0000000..4f42d14 --- /dev/null +++ b/docs/tutorials/run-local-mail.md @@ -0,0 +1,173 @@ +# Run MAIL Locally + +Status: draft + +## Outcome + +The reader starts a local MAIL v2 deployment with the memory backend, logs in as +at least two user-agents, runs a daemon, sends one message, and verifies that +the message was delivered. + +## Audience + +New contributors and first-time users who have cloned the repository and want a +working local loop before reading deeper docs. + +## Not Here + +- Exhaustive command flags belong in reference pages. +- Deployment hardening belongs in how-to guides and explanations. + +## Steps + +### 1. Install workspace dependencies + +With the `mail` repository downloaded, navigate into it and use `uv` to install workspace dependencies: + +```bash +cd mail +uv sync +``` + +### 2. Configure the server environment + +The MAIL server expects a number of environment variables to be set in order to run. +These are: +- `MAIL_HOST`: The host domain or IP address for this MAIL server. Use `localhost` for this tutorial. +- `MAIL_JWT_SECRET_KEY`: The secret key for the MAIL server to use for JWT auth. Run `openssl rand -hex 32` and use that value for this tutorial. +- `MAIL_JWT_ALGORITHM`: The JWT algorithm used by the MAIL server. Use `HS256` for this tutorial. +- `MAIL_JWT_EXPIRE_MINUTES`: The lifetime to use for JWTs on this MAIL server. Use `30` for this tutorial. + +### 3. Initialize the memory backend + +The MAIL server uses an in-memory backend by default that saves data to the local filesystem. +This backend must be initialized prior to running `mail-server`. +To initialize the memory backend, run: + +```bash +uv run backend-init --type memory --host localhost +``` + +This script will set up a new MAIL memory backend deployment in the local filesystem. +It will also generate credentials for one user-agent of each type: `agent`, `admin`, `daemon`, and `user`. +For each user-agent, the password will be written to the filepath printed to the console. +Copy these credentials into a safe place and then delete the files. + +### 4. Start `mail-server` + +With your environment configured as described in step 2, you can now start up the MAIL server: + +```bash +uv run mail-server --backend memory +``` + +The MAIL server (hosted on `http://127.0.0.1:8865`) will start up using the memory backend you just initialized. + +### 5. Log in as a sender and recipient + +Once the MAIL server is up and running, open a new terminal and log in as the `admin` user-agent that was just created: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_ADDRESS=admin:dummy@localhost +MAIL_PASSWORD={admin_password} +uv run mail login +``` + +Use the `admin` password that was generated in step 3. We'll use this user-agent to send a MAIL message. Running `login` should print a generated JWT for this admin that can be used in subsequent operations. + +Then, open another terminal and log in as the `agent` user-agent that was just created: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_ADDRESS=supervisor@default@localhost +MAIL_PASSWORD={agent_password} +uv run mail login +``` + +Use the `agent` password that was generated in step 3. We'll use this user-agent to receive the MAIL message sent by the `admin`. Running `login` should print a generated JWT for this agent that can be used in subsequent operations. + +### 6. Start `mail-daemon` with daemon credentials + +In order for MAIL messages to be delivered between user-agents, an authenticated daemon must be connected to the server. +Open another terminal and run `mail-daemon` with the generated credentials from step 3: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_ADDRESS=daemon:dummy@localhost +MAIL_PASSWORD={daemon_password} +uv run mail-daemon +``` + +The MAIL daemon should then start up and log in to the server. When there are new messages on the server to deliver, the daemon will deliver them to the specified recipient(s). + +### 7. Compose and send a message + +We will now attempt to compose and send a message as the `admin` that was authenticated in step 5. To compose a new message draft as `admin:dummy@localhost`, run: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={admin_jwt} +uv run mail compose "Test subject" "This is a message body" +``` + +You should see the new draft printed to the console, including its unique draft ID (a UUID). We can now send a MAIL message to the `agent` by specifying the draft ID and the agent's address: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={admin_jwt} +uv run mail send {draft_id} supervisor@default@localhost +``` + +You should see a MAIL message created from the draft that you just composed, including its unique message ID (a UUID). This will be delivered by the daemon (from step 6) to `supervisor@default@localhost`. Note that this process may take up to 30 seconds. + +### 8. Open inbox and outbox entries to confirm delivery + +To check that the `admin`'s message has been delivered, open the `agent`'s inbox using their JWT from step 5: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={agent_jwt} +uv run mail inbox +``` + +Once the message has been delivered, you should see it in the `agent` inbox. Open and read the full message: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={agent_jwt} +uv run mail open {message_id} +``` + +You should now see the message composed by the `admin` with the subject "Test subject" and body "This is a message body". At the bottom of the message, you should also see: + +```text +Delivered By: daemon:dummy@localhost +``` + +You can also access this message in the sending `admin`'s outbox. To do so, use the `admin`'s JWT from step 5: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={admin_jwt} +uv run mail outbox +``` + +Assuming the composed message ID is present, you can open and read it with: + +```bash +MAIL_SERVER=http://127.0.0.1:8865 +MAIL_TOKEN={admin_jwt} +uv run mail outbox-open {message_id} +``` + +Like in the `agent`'s inbox, you should see the message contents as you composed them, with a subject of "Test subject" and a body of "This is a message body". + +## Source Material + +- `README.md` +- `src/mail/server/docs/tutorials/quickstart.md` +- `src/mail/client/docs/tutorials/quickstart.md` +- `src/mail/server/.env.example` +- `src/mail/server/src/mail_server/backend_init.py` +- `src/mail/daemon/src/mail_daemon/maild/api.py` diff --git a/docs/tutorials/send-first-message.md b/docs/tutorials/send-first-message.md new file mode 100644 index 0000000..188969b --- /dev/null +++ b/docs/tutorials/send-first-message.md @@ -0,0 +1,147 @@ +# Send Your First MAIL Message + +Status: draft + +## Outcome + +The reader uses an existing MAIL server account to create a draft, send it to a +recipient, and inspect the message in the CLI. + +## Audience + +Users who already have a MAIL server URL and credentials but have not used the +`mail` CLI before. + +## Not Here + +- Admin account creation belongs in [Manage User-Agents](../howtos/manage-user-agents.md). +- Complete CLI option tables belong in [Client CLI](../references/client-cli.md). + +## Steps + +### 1. Verify `mail --help` works + +With the `mail` repository installed, ensure the MAIL client CLI is accessible: + +```bash +uv run mail --help +``` + +You should see a list of CLI commands (e.g. `login`, `compose`, `inbox`) and usage examples. + +### 2. Log in with credentials in env vars + +To log into a MAIL server at a specified address with valid credentials, set the environment variables for the server URL, user-agent address, and user-agent password, and then run the `login` command: + +```bash +MAIL_SERVER={server_url} +MAIL_ADDRESS={ua_address} +MAIL_PASSWORD={ua_password} +uv run mail login +``` + +This should print a JWT that can be used in subsequent operations with the `mail` CLI. Note that JWTs will expire after a predetermined period of time; simply run the `login` command again to obtain a fresh token. + +### 3. Store the returned token in `MAIL_TOKEN` + +Rather than requiring a `MAIL_ADDRESS` and `MAIL_PASSWORD` for every single command, the `mail` CLI expects the token obtained in step 2 as an environment variable for all non-`login` operations: + +```env +MAIL_TOKEN={jwt} +``` + +### 4. Run `mail whoami` + +Once you're logged into a MAIL server, you can view basic information about your account with the `whoami` command: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail whoami +``` + +You should see your MAIL address (e.g. `user:example@example.com`) as well as your user-agent type (which must be `agent`, `admin`, `daemon`, or `user`). + +### 5. Compose a draft + +To send a new MAIL message, you must first compose a draft that can be sent. +Use your credentials with the `compose` command to do so: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail compose "Message Subject" "This is a message body" +``` + +You should see the newly-created draft with the subject "Message Subject", body "This is a message body", and a unique draft ID (UUID) associated with it. + +### 6. Send the draft to one recipient + +With a draft composed, you can now send it to another MAIL user-agent by their address and the ID of the draft that was just created: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail send {draft_id} {ua_address} +``` + +You should see the newly-created MAIL message from your draft, with the subject "Message Subject", body "This is a message body", and a unique message ID (a UUID, but NOT the same as the draft ID). + +### 7. Open the outbox message + +You can verify that the sent message is now in your outbox: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail outbox +``` + +You should see the new message ID in your outbox. You can open the full message with `outbox-open`: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={jwt} +uv run mail outbox-open {message_id} +``` + +You should now see a MAIL message with the same ID as the one you sent, as well as the subject "Message Subject" and body "This is a message body". You may also see something like: + +```text +Delivered By: daemon:{daemon_name}@example.com +``` + +This is the address of the MAIL daemon that has delivered your message to the specified recipient(s). If you don't see it immediately, don't worry--the delivery process can take time, especially if there is a backlog of messages on the server to deliver. + +### (optional) If testing with a second account, open the recipient inbox + +If you decided to send a message to a MAIL address that you also have credentials for, you can check that the message has been delivered to its inbox. First, log into your recipient user-agent account by following the process in steps 2-3 to obtain a JWT. Then, check the user-agent's inbox: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={recipient_jwt} +uv run mail inbox +``` + +You should see the ID of the message that was delivered in your inbox. You can open it and read the message contents with the `open` command: + +```bash +MAIL_SERVER={server_url} +MAIL_TOKEN={recipient_jwt} +uv run mail open {message_id} +``` + +You should now see a MAIL message with the same ID as the one you sent, as well as the subject "Message Subject" and body "This is a message body". You should also see the address of the MAIL daemon that has delivered your message: + +```text +Delivered By: daemon:{daemon_name}@example.com +``` + +## Source Material + +- `src/mail/client/docs/tutorials/quickstart.md` +- `src/mail/client/src/mail_client/cli.py` +- `src/mail/client/src/mail_client/commands/compose.py` +- `src/mail/client/src/mail_client/commands/send.py` +- `src/mail/client/src/mail_client/commands/inbox_open.py` +- `src/mail/client/src/mail_client/commands/outbox_open.py` diff --git a/scripts/build_cli_docs.py b/scripts/build_cli_docs.py new file mode 100644 index 0000000..723e4cc --- /dev/null +++ b/scripts/build_cli_docs.py @@ -0,0 +1,179 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 MAIL Contributors +"""Generate the CLI reference pages under docs/references/ from the argparse +parsers that each MAIL command builds. + +Each MAIL CLI exposes a ``build_parser() -> argparse.ArgumentParser``. This +script imports those parsers and renders one Markdown reference per CLI, so the +docs cannot drift from the actual flags and subcommands. Regenerate with: + + uv run python scripts/build_cli_docs.py + +The generated pages are committed; do not edit them by hand. +""" + +from __future__ import annotations + +import argparse +import importlib +from pathlib import Path + +DOCS_DIR = Path(__file__).resolve().parent.parent / "docs" / "references" + +# (output filename, page title, dotted path to build_parser, console script name) +CLIS = [ + ("client-cli.md", "Client CLI", "mail_client.cli", "mail"), + ("admin-cli.md", "Admin CLI", "mail_client.admin_panel", "mail-admin"), + ("server-cli.md", "Server CLI", "mail_server.cli", "mail-server"), + ("daemon-cli.md", "Daemon CLI", "mail_daemon.cli", "mail-daemon"), +] + +BANNER = ( + "> **Generated file — do not edit by hand.** Regenerate with " + "`uv run python scripts/build_cli_docs.py` after changing the CLI. " + "See [Regenerate API Artifacts](../howtos/regenerate-api-artifacts.md)." +) + + +def _metavar(action: argparse.Action) -> str: + """A display placeholder for an option/positional that takes a value.""" + if action.nargs == 0: + return "" + if action.metavar: + return str(action.metavar) + if action.choices: + return "{" + ",".join(str(c) for c in action.choices) + "}" + return action.dest.upper() + + +def _help_text(action: argparse.Action, prog: str) -> str: + """Expand argparse %-substitutions (e.g. %(default)s) the way --help does.""" + raw = (action.help or "").strip() + if "%" not in raw: + return raw + params = {**vars(action), "prog": prog} + if params.get("choices") is not None: + params["choices"] = ", ".join(str(c) for c in params["choices"]) + try: + return raw % params + except (KeyError, ValueError, TypeError): + return raw + + +def _render_option(action: argparse.Action, prog: str) -> str: + flags = ", ".join(f"`{opt}`" for opt in action.option_strings) + meta = _metavar(action) + if meta: + flags += f" `{meta}`" + help_text = _help_text(action, prog) + return f"- {flags} — {help_text}".rstrip(" —") + + +def _render_positional(action: argparse.Action, prog: str) -> str: + name = action.metavar or action.dest + help_text = _help_text(action, prog) + return f"- `{name}` — {help_text}".rstrip(" —") + + +def _user_actions(parser: argparse.ArgumentParser): + """Actions worth documenting: skip -h/--help and the subparsers action.""" + for action in parser._actions: + if isinstance(action, argparse._HelpAction): + continue + if isinstance(action, argparse._SubParsersAction): + continue + yield action + + +def _subparsers_action(parser: argparse.ArgumentParser): + for action in parser._actions: + if isinstance(action, argparse._SubParsersAction): + return action + return None + + +def _arg_items(parser: argparse.ArgumentParser) -> tuple[list[str], list[str]]: + prog = parser.prog + positionals = [ + _render_positional(a, prog) + for a in _user_actions(parser) + if not a.option_strings + ] + options = [ + _render_option(a, prog) for a in _user_actions(parser) if a.option_strings + ] + return positionals, options + + +def _labeled_block(parser: argparse.ArgumentParser) -> list[str]: + """Argument/option lists under a bold mini-label (used inside subcommands).""" + positionals, options = _arg_items(parser) + lines: list[str] = [] + if positionals: + lines += ["**Arguments:**", "", *positionals, ""] + if options: + lines += ["**Options:**", "", *options, ""] + return lines + + +def _render_subcommands(sub_action: argparse._SubParsersAction) -> list[str]: + # Map each subparser object to all the names (primary + aliases) that reach it. + names_by_parser: dict[int, list[str]] = {} + for name, subparser in sub_action.choices.items(): + names_by_parser.setdefault(id(subparser), []).append(name) + + lines: list[str] = ["## Commands", ""] + for pseudo in sub_action._choices_actions: + primary = pseudo.dest + subparser = sub_action.choices[primary] + aliases = [n for n in names_by_parser[id(subparser)] if n != primary] + heading = f"### `{primary}`" + if aliases: + heading += " (aliases: " + ", ".join(f"`{a}`" for a in aliases) + ")" + lines.append(heading) + lines.append("") + summary = (pseudo.help or subparser.description or "").strip() + if summary: + lines.append(summary) + lines.append("") + lines.extend(_labeled_block(subparser)) + return lines + + +def render_page(title: str, module_path: str, script: str) -> str: + module = importlib.import_module(module_path) + parser: argparse.ArgumentParser = module.build_parser() + + lines = [f"# {title}", "", "Status: generated", "", BANNER, ""] + description = (parser.description or "").strip() + if description: + lines.append(description) + lines.append("") + lines.append(f"Invoke as `{script}` (or `uv run {script}` from a workspace " + f"checkout). Source: `{module_path.replace('.', '/')}.py`.") + lines.append("") + + sub_action = _subparsers_action(parser) + positionals, options = _arg_items(parser) + if positionals: + lines += ["## Arguments", "", *positionals, ""] + if options: + lines += ["## Global options" if sub_action else "## Options", "", *options, ""] + + if sub_action is not None: + lines.extend(_render_subcommands(sub_action)) + + text = "\n".join(lines).rstrip() + "\n" + return text + + +def main() -> None: + for filename, title, module_path, script in CLIS: + page = render_page(title, module_path, script) + out = DOCS_DIR / filename + out.write_text(page, encoding="utf-8") + print(f"wrote {out.relative_to(DOCS_DIR.parent.parent)}") + + +if __name__ == "__main__": + main() diff --git a/src/mail/protocol/src/mail_protocol/core/constants.py b/src/mail/protocol/src/mail_protocol/core/constants.py index dd3d4c9..ff57c5d 100644 --- a/src/mail/protocol/src/mail_protocol/core/constants.py +++ b/src/mail/protocol/src/mail_protocol/core/constants.py @@ -11,26 +11,26 @@ MESSAGE_TAG_LEN_MAX = 32 AGENT_NAME_LEN_MIN = 1 -AGENT_NAME_LEN_MAX = 31 +AGENT_NAME_LEN_MAX = 32 USER_NAME_LEN_MIN = 1 -USER_NAME_LEN_MAX = 31 +USER_NAME_LEN_MAX = 32 DAEMON_WORKER_NAME_LEN_MIN = 1 -DAEMON_WORKER_NAME_LEN_MAX = 31 +DAEMON_WORKER_NAME_LEN_MAX = 32 SWARM_NAME_LEN_MIN = 1 -SWARM_NAME_LEN_MAX = 31 +SWARM_NAME_LEN_MAX = 32 SWARM_DESCRIPTION_LEN_MIN = 0 SWARM_DESCRIPTION_LEN_MAX = 255 SWARM_KEYWORD_LEN_MIN = 1 -SWARM_KEYWORD_LEN_MAX = 31 +SWARM_KEYWORD_LEN_MAX = 32 # List length limits mirror agent names: short slugs, swarm-scoped. LIST_NAME_LEN_MIN = 1 -LIST_NAME_LEN_MAX = 31 +LIST_NAME_LEN_MAX = 32 # The literal prefix that identifies a list-shaped MAIL address. # Example address: ``list:welfare-discourse@chorus@localhost``.