MAIL v2: documentation overhaul (+ accumulated v2 feature line)

README.md

@@ -1,91 +1,163 @@
 # Multi-Agent Interface Layer (MAIL)
 
-MAIL is a protocol and Python implementation for message-oriented coordination
-between humans, agents, daemons, and swarms.
+[![PyPI](https://img.shields.io/pypi/v/mail-swarms)](https://pypi.org/project/mail-swarms/)
+[![Python](https://img.shields.io/badge/python-3.12%2B-blue)](https://www.python.org/)
+[![License](https://img.shields.io/badge/license-Apache%202.0-green)](LICENSE)
+[![Spec](https://img.shields.io/badge/MAIL%20spec-v2.0-blueviolet)](spec/SPEC.md)
+
+**MAIL is an open protocol — and a Python implementation — for email-like
+communication between humans and AI agents.** Every participant (a human user, an
+AI agent, or a delivery daemon) is an addressable *user-agent* with its own
+inbox, and they exchange messages much like people exchange email: compose a
+draft, send it to one or more addresses, and let a daemon deliver it.
+
+MAIL deliberately covers the **communication layer and little else** — not an
+agent runtime, not tool execution. If you already have agents, MAIL gives them a
+shared, standard way to talk. See [What is MAIL?](docs/explanations/mail-v2-overview.md).
+
+## Highlights
+
+- **Email-like model** — addresses, inboxes, outboxes, drafts, trash, and
+  mailing lists, all defined by an open [specification](spec/SPEC.md).
+- **HTTP-native** — a FastAPI server with an authoritative
+  [OpenAPI contract](spec/openapi.yaml); any client that speaks the contract works.
+- **Separation of concerns** — the server owns state, daemons deliver messages,
+  clients are just authenticated user-agents.
+- **Pluggable storage** — an in-memory backend for development and a
+  transactional SQLite backend for durability.
+- **Batteries included** — a CLI client (`mail`), an admin CLI (`mail-admin`), a
+  delivery daemon, and webhook delivery for push notifications.
+
+## Installation
+
+MAIL ships as five lockstep packages on PyPI under `mail-swarms-*`. Install the
+components you need:
 
-This repository is being reorganized for MAIL v2. The active implementation is
-split into package-specific workspaces, while the older MAIL v1 reference
-runtime is archived under `src/mail/legacy`.
-
-## Active v2 Packages
-
-- `src/mail/protocol` - shared protocol types and constants (`mail-swarms-protocol`)
-- `src/mail/server` - FastAPI server implementation (`mail-swarms-server`)
-- `src/mail/client` - command-line client (`mail-swarms-client`)
-- `src/mail/daemon` - daemon implementation (`mail-swarms-daemon`)
-
-## Repository Layout
-
-```text
-mail/
-├── docs/                 # v2 repository-level docs
-├── spec/                 # protocol specification and schemas
-├── src/mail/
-│   ├── protocol/         # mail-swarms-protocol package
-│   ├── server/           # mail-swarms-server package
-│   ├── client/           # mail-swarms-client package
-│   ├── daemon/           # mail-swarms-daemon package
-│   └── legacy/           # archived MAIL v1 runtime, docs, config, and UI
-├── tests/                # active MAIL v2 test suite
-├── scripts/              # repository maintenance scripts
-└── pyproject.toml        # uv workspace and meta-package configuration
+```bash
+pip install mail-swarms-server   # the FastAPI server + backend-init
+pip install mail-swarms-client   # the `mail` and `mail-admin` CLIs
+pip install mail-swarms-daemon   # the delivery daemon
 ```
 
-## Development
-
-Install the workspace dependencies:
+To work from a source checkout, use [uv](https://docs.astral.sh/uv/):
 
 ```bash
+git clone https://github.com/charonlabs/mail.git
+cd mail
 uv sync
 ```
 
-Run the v2 server:
+Requires Python 3.12+.
 
-```bash
-uv run mail-server
-```
+## Quickstart
 
-Use the v2 client:
+Bring up a local deployment and send your first message. (Prefix commands with
+`uv run` when working from a source checkout.)
 
 ```bash
-uv run mail --help
+# 1. Initialize a local memory backend (creates a swarm + starter user-agents)
+uv run backend-init --type memory --host localhost
+
+# 2. Configure and start the server
+export MAIL_HOST=localhost
+export MAIL_JWT_SECRET_KEY=$(openssl rand -hex 32)
+export MAIL_JWT_ALGORITHM=HS256
+export MAIL_JWT_EXPIRE_MINUTES=30
+export MAIL_REFRESH_TOKEN_EXPIRE_DAYS=30
+uv run mail-server --backend memory          # http://127.0.0.1:8865
+
+# 3. In another terminal, start the delivery daemon (with daemon credentials)
+uv run mail-daemon
+
+# 4. In a third terminal, log in and send a message
+export MAIL_SERVER=http://127.0.0.1:8865
+uv run mail login
+uv run mail compose "Hello" "My first MAIL message"
+uv run mail send <draft_id> supervisor@default@localhost
 ```
 
-Run active v2 tests:
+The full walkthrough — including where the generated credentials live — is in
+[Run MAIL Locally](docs/tutorials/run-local-mail.md).
 
-```bash
-uv run pytest
-```
+## Packages
 
-During the transition, some root-level scripts still target the legacy runtime.
-Legacy tests and other v1 material live under `src/mail/legacy`.
+| Package | Directory | Provides |
+| --- | --- | --- |
+| [`mail-swarms-protocol`](https://pypi.org/project/mail-swarms-protocol/) | `src/mail/protocol` | Shared protocol types, constants, and validators |
+| [`mail-swarms-server`](https://pypi.org/project/mail-swarms-server/) | `src/mail/server` | FastAPI server, storage backends, `backend-init` |
+| [`mail-swarms-client`](https://pypi.org/project/mail-swarms-client/) | `src/mail/client` | `mail` and `mail-admin` CLIs |
+| [`mail-swarms-daemon`](https://pypi.org/project/mail-swarms-daemon/) | `src/mail/daemon` | The delivery daemon (`mail-daemon`) |
 
-Run archived legacy tests explicitly:
+## Documentation
 
-```bash
-uv run --extra legacy pytest src/mail/legacy/tests
+Full docs live in [`docs/`](docs/README.md), organized by the
+[Divio system](docs/explanations/documentation-system.md):
+
+- **Tutorials** — [Run MAIL Locally](docs/tutorials/run-local-mail.md) ·
+  [Send Your First Message](docs/tutorials/send-first-message.md) ·
+  [Build a Minimal HTTP Client](docs/tutorials/build-minimal-http-client.md) ·
+  [Build a Webhook Receiver](docs/tutorials/build-webhook-receiver.md)
+- **How-to guides** — [running the server](docs/howtos/run-server.md),
+  [daemon](docs/howtos/run-daemon.md), [authentication](docs/howtos/authenticate-user-agent.md),
+  [sending messages](docs/howtos/send-message-cli.md),
+  [swarms](docs/howtos/manage-swarms.md),
+  [mailing lists](docs/howtos/manage-mailing-lists.md),
+  [webhooks](docs/howtos/manage-webhooks.md), and more.
+- **Reference** — [HTTP API](docs/references/http-api.md) ·
+  [Data Models](docs/references/data-models.md) ·
+  [Configuration](docs/references/configuration.md) ·
+  [Storage Backends](docs/references/storage-backends.md) ·
+  [CLIs](docs/references/client-cli.md)
+- **Explanations** — [Architecture](docs/explanations/architecture.md) ·
+  [Addressing](docs/explanations/addressing-model.md) ·
+  [Delivery](docs/explanations/delivery-model.md) ·
+  [Security](docs/explanations/security-model.md)
+
+The protocol itself is specified in [`spec/SPEC.md`](spec/SPEC.md), with the
+authoritative HTTP contract in [`spec/openapi.yaml`](spec/openapi.yaml).
+
+## Repository layout
+
+```text
+mail/
+├── docs/              # documentation (tutorials / howtos / references / explanations)
+├── spec/              # SPEC.md + generated openapi.yaml
+├── src/mail/
+│   ├── protocol/      # mail-swarms-protocol
+│   ├── server/        # mail-swarms-server
+│   ├── client/        # mail-swarms-client
+│   ├── daemon/        # mail-swarms-daemon
+│   └── legacy/        # archived MAIL v1 runtime (reference only)
+├── tests/             # active v2 test suite (contract / e2e / integration / unit)
+├── scripts/           # artifact generation + maintenance
+└── pyproject.toml     # uv workspace + meta-package
 ```
 
-## Documentation
+See [Repository Layout](docs/references/repository-layout.md) for the full map.
+
+## Development
 
-- Root v2 docs: `docs/README.md`
-- Protocol/specification: `spec/`
-- Server docs: `src/mail/server/docs/`
-- Client docs: `src/mail/client/docs/`
-- Legacy runtime notes: `src/mail/legacy/README.md`
-- Archived v1 docs: `src/mail/legacy/docs/`
+```bash
+uv sync              # install the workspace
+uv run pytest        # run the active v2 test suite
+uv run mail --help   # explore the client CLI
+```
 
-## Legacy Runtime
+More: [Run the Test Suite](docs/howtos/run-tests.md) ·
+[Regenerate API Artifacts](docs/howtos/regenerate-api-artifacts.md).
 
-The MAIL v1 runtime is kept for compatibility and historical reference. Use
-`mail.legacy.*` imports for archived code as it is migrated into the legacy
-namespace.
+The MAIL v1 runtime is archived under `src/mail/legacy/` for reference and is not
+part of the v2 packages — see [MAIL v1 Legacy Runtime](docs/explanations/mail-v1-legacy.md).
 
-Do not add new v2 behavior to the legacy runtime unless it is needed for a
-specific compatibility or migration task.
+## Contributing
 
-## Licensing
+Contributions are welcome. Please read [CONTRIBUTING.md](CONTRIBUTING.md); commits
+must be signed off under the [Developer Certificate of Origin](DCO).
 
-Reference implementation code is licensed under Apache License 2.0. Protocol
-specification materials are covered by their repository license files.
+## License
 
+Reference implementation code is licensed under the
+[Apache License 2.0](LICENSE). The protocol specification and patent grant are
+covered by [SPEC-LICENSE](SPEC-LICENSE) and
+[SPEC-PATENT-LICENSE](SPEC-PATENT-LICENSE). "MAIL" and related marks are subject
+to the [trademark policy](TRADEMARKS.md).

docs/README.md

@@ -1,34 +1,101 @@
-# MAIL v2 Documentation
+# MAIL Documentation
 
-This directory is the root documentation entry point for MAIL v2.
+This directory is the canonical documentation home for active MAIL v2 work.
+MAIL uses the [Divio documentation system][divio-about], so each page belongs
+to exactly one of four categories.
 
-The old MAIL v1 reference implementation docs have been archived under
-`src/mail/legacy/docs/`. Use those archived docs only when maintaining or
-studying the legacy runtime.
+## Start Here
 
-## Repository-Level Docs
+- New to MAIL: follow [Run MAIL Locally](tutorials/run-local-mail.md).
+- Trying to complete a known task: browse [How-To Guides](howtos/README.md).
+- Looking up commands, models, or endpoints: browse [Reference](references/README.md).
+- Trying to understand concepts and tradeoffs: browse [Explanations](explanations/README.md).
 
-- [`testing-plan.md`](testing-plan.md) - v2 testing suite overhaul plan
-  (categories, phases, conventions)
+## Categories
 
-## Active Package Docs
+- [Tutorials](tutorials/README.md) are learning-oriented lessons. They walk a
+  beginner through a concrete project and should be tested end to end.
+- [How-To Guides](howtos/README.md) are goal-oriented recipes. They answer
+  "How do I ...?" questions for readers who already know the basics.
+- [Reference](references/README.md) is information-oriented lookup material. It
+  describes commands, APIs, models, configuration, and repository structure.
+- [Explanations](explanations/README.md) are understanding-oriented discussions.
+  They explain why MAIL works the way it does and how its pieces fit together.
 
-- `src/mail/protocol/README.md` - protocol package documentation
-- `src/mail/server/docs/` - v2 server documentation
-- `src/mail/client/docs/` - v2 client documentation
-- `src/mail/daemon/README.md` - daemon package documentation
+## Proposed Layout
 
-## Repository-Level Docs To Add
+### Tutorials
 
-This root docs area should contain project-wide v2 material that is not owned by
-a single package, such as:
+- [Run MAIL Locally](tutorials/run-local-mail.md)
+- [Send Your First MAIL Message](tutorials/send-first-message.md)
+- [Build a Minimal HTTP Client](tutorials/build-minimal-http-client.md)
+- [Build a Webhook Receiver](tutorials/build-webhook-receiver.md)
 
-- repository layout
-- release process
-- local development workflow
-- compatibility and migration notes
-- protocol governance and specification process
+### How-To Guides
 
-Keep package-specific usage and reference material with the package that owns
-the code.
+- [Initialize the Memory Backend](howtos/initialize-memory-backend.md)
+- [Run the MAIL Server](howtos/run-server.md)
+- [Run the MAIL Daemon](howtos/run-daemon.md)
+- [Authenticate a User-Agent](howtos/authenticate-user-agent.md)
+- [Send a Message with the CLI](howtos/send-message-cli.md)
+- [Manage User-Agents](howtos/manage-user-agents.md)
+- [Manage Swarms](howtos/manage-swarms.md)
+- [Manage Mailing Lists](howtos/manage-mailing-lists.md)
+- [Manage Webhooks](howtos/manage-webhooks.md)
+- [Regenerate API Artifacts](howtos/regenerate-api-artifacts.md)
+- [Run the Test Suite](howtos/run-tests.md)
 
+### Reference
+
+- [Repository Layout](references/repository-layout.md)
+- [Configuration](references/configuration.md)
+- [Protocol Specification](references/protocol-specification.md)
+- [HTTP API](references/http-api.md)
+- [Client CLI](references/client-cli.md)
+- [Admin CLI](references/admin-cli.md)
+- [Server CLI](references/server-cli.md)
+- [Daemon CLI](references/daemon-cli.md)
+- [Data Models](references/data-models.md)
+- [Storage Backends](references/storage-backends.md)
+
+### Explanations
+
+- [MAIL v2 Overview](explanations/mail-v2-overview.md)
+- [Architecture](explanations/architecture.md)
+- [Addressing Model](explanations/addressing-model.md)
+- [Delivery Model](explanations/delivery-model.md)
+- [Security Model](explanations/security-model.md)
+- [Mailing Lists](explanations/mailing-lists.md)
+- [Webhook Delivery](explanations/webhook-delivery.md)
+- [MAIL v1 Legacy Runtime](explanations/mail-v1-legacy.md)
+- [Documentation System](explanations/documentation-system.md)
+
+## Existing Source Material
+
+- Protocol source of truth: [../spec/SPEC.md](../spec/SPEC.md) and
+  [../spec/openapi.yaml](../spec/openapi.yaml)
+- Active package docs to migrate or consolidate:
+  [client docs](../src/mail/client/docs/README.md) and
+  [server docs](../src/mail/server/docs/README.md)
+- Archived MAIL v1 material:
+  [legacy README](../src/mail/legacy/README.md) and
+  [legacy docs](../src/mail/legacy/docs/README.md)
+
+## Writing Rules
+
+- Keep a page in one category. If a page starts teaching, solving, describing,
+  and discussing at once, split it.
+- Keep tutorials robust and repeatable. They should avoid optional branches and
+  should show visible progress quickly.
+- Keep how-to guides task-focused. Link to explanations instead of pausing for
+  conceptual discussion.
+- Keep reference pages close to the implementation and generated contracts.
+  Command and API references should point at their source files.
+- Keep explanations free to discuss motivation, alternatives, and tradeoffs, but
+  link out to tutorials, how-tos, and reference pages for action or lookup.
+
+[divio-about]: https://docs.divio.com/documentation-system/
+[divio-tutorials]: https://docs.divio.com/documentation-system/tutorials/
+[divio-howtos]: https://docs.divio.com/documentation-system/how-to-guides/
+[divio-references]: https://docs.divio.com/documentation-system/reference/
+[divio-explanations]: https://docs.divio.com/documentation-system/explanation/

docs/explanations/README.md

@@ -0,0 +1,26 @@
+# Explanations
+
+Explanations discuss MAIL concepts, motivation, architecture, and tradeoffs.
+They are for understanding, not for step-by-step tasks or exhaustive lookup.
+
+## Planned Explanations
+
+| Page | Question it answers |
+| --- | --- |
+| [MAIL v2 Overview](mail-v2-overview.md) | What is MAIL and what problem is v2 trying to solve? |
+| [Architecture](architecture.md) | How do protocol, server, client, daemon, and backend pieces fit together? |
+| [Addressing Model](addressing-model.md) | Why does MAIL use host-scoped and swarm-scoped addresses? |
+| [Delivery Model](delivery-model.md) | Why are daemons responsible for message delivery? |
+| [Security Model](security-model.md) | What are the main trust boundaries and risks? |
+| [Mailing Lists](mailing-lists.md) | What is a list? How does it expand, what does its policy mean, and how do admin and user-agent permissions split? |
+| [Webhook Delivery](webhook-delivery.md) | What is the webhook contract — payload shape, HMAC signing, retry behavior, and the inbox-is-source-of-truth assumption? |
+| [MAIL v1 Legacy Runtime](mail-v1-legacy.md) | How should readers interpret the archived v1 runtime and docs? |
+| [Documentation System](documentation-system.md) | How should maintainers decide where a new page belongs? |
+
+## Explanation Checklist
+
+- Start with a concrete question or tension.
+- Discuss tradeoffs and alternatives.
+- Link to tutorials for learning paths.
+- Link to how-to guides for tasks.
+- Link to reference pages for commands, fields, and exact contracts.