diff --git a/docs/testing-plan.md b/docs/testing-plan.md index 462a2ce..79096a2 100644 --- a/docs/testing-plan.md +++ b/docs/testing-plan.md @@ -85,11 +85,15 @@ three files moves here, alongside: - `deployment_dir` — `tmp_path`-backed deployment tree; monkeypatches `mail_server.backends.memory.fs.DEPLOYMENT_PATH` -- `backend` — started `MemoryBackend` seeded with a standard cast: - one admin, two users, one agent, one daemon, one swarm +- `backend` — the started backend behind `app_client`, seeded with a standard + cast: one admin, two users, one agent, one daemon, one swarm. Parametrized + over both backends (`memory` and `sqlite`) via `backend_kind`; tests never + touch backend internals — they seed/assert through the public API or the + backend-agnostic `seed_trash` / `seed_list` / `list_members` fixtures - `app_client` — `TestClient` over the **real** `mail_server.server.app` (env vars `MAIL_HOST`, `MAIL_JWT_SECRET_KEY`, `MAIL_JWT_ALGORITHM` set - before import), wired to `backend` + before import), wired to `backend`. A module may override `backend_kind` to + pin one backend (e.g. `test_stubs.py` → memory, `test_gap_fill.py` → sqlite) - `token_for(address)` — factory issuing real JWTs via `POST /auth/token`, so integration tests exercise real auth instead of monkeypatching it - `webhook_receiver` — in-process ASGI app that records deliveries and can be diff --git a/src/mail/server/docs/README.md b/src/mail/server/docs/README.md index f1d46bb..96b3d9c 100644 --- a/src/mail/server/docs/README.md +++ b/src/mail/server/docs/README.md @@ -9,4 +9,5 @@ This document serves as the root documentation file for the `mail-swarms-server` ## Reference Docs - **`mail-server` CLI reference**: [reference/cli.md](reference/cli.md) +- **Server backends (`memory` vs `sqlite`)**: [reference/backends.md](reference/backends.md) - **MAIL HTTP API reference**: [reference/http.md](reference/http.md) diff --git a/src/mail/server/docs/reference/backends.md b/src/mail/server/docs/reference/backends.md new file mode 100644 index 0000000..dc1d1a4 --- /dev/null +++ b/src/mail/server/docs/reference/backends.md @@ -0,0 +1,114 @@ +# MAIL Server Backends + +`mail-server` stores all of its state — user-agents, swarms, messages, the four +boxes (inbox/outbox/drafts/trash), the delivery buffer, webhooks, and lists — +through a pluggable backend. Two backends ship today, selected with +`--backend` (see [cli.md](cli.md)): + +| | `memory` (default) | `sqlite` | +|---|---|---| +| Store | process-local dicts | SQLite file (SQLAlchemy async + `aiosqlite`) | +| Durability | periodic checkpoint + shutdown flush | per-commit (transactional) | +| Survives `kill -9` | only up to the last checkpoint | yes — committed writes are durable | +| Pagination / sorting | in Python over the whole box | pushed into SQL (`ORDER BY ... LIMIT`) | +| Method coverage | core; some endpoints are stubs | full parity (implements the stubs too) | +| Scaling | single process | single node | + +Both implement the same `MAILServerBackend` protocol, so the HTTP API is +identical regardless of which one is selected. + +## `memory` backend + +The default. Holds all state in process-local dictionaries and persists it to a +directory tree under `~/.mail-swarms/deployments//` on shutdown and +on a periodic checkpoint (`--memory-save-interval`, default 60s). It is the +reference proof-of-concept: simple and dependency-light, but durability is +bounded by the checkpoint interval — an abrupt `kill -9` loses everything +written since the last checkpoint. + +A handful of endpoints (`DELETE /inbox/{id}`, `DELETE /drafts/{id}`, +`DELETE /trash/{id}`, `POST /trash/clear`, `POST /daemon/deliver/remote`, +`PATCH /admin/webhooks/{id}`) raise `NotImplementedError` on this backend. + +## `sqlite` backend + +A durable, transactional backend over a single SQLite file. Every write commits +in its own short transaction, so a committed message survives an abrupt +`kill -9` — the window the memory backend's checkpoint cannot close. +Pagination, sorting, and filtering are pushed into SQL rather than loaded into +Python. It implements **every** protocol method, including the ones the memory +backend leaves as stubs, making it the more complete backend. + +### Database location + +Resolution precedence (highest first): + +1. `--database-url` / `MAIL_DATABASE_URL` — a full URL, e.g. + `sqlite:////absolute/path/mail.db`. +2. `--sqlite-path` / `MAIL_SQLITE_PATH` — a file path. +3. Default: `~/.mail-swarms/deployments/default/mail.db`. + +A `sqlite://` URL is normalized to the async `sqlite+aiosqlite://` driver +automatically, and the parent directory is created if missing. + +### Connection settings + +Each connection is opened with: + +- `journal_mode=WAL` — readers never block the writer. +- `foreign_keys=ON` — referential integrity is enforced (cascade deletes work). +- `busy_timeout=5000` — brief write contention retries for up to 5s instead of + immediately raising `database is locked`. + +### Initialization + +Provision a SQLite deployment with `backend-init --type sqlite` (same argument +surface as the memory initializer — deployment, swarm, agents, daemons, users, +admins, host): + +```bash +backend-init --type sqlite --swarm chorus --host localhost \ + --agents supervisor --users alice --admins root --daemons dummy +``` + +This creates the database file and schema and seeds the swarm and user-agents, +writing each generated password to `~/.mail-swarms/deployments//.secrets/
`. +Re-running is safe: existing swarms and user-agents are left untouched. No box +files are created — per-owner box membership is created lazily on first +delivery. + +Then run the server against the same database: + +```bash +mail-server --backend sqlite +``` + +(The startup lifespan creates the schema if it does not already exist, so +running `mail-server --backend sqlite` against a fresh path also works; use +`backend-init` when you want a seeded cast.) + +### Migrating an existing `memory` deployment + +If you already have a filesystem (`memory`) deployment, you can import it into a +new SQLite database of the same name instead of seeding a fresh cast: + +```bash +backend-init --type sqlite --import-fs +``` + +This reads the existing `~/.mail-swarms/deployments//` tree (user-agents, +swarms, messages, all four boxes with their ordering, the delivery buffer, +webhooks, and lists) and writes it into `/mail.db`. Existing +`.secrets/` files are untouched, so credentials carry over. The import runs in a +single transaction and **refuses to run against a non-empty database**, so it +can't clobber an existing SQLite deployment. + +### Single-node caveat + +SQLite serializes writers even in WAL mode, and `aiosqlite` runs each connection +on a thread, so concurrent requests can still contend on the write lock (the +`busy_timeout` retry absorbs brief contention). This makes the `sqlite` backend +a good fit for **single-node durable deployments**. Horizontal, multi-process +scaling wants a client/server database such as PostgreSQL; the URL-normalization +seam leaves the door open for a future Postgres backend, but that is not part of +this release. diff --git a/src/mail/server/docs/reference/cli.md b/src/mail/server/docs/reference/cli.md index 461ad97..c2b2de2 100644 --- a/src/mail/server/docs/reference/cli.md +++ b/src/mail/server/docs/reference/cli.md @@ -18,10 +18,21 @@ mail-server [option]... - **Example**: `mail-server --port 8000` - `-b`/`--backend`: The MAIL server backend to use. - **Default**: `memory` - - **Choices**: `memory` - - **Example**: `mail-server --backend memory` + - **Choices**: `memory`, `sqlite` + - **Example**: `mail-server --backend sqlite` + - See [backends.md](backends.md) for how the two backends differ. - `--memory-save-interval`: Seconds between memory backend filesystem checkpoints. - **Default**: `60` - **Environment**: `MAIL_MEMORY_SAVE_INTERVAL_SECONDS` - **Disable**: Set to `0` to rely only on startup/shutdown persistence. - **Example**: `mail-server --memory-save-interval 30` + - Ignored unless `--backend memory`. +- `--sqlite-path`: Path to the SQLite database file (`sqlite` backend only). + - **Default**: `~/.mail-swarms/deployments/default/mail.db` + - **Environment**: `MAIL_SQLITE_PATH` + - **Example**: `mail-server --backend sqlite --sqlite-path /var/lib/mail/mail.db` +- `--database-url`: Full database URL (`sqlite` backend only); takes precedence + over `--sqlite-path`. + - **Default**: unset (falls back to `--sqlite-path`, then the default path) + - **Environment**: `MAIL_DATABASE_URL` + - **Example**: `mail-server --backend sqlite --database-url sqlite:////abs/path/mail.db` diff --git a/src/mail/server/docs/tutorials/quickstart.md b/src/mail/server/docs/tutorials/quickstart.md index c331585..8b06c88 100644 --- a/src/mail/server/docs/tutorials/quickstart.md +++ b/src/mail/server/docs/tutorials/quickstart.md @@ -49,6 +49,30 @@ All four user-agents listed above have associated plain-text password stored in > Copy the generated passwords and keep them in a safe place. > Afterwards, remove the files generated from the backend filesystem. +## `sqlite` Backend Setup + +The `sqlite` backend is a durable, transactional alternative to `memory`: a +committed message survives an abrupt `kill -9`, not just a clean shutdown. To +initialize one, pass `--type sqlite`: + +```bash +uv run backend-init --type sqlite +``` + +This creates a SQLite database at +`~/.mail-swarms/deployments/default/mail.db` and seeds the same cast as the +`memory` initializer, writing each generated password to the printed +`.secrets/` paths. Then run the server against it: + +```bash +uv run mail-server --backend sqlite +``` + +The database path can be overridden with `--sqlite-path` / `MAIL_SQLITE_PATH` +or `--database-url` / `MAIL_DATABASE_URL`. See +[reference/backends.md](../reference/backends.md) for the full comparison, +connection settings, and the single-node caveat. + ## Running the Server With your environment variables configured, try running `mail-server`: diff --git a/src/mail/server/pyproject.toml b/src/mail/server/pyproject.toml index 46c65bf..d0733bd 100644 --- a/src/mail/server/pyproject.toml +++ b/src/mail/server/pyproject.toml @@ -9,6 +9,7 @@ authors = [ requires-python = ">=3.12" dependencies = [ "aiohttp>=3.12.15", + "aiosqlite>=0.20", "fastapi>=0.116.1", "mail-swarms-protocol==2.0.1", "pwdlib[argon2]>=0.3.0", @@ -16,6 +17,7 @@ dependencies = [ "pyjwt>=2.10.1", "python-dotenv>=1.1.1", "python-multipart>=0.0.20", + "sqlalchemy[asyncio]>=2.0", "uvicorn>=0.35.0", ] diff --git a/src/mail/server/src/mail_server/backend_init.py b/src/mail/server/src/mail_server/backend_init.py index 9bcd888..7ae5b98 100644 --- a/src/mail/server/src/mail_server/backend_init.py +++ b/src/mail/server/src/mail_server/backend_init.py @@ -2,6 +2,7 @@ # Copyright (c) 2026 Addison Kline import argparse +import asyncio from mail_protocol.cli_help import add_license_argument from mail_protocol.core.validators import ( @@ -29,7 +30,7 @@ def main() -> None: "-t", "--type", default="memory", - choices=["memory"], + choices=["memory", "sqlite"], help="the type of backend to initialize (default: %(default)s)", ) parser.add_argument( @@ -87,10 +88,20 @@ def main() -> None: default="example.com", help="the host domain or IP address to use (default: %(default)s)", ) + parser.add_argument( + "--import-fs", + action="store_true", + help=( + "for --type sqlite: import the existing filesystem (memory) " + "deployment of the same name into the new SQLite database instead " + "of seeding a fresh cast" + ), + ) # parse and handle args args = parser.parse_args() be_type = args.type + import_fs = args.import_fs deployment = args.deployment swarm = args.swarm swarm_description = args.swarm_description @@ -142,6 +153,9 @@ def main() -> None: except ValueError as e: print(f"invalid host {host}: {e}") exit(1) + if import_fs and be_type != "sqlite": + print("--import-fs is only valid with --type sqlite") + exit(1) # initialize backend match be_type: @@ -157,5 +171,33 @@ def main() -> None: admins=admins, host=host, ) + case "sqlite": + # Imported lazily so SQLAlchemy stays off the import path for + # memory-only initialization. + if import_fs: + from mail_server.backends.sqlite.migrate import ( + import_memory_deployment, + ) + + counts = asyncio.run( + import_memory_deployment(deployment=deployment) + ) + print(f"imported filesystem deployment {deployment}: {counts}") + else: + from mail_server.backends.sqlite.init import init_sqlite_backend + + asyncio.run( + init_sqlite_backend( + deployment=deployment, + swarm=swarm, + swarm_description=swarm_description, + swarm_keywords=swarm_keywords, + agents=agents, + daemons=daemons, + users=users, + admins=admins, + host=host, + ) + ) case _: raise ValueError(f"invalid backend type: {be_type}") diff --git a/src/mail/server/src/mail_server/backends/sqlite/__init__.py b/src/mail/server/src/mail_server/backends/sqlite/__init__.py new file mode 100644 index 0000000..dbdfa38 --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/__init__.py @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline diff --git a/src/mail/server/src/mail_server/backends/sqlite/api.py b/src/mail/server/src/mail_server/backends/sqlite/api.py new file mode 100644 index 0000000..7b76d3f --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/api.py @@ -0,0 +1,1162 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +``SQLiteBackend`` — a durable, transactional ``MAILServerBackend``. + +Each protocol method opens one ``Database.session()`` and delegates to the +repository layer; the session context manager owns the transaction, so +multi-write operations (``send_draft``, daemon delivery, inbox→trash moves) +commit atomically. Error semantics mirror the memory backend exactly — the same +human-readable ``ValueError`` messages the routers translate to HTTP errors and +the integration suite asserts on. + +Two deliberate refinements over the memory backend, both enabled by lazy +membership rows (no per-agent box dicts to pre-create): + +- Box reads treat "no membership rows" as an *empty box for a known agent* + rather than raising "no inbox found"; the agent is already authenticated. +- Delivery is idempotent per (owner, box, message): re-delivering the same + message to a recipient is a no-op instead of a duplicate row. + +Webhooks fire *after* the delivery transaction commits (never inside it), via +``asyncio.create_task`` — matching memory and keeping DB transactions short. + +The gap-fill methods (``delete_inbox_message``, ``delete_draft``, +``delete_trash_message``, ``clear_trash``, ``admin_webhook_patch``, +``daemon_deliver_remote``) that memory leaves as ``NotImplementedError`` are +fully implemented here. See ``src/mail/server/docs/reference/backends.md``. +""" + +from __future__ import annotations + +import asyncio +import logging +import uuid +from datetime import UTC, datetime +from typing import Any, NamedTuple + +from mail_protocol.core.constants import LIST_ADDRESS_PREFIX +from mail_protocol.core.drafts import MAILDraft, MAILDraftsEntry, MAILDraftsEntrySummary +from mail_protocol.core.inbox import MAILInboxEntry, MAILInboxEntrySummary +from mail_protocol.core.lists import MAILList, MAILListInBackend +from mail_protocol.core.messages import MAILMessage, MAILMessageSummary +from mail_protocol.core.outbox import MAILOutboxEntry, MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm, MAILSwarmSummary +from mail_protocol.core.trash import MAILTrashEntry, MAILTrashEntrySummary +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILAgent, + MAILDaemon, + MAILUser, + MAILUserAgent, + MAILUserAgentInBackend, +) +from mail_protocol.core.webhooks import MAILWebhook +from mail_protocol.network.requests import ( + AdminAgentPostRequest, + AdminDaemonPostRequest, + AdminListPatchRequest, + AdminListPostRequest, + AdminSwarmPostRequest, + AdminUserPostRequest, + AdminWebhooksPatchRequest, + AdminWebhooksPostRequest, + AuthPasswordResetRequest, + BoxFilterParams, + DaemonDeliverLocalRequest, + DaemonDeliverRemoteRequest, + DraftPatchRequest, + DraftPostRequest, + DraftSendPostRequest, +) + +from mail_server.auth import get_password_hash, verify_password +from mail_server.backends.base import MAILServerBackend +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.repositories import ( + BOX_DRAFTS, + BOX_INBOX, + BOX_OUTBOX, + BOX_TRASH, + MailStore, +) + +logger = logging.getLogger(__name__) + + +def _is_agent_recipient(address: str) -> bool: + """ + Return True iff ``address`` is an *agent* address (``name@swarm@host``). + + Mirrors the memory backend: webhooks fire only for agent recipients, never + for ``list:`` fan-out targets or 2-segment user/admin/daemon addresses. + """ + + if address.startswith(f"{LIST_ADDRESS_PREFIX}:"): + return False + return address.count("@") == 2 + + +class _WebhookFire(NamedTuple): + """A ``mail.delivered`` POST to schedule once the delivery txn commits.""" + + url: str + recipient: str + message: MAILMessage + secret: str + list_address: str | None + + +class SQLiteBackend(MAILServerBackend): + """A transactional ``MAILServerBackend`` over SQLite (SQLAlchemy async).""" + + def __init__(self, url: str) -> None: + self._db = Database(url) + # Set from the ``host`` kwarg on startup; admin CRUD builds full + # addresses as ``f"{local}@{self.host}"``, matching the memory backend. + self.host: str = "" + # Retain references to in-flight webhook tasks so they are not GC'd + # mid-flight; entries are discarded when each task completes. + self._delivery_tasks: set[asyncio.Task[None]] = set() + + # + # Lifecycle handlers + # + async def on_server_startup(self, **kwargs: Any) -> None: + logger.info("initializing sqlite backend...") + host = kwargs.get("host") + if isinstance(host, str): + self.host = host + await self._db.create_schema() + logger.info("sqlite backend initialization complete") + + async def on_server_shutdown(self, **kwargs: Any) -> None: + logger.info("shutting down sqlite backend...") + await self._db.dispose() + logger.info("sqlite backend shutdown complete") + + # + # User-agent handlers + # + async def get_user_agent(self, address: str) -> MAILUserAgentInBackend: + async with self._db.session() as session: + user_agent = await MailStore(session).user_agents.get(address) + if user_agent is None: + raise ValueError(f"user-agent with address {address} not found") + return user_agent + + async def user_agent_exists(self, address: str) -> bool: + async with self._db.session() as session: + return await MailStore(session).user_agents.exists(address) + + async def reset_password( + self, user_agent: MAILUserAgent, payload: AuthPasswordResetRequest + ) -> str: + ua_addr = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + ua_in_be = await store.user_agents.get(ua_addr) + if ua_in_be is None: + raise ValueError(f"user-agent with address {ua_addr} not found") + if not verify_password( + plain_password=payload.current_password, + hashed_password=ua_in_be.hashed_password, + ): + raise ValueError("incorrect password") + await store.user_agents.set_password( + ua_addr, get_password_hash(payload.new_password) + ) + return "success" + + # + # Swarm endpoint handlers + # + async def get_swarms(self) -> list[MAILSwarmSummary]: + async with self._db.session() as session: + swarms = await MailStore(session).swarms.list_all() + return [swarm.summarize() for swarm in swarms] + + async def get_swarm(self, swarm_name: str) -> MAILSwarm: + async with self._db.session() as session: + swarm = await MailStore(session).swarms.get(swarm_name) + if swarm is None: + raise ValueError(f"swarm with name {swarm_name} not found") + return swarm + + async def get_swarm_health(self, swarm_name: str) -> str: + async with self._db.session() as session: + swarm = await MailStore(session).swarms.get(swarm_name) + if swarm is None: + raise ValueError(f"swarm with name {swarm_name} not found") + return "ok" + + # + # Inbox endpoint handlers + # + async def get_inbox( + self, user_agent: MAILUserAgent, filters: BoxFilterParams + ) -> tuple[list[MAILInboxEntrySummary], int]: + async with self._db.session() as session: + return await MailStore(session).boxes.list_inbox( + user_agent.get_address(), filters + ) + + async def get_inbox_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILInboxEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_INBOX, message_id): + raise ValueError( + f"message with ID {message_id} not found in inbox at " + f"address {ua_address}" + ) + inbox_entry = await store.boxes.get_inbox_entry(message_id) + if inbox_entry is None: + raise ValueError( + f"message with ID {message_id} not found in inbox entries" + ) + message = await store.messages.get(message_id) + if message is None: + raise ValueError(f"message with ID {message_id} not found in messages") + return MAILInboxEntry( + message=message, + received_at=inbox_entry.received_at, + delivered_by=inbox_entry.delivered_by, + ) + + async def delete_inbox_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILInboxEntry: + """Move a message from the owner's inbox to their trash (one txn).""" + + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_INBOX, message_id): + raise ValueError( + f"message with ID {message_id} not found in inbox at " + f"address {ua_address}" + ) + inbox_entry = await store.boxes.get_inbox_entry(message_id) + if inbox_entry is None: + raise ValueError( + f"message with ID {message_id} not found in inbox entries" + ) + message = await store.messages.get(message_id) + if message is None: + raise ValueError(f"message with ID {message_id} not found in messages") + + result = MAILInboxEntry( + message=message, + received_at=inbox_entry.received_at, + delivered_by=inbox_entry.delivered_by, + ) + + trashed_at = datetime.now(UTC) + await store.boxes.remove_membership(ua_address, BOX_INBOX, message_id) + await store.boxes.add_membership( + ua_address, BOX_TRASH, message_id, trashed_at + ) + await store.boxes.upsert_trash_entry( + MAILTrashEntry(message=message, trashed_at=trashed_at) + ) + # Drop the shared inbox entry once no inbox references it anymore. + if await store.boxes.count_item_members(BOX_INBOX, message_id) == 0: + await store.boxes.delete_inbox_entry(message_id) + + return result + + # + # Outbox endpoint handlers + # + async def get_outbox( + self, user_agent: MAILUserAgent, filters: BoxFilterParams + ) -> tuple[list[MAILOutboxEntrySummary], int]: + async with self._db.session() as session: + return await MailStore(session).boxes.list_outbox( + user_agent.get_address(), filters + ) + + async def get_outbox_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILOutboxEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_OUTBOX, message_id): + raise ValueError( + f"message with ID {message_id} not found in outbox at " + f"address {ua_address}" + ) + outbox_entry = await store.boxes.get_outbox_entry(message_id) + if outbox_entry is None: + raise ValueError( + f"message with ID {message_id} not found in outbox entries" + ) + message = await store.messages.get(message_id) + if message is None: + raise ValueError(f"message with ID {message_id} not found in messages") + return MAILOutboxEntry( + message=message, + delivered_at=outbox_entry.delivered_at, + ) + + # + # Drafts box endpoints + # + async def get_drafts( + self, user_agent: MAILUserAgent, filters: BoxFilterParams + ) -> tuple[list[MAILDraftsEntrySummary], int]: + async with self._db.session() as session: + return await MailStore(session).boxes.list_drafts( + user_agent.get_address(), filters + ) + + async def post_draft( + self, user_agent: MAILUserAgent, payload: DraftPostRequest + ) -> MAILDraftsEntry: + ua_address = user_agent.get_address() + draft_id = str(uuid.uuid4()) + draft = MAILDraft( + draft_id=draft_id, + subject=payload.subject, + body=payload.body, + created_at=datetime.now(UTC), + updated_at=None, + reply_to=payload.reply_to, + tags=payload.tags, + ) + draft_entry = MAILDraftsEntry(draft=draft, sent_at=None) + async with self._db.session() as session: + store = MailStore(session) + await store.boxes.upsert_draft_entry(draft_entry) + await store.boxes.add_membership( + ua_address, BOX_DRAFTS, draft_id, draft.created_at + ) + return draft_entry + + async def get_draft( + self, user_agent: MAILUserAgent, draft_id: str + ) -> MAILDraftsEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_DRAFTS, draft_id): + raise ValueError( + f"draft with ID {draft_id} not found in draft box at " + f"address {ua_address}" + ) + draft_entry = await store.boxes.get_draft_entry(draft_id) + if draft_entry is None: + raise ValueError(f"draft with ID {draft_id} not found in draft box entries") + return draft_entry + + async def patch_draft( + self, + user_agent: MAILUserAgent, + draft_id: str, + payload: DraftPatchRequest, + ) -> MAILDraftsEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_DRAFTS, draft_id): + raise ValueError( + f"draft with ID {draft_id} not found in draft box at " + f"address {ua_address}" + ) + draft_entry = await store.boxes.get_draft_entry(draft_id) + if draft_entry is None: + raise ValueError( + f"draft with ID {draft_id} not found in draft box entries" + ) + + # Only fields explicitly supplied are modified. ``tags=[]`` is a + # deliberate "clear all tags"; an unset field (None) is left alone. + updated_fields: dict[str, Any] = {} + if payload.subject is not None: + updated_fields["subject"] = payload.subject + if payload.body is not None: + updated_fields["body"] = payload.body + if payload.reply_to is not None: + updated_fields["reply_to"] = payload.reply_to + if payload.tags is not None: + updated_fields["tags"] = payload.tags + + if not updated_fields: + return draft_entry + + updated_draft = draft_entry.draft.model_copy( + update={**updated_fields, "updated_at": datetime.now(UTC)} + ) + updated_entry = draft_entry.model_copy(update={"draft": updated_draft}) + await store.boxes.upsert_draft_entry(updated_entry) + return updated_entry + + async def delete_draft( + self, user_agent: MAILUserAgent, draft_id: str + ) -> MAILDraftsEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_DRAFTS, draft_id): + raise ValueError( + f"draft with ID {draft_id} not found in draft box at " + f"address {ua_address}" + ) + draft_entry = await store.boxes.get_draft_entry(draft_id) + if draft_entry is None: + raise ValueError( + f"draft with ID {draft_id} not found in draft box entries" + ) + await store.boxes.remove_membership(ua_address, BOX_DRAFTS, draft_id) + await store.boxes.delete_draft_entry(draft_id) + return draft_entry + + async def send_draft( + self, + user_agent: MAILUserAgent, + draft_id: str, + payload: DraftSendPostRequest, + ) -> MAILMessage: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_DRAFTS, draft_id): + raise ValueError( + f"draft with ID {draft_id} not found in draft box at " + f"address {ua_address}" + ) + draft_entry = await store.boxes.get_draft_entry(draft_id) + if draft_entry is None: + raise ValueError( + f"draft with ID {draft_id} not found in draft box entries" + ) + draft = draft_entry.draft + + message_id = str(uuid.uuid4()) # distinct from draft_id + # Draft tags + send-time tags, order-preserving union. + tags = list(draft.tags) + for tag in payload.tags: + if tag not in tags: + tags.append(tag) + now = datetime.now(UTC) + message = MAILMessage( + mail_version="2.0", + message_id=message_id, + reply_to=draft.reply_to, + sender=ua_address, + recipients=payload.recipients, + subject=draft.subject, + body=draft.body, + tags=tags, + sent_at=now, + metadata={}, + ) + outbox_entry = MAILOutboxEntrySummary( + message_id=message_id, + recipients=message.recipients, + subject=message.subject, + body_size=len(message.body), + sent_at=now, + delivered_at=None, + delivered_by=None, + ) + + await store.messages.add(message) + await store.boxes.upsert_outbox_entry(outbox_entry) + await store.boxes.add_membership(ua_address, BOX_OUTBOX, message_id, now) + await store.buffer.enqueue(message_id) + return message + + # + # Trash box endpoints + # + async def get_trash( + self, user_agent: MAILUserAgent, filters: BoxFilterParams + ) -> tuple[list[MAILTrashEntrySummary], int]: + async with self._db.session() as session: + return await MailStore(session).boxes.list_trash( + user_agent.get_address(), filters + ) + + async def get_trash_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILTrashEntry: + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_TRASH, message_id): + raise ValueError( + f"message with ID {message_id} not found in trash box at " + f"address {ua_address}" + ) + trash_entry = await store.boxes.get_trash_entry(message_id) + if trash_entry is None: + raise ValueError( + f"message with ID {message_id} not found in trash entries" + ) + return trash_entry + + async def delete_trash_message( + self, user_agent: MAILUserAgent, message_id: str + ) -> MAILTrashEntry: + """Hard-delete a trashed message from the owner's trash box.""" + + ua_address = user_agent.get_address() + async with self._db.session() as session: + store = MailStore(session) + if not await store.boxes.is_member(ua_address, BOX_TRASH, message_id): + raise ValueError( + f"message with ID {message_id} not found in trash box at " + f"address {ua_address}" + ) + trash_entry = await store.boxes.get_trash_entry(message_id) + if trash_entry is None: + raise ValueError( + f"message with ID {message_id} not found in trash entries" + ) + await store.boxes.remove_membership(ua_address, BOX_TRASH, message_id) + # Drop the shared trash entry once no trash references it; the + # canonical ``messages`` row is retained (mirrors memory, which + # never deletes messages). + if await store.boxes.count_item_members(BOX_TRASH, message_id) == 0: + await store.boxes.delete_trash_entry(message_id) + return trash_entry + + async def clear_trash( + self, user_agent: MAILUserAgent + ) -> list[MAILTrashEntrySummary]: + ua_address = user_agent.get_address() + summaries: list[MAILTrashEntrySummary] = [] + async with self._db.session() as session: + store = MailStore(session) + for message_id in await store.boxes.list_item_ids(ua_address, BOX_TRASH): + trash_entry = await store.boxes.get_trash_entry(message_id) + if trash_entry is not None: + summaries.append(trash_entry.summarize()) + await store.boxes.remove_membership(ua_address, BOX_TRASH, message_id) + if await store.boxes.count_item_members(BOX_TRASH, message_id) == 0: + await store.boxes.delete_trash_entry(message_id) + return summaries + + # + # Daemon-only endpoints + # + async def daemon_clear_message_buffer(self, daemon: MAILDaemon) -> list[str]: + async with self._db.session() as session: + return await MailStore(session).buffer.drain() + + async def daemon_deliver_local( + self, daemon: MAILDaemon, payload: DaemonDeliverLocalRequest + ) -> list[MAILMessageSummary]: + delivered: list[MAILMessageSummary] = [] + fires: list[_WebhookFire] = [] + async with self._db.session() as session: + store = MailStore(session) + webhooks = await self._delivered_webhooks(store) + for message_id in payload.message_ids: + message = await store.messages.get(message_id) + if message is None: + logger.warning(f"failed to get message by ID {message_id}") + continue + + delivered_time = datetime.now(UTC) + # Mark the shared outbox entry delivered (it must already exist). + outbox_entry = await store.boxes.get_outbox_entry(message_id) + if outbox_entry is not None: + outbox_entry.delivered_at = delivered_time + outbox_entry.delivered_by = daemon.get_address() + await store.boxes.upsert_outbox_entry(outbox_entry) + + await self._deliver_one( + store, + daemon=daemon, + message=message, + delivered_time=delivered_time, + webhooks=webhooks, + fires=fires, + ) + delivered.append(message.summarize()) + self._schedule_webhooks(fires) + return delivered + + async def daemon_deliver_remote( + self, daemon: MAILDaemon, payload: DaemonDeliverRemoteRequest + ) -> list[MAILMessageSummary]: + """ + Deliver messages authored by *remote* agents to local recipients. + + Mirrors ``daemon_deliver_local`` but the messages arrive in full from + off-server, so they are persisted into the canonical ``messages`` store + first; there is no local outbox to update (the sender is remote). + + TODO: the ``/daemon/deliver/remote`` HTTP route is still a + router-level ``NotImplementedError`` stub, so this method is currently + reachable only via direct backend calls/tests, not over the wire. + """ + + delivered: list[MAILMessageSummary] = [] + fires: list[_WebhookFire] = [] + async with self._db.session() as session: + store = MailStore(session) + webhooks = await self._delivered_webhooks(store) + for message in payload.messages: + if await store.messages.get(message.message_id) is None: + await store.messages.add(message) + await self._deliver_one( + store, + daemon=daemon, + message=message, + delivered_time=datetime.now(UTC), + webhooks=webhooks, + fires=fires, + ) + delivered.append(message.summarize()) + self._schedule_webhooks(fires) + return delivered + + # + # Delivery helpers (session-scoped; webhooks collected, fired post-commit) + # + async def _delivered_webhooks(self, store: MailStore) -> list[MAILWebhook]: + return [ + wh + for wh in await store.webhooks.list_all() + if "mail.delivered" in wh.events + ] + + async def _deliver_one( + self, + store: MailStore, + *, + daemon: MAILDaemon, + message: MAILMessage, + delivered_time: datetime, + webhooks: list[MAILWebhook], + fires: list[_WebhookFire], + ) -> None: + """Upsert the shared inbox entry and deliver to each recipient.""" + + inbox_entry = MAILInboxEntrySummary( + message_id=message.message_id, + sender=message.sender, + subject=message.subject, + body_size=len(message.body), + received_at=delivered_time, + delivered_by=daemon.get_address(), + ) + await store.boxes.upsert_inbox_entry(inbox_entry) + + for recipient in message.recipients: + if recipient.startswith(f"{LIST_ADDRESS_PREFIX}:"): + await self._fan_out_to_list( + store, + list_address=recipient, + message=message, + delivered_time=delivered_time, + webhooks=webhooks, + fires=fires, + ) + continue + await self._deliver_to_address( + store, + address=recipient, + message=message, + delivered_time=delivered_time, + list_address=None, + webhooks=webhooks, + fires=fires, + ) + + async def _deliver_to_address( + self, + store: MailStore, + *, + address: str, + message: MAILMessage, + delivered_time: datetime, + list_address: str | None, + webhooks: list[MAILWebhook], + fires: list[_WebhookFire], + ) -> None: + user_agent = await store.user_agents.get(address) + if user_agent is None: + logger.warning(f"failed to validate recipient address {address}") + return + ua_address = user_agent.get_address() + # Idempotent: re-delivering the same message is a no-op. + if not await store.boxes.is_member(ua_address, BOX_INBOX, message.message_id): + await store.boxes.add_membership( + ua_address, BOX_INBOX, message.message_id, delivered_time + ) + + # ``mail.delivered`` is agent-scoped: only ``name@swarm@host`` recipients + # carry the swarm the webhook payload requires. + if user_agent.user_agent.ua_type != "agent" or not _is_agent_recipient( + address + ): + logger.debug( + f"skipping `mail.delivered` webhooks for non-agent recipient {address}" + ) + return + for webhook in webhooks: + fires.append( + _WebhookFire( + url=webhook.url, + recipient=address, + message=message, + secret=webhook.secret, + list_address=list_address, + ) + ) + + async def _fan_out_to_list( + self, + store: MailStore, + *, + list_address: str, + message: MAILMessage, + delivered_time: datetime, + webhooks: list[MAILWebhook], + fires: list[_WebhookFire], + ) -> None: + mail_list = await store.lists.get_by_address(list_address) + if mail_list is None: + logger.warning( + f"unknown list address in recipients; skipping: {list_address}" + ) + return + for member in mail_list.members: + if member.startswith(f"{LIST_ADDRESS_PREFIX}:"): + logger.warning( + f"nested list members are not supported in v1; " + f"skipping {member!r} in {list_address!r}" + ) + continue + await self._deliver_to_address( + store, + address=member, + message=message, + delivered_time=delivered_time, + list_address=list_address, + webhooks=webhooks, + fires=fires, + ) + + def _schedule_webhooks(self, fires: list[_WebhookFire]) -> None: + """Fire collected ``mail.delivered`` POSTs after the txn has committed.""" + + for fire in fires: + task = asyncio.create_task( + self.handle_webhook_delivered_for_url( + url=fire.url, + recipient=fire.recipient, + message=fire.message, + secret=fire.secret, + list_address=fire.list_address, + ) + ) + self._delivery_tasks.add(task) + task.add_done_callback(self._delivery_tasks.discard) + + # + # Administrator endpoints — agents + # + async def admin_get_agents(self, admin: MAILAdmin) -> list[str]: + async with self._db.session() as session: + agents = await MailStore(session).user_agents.list_by_type("agent") + local_addrs: list[str] = [] + for agent in agents: + name, swarm, _host = agent.get_address().split("@") + local_addrs.append(f"{name}@{swarm}") + return local_addrs + + async def admin_get_agent( + self, admin: MAILAdmin, agent_address: str + ) -> MAILAgent: + full_address = f"{agent_address}@{self.host}" + async with self._db.session() as session: + agent = await MailStore(session).user_agents.get(full_address) + if agent is None: + raise ValueError(f"no agent found with address {agent_address}") + inner = agent.user_agent + if not isinstance(inner, MAILAgent): + raise ValueError(f"invalid agent address: {agent_address}") + return inner + + async def admin_post_agent( + self, admin: MAILAdmin, payload: AdminAgentPostRequest + ) -> MAILAgent: + full_address = f"{payload.agent_name}@{payload.swarm_name}@{self.host}" + agent = MAILAgent( + ua_type="agent", + name=payload.agent_name, + swarm=payload.swarm_name, + host=self.host, + ) + async with self._db.session() as session: + store = MailStore(session) + if await store.user_agents.exists(full_address): + raise ValueError(f"agent address already taken: {full_address}") + await store.user_agents.add( + MAILUserAgentInBackend( + user_agent=agent, + hashed_password=get_password_hash(payload.agent_password), + ) + ) + return agent + + async def admin_delete_agent( + self, admin: MAILAdmin, agent_address: str + ) -> MAILAgent: + full_address = f"{agent_address}@{self.host}" + async with self._db.session() as session: + store = MailStore(session) + agent = await store.user_agents.get(full_address) + if agent is None: + raise ValueError(f"agent not found: {agent_address}") + inner = agent.user_agent + if not isinstance(inner, MAILAgent): + raise ValueError(f"invalid agent address: {agent_address}") + await store.user_agents.delete(full_address) + return inner + + # + # Administrator endpoints — daemons + # + async def admin_get_daemons(self, admin: MAILAdmin) -> list[str]: + async with self._db.session() as session: + daemons = await MailStore(session).user_agents.list_by_type("daemon") + worker_names: list[str] = [] + for daemon in daemons: + name, _host = daemon.get_address().split("@") + worker_names.append(name.removeprefix("daemon:")) + return worker_names + + async def admin_get_daemon( + self, admin: MAILAdmin, worker_name: str + ) -> MAILDaemon: + full_address = f"daemon:{worker_name}@{self.host}" + async with self._db.session() as session: + daemon = await MailStore(session).user_agents.get(full_address) + if daemon is None: + raise ValueError(f"no daemon found with worker name {worker_name}") + inner = daemon.user_agent + if not isinstance(inner, MAILDaemon): + raise ValueError(f"invalid worker name: {worker_name}") + return inner + + async def admin_post_daemon( + self, admin: MAILAdmin, payload: AdminDaemonPostRequest + ) -> MAILDaemon: + full_address = f"daemon:{payload.worker_name}@{self.host}" + daemon = MAILDaemon( + ua_type="daemon", + worker_name=payload.worker_name, + host=self.host, + ) + async with self._db.session() as session: + store = MailStore(session) + if await store.user_agents.exists(full_address): + raise ValueError(f"daemon address already taken: {full_address}") + await store.user_agents.add( + MAILUserAgentInBackend( + user_agent=daemon, + hashed_password=get_password_hash(payload.daemon_password), + ) + ) + return daemon + + async def admin_delete_daemon( + self, admin: MAILAdmin, worker_name: str + ) -> MAILDaemon: + full_address = f"daemon:{worker_name}@{self.host}" + async with self._db.session() as session: + store = MailStore(session) + daemon = await store.user_agents.get(full_address) + if daemon is None: + raise ValueError(f"daemon not found: {worker_name}") + inner = daemon.user_agent + if not isinstance(inner, MAILDaemon): + raise ValueError(f"invalid daemon worker name: {worker_name}") + await store.user_agents.delete(full_address) + return inner + + # + # Administrator endpoints — users + # + async def admin_get_users(self, admin: MAILAdmin) -> list[str]: + async with self._db.session() as session: + users = await MailStore(session).user_agents.list_by_type("user") + user_ids: list[str] = [] + for user in users: + name, _host = user.get_address().split("@") + user_ids.append(name.removeprefix("user:")) + return user_ids + + async def admin_get_user(self, admin: MAILAdmin, user_id: str) -> MAILUser: + full_address = f"user:{user_id}@{self.host}" + async with self._db.session() as session: + user = await MailStore(session).user_agents.get(full_address) + if user is None: + raise ValueError(f"no user found with ID {user_id}") + inner = user.user_agent + if not isinstance(inner, MAILUser): + raise ValueError(f"invalid user ID: {user_id}") + return inner + + async def admin_post_user( + self, admin: MAILAdmin, payload: AdminUserPostRequest + ) -> MAILUser: + full_address = f"user:{payload.user_id}@{self.host}" + user = MAILUser(ua_type="user", user_id=payload.user_id, host=self.host) + async with self._db.session() as session: + store = MailStore(session) + if await store.user_agents.exists(full_address): + raise ValueError(f"user address already taken: {full_address}") + await store.user_agents.add( + MAILUserAgentInBackend( + user_agent=user, + hashed_password=get_password_hash(payload.user_password), + ) + ) + return user + + async def admin_delete_user(self, admin: MAILAdmin, user_id: str) -> MAILUser: + full_address = f"user:{user_id}@{self.host}" + async with self._db.session() as session: + store = MailStore(session) + user = await store.user_agents.get(full_address) + if user is None: + raise ValueError(f"user not found: {user_id}") + inner = user.user_agent + if not isinstance(inner, MAILUser): + raise ValueError(f"invalid user ID: {user_id}") + await store.user_agents.delete(full_address) + return inner + + # + # Administrator endpoints — swarms + # + async def admin_post_swarm( + self, admin: MAILAdmin, payload: AdminSwarmPostRequest + ) -> MAILSwarm: + new_swarm = MAILSwarm( + name=payload.name, + description=payload.description, + keywords=payload.keywords, + agents=[], + metadata={}, + ) + async with self._db.session() as session: + store = MailStore(session) + if await store.swarms.get(payload.name) is not None: + raise ValueError(f"swarm with name {payload.name} already exists") + await store.swarms.add(new_swarm) + return new_swarm + + async def admin_delete_swarm( + self, admin: MAILAdmin, swarm_name: str + ) -> MAILSwarm: + async with self._db.session() as session: + store = MailStore(session) + swarm = await store.swarms.delete(swarm_name) + if swarm is None: + raise ValueError(f"swarm with name {swarm_name} not found") + return swarm + + # + # Webhook handlers + # + async def admin_webhooks_get(self, admin: MAILAdmin) -> list[str]: + async with self._db.session() as session: + webhooks = await MailStore(session).webhooks.list_all() + return [wh.webhook_id for wh in webhooks] + + async def admin_webhook_get( + self, admin: MAILAdmin, webhook_id: str + ) -> MAILWebhook: + async with self._db.session() as session: + webhook = await MailStore(session).webhooks.get_by_id(webhook_id) + if webhook is None: + raise ValueError(f"webhook with ID {webhook_id} not found") + return webhook + + async def admin_webhook_post( + self, admin: MAILAdmin, payload: AdminWebhooksPostRequest + ) -> MAILWebhook: + async with self._db.session() as session: + store = MailStore(session) + # Idempotent on URL: an existing webhook is returned unchanged. + existing = await store.webhooks.get_by_url(payload.url) + if existing is not None: + return existing + new_webhook = MAILWebhook( + webhook_id=f"wh_{uuid.uuid4()}", + url=payload.url, + events=payload.events, + secret=payload.secret, + ) + await store.webhooks.add(new_webhook) + return new_webhook + + async def admin_webhook_patch( + self, + admin: MAILAdmin, + webhook_id: str, + payload: AdminWebhooksPatchRequest, + ) -> MAILWebhook: + """Update an existing webhook's URL and/or secret, preserving its id.""" + + async with self._db.session() as session: + store = MailStore(session) + existing = await store.webhooks.get_by_id(webhook_id) + if existing is None: + raise ValueError(f"webhook with ID {webhook_id} not found") + updated = existing.model_copy( + update={"url": payload.url, "secret": payload.secret} + ) + # The table is keyed by URL; delete + re-add handles both an + # in-place secret change and a URL (PK) move uniformly. + await store.webhooks.delete_by_url(existing.url) + await store.webhooks.add(updated) + return updated + + async def admin_webhook_delete( + self, admin: MAILAdmin, webhook_id: str + ) -> MAILWebhook: + async with self._db.session() as session: + store = MailStore(session) + webhook = await store.webhooks.get_by_id(webhook_id) + if webhook is None: + raise ValueError(f"webhook with ID {webhook_id} not found") + await store.webhooks.delete_by_url(webhook.url) + return webhook + + # + # List endpoints + # + async def get_lists(self) -> list[MAILListInBackend]: + async with self._db.session() as session: + return await MailStore(session).lists.list_all() + + async def get_list(self, list_address: str) -> MAILListInBackend: + async with self._db.session() as session: + mail_list = await MailStore(session).lists.get_by_address(list_address) + if mail_list is None: + raise ValueError(f"list not found: {list_address}") + return mail_list + + async def admin_get_lists(self, admin: MAILAdmin) -> list[MAILListInBackend]: + return await self.get_lists() + + async def admin_get_list( + self, admin: MAILAdmin, list_address: str + ) -> MAILListInBackend: + return await self.get_list(list_address) + + async def admin_post_list( + self, admin: MAILAdmin, payload: AdminListPostRequest + ) -> MAILListInBackend: + mail_list = MAILList( + name=payload.name, + swarm=payload.swarm_name, + host=self.host, + owner=payload.owner, + members=payload.members, + policy=payload.policy, + ) + address = mail_list.get_address() + now = datetime.now(UTC) + record = MAILListInBackend( + **mail_list.model_dump(), + list_id=str(uuid.uuid4()), + created_at=now, + updated_at=now, + ) + async with self._db.session() as session: + store = MailStore(session) + if await store.lists.get_by_address(address) is not None: + raise ValueError(f"list address already taken: {address}") + await store.lists.add(record) + return record + + async def admin_patch_list( + self, + admin: MAILAdmin, + list_address: str, + payload: AdminListPatchRequest, + ) -> MAILListInBackend: + async with self._db.session() as session: + store = MailStore(session) + existing = await store.lists.get_by_address(list_address) + if existing is None: + raise ValueError(f"list not found: {list_address}") + if payload.policy is None: + return existing + updated = existing.model_copy( + update={"policy": payload.policy, "updated_at": datetime.now(UTC)} + ) + await store.lists.update(updated) + return updated + + async def admin_delete_list( + self, admin: MAILAdmin, list_address: str + ) -> MAILListInBackend: + async with self._db.session() as session: + mail_list = await MailStore(session).lists.delete(list_address) + if mail_list is None: + raise ValueError(f"list not found: {list_address}") + return mail_list + + async def add_list_member( + self, list_address: str, member_address: str + ) -> MAILListInBackend: + async with self._db.session() as session: + store = MailStore(session) + existing = await store.lists.get_by_address(list_address) + if existing is None: + raise ValueError(f"list not found: {list_address}") + if member_address in existing.members: + return existing + updated = existing.model_copy( + update={ + "members": [*existing.members, member_address], + "updated_at": datetime.now(UTC), + } + ) + await store.lists.update(updated) + return updated + + async def remove_list_member( + self, list_address: str, member_address: str + ) -> MAILListInBackend: + async with self._db.session() as session: + store = MailStore(session) + existing = await store.lists.get_by_address(list_address) + if existing is None: + raise ValueError(f"list not found: {list_address}") + if member_address not in existing.members: + return existing + updated = existing.model_copy( + update={ + "members": [m for m in existing.members if m != member_address], + "updated_at": datetime.now(UTC), + } + ) + await store.lists.update(updated) + return updated + + # + # Message endpoints + # + async def get_message(self, message_id: str) -> MAILMessage: + async with self._db.session() as session: + message = await MailStore(session).messages.get(message_id) + if message is None: + raise ValueError(f"undefined message ID: {message_id}") + return message diff --git a/src/mail/server/src/mail_server/backends/sqlite/database.py b/src/mail/server/src/mail_server/backends/sqlite/database.py new file mode 100644 index 0000000..f0cb552 --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/database.py @@ -0,0 +1,152 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Database plumbing for the MAIL SQLite backend. + +``Database`` owns the async engine, the session factory, and the SQLite +pragmas. It mirrors the SQLAlchemy async stack used in chorus' ``db`` package: +``create_async_engine`` over ``sqlite+aiosqlite``, an ``async_sessionmaker`` +with ``expire_on_commit=False``, WAL + ``foreign_keys=ON`` + ``busy_timeout`` +configured via a ``connect`` event listener, and a ``session()`` context +manager that commits on success and rolls back on error. + +All backend mutations are expected to run inside a single ``session()`` block so +that multi-step operations (e.g. ``send_draft``: insert message + outbox entry + +membership + buffer row) commit atomically. Keep session scopes short and never +hold a transaction open across non-DB ``await``s — in particular, webhook POSTs +must fire *after* the delivery transaction commits, never inside it. + +See ``src/mail/server/docs/reference/backends.md`` for the backend overview. +""" + +from __future__ import annotations + +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager +from pathlib import Path +from typing import Protocol + +from sqlalchemy import event +from sqlalchemy.engine import Connection, make_url +from sqlalchemy.ext.asyncio import ( + AsyncEngine, + AsyncSession, + async_sessionmaker, + create_async_engine, +) + +from mail_server.backends.sqlite.schema import Base + +# Milliseconds SQLite waits on a held write lock before raising +# ``database is locked``. WAL lets readers proceed without blocking the writer, +# but writers still serialize; a non-zero busy timeout turns brief contention +# into a retry instead of an immediate error. +_BUSY_TIMEOUT_MS = 5000 + + +class _Cursor(Protocol): + def execute(self, statement: str) -> object: ... + + def close(self) -> None: ... + + +class _SQLiteConnection(Protocol): + def cursor(self) -> _Cursor: ... + + +class Database: + """Async engine + session factory for the SQLite backend.""" + + def __init__(self, url: str): + self.url = normalize_database_url(url) + _ensure_sqlite_parent(self.url) + self.engine = create_async_engine(self.url) + _configure_sqlite(self.engine) + self._sessions = async_sessionmaker( + self.engine, + expire_on_commit=False, + ) + + async def create_schema(self) -> None: + """Create every table/index, then run additive forward-compat guards.""" + + async with self.engine.begin() as connection: + await connection.run_sync(Base.metadata.create_all) + await connection.run_sync(_ensure_schema_columns) + + @asynccontextmanager + async def session(self) -> AsyncIterator[AsyncSession]: + """Yield a session that commits on success and rolls back on error.""" + + async with self._sessions() as session: + try: + yield session + await session.commit() + except Exception: + await session.rollback() + raise + + async def dispose(self) -> None: + """Dispose the engine and its connection pool (server shutdown).""" + + await self.engine.dispose() + + +def normalize_database_url(url: str) -> str: + """ + Map a plain driver URL onto its async driver. + + ``sqlite://`` → ``sqlite+aiosqlite://``. The ``postgresql://`` → + ``postgresql+psycopg://`` rewrite is kept as the seam for a future Postgres + backend (out of scope here) so the same repositories can target it later. + """ + + if url.startswith("sqlite://") and not url.startswith("sqlite+"): + return url.replace("sqlite://", "sqlite+aiosqlite://", 1) + if url.startswith("postgresql://"): + return url.replace("postgresql://", "postgresql+psycopg://", 1) + return url + + +def _ensure_schema_columns(connection: Connection) -> None: + """ + Additive, ``create_all``-friendly schema guard. + + This is the forward-compatibility hook mirroring chorus' approach: when a + queryable column is added to a ``*Row`` table in a later release, add an + idempotent ``ALTER TABLE ... ADD COLUMN`` here so existing databases pick it + up without a migration framework. There are no such additions yet, so this + is currently a no-op. + """ + + del connection # no additive columns yet; hook retained for forward-compat + + +def _ensure_sqlite_parent(url: str) -> None: + """Create the parent directory for a file-backed SQLite database.""" + + parsed = make_url(url) + if not parsed.drivername.startswith("sqlite"): + return + if parsed.database is None or parsed.database == ":memory:": + return + Path(parsed.database).expanduser().parent.mkdir(parents=True, exist_ok=True) + + +def _configure_sqlite(engine: AsyncEngine) -> None: + """Apply WAL, foreign-key enforcement, and a busy timeout per connection.""" + + if not engine.url.drivername.startswith("sqlite"): + return + + @event.listens_for(engine.sync_engine, "connect") + def _set_sqlite_pragmas( + dbapi_connection: _SQLiteConnection, + _connection_record: object, + ) -> None: + cursor = dbapi_connection.cursor() + cursor.execute("PRAGMA foreign_keys=ON") + cursor.execute("PRAGMA journal_mode=WAL") + cursor.execute(f"PRAGMA busy_timeout={_BUSY_TIMEOUT_MS}") + cursor.close() diff --git a/src/mail/server/src/mail_server/backends/sqlite/init.py b/src/mail/server/src/mail_server/backends/sqlite/init.py new file mode 100644 index 0000000..89e98fb --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/init.py @@ -0,0 +1,159 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +One-time initialization for the SQLite backend, used by ``backend-init``. + +Creates the deployment's database file + schema and seeds the same initial +state ``init_memory_backend`` writes — one swarm plus the requested agents / +daemons / users / admins, each with a generated password (hashed via +``PasswordHash``) and its plaintext written to the existing +``/.secrets/
`` files so the rest of the tooling is +unchanged. No empty box files are needed: ``mailbox_items`` membership is +created lazily on first delivery. + +Re-running against an existing deployment is safe — the swarm and any +user-agents that already exist are left untouched (their stored password hash +and secret file are preserved) rather than duplicated. +""" + +from __future__ import annotations + +import secrets +from pathlib import Path + +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILAgent, + MAILDaemon, + MAILUser, + MAILUserAgentInBackend, +) +from pwdlib import PasswordHash + +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.repositories import MailStore + +# The concrete user-agent variants ``backend-init`` seeds (the members of the +# ``MAILUserAgent.user_agent`` discriminated union). +type _UserAgentVariant = MAILAgent | MAILUser | MAILAdmin | MAILDaemon + + +def default_sqlite_path(deployment: str = "default") -> Path: + """Default DB file for a deployment: ``~/.mail-swarms/...//mail.db``.""" + + return ( + Path.home() + .joinpath(".mail-swarms", "deployments", deployment, "mail.db") + ) + + +async def _seed_user_agent( + store: MailStore, + password_hash: PasswordHash, + secrets_path: Path, + user_agent: _UserAgentVariant, + label: str, +) -> None: + """Insert one user-agent (if absent) and write its plaintext secret.""" + + address = user_agent.get_address() + if await store.user_agents.exists(address): + print(f"{label} already exists, skipping: {address}") + return + + password = secrets.token_urlsafe(32) + await store.user_agents.add( + MAILUserAgentInBackend( + user_agent=user_agent, + hashed_password=password_hash.hash(password), + ) + ) + secrets_path.joinpath(address).write_text(password, encoding="utf-8") + print(f"wrote new {label}: {address} (secret in {secrets_path})") + + +async def init_sqlite_backend( + deployment: str = "default", + swarm: str = "default", + swarm_description: str = "A MAIL swarm", + swarm_keywords: list[str] = [], + agents: list[str] = ["supervisor"], + daemons: list[str] = ["dummy"], + users: list[str] = ["dummy"], + admins: list[str] = ["dummy"], + host: str = "example.com", + db_path: Path | None = None, +) -> None: + """Initialize a fresh SQLite backend for ``mail-server``.""" + + db_path = db_path or default_sqlite_path(deployment) + deployment_path = db_path.parent + secrets_path = deployment_path.joinpath(".secrets") + secrets_path.mkdir(parents=True, exist_ok=True) + print(f"ensured deployment path: {deployment_path}") + print(f"ensured secrets path: {secrets_path}") + + # ``Database`` creates the db file's parent dir and applies the schema. + db = Database(f"sqlite:///{db_path}") + await db.create_schema() + print(f"ensured sqlite database + schema: {db_path}") + + password_hash = PasswordHash.recommended() + try: + async with db.session() as session: + store = MailStore(session) + + if await store.swarms.get(swarm) is None: + await store.swarms.add( + MAILSwarm( + name=swarm, + description=swarm_description, + keywords=swarm_keywords, + agents=agents, + metadata={}, + ) + ) + print(f"wrote swarm: {swarm}") + else: + print(f"swarm already exists, skipping: {swarm}") + + for agent_name in agents: + await _seed_user_agent( + store, + password_hash, + secrets_path, + MAILAgent( + ua_type="agent", name=agent_name, swarm=swarm, host=host + ), + "agent", + ) + for daemon_name in daemons: + await _seed_user_agent( + store, + password_hash, + secrets_path, + MAILDaemon(ua_type="daemon", worker_name=daemon_name, host=host), + "daemon", + ) + for user_name in users: + await _seed_user_agent( + store, + password_hash, + secrets_path, + MAILUser(ua_type="user", user_id=user_name, host=host), + "user", + ) + for admin_name in admins: + await _seed_user_agent( + store, + password_hash, + secrets_path, + MAILAdmin(ua_type="admin", admin_id=admin_name, host=host), + "admin", + ) + finally: + await db.dispose() + + print(f"sqlite backend initialization complete: {db_path}") diff --git a/src/mail/server/src/mail_server/backends/sqlite/migrate.py b/src/mail/server/src/mail_server/backends/sqlite/migrate.py new file mode 100644 index 0000000..008aced --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/migrate.py @@ -0,0 +1,223 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +One-time import of a filesystem (memory-backend) deployment into SQLite. + +The memory backend persists its state as a directory tree under +``~/.mail-swarms/deployments//``. This module reads that tree with +the memory backend's own loaders (so the on-disk format can never drift) and +writes every collection into the SQLite store in a single transaction. + +The per-owner box ordering that the memory backend kept implicitly (Python list +order) is reconstructed by inserting ``mailbox_items`` membership rows in list +order, each stamped with the box-arrival timestamp drawn from its entry +(``received_at`` / ``sent_at`` / ``draft.created_at`` / ``trashed_at``), so the +autoincrement ``id`` tiebreaker reproduces the original order. + +The import refuses to run against a non-empty database, so it can't clobber an +existing SQLite deployment. See ``src/mail/server/docs/reference/backends.md``. +""" + +from __future__ import annotations + +import logging +from collections.abc import Callable +from datetime import datetime +from pathlib import Path +from typing import Any + +from sqlalchemy import func, select +from sqlalchemy.ext.asyncio import AsyncSession + +import mail_server.backends.memory.fs as memory_fs +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.init import default_sqlite_path +from mail_server.backends.sqlite.repositories import ( + BOX_DRAFTS, + BOX_INBOX, + BOX_OUTBOX, + BOX_TRASH, + MailStore, +) +from mail_server.backends.sqlite.schema import ( + ListRow, + MessageRow, + SwarmRow, + UserAgentRow, + WebhookRow, +) + +logger = logging.getLogger(__name__) + + +def _memory_deployment_dir(deployment: str) -> Path: + return Path.home().joinpath(".mail-swarms", "deployments", deployment) + + +async def _is_empty(session: AsyncSession) -> bool: + """True if no top-level collection has any rows yet.""" + + for row_cls in (UserAgentRow, SwarmRow, MessageRow, WebhookRow, ListRow): + count = await session.scalar(select(func.count()).select_from(row_cls)) + if count: + return False + return True + + +async def import_memory_deployment( + deployment: str = "default", + source_dir: Path | None = None, + db_path: Path | None = None, +) -> dict[str, int]: + """ + Import a filesystem deployment into a fresh SQLite database. + + ``source_dir`` defaults to ``~/.mail-swarms/deployments/`` (the + memory backend's tree); ``db_path`` defaults to ``/mail.db``. + Returns per-collection row counts. Raises if the source is missing or the + target database already holds data. + """ + + source_dir = source_dir or _memory_deployment_dir(deployment) + if not source_dir.is_dir(): + raise FileNotFoundError( + f"no filesystem deployment to import at {source_dir}" + ) + db_path = db_path or default_sqlite_path(deployment) + + # Load every collection via the memory backend's loaders by pointing them at + # the source tree (mirrors how the test harness redirects persistence). + previous_path = memory_fs.DEPLOYMENT_PATH + memory_fs.DEPLOYMENT_PATH = source_dir + try: + user_agents = await memory_fs.load_user_agents() + swarms = await memory_fs.load_swarms() + messages = await memory_fs.load_messages() + inbox_entries = await memory_fs.load_inbox_entries() + inboxes = await memory_fs.load_inboxes() + outbox_entries = await memory_fs.load_outbox_entries() + outboxes = await memory_fs.load_outboxes() + draft_entries = await memory_fs.load_draft_entries() + drafts = await memory_fs.load_drafts() + trash_entries = await memory_fs.load_trash_entries() + trashes = await memory_fs.load_trashes() + message_buffer = await memory_fs.load_message_buffer() + webhooks = await memory_fs.load_webhooks() + lists = await memory_fs.load_lists() + finally: + memory_fs.DEPLOYMENT_PATH = previous_path + + db = Database(f"sqlite:///{db_path}") + await db.create_schema() + try: + async with db.session() as session: + if not await _is_empty(session): + raise ValueError( + f"target sqlite database {db_path} is not empty; " + "refusing to import" + ) + store = MailStore(session) + + for ua_in_be in user_agents.values(): + await store.user_agents.add(ua_in_be) + for swarm in swarms.values(): + await store.swarms.add(swarm) + + # Canonical messages first (box entries FK onto them). Track which + # ids exist so we never insert an entry whose message is absent. + present: set[str] = set() + for message in messages.values(): + await store.messages.add(message) + present.add(message.message_id) + + for entry in inbox_entries.values(): + if entry.message_id in present: + await store.boxes.upsert_inbox_entry(entry) + for entry in outbox_entries.values(): + if entry.message_id in present: + await store.boxes.upsert_outbox_entry(entry) + for trash_entry in trash_entries.values(): + # Trash entries carry the full message; restore it if the + # canonical row was already gone. + if trash_entry.message.message_id not in present: + await store.messages.add(trash_entry.message) + present.add(trash_entry.message.message_id) + await store.boxes.upsert_trash_entry(trash_entry) + for draft_entry in draft_entries.values(): + await store.boxes.upsert_draft_entry(draft_entry) + + await _import_membership( + store, BOX_INBOX, inboxes, inbox_entries, lambda e: e.received_at + ) + await _import_membership( + store, BOX_OUTBOX, outboxes, outbox_entries, lambda e: e.sent_at + ) + await _import_membership( + store, + BOX_DRAFTS, + drafts, + draft_entries, + lambda e: e.draft.created_at, + ) + await _import_membership( + store, BOX_TRASH, trashes, trash_entries, lambda e: e.trashed_at + ) + + for message_id in message_buffer: + await store.buffer.enqueue(message_id) + for webhook in webhooks.values(): + await store.webhooks.add(webhook) + for mail_list in lists.values(): + await store.lists.add(mail_list) + finally: + await db.dispose() + + counts = { + "user_agents": len(user_agents), + "swarms": len(swarms), + "messages": len(messages), + "inbox_entries": len(inbox_entries), + "outbox_entries": len(outbox_entries), + "draft_entries": len(draft_entries), + "trash_entries": len(trash_entries), + "webhooks": len(webhooks), + "lists": len(lists), + "buffered": len(message_buffer), + } + logger.info("imported filesystem deployment %s into %s: %s", deployment, db_path, counts) + return counts + + +async def _import_membership( + store: MailStore, + box: str, + boxes: dict[str, list[str]], + entries: dict[str, Any], + entered_at_of: Callable[[Any], datetime], +) -> None: + """ + Recreate ``mailbox_items`` rows for one box from its per-owner id lists. + + Insertion follows list order, so the autoincrement ``id`` reproduces the + memory backend's insertion-order tiebreaker. The arrival timestamp is read + from each item's entry via ``entered_at_of``; items missing an entry (or + already present) are skipped. + """ + + for owner, item_ids in boxes.items(): + for item_id in item_ids: + entry = entries.get(item_id) + if entry is None: + logger.warning( + "skipping %s membership for %s: no entry for item %s", + box, + owner, + item_id, + ) + continue + if await store.boxes.is_member(owner, box, item_id): + continue + await store.boxes.add_membership( + owner, box, item_id, entered_at_of(entry) + ) diff --git a/src/mail/server/src/mail_server/backends/sqlite/repositories.py b/src/mail/server/src/mail_server/backends/sqlite/repositories.py new file mode 100644 index 0000000..e5d2df7 --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/repositories.py @@ -0,0 +1,631 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Session-scoped repositories for the MAIL SQLite backend. + +Mirrors the chorus repository pattern: a top-level ``MailStore(session)`` frozen +dataclass exposes sub-repositories as properties, each a frozen dataclass +wrapping the *same* ``AsyncSession``. Repositories own **SQL only** — +``select`` / ``insert`` / ``update`` / ``delete``, pagination, and ordering — +and return MAIL Pydantic models (via ``serializers``), never ORM rows. They call +``session.flush()`` (never ``commit()``); the ``Database.session()`` context +manager owns the transaction boundary, so several repository calls compose into +one atomic operation. + +``MailboxRepository`` is the workhorse: it unifies the four boxes (inbox, +outbox, drafts, trash) over the shared ``mailbox_items`` membership table plus +the per-box entry tables, pushing sorting/pagination into ``ORDER BY ... LIMIT +/ OFFSET`` instead of loading whole boxes into Python. + +See ``src/mail/server/docs/reference/backends.md`` for the backend overview. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from typing import Any + +from mail_protocol.core.drafts import MAILDraftsEntry, MAILDraftsEntrySummary +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.outbox import MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.trash import MAILTrashEntry, MAILTrashEntrySummary +from mail_protocol.core.user_agents import MAILUserAgentInBackend +from mail_protocol.core.webhooks import MAILWebhook +from mail_protocol.network.requests import BoxFilterParams +from sqlalchemy import asc, delete, func, select +from sqlalchemy.ext.asyncio import AsyncSession + +from mail_server.backends.sqlite import serializers as ser +from mail_server.backends.sqlite.schema import ( + DraftEntryRow, + InboxEntryRow, + ListRow, + MailboxItemRow, + MessageBufferRow, + MessageRow, + OutboxEntryRow, + SwarmRow, + TrashEntryRow, + UserAgentRow, + WebhookRow, +) + +# Box discriminators stored in ``mailbox_items.box``. +BOX_INBOX = "inbox" +BOX_OUTBOX = "outbox" +BOX_DRAFTS = "drafts" +BOX_TRASH = "trash" + + +@dataclass(frozen=True) +class MailStore: + """Root handle wrapping a single session; hands out sub-repositories.""" + + session: AsyncSession + + @property + def user_agents(self) -> UserAgentRepository: + return UserAgentRepository(self.session) + + @property + def swarms(self) -> SwarmRepository: + return SwarmRepository(self.session) + + @property + def messages(self) -> MessageRepository: + return MessageRepository(self.session) + + @property + def boxes(self) -> MailboxRepository: + return MailboxRepository(self.session) + + @property + def buffer(self) -> MessageBufferRepository: + return MessageBufferRepository(self.session) + + @property + def webhooks(self) -> WebhookRepository: + return WebhookRepository(self.session) + + @property + def lists(self) -> ListRepository: + return ListRepository(self.session) + + +# --------------------------------------------------------------------------- # +# user_agents +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class UserAgentRepository: + session: AsyncSession + + async def get(self, address: str) -> MAILUserAgentInBackend | None: + row = await self.session.get(UserAgentRow, address) + if row is None: + return None + return ser.user_agent_from_row(row) + + async def exists(self, address: str) -> bool: + result = await self.session.scalar( + select(UserAgentRow.address).where(UserAgentRow.address == address) + ) + return result is not None + + async def list_by_type(self, ua_type: str) -> list[MAILUserAgentInBackend]: + rows = await self.session.scalars( + select(UserAgentRow) + .where(UserAgentRow.ua_type == ua_type) + .order_by(UserAgentRow.created_at, UserAgentRow.address) + ) + return [ser.user_agent_from_row(row) for row in rows] + + async def add(self, model: MAILUserAgentInBackend) -> MAILUserAgentInBackend: + self.session.add(UserAgentRow(**ser.user_agent_to_columns(model))) + await self.session.flush() + return model + + async def delete(self, address: str) -> MAILUserAgentInBackend | None: + row = await self.session.get(UserAgentRow, address) + if row is None: + return None + model = ser.user_agent_from_row(row) + await self.session.delete(row) + await self.session.flush() + return model + + async def set_password( + self, address: str, hashed_password: str + ) -> MAILUserAgentInBackend | None: + """Rewrite the password hash in both the typed column and the body.""" + + row = await self.session.get(UserAgentRow, address) + if row is None: + return None + model = ser.user_agent_from_row(row) + model.hashed_password = hashed_password + cols = ser.user_agent_to_columns(model) + row.hashed_password = cols["hashed_password"] + row.body = cols["body"] + await self.session.flush() + return model + + +# --------------------------------------------------------------------------- # +# swarms +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class SwarmRepository: + session: AsyncSession + + async def list_all(self) -> list[MAILSwarm]: + rows = await self.session.scalars( + select(SwarmRow).order_by(SwarmRow.created_at, SwarmRow.name) + ) + return [ser.swarm_from_row(row) for row in rows] + + async def get(self, name: str) -> MAILSwarm | None: + row = await self.session.get(SwarmRow, name) + if row is None: + return None + return ser.swarm_from_row(row) + + async def add(self, model: MAILSwarm) -> MAILSwarm: + self.session.add(SwarmRow(**ser.swarm_to_columns(model))) + await self.session.flush() + return model + + async def delete(self, name: str) -> MAILSwarm | None: + row = await self.session.get(SwarmRow, name) + if row is None: + return None + model = ser.swarm_from_row(row) + await self.session.delete(row) + await self.session.flush() + return model + + +# --------------------------------------------------------------------------- # +# messages (canonical store) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class MessageRepository: + session: AsyncSession + + async def get(self, message_id: str) -> MAILMessage | None: + row = await self.session.get(MessageRow, message_id) + if row is None: + return None + return ser.message_from_row(row) + + async def add(self, model: MAILMessage) -> MAILMessage: + self.session.add(MessageRow(**ser.message_to_columns(model))) + await self.session.flush() + return model + + async def delete(self, message_id: str) -> bool: + row = await self.session.get(MessageRow, message_id) + if row is None: + return False + await self.session.delete(row) + await self.session.flush() + return True + + +# --------------------------------------------------------------------------- # +# boxes — mailbox_items membership + the four entry tables +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class MailboxRepository: + session: AsyncSession + + # + # Membership (mailbox_items) + # + async def is_member(self, owner: str, box: str, item_id: str) -> bool: + result = await self.session.scalar( + select(MailboxItemRow.id).where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + MailboxItemRow.item_id == item_id, + ) + ) + return result is not None + + async def add_membership( + self, owner: str, box: str, item_id: str, entered_at: datetime + ) -> None: + self.session.add( + MailboxItemRow( + owner_address=owner, + box=box, + item_id=item_id, + entered_at=entered_at, + ) + ) + await self.session.flush() + + async def remove_membership(self, owner: str, box: str, item_id: str) -> bool: + """Delete one membership row; return whether it existed.""" + + if not await self.is_member(owner, box, item_id): + return False + await self.session.execute( + delete(MailboxItemRow).where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + MailboxItemRow.item_id == item_id, + ) + ) + await self.session.flush() + return True + + async def list_item_ids(self, owner: str, box: str) -> list[str]: + """Item ids in a box, in insertion order (used by ``clear_trash``).""" + + rows = await self.session.scalars( + select(MailboxItemRow.item_id) + .where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + ) + .order_by(asc(MailboxItemRow.id)) + ) + return list(rows) + + async def count_item_members(self, box: str, item_id: str) -> int: + """How many owners still reference ``item_id`` in ``box`` (orphan check).""" + + result = await self.session.scalar( + select(func.count()) + .select_from(MailboxItemRow) + .where(MailboxItemRow.box == box, MailboxItemRow.item_id == item_id) + ) + return result or 0 + + async def _count(self, owner: str, box: str) -> int: + result = await self.session.scalar( + select(func.count()) + .select_from(MailboxItemRow) + .where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + ) + ) + return result or 0 + + # + # Paginated reads + # + async def _page( + self, + *, + owner: str, + box: str, + entry_cls: Any, + entry_pk: Any, + filters: BoxFilterParams, + allow_message_sort: bool, + ) -> tuple[list[Any], int]: + """ + Return one ordered, sliced page of entry rows + the full box count. + + ``entered_at`` sorts by ``mailbox_items.entered_at`` (the box-arrival + time); ``sent_at`` joins ``messages`` and sorts by the original send + time — only valid for boxes whose items are real messages + (``allow_message_sort``). ``mailbox_items.id`` is the always-ascending + tiebreaker, reproducing the memory backend's stable insertion order. + """ + + total = await self._count(owner, box) + if total == 0: + return [], total + + stmt = ( + select(entry_cls) + .join(MailboxItemRow, MailboxItemRow.item_id == entry_pk) + .where( + MailboxItemRow.owner_address == owner, + MailboxItemRow.box == box, + ) + ) + + if allow_message_sort and filters.sort_by == "sent_at": + stmt = stmt.join( + MessageRow, MessageRow.message_id == MailboxItemRow.item_id + ) + sort_col: Any = MessageRow.sent_at + else: + sort_col = MailboxItemRow.entered_at + + sort_col = sort_col.desc() if filters.order == "desc" else sort_col.asc() + stmt = ( + stmt.order_by(sort_col, asc(MailboxItemRow.id)) + .limit(filters.limit) + .offset(filters.offset) + ) + + rows = list(await self.session.scalars(stmt)) + return rows, total + + async def list_inbox( + self, owner: str, filters: BoxFilterParams + ) -> tuple[list[MAILInboxEntrySummary], int]: + rows, total = await self._page( + owner=owner, + box=BOX_INBOX, + entry_cls=InboxEntryRow, + entry_pk=InboxEntryRow.message_id, + filters=filters, + allow_message_sort=True, + ) + return [ser.inbox_entry_from_row(row) for row in rows], total + + async def list_outbox( + self, owner: str, filters: BoxFilterParams + ) -> tuple[list[MAILOutboxEntrySummary], int]: + rows, total = await self._page( + owner=owner, + box=BOX_OUTBOX, + entry_cls=OutboxEntryRow, + entry_pk=OutboxEntryRow.message_id, + filters=filters, + allow_message_sort=True, + ) + return [ser.outbox_entry_from_row(row) for row in rows], total + + async def list_drafts( + self, owner: str, filters: BoxFilterParams + ) -> tuple[list[MAILDraftsEntrySummary], int]: + # Drafts have no send time; ``sort_by=sent_at`` is rejected at the + # router, so only ``entered_at`` (== created_at) ordering reaches here. + rows, total = await self._page( + owner=owner, + box=BOX_DRAFTS, + entry_cls=DraftEntryRow, + entry_pk=DraftEntryRow.draft_id, + filters=filters, + allow_message_sort=False, + ) + return [ser.draft_entry_from_row(row).summarize() for row in rows], total + + async def list_trash( + self, owner: str, filters: BoxFilterParams + ) -> tuple[list[MAILTrashEntrySummary], int]: + rows, total = await self._page( + owner=owner, + box=BOX_TRASH, + entry_cls=TrashEntryRow, + entry_pk=TrashEntryRow.message_id, + filters=filters, + allow_message_sort=True, + ) + return [ser.trash_entry_from_row(row).summarize() for row in rows], total + + # + # inbox_entries (shared, keyed by message id) + # + async def get_inbox_entry(self, message_id: str) -> MAILInboxEntrySummary | None: + row = await self.session.get(InboxEntryRow, message_id) + if row is None: + return None + return ser.inbox_entry_from_row(row) + + async def upsert_inbox_entry(self, summary: MAILInboxEntrySummary) -> None: + cols = ser.inbox_entry_to_columns(summary) + row = await self.session.get(InboxEntryRow, summary.message_id) + if row is None: + self.session.add(InboxEntryRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + + async def delete_inbox_entry(self, message_id: str) -> None: + row = await self.session.get(InboxEntryRow, message_id) + if row is not None: + await self.session.delete(row) + await self.session.flush() + + # + # outbox_entries (shared, keyed by message id) + # + async def get_outbox_entry(self, message_id: str) -> MAILOutboxEntrySummary | None: + row = await self.session.get(OutboxEntryRow, message_id) + if row is None: + return None + return ser.outbox_entry_from_row(row) + + async def upsert_outbox_entry(self, summary: MAILOutboxEntrySummary) -> None: + cols = ser.outbox_entry_to_columns(summary) + row = await self.session.get(OutboxEntryRow, summary.message_id) + if row is None: + self.session.add(OutboxEntryRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + + # + # draft_entries (keyed by draft id) + # + async def get_draft_entry(self, draft_id: str) -> MAILDraftsEntry | None: + row = await self.session.get(DraftEntryRow, draft_id) + if row is None: + return None + return ser.draft_entry_from_row(row) + + async def upsert_draft_entry(self, entry: MAILDraftsEntry) -> None: + cols = ser.draft_entry_to_columns(entry) + row = await self.session.get(DraftEntryRow, entry.draft.draft_id) + if row is None: + self.session.add(DraftEntryRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + + async def delete_draft_entry(self, draft_id: str) -> None: + row = await self.session.get(DraftEntryRow, draft_id) + if row is not None: + await self.session.delete(row) + await self.session.flush() + + # + # trash_entries (shared, keyed by message id) + # + async def get_trash_entry(self, message_id: str) -> MAILTrashEntry | None: + row = await self.session.get(TrashEntryRow, message_id) + if row is None: + return None + return ser.trash_entry_from_row(row) + + async def upsert_trash_entry(self, entry: MAILTrashEntry) -> None: + cols = ser.trash_entry_to_columns(entry) + row = await self.session.get(TrashEntryRow, entry.message.message_id) + if row is None: + self.session.add(TrashEntryRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + + async def delete_trash_entry(self, message_id: str) -> None: + row = await self.session.get(TrashEntryRow, message_id) + if row is not None: + await self.session.delete(row) + await self.session.flush() + + +# --------------------------------------------------------------------------- # +# message_buffer (FIFO delivery queue) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class MessageBufferRepository: + session: AsyncSession + + async def enqueue(self, message_id: str) -> None: + self.session.add(MessageBufferRow(message_id=message_id)) + await self.session.flush() + + async def list_ids(self) -> list[str]: + rows = await self.session.scalars( + select(MessageBufferRow.message_id).order_by(asc(MessageBufferRow.id)) + ) + return list(rows) + + async def drain(self) -> list[str]: + """Return every buffered id in FIFO order and clear the buffer atomically.""" + + ids = await self.list_ids() + if ids: + await self.session.execute(delete(MessageBufferRow)) + await self.session.flush() + return ids + + +# --------------------------------------------------------------------------- # +# webhooks (keyed by URL) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class WebhookRepository: + session: AsyncSession + + async def list_all(self) -> list[MAILWebhook]: + rows = await self.session.scalars( + select(WebhookRow).order_by(WebhookRow.created_at, WebhookRow.url) + ) + return [ser.webhook_from_row(row) for row in rows] + + async def get_by_url(self, url: str) -> MAILWebhook | None: + row = await self.session.get(WebhookRow, url) + if row is None: + return None + return ser.webhook_from_row(row) + + async def get_by_id(self, webhook_id: str) -> MAILWebhook | None: + row = await self.session.scalar( + select(WebhookRow).where(WebhookRow.webhook_id == webhook_id) + ) + if row is None: + return None + return ser.webhook_from_row(row) + + async def add(self, model: MAILWebhook) -> MAILWebhook: + self.session.add(WebhookRow(**ser.webhook_to_columns(model))) + await self.session.flush() + return model + + async def delete_by_url(self, url: str) -> MAILWebhook | None: + row = await self.session.get(WebhookRow, url) + if row is None: + return None + model = ser.webhook_from_row(row) + await self.session.delete(row) + await self.session.flush() + return model + + +# --------------------------------------------------------------------------- # +# lists (members live inside the body) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class ListRepository: + session: AsyncSession + + async def list_all(self) -> list[MAILListInBackend]: + rows = await self.session.scalars( + select(ListRow).order_by(ListRow.created_at, ListRow.address) + ) + return [ser.list_from_row(row) for row in rows] + + async def get_by_address(self, address: str) -> MAILListInBackend | None: + row = await self.session.get(ListRow, address) + if row is None: + return None + return ser.list_from_row(row) + + async def add(self, model: MAILListInBackend) -> MAILListInBackend: + self.session.add(ListRow(**ser.list_to_columns(model))) + await self.session.flush() + return model + + async def update(self, model: MAILListInBackend) -> MAILListInBackend: + """Persist a mutated list (member edits, policy patch) by address.""" + + cols = ser.list_to_columns(model) + row = await self.session.get(ListRow, model.get_address()) + if row is None: + self.session.add(ListRow(**cols)) + else: + for key, value in cols.items(): + setattr(row, key, value) + await self.session.flush() + return model + + async def delete(self, address: str) -> MAILListInBackend | None: + row = await self.session.get(ListRow, address) + if row is None: + return None + model = ser.list_from_row(row) + await self.session.delete(row) + await self.session.flush() + return model diff --git a/src/mail/server/src/mail_server/backends/sqlite/schema.py b/src/mail/server/src/mail_server/backends/sqlite/schema.py new file mode 100644 index 0000000..9c469fc --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/schema.py @@ -0,0 +1,250 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Declarative SQLAlchemy schema for the MAIL SQLite backend. + +Every entity row follows the *hybrid* convention: typed, indexed columns for +the handful of fields that the backend filters, sorts, or paginates on, plus a +``body`` JSON column holding the full MAIL Pydantic model +(``model.model_dump(mode="json")``). Reads always rehydrate the model from +``body`` via ``Model.model_validate(...)`` — the typed columns exist only for +``WHERE`` / ``ORDER BY``. Adding a field to a MAIL model therefore needs a +schema change only when that field must become queryable. + +See ``src/mail/server/docs/reference/backends.md`` for the backend overview. +""" + +from __future__ import annotations + +from datetime import UTC, datetime +from typing import Any + +from sqlalchemy import ( + JSON, + DateTime, + ForeignKey, + Index, + Integer, + String, + Text, + UniqueConstraint, +) +from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column + + +def utc_now() -> datetime: + """Return the current time as a timezone-aware UTC ``datetime``.""" + + return datetime.now(UTC) + + +class Base(DeclarativeBase): + pass + + +class UserAgentRow(Base): + """A MAIL user-agent (agent / user / admin / daemon).""" + + __tablename__ = "user_agents" + + # Full MAIL address: ``name@swarm@host`` for agents, + # ``prefix:name@host`` for users / admins / daemons. + address: Mapped[str] = mapped_column(String(512), primary_key=True) + ua_type: Mapped[str] = mapped_column(String(16), index=True) + # Swarm is only meaningful for agents; null for user / admin / daemon. + swarm: Mapped[str | None] = mapped_column(String(128), nullable=True, index=True) + host: Mapped[str] = mapped_column(String(255), index=True) + hashed_password: Mapped[str] = mapped_column(Text) + # Full ``MAILUserAgentInBackend``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now, onupdate=utc_now + ) + + +class SwarmRow(Base): + """A MAIL swarm exposed by this server.""" + + __tablename__ = "swarms" + + name: Mapped[str] = mapped_column(String(128), primary_key=True) + # Full ``MAILSwarm``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now, onupdate=utc_now + ) + + +class MessageRow(Base): + """The canonical store of every MAIL message known to this server.""" + + __tablename__ = "messages" + + # Bare UUID, as stored on ``MAILMessage.message_id`` (no ``msg_`` prefix). + message_id: Mapped[str] = mapped_column(String(64), primary_key=True) + sender: Mapped[str] = mapped_column(String(512), index=True) + subject: Mapped[str] = mapped_column(Text) + reply_to: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True) + sent_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + # Full ``MAILMessage`` (recipients, tags, metadata, body text). + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class InboxEntryRow(Base): + """ + A shared inbox-entry summary, keyed globally by message id. + + Mirrors the memory backend: when a message fans out to N recipients they + share one entry row; the per-owner data is the *membership* + (``mailbox_items``), not the entry. + """ + + __tablename__ = "inbox_entries" + + message_id: Mapped[str] = mapped_column( + String(64), + ForeignKey("messages.message_id", ondelete="CASCADE"), + primary_key=True, + ) + sender: Mapped[str] = mapped_column(String(512)) + subject: Mapped[str] = mapped_column(Text) + body_size: Mapped[int] = mapped_column(Integer) + received_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + delivered_by: Mapped[str | None] = mapped_column(String(512), nullable=True) + # Full ``MAILInboxEntrySummary``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class OutboxEntryRow(Base): + """A shared outbox-entry summary, keyed globally by message id.""" + + __tablename__ = "outbox_entries" + + message_id: Mapped[str] = mapped_column( + String(64), + ForeignKey("messages.message_id", ondelete="CASCADE"), + primary_key=True, + ) + sent_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + delivered_at: Mapped[datetime | None] = mapped_column( + DateTime(timezone=True), nullable=True + ) + delivered_by: Mapped[str | None] = mapped_column(String(512), nullable=True) + # Full ``MAILOutboxEntrySummary``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class DraftEntryRow(Base): + """A draft-box entry, keyed by draft id.""" + + __tablename__ = "draft_entries" + + draft_id: Mapped[str] = mapped_column(String(64), primary_key=True) + created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + updated_at: Mapped[datetime | None] = mapped_column( + DateTime(timezone=True), nullable=True + ) + # Full ``MAILDraftsEntry``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class TrashEntryRow(Base): + """A shared trash-entry, keyed globally by message id.""" + + __tablename__ = "trash_entries" + + message_id: Mapped[str] = mapped_column( + String(64), + ForeignKey("messages.message_id", ondelete="CASCADE"), + primary_key=True, + ) + trashed_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + # Full ``MAILTrashEntry``. + body: Mapped[dict[str, Any]] = mapped_column(JSON) + + +class MailboxItemRow(Base): + """ + Unified per-owner box membership + ordering for all four boxes. + + One row links an owner's box to an entry. ``box`` discriminates between + ``inbox`` / ``outbox`` / ``drafts`` / ``trash``; ``item_id`` is a message + id (inbox / outbox / trash) or a draft id (drafts). The autoincrement + ``id`` reproduces the insertion order that the memory backend got for free + from Python lists, and serves as the stable tiebreaker when two entries + share an ``entered_at``. + """ + + __tablename__ = "mailbox_items" + __table_args__ = ( + UniqueConstraint( + "owner_address", + "box", + "item_id", + name="uq_mailbox_items_owner_box_item", + ), + Index("ix_mailbox_items_owner_box", "owner_address", "box"), + ) + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + owner_address: Mapped[str] = mapped_column( + String(512), + ForeignKey("user_agents.address", ondelete="CASCADE"), + index=True, + ) + box: Mapped[str] = mapped_column(String(8)) + item_id: Mapped[str] = mapped_column(String(64)) + entered_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True) + + +class MessageBufferRow(Base): + """The FIFO message delivery queue. Autoincrement ``id`` preserves order.""" + + __tablename__ = "message_buffer" + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + message_id: Mapped[str] = mapped_column(String(64), unique=True, index=True) + + +class WebhookRow(Base): + """A server webhook, keyed by URL (matching the memory backend).""" + + __tablename__ = "webhooks" + + url: Mapped[str] = mapped_column(String(512), primary_key=True) + webhook_id: Mapped[str] = mapped_column(String(128), unique=True, index=True) + # Full ``MAILWebhook`` (events, secret). + body: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now, onupdate=utc_now + ) + + +class ListRow(Base): + """A MAIL list. Members live inside ``body``, mirroring the memory backend.""" + + __tablename__ = "lists" + + # ``list:@@``. + address: Mapped[str] = mapped_column(String(512), primary_key=True) + list_id: Mapped[str] = mapped_column(String(64), unique=True, index=True) + swarm: Mapped[str] = mapped_column(String(128), index=True) + host: Mapped[str] = mapped_column(String(255)) + # Full ``MAILListInBackend`` (members, policy, owner). + body: Mapped[dict[str, Any]] = mapped_column(JSON) + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now + ) + updated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), default=utc_now, onupdate=utc_now + ) diff --git a/src/mail/server/src/mail_server/backends/sqlite/serializers.py b/src/mail/server/src/mail_server/backends/sqlite/serializers.py new file mode 100644 index 0000000..30bffea --- /dev/null +++ b/src/mail/server/src/mail_server/backends/sqlite/serializers.py @@ -0,0 +1,223 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Row <-> MAIL Pydantic model conversion for the SQLite backend. + +This module centralizes the *hybrid* invariant in one place: every entity row +carries a ``body`` JSON column equal to ``model.model_dump(mode="json")`` plus a +handful of typed columns used only for ``WHERE`` / ``ORDER BY``. Each table gets +a pair of helpers: + +- ``_to_columns(model)`` — the keyword arguments to construct (or update) + a ``*Row``: the derived typed columns *and* ``body``. The typed columns are + always computed from the model, never the other way around, so they can never + drift from the body. +- ``
_from_row(row)`` — rehydrate the model. This reads **only** + ``row.body`` via ``Model.model_validate(...)``; the typed columns are never + consulted on read. That is what makes the schema tolerant of model evolution: + adding a field to a MAIL model needs a schema change only if the new field + must become queryable. + +``mailbox_items`` and ``message_buffer`` carry no JSON body and have no MAIL +model — they are pure membership / ordering rows constructed directly by the +repositories, so they have no serializer here. + +See ``src/mail/server/docs/reference/backends.md`` for the backend overview. +""" + +from __future__ import annotations + +from typing import Any + +from mail_protocol.core.drafts import MAILDraftsEntry +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.outbox import MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.trash import MAILTrashEntry +from mail_protocol.core.user_agents import MAILUserAgentInBackend +from mail_protocol.core.webhooks import MAILWebhook + +from mail_server.backends.sqlite.schema import ( + DraftEntryRow, + InboxEntryRow, + ListRow, + MessageRow, + OutboxEntryRow, + SwarmRow, + TrashEntryRow, + UserAgentRow, + WebhookRow, +) + +# --------------------------------------------------------------------------- # +# user_agents +# --------------------------------------------------------------------------- # + + +def user_agent_to_columns(model: MAILUserAgentInBackend) -> dict[str, Any]: + ua = model.user_agent + return { + "address": model.get_address(), + "ua_type": ua.ua_type, + # ``swarm`` is meaningful only for agents; users/admins/daemons have none. + "swarm": getattr(ua, "swarm", None), + "host": ua.host, + "hashed_password": model.hashed_password, + "body": model.model_dump(mode="json"), + } + + +def user_agent_from_row(row: UserAgentRow) -> MAILUserAgentInBackend: + return MAILUserAgentInBackend.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# swarms +# --------------------------------------------------------------------------- # + + +def swarm_to_columns(model: MAILSwarm) -> dict[str, Any]: + return { + "name": model.name, + "body": model.model_dump(mode="json"), + } + + +def swarm_from_row(row: SwarmRow) -> MAILSwarm: + return MAILSwarm.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# messages +# --------------------------------------------------------------------------- # + + +def message_to_columns(model: MAILMessage) -> dict[str, Any]: + return { + "message_id": model.message_id, + "sender": model.sender, + "subject": model.subject, + "reply_to": model.reply_to, + "sent_at": model.sent_at, + "body": model.model_dump(mode="json"), + } + + +def message_from_row(row: MessageRow) -> MAILMessage: + return MAILMessage.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# inbox_entries (shared summary, keyed by message id) +# --------------------------------------------------------------------------- # + + +def inbox_entry_to_columns(model: MAILInboxEntrySummary) -> dict[str, Any]: + return { + "message_id": model.message_id, + "sender": model.sender, + "subject": model.subject, + "body_size": model.body_size, + "received_at": model.received_at, + "delivered_by": model.delivered_by, + "body": model.model_dump(mode="json"), + } + + +def inbox_entry_from_row(row: InboxEntryRow) -> MAILInboxEntrySummary: + return MAILInboxEntrySummary.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# outbox_entries (shared summary, keyed by message id) +# --------------------------------------------------------------------------- # + + +def outbox_entry_to_columns(model: MAILOutboxEntrySummary) -> dict[str, Any]: + return { + "message_id": model.message_id, + "sent_at": model.sent_at, + "delivered_at": model.delivered_at, + "delivered_by": model.delivered_by, + "body": model.model_dump(mode="json"), + } + + +def outbox_entry_from_row(row: OutboxEntryRow) -> MAILOutboxEntrySummary: + return MAILOutboxEntrySummary.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# draft_entries +# --------------------------------------------------------------------------- # + + +def draft_entry_to_columns(model: MAILDraftsEntry) -> dict[str, Any]: + return { + "draft_id": model.draft.draft_id, + "created_at": model.draft.created_at, + "updated_at": model.draft.updated_at, + "body": model.model_dump(mode="json"), + } + + +def draft_entry_from_row(row: DraftEntryRow) -> MAILDraftsEntry: + return MAILDraftsEntry.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# trash_entries (shared, keyed by message id) +# --------------------------------------------------------------------------- # + + +def trash_entry_to_columns(model: MAILTrashEntry) -> dict[str, Any]: + return { + "message_id": model.message.message_id, + "trashed_at": model.trashed_at, + "body": model.model_dump(mode="json"), + } + + +def trash_entry_from_row(row: TrashEntryRow) -> MAILTrashEntry: + return MAILTrashEntry.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# webhooks (keyed by URL) +# --------------------------------------------------------------------------- # + + +def webhook_to_columns(model: MAILWebhook) -> dict[str, Any]: + return { + "url": model.url, + "webhook_id": model.webhook_id, + "body": model.model_dump(mode="json"), + } + + +def webhook_from_row(row: WebhookRow) -> MAILWebhook: + return MAILWebhook.model_validate(row.body) + + +# --------------------------------------------------------------------------- # +# lists (members live inside the body) +# --------------------------------------------------------------------------- # + + +def list_to_columns(model: MAILListInBackend) -> dict[str, Any]: + return { + "address": model.get_address(), + "list_id": model.list_id, + "swarm": model.swarm, + "host": model.host, + "created_at": model.created_at, + "updated_at": model.updated_at, + "body": model.model_dump(mode="json"), + } + + +def list_from_row(row: ListRow) -> MAILListInBackend: + return MAILListInBackend.model_validate(row.body) diff --git a/src/mail/server/src/mail_server/cli.py b/src/mail/server/src/mail_server/cli.py index ae9c1a7..3fc1bf8 100644 --- a/src/mail/server/src/mail_server/cli.py +++ b/src/mail/server/src/mail_server/cli.py @@ -12,6 +12,9 @@ "mail-server --host 0.0.0.0 --port 8865", "mail-server --backend memory", "mail-server --memory-save-interval 30", + "mail-server --backend sqlite", + "mail-server --backend sqlite --sqlite-path /var/lib/mail/mail.db", + "mail-server --backend sqlite --database-url sqlite:////abs/path/mail.db", ] DEFAULT_MEMORY_SAVE_INTERVAL_SECONDS = 60.0 @@ -60,7 +63,7 @@ def build_parser() -> argparse.ArgumentParser: "-b", "--backend", metavar="BACKEND", - choices=["memory"], + choices=["memory", "sqlite"], default="memory", help="the MAIL server backend to use (default: %(default)s)", ) @@ -74,6 +77,24 @@ def build_parser() -> argparse.ArgumentParser: "set 0 to disable (default: %(default)s)" ), ) + parser.add_argument( + "--sqlite-path", + metavar="PATH", + default=os.getenv("MAIL_SQLITE_PATH"), + help=( + "sqlite backend database file (env: MAIL_SQLITE_PATH; default: " + "~/.mail-swarms/deployments/default/mail.db)" + ), + ) + parser.add_argument( + "--database-url", + metavar="URL", + default=os.getenv("MAIL_DATABASE_URL"), + help=( + "sqlite backend database URL; takes precedence over --sqlite-path " + "(env: MAIL_DATABASE_URL)" + ), + ) return parser diff --git a/src/mail/server/src/mail_server/routers/daemon.py b/src/mail/server/src/mail_server/routers/daemon.py index 85ac0f6..8d2d772 100644 --- a/src/mail/server/src/mail_server/routers/daemon.py +++ b/src/mail/server/src/mail_server/routers/daemon.py @@ -9,7 +9,10 @@ ) from mail_server.auth import validate_daemon -from mail_server.validators import validate_deliver_local_request +from mail_server.validators import ( + validate_deliver_local_request, + validate_deliver_remote_request, +) router = APIRouter(prefix="/daemon", tags=["daemon"]) @@ -53,4 +56,11 @@ async def deliver_local_messages(request: Request) -> DaemonDeliverLocalResponse response_model=DaemonDeliverRemoteResponse, ) async def deliver_remote_messages(request: Request) -> DaemonDeliverRemoteResponse: - raise NotImplementedError + backend = request.app.state.backend + daemon = await validate_daemon(backend=backend, request=request) + payload = await validate_deliver_remote_request(request=request) + result = await backend.daemon_deliver_remote(daemon=daemon, payload=payload) + return DaemonDeliverRemoteResponse( + messages=result, + metadata={}, + ) diff --git a/src/mail/server/src/mail_server/routers/drafts.py b/src/mail/server/src/mail_server/routers/drafts.py index a3e7eb7..9a1f670 100644 --- a/src/mail/server/src/mail_server/routers/drafts.py +++ b/src/mail/server/src/mail_server/routers/drafts.py @@ -114,7 +114,20 @@ async def patch_draft(request: Request) -> DraftPatchResponse: response_model=DraftDeleteResponse, ) async def delete_draft(request: Request) -> DraftDeleteResponse: - raise NotImplementedError + backend = request.app.state.backend + user_agent = await validate_user_agent(backend=backend, request=request) + draft_id = request.path_params.get("draft_id") + try: + result = await backend.delete_draft(user_agent=user_agent, draft_id=draft_id) + except ValueError: + raise HTTPException( + status_code=404, detail=f"draft with ID {draft_id} not found" + ) + + return DraftDeleteResponse( + entry=result, + metadata={}, + ) @router.post( diff --git a/src/mail/server/src/mail_server/routers/inbox.py b/src/mail/server/src/mail_server/routers/inbox.py index e754a5c..3702737 100644 --- a/src/mail/server/src/mail_server/routers/inbox.py +++ b/src/mail/server/src/mail_server/routers/inbox.py @@ -68,4 +68,19 @@ async def open_inbox_message(request: Request) -> InboxMessageGetResponse: response_model=InboxMessageDeleteResponse, ) async def delete_inbox_message(request: Request) -> InboxMessageDeleteResponse: - raise NotImplementedError + backend = request.app.state.backend + user_agent = await validate_user_agent(backend=backend, request=request) + message_id = request.path_params.get("message_id") + try: + result = await backend.delete_inbox_message( + user_agent=user_agent, message_id=message_id + ) + except ValueError: + raise HTTPException( + status_code=404, detail=f"message with ID {message_id} not found in inbox" + ) + + return InboxMessageDeleteResponse( + entry=result, + metadata={}, + ) diff --git a/src/mail/server/src/mail_server/routers/trash.py b/src/mail/server/src/mail_server/routers/trash.py index 2a2f854..6451b8d 100644 --- a/src/mail/server/src/mail_server/routers/trash.py +++ b/src/mail/server/src/mail_server/routers/trash.py @@ -67,8 +67,25 @@ async def get_trashed_message(request: Request) -> TrashMessageGetResponse: summary="Delete a specific trashed message by ID", response_model=TrashMessageDeleteResponse, ) -async def delete_trashed_message(message_id: str) -> TrashMessageDeleteResponse: - raise NotImplementedError +async def delete_trashed_message( + request: Request, message_id: str +) -> TrashMessageDeleteResponse: + backend = request.app.state.backend + user_agent = await validate_user_agent(backend=backend, request=request) + try: + result = await backend.delete_trash_message( + user_agent=user_agent, message_id=message_id + ) + except ValueError: + raise HTTPException( + status_code=404, + detail=f"no message with ID {message_id} found in trash box", + ) + + return TrashMessageDeleteResponse( + entry=result, + metadata={}, + ) @router.post( @@ -76,5 +93,12 @@ async def delete_trashed_message(message_id: str) -> TrashMessageDeleteResponse: summary="Remove all exisisting messages from trash", response_model=TrashClearPostResponse, ) -async def post_trash_clear() -> TrashClearPostResponse: - raise NotImplementedError +async def post_trash_clear(request: Request) -> TrashClearPostResponse: + backend = request.app.state.backend + user_agent = await validate_user_agent(backend=backend, request=request) + result = await backend.clear_trash(user_agent=user_agent) + + return TrashClearPostResponse( + entries=result, + metadata={}, + ) diff --git a/src/mail/server/src/mail_server/server.py b/src/mail/server/src/mail_server/server.py index 27b2372..8659393 100644 --- a/src/mail/server/src/mail_server/server.py +++ b/src/mail/server/src/mail_server/server.py @@ -6,6 +6,7 @@ import time from argparse import Namespace from contextlib import asynccontextmanager +from pathlib import Path import uvicorn from fastapi import FastAPI @@ -125,17 +126,45 @@ async def get_health() -> HealthGetResponse: # +def _resolve_sqlite_url(args: Namespace) -> str: + """ + Resolve the sqlite backend database URL from CLI args / env. + + Precedence: ``--database-url`` (full URL) > ``--sqlite-path`` (file path) > + the per-deployment default ``~/.mail-swarms/deployments/default/mail.db``. + Env fallbacks (``MAIL_DATABASE_URL`` / ``MAIL_SQLITE_PATH``) are applied as + the argparse defaults in ``cli.py``. + """ + + url = getattr(args, "database_url", None) + if url: + return url + path = getattr(args, "sqlite_path", None) + if path: + return f"sqlite:///{Path(path).expanduser()}" + + # Lazy import keeps SQLAlchemy off the import path for memory-only runs. + from mail_server.backends.sqlite.init import default_sqlite_path + + return f"sqlite:///{default_sqlite_path()}" + + def run_server(args: Namespace) -> None: """ Run the MAIL server from the CLI. """ + global _backend match args.backend: case "memory" | "mem": - global _backend _backend = MemoryBackend( persistence_interval_seconds=getattr(args, "memory_save_interval", 0) ) + case "sqlite": + # Lazy import so SQLAlchemy is only loaded when actually selected. + from mail_server.backends.sqlite.api import SQLiteBackend + + _backend = SQLiteBackend(url=_resolve_sqlite_url(args)) case _: raise ValueError(f"invalid backend type: {args.backend}") diff --git a/src/mail/server/src/mail_server/validators.py b/src/mail/server/src/mail_server/validators.py index 4ce6f36..9b182bb 100644 --- a/src/mail/server/src/mail_server/validators.py +++ b/src/mail/server/src/mail_server/validators.py @@ -14,6 +14,7 @@ AuthPasswordResetRequest, BoxFilterParams, DaemonDeliverLocalRequest, + DaemonDeliverRemoteRequest, DraftPatchRequest, DraftPostRequest, DraftSendPostRequest, @@ -110,6 +111,22 @@ async def validate_deliver_local_request( ) +async def validate_deliver_remote_request( + request: Request, +) -> DaemonDeliverRemoteRequest: + """ + Ensure that the request payload is valid for `POST /daemon/deliver/remote`. + """ + + try: + body = await request.json() + return DaemonDeliverRemoteRequest.model_validate(body) + except ValueError as e: + raise HTTPException( + status_code=422, detail=f"request body validation failed: {e}" + ) + + # # Admin endpoint validators # diff --git a/tests/e2e/conftest.py b/tests/e2e/conftest.py index 0c7a9d9..2d6d573 100644 --- a/tests/e2e/conftest.py +++ b/tests/e2e/conftest.py @@ -42,6 +42,7 @@ class E2EStack: def __init__(self, home: Path) -> None: self.home = home + self.backend = "memory" self.port = _free_port() self.base_url = f"http://127.0.0.1:{self.port}" self.env = { @@ -57,10 +58,13 @@ def __init__(self, home: Path) -> None: # ─── provisioning and lifecycle ──────────────────────────────── - def provision(self) -> None: + def provision(self, backend: str = "memory") -> None: + self.backend = backend subprocess.run( [ str(VENV_BIN / "backend-init"), + "--type", + backend, "--swarm", SWARM, "--host", @@ -93,6 +97,8 @@ def start_server( ) -> None: command = [ str(VENV_BIN / "mail-server"), + "--backend", + self.backend, "--host", "127.0.0.1", "--port", @@ -217,3 +223,16 @@ def e2e_stack(tmp_path: Path) -> E2EStack: stack.start_server() yield stack stack.stop_server() + + +@pytest.fixture +def sqlite_e2e_stack(tmp_path: Path) -> E2EStack: + """An e2e stack provisioned and served on the sqlite backend.""" + + home = tmp_path / "home" + home.mkdir() + stack = E2EStack(home) + stack.provision(backend="sqlite") + stack.start_server() + yield stack + stack.stop_server() diff --git a/tests/e2e/test_sqlite_durability.py b/tests/e2e/test_sqlite_durability.py new file mode 100644 index 0000000..e5ffaeb --- /dev/null +++ b/tests/e2e/test_sqlite_durability.py @@ -0,0 +1,44 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +The durability property the sqlite backend exists to provide: a committed +message survives an abrupt ``kill -9`` with no graceful shutdown — the window +the memory backend's checkpoint cannot close. +""" + +HOST = "localhost" +USER = f"user:alice@{HOST}" +OTHER_USER = f"user:bob@{HOST}" + + +def test_sqlite_committed_message_survives_sigkill(sqlite_e2e_stack) -> None: + stack = sqlite_e2e_stack + alice = stack.login(USER) + bob = stack.login(OTHER_USER) + + draft = stack.cli_json( + "compose", "Durable", "Survives kill -9.", token=alice + ) + sent = stack.cli_json( + "send", draft["entry"]["draft"]["draft_id"], OTHER_USER, token=alice + ) + message_id = sent["message"]["message_id"] + + def bob_has_mail() -> bool: + inbox = stack.cli_json("inbox", token=bob) + return any(e["message_id"] == message_id for e in inbox["entries"]) + + with stack.daemon_running(): + stack.wait_for(bob_has_mail) + + # SIGKILL: no lifespan shutdown, no checkpoint — only what is already + # committed to the sqlite file can survive. + stack.kill_server() + stack.start_server() + + # JWTs are stateless, so the tokens remain valid across the restart. + opened = stack.cli_json("inbox-open", message_id, token=bob) + assert opened["entry"]["message"]["body"] == "Survives kill -9." + outbox = stack.cli_json("outbox", token=alice) + assert any(e["message_id"] == message_id for e in outbox["entries"]) diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 5c94c85..707c210 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -1,7 +1,10 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Charon Labs (contribution PR) +import asyncio import os +from collections.abc import Awaitable, Callable, Iterator +from datetime import datetime from pathlib import Path # mail_server.server reads MAIL_HOST and mail_server.routers.auth reads @@ -12,7 +15,10 @@ import pytest # noqa: E402 from fastapi.testclient import TestClient # noqa: E402 +from mail_protocol.core.lists import MAILListInBackend # noqa: E402 +from mail_protocol.core.messages import MAILMessage # noqa: E402 from mail_protocol.core.swarms import MAILSwarm # noqa: E402 +from mail_protocol.core.trash import MAILTrashEntry # noqa: E402 from mail_protocol.core.user_agents import ( # noqa: E402 MAILAdmin, MAILAgent, @@ -22,7 +28,14 @@ ) from mail_server import server as mail_server_module # noqa: E402 from mail_server.auth import get_password_hash # noqa: E402 +from mail_server.backends.base import MAILServerBackend # noqa: E402 from mail_server.backends.memory.api import MemoryBackend # noqa: E402 +from mail_server.backends.sqlite.api import SQLiteBackend # noqa: E402 +from mail_server.backends.sqlite.database import Database # noqa: E402 +from mail_server.backends.sqlite.repositories import ( # noqa: E402 + BOX_TRASH, + MailStore, +) HOST = "localhost" SWARM = "chorus" @@ -39,32 +52,33 @@ # per session instead of once per seeded user-agent per test. PASSWORD_HASH = get_password_hash(PASSWORD) +# The integration suite runs against every backend in this list. Backend +# internals are never touched directly — seeding/assertions go through the +# public API or the backend-agnostic ``seed_*`` fixtures below — so each test +# exercises identical behavior on memory and sqlite. +BACKENDS = ["memory", "sqlite"] -def _seed_cast(backend: MemoryBackend) -> None: - """ - Seed the standard cast: one admin, two users, one agent, one daemon, - and one swarm — all sharing PASSWORD. Mirrors what `backend-init` - plus admin CRUD calls would provision. - """ - cast = { +def _cast() -> dict[str, MAILUserAgentInBackend]: + """The standard cast (one admin, two users, one agent, one daemon).""" + + members = { ADMIN: MAILAdmin(ua_type="admin", admin_id="ryan", host=HOST), USER: MAILUser(ua_type="user", user_id="alice", host=HOST), OTHER_USER: MAILUser(ua_type="user", user_id="bob", host=HOST), AGENT: MAILAgent(ua_type="agent", name="sage", swarm=SWARM, host=HOST), DAEMON: MAILDaemon(ua_type="daemon", worker_name="dummy", host=HOST), } - for address, user_agent in cast.items(): - backend.user_agents[address] = MAILUserAgentInBackend( - user_agent=user_agent, - hashed_password=PASSWORD_HASH, + return { + address: MAILUserAgentInBackend( + user_agent=user_agent, hashed_password=PASSWORD_HASH ) - backend.inboxes[address] = [] - backend.outboxes[address] = [] - backend.drafts[address] = [] - backend.trashes[address] = [] + for address, user_agent in members.items() + } + - backend.swarms[SWARM] = MAILSwarm( +def _swarm() -> MAILSwarm: + return MAILSwarm( name=SWARM, description="integration test swarm", keywords=["testing"], @@ -73,28 +87,159 @@ def _seed_cast(backend: MemoryBackend) -> None: ) +def _seed_memory_cast(backend: MemoryBackend) -> None: + for address, ua_in_be in _cast().items(): + backend.user_agents[address] = ua_in_be + backend.inboxes[address] = [] + backend.outboxes[address] = [] + backend.drafts[address] = [] + backend.trashes[address] = [] + backend.swarms[SWARM] = _swarm() + + +def _run_sqlite_write( + url: str, + mutate: Callable[[MailStore], Awaitable[object]], + *, + create_schema: bool = False, +) -> None: + """ + Apply a write to a file-backed sqlite db via a throwaway engine. + + Per-test seeding can't reuse the app's engine (it is bound to the + TestClient's event loop), so we open a short-lived ``Database`` on the same + file in a fresh loop. WAL makes the committed rows visible to the app, and + seeding is sequential with the HTTP calls, so there is no write contention. + """ + + async def _run() -> None: + db = Database(url) + try: + if create_schema: + await db.create_schema() + async with db.session() as session: + await mutate(MailStore(session)) + finally: + await db.dispose() + + asyncio.run(_run()) + + +async def _seed_sqlite_cast(store: MailStore) -> None: + for ua_in_be in _cast().values(): + await store.user_agents.add(ua_in_be) + await store.swarms.add(_swarm()) + + +@pytest.fixture(params=BACKENDS) +def backend_kind(request: pytest.FixtureRequest) -> str: + """The backend under test for this parametrization (``memory``/``sqlite``).""" + + return request.param + + @pytest.fixture def app_client( + backend_kind: str, deployment_dir: Path, + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, -) -> TestClient: +) -> Iterator[TestClient]: """ - The real composed FastAPI app over ASGI with a fresh MemoryBackend, - seeded with the standard cast. Auth is NOT monkeypatched — requests - must carry real JWTs (see ``token_for`` / ``headers_for``). + The real composed FastAPI app over ASGI, seeded with the standard cast on + the selected backend. Auth is NOT monkeypatched — requests must carry real + JWTs (see ``token_for`` / ``headers_for``). """ - monkeypatch.setattr(mail_server_module, "_backend", MemoryBackend()) + if backend_kind == "memory": + monkeypatch.setattr(mail_server_module, "_backend", MemoryBackend()) + with TestClient(mail_server_module.app) as client: + _seed_memory_cast(mail_server_module.app.state.backend) + yield client + return + + # sqlite: seed the cast (and create the schema) before the app starts, so + # the rows are already committed when the lifespan runs. + db_url = f"sqlite:///{tmp_path / 'mail.db'}" + _run_sqlite_write(db_url, _seed_sqlite_cast, create_schema=True) + monkeypatch.setattr(mail_server_module, "_backend", SQLiteBackend(url=db_url)) with TestClient(mail_server_module.app) as client: - _seed_cast(client.app.state.backend) yield client @pytest.fixture -def backend(app_client: TestClient) -> MemoryBackend: +def backend(app_client: TestClient) -> MAILServerBackend: """The backend behind ``app_client`` (overrides the root fixture).""" - return app_client.app.state.backend + backend: MAILServerBackend = mail_server_module.app.state.backend + return backend + + +@pytest.fixture +def seed_trash(backend: MAILServerBackend) -> Callable[..., str]: + """ + Backend-agnostic: place ``message`` directly in ``owner``'s trash with the + given ``trashed_at`` (and register the message in the canonical store, which + the ``sent_at`` sort resolves against). Returns the message id. + + Bypassing the API is deliberate — it lets a test pin ``trashed_at`` and the + message's ``sent_at`` independently, which the natural inbox→trash flow + (both stamped with the wall clock) cannot. + """ + + def _seed(owner: str, *, message: MAILMessage, trashed_at: datetime) -> str: + entry = MAILTrashEntry(message=message, trashed_at=trashed_at) + if isinstance(backend, MemoryBackend): + backend.messages[message.message_id] = message + backend.trash_entries[message.message_id] = entry + backend.trashes.setdefault(owner, []).append(message.message_id) + else: + assert isinstance(backend, SQLiteBackend) + + async def mutate(store: MailStore) -> None: + await store.messages.add(message) + await store.boxes.upsert_trash_entry(entry) + await store.boxes.add_membership( + owner, BOX_TRASH, message.message_id, trashed_at + ) + + _run_sqlite_write(backend._db.url, mutate) + return message.message_id + + return _seed + + +@pytest.fixture +def seed_list(backend: MAILServerBackend) -> Callable[[MAILListInBackend], str]: + """Backend-agnostic: persist a prebuilt list. Returns its address.""" + + def _seed(record: MAILListInBackend) -> str: + if isinstance(backend, MemoryBackend): + backend.lists[record.get_address()] = record + else: + assert isinstance(backend, SQLiteBackend) + + async def mutate(store: MailStore) -> None: + await store.lists.add(record) + + _run_sqlite_write(backend._db.url, mutate) + return record.get_address() + + return _seed + + +@pytest.fixture +def list_members( + app_client: TestClient, headers_for: Callable[..., dict[str, str]] +) -> Callable[..., list[str]]: + """Read a list's members through the public API (backend-agnostic).""" + + def _members(address: str, viewer: str = USER) -> list[str]: + response = app_client.get(f"/lists/{address}", headers=headers_for(viewer)) + assert response.status_code == 200, response.text + return response.json()["mail_list"]["members"] + + return _members @pytest.fixture diff --git a/tests/integration/test_admin.py b/tests/integration/test_admin.py index 3213264..b1f0d7f 100644 --- a/tests/integration/test_admin.py +++ b/tests/integration/test_admin.py @@ -2,7 +2,6 @@ # Copyright (c) 2026 Charon Labs (contribution PR) from fastapi.testclient import TestClient -from mail_server.backends.memory.api import MemoryBackend ADMIN = "admin:ryan@localhost" SWARM = "chorus" @@ -81,18 +80,20 @@ def test_post_agent_invalid_name_returns_422( def test_delete_agent_removes_account_and_boxes( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: address = f"sage@{SWARM}@localhost" response = app_client.delete( f"/admin/agents/sage@{SWARM}", headers=headers_for(ADMIN) ) assert response.status_code == 200 - assert address not in backend.user_agents - assert address not in backend.inboxes - assert address not in backend.outboxes - assert address not in backend.drafts - assert address not in backend.trashes + # The account is gone: a follow-up admin read 404s. + assert ( + app_client.get( + f"/admin/agents/sage@{SWARM}", headers=headers_for(ADMIN) + ).status_code + == 404 + ) # Credentials no longer authenticate. response = app_client.post( @@ -152,11 +153,17 @@ def test_post_daemon_duplicate_returns_409(app_client: TestClient, headers_for) def test_delete_daemon_removes_account( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: response = app_client.delete("/admin/daemons/dummy", headers=headers_for(ADMIN)) assert response.status_code == 200 - assert "daemon:dummy@localhost" not in backend.user_agents + # The account is gone: a follow-up admin read 404s. + assert ( + app_client.get( + "/admin/daemons/dummy", headers=headers_for(ADMIN) + ).status_code + == 404 + ) def test_delete_daemon_unknown_returns_404(app_client: TestClient, headers_for) -> None: @@ -208,12 +215,17 @@ def test_post_user_duplicate_returns_409(app_client: TestClient, headers_for) -> def test_delete_user_removes_account_and_boxes( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: response = app_client.delete("/admin/users/bob", headers=headers_for(ADMIN)) assert response.status_code == 200 - assert "user:bob@localhost" not in backend.user_agents - assert "user:bob@localhost" not in backend.inboxes + # The account is gone: a follow-up admin read 404s. + assert ( + app_client.get( + "/admin/users/bob", headers=headers_for(ADMIN) + ).status_code + == 404 + ) def test_delete_user_unknown_returns_404(app_client: TestClient, headers_for) -> None: diff --git a/tests/integration/test_daemon.py b/tests/integration/test_daemon.py index 89e101e..dea499a 100644 --- a/tests/integration/test_daemon.py +++ b/tests/integration/test_daemon.py @@ -2,7 +2,6 @@ # Copyright (c) 2026 Charon Labs (contribution PR) from fastapi.testclient import TestClient -from mail_server.backends.memory.api import MemoryBackend USER = "user:alice@localhost" OTHER_USER = "user:bob@localhost" @@ -42,7 +41,7 @@ def test_clear_message_buffer_returns_pending_ids_once( def test_deliver_local_updates_recipient_inbox( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: message_id = _compose_and_send(app_client, headers_for(USER)) daemon_headers = headers_for(DAEMON) @@ -58,10 +57,15 @@ def test_deliver_local_updates_recipient_inbox( assert len(summaries) == 1 assert summaries[0]["message_id"] == message_id - assert message_id in backend.inboxes[OTHER_USER] - outbox_entry = backend.outbox_entries[message_id] - assert outbox_entry.delivered_at is not None - assert outbox_entry.delivered_by == DAEMON + # The recipient can now open it from their inbox... + opened = app_client.get( + f"/inbox/{message_id}", headers=headers_for(OTHER_USER) + ) + assert opened.status_code == 200 + # ...and the sender's outbox entry is marked delivered. + outbox = app_client.get(f"/outbox/{message_id}", headers=headers_for(USER)) + assert outbox.status_code == 200 + assert outbox.json()["entry"]["delivered_at"] is not None def test_deliver_local_skips_unknown_message_ids( @@ -88,7 +92,7 @@ def test_deliver_local_rejects_non_uuid_ids( def test_deliver_local_skips_unknown_recipient( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: """ A recipient address that doesn't resolve to a registered user-agent @@ -117,5 +121,8 @@ def test_deliver_local_skips_unknown_recipient( headers=daemon_headers, ) assert response.status_code == 200 - assert message_id in backend.inboxes[OTHER_USER] - assert "user:ghost@localhost" not in backend.inboxes + # The known recipient received it; the unknown one was skipped without error. + opened = app_client.get( + f"/inbox/{message_id}", headers=headers_for(OTHER_USER) + ) + assert opened.status_code == 200 diff --git a/tests/integration/test_gap_fill.py b/tests/integration/test_gap_fill.py new file mode 100644 index 0000000..189f0c9 --- /dev/null +++ b/tests/integration/test_gap_fill.py @@ -0,0 +1,137 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +End-to-end HTTP coverage for the endpoints that are ``NotImplementedError`` +stubs on the memory backend but fully implemented on sqlite (the routers were +wired to delegate to the backend). Pinned to the sqlite backend; the memory +backend's stub behavior is asserted in ``test_stubs.py``. +""" + +import pytest +from fastapi.testclient import TestClient + +USER = "user:alice@localhost" +OTHER_USER = "user:bob@localhost" +DAEMON = "daemon:dummy@localhost" +ADMIN = "admin:ryan@localhost" + + +@pytest.fixture +def backend_kind() -> str: + """Pin this suite to sqlite, where these endpoints are implemented.""" + + return "sqlite" + + +def test_delete_inbox_message_moves_to_trash( + app_client: TestClient, headers_for, deliver_message +) -> None: + message_id = deliver_message(USER, [OTHER_USER]) + bob = headers_for(OTHER_USER) + + response = app_client.delete(f"/inbox/{message_id}", headers=bob) + assert response.status_code == 200 + + # Gone from the inbox, now readable from trash. + assert app_client.get(f"/inbox/{message_id}", headers=bob).status_code == 404 + assert app_client.get(f"/trash/{message_id}", headers=bob).status_code == 200 + + +def test_delete_draft_removes_it(app_client: TestClient, headers_for) -> None: + headers = headers_for(USER) + response = app_client.post( + "/drafts", + json={"subject": "Disposable", "body": "Delete me"}, + headers=headers, + ) + draft_id = response.json()["entry"]["draft"]["draft_id"] + + response = app_client.delete(f"/drafts/{draft_id}", headers=headers) + assert response.status_code == 200 + assert app_client.get(f"/drafts/{draft_id}", headers=headers).status_code == 404 + + +def test_delete_trashed_message_removes_it( + app_client: TestClient, headers_for, deliver_message +) -> None: + message_id = deliver_message(USER, [OTHER_USER]) + bob = headers_for(OTHER_USER) + app_client.delete(f"/inbox/{message_id}", headers=bob) # inbox -> trash + + response = app_client.delete(f"/trash/{message_id}", headers=bob) + assert response.status_code == 200 + assert app_client.get(f"/trash/{message_id}", headers=bob).status_code == 404 + + +def test_trash_clear_empties_box( + app_client: TestClient, headers_for, deliver_message +) -> None: + bob = headers_for(OTHER_USER) + for _ in range(2): + message_id = deliver_message(USER, [OTHER_USER]) + app_client.delete(f"/inbox/{message_id}", headers=bob) + + response = app_client.post("/trash/clear", headers=bob) + assert response.status_code == 200 + assert len(response.json()["entries"]) == 2 + assert app_client.get("/trash", headers=bob).json()["entries"] == [] + + +def test_daemon_deliver_remote_delivers_to_local_inbox( + app_client: TestClient, headers_for +) -> None: + message_id = "99999999-9999-4999-8999-999999999999" + response = app_client.post( + "/daemon/deliver/remote", + json={ + "messages": [ + { + "mail_version": "2.0", + "message_id": message_id, + "sender": "echo@otherswarm@remote.example.com", + "recipients": [OTHER_USER], + "subject": "Remote", + "body": "from afar", + "tags": [], + "sent_at": "2026-06-24T00:00:00Z", + "metadata": {}, + } + ] + }, + headers=headers_for(DAEMON), + ) + assert response.status_code == 200 + assert response.json()["messages"][0]["message_id"] == message_id + + opened = app_client.get(f"/inbox/{message_id}", headers=headers_for(OTHER_USER)) + assert opened.status_code == 200 + assert opened.json()["entry"]["message"]["body"] == "from afar" + + +def test_patch_webhook_updates(app_client: TestClient, headers_for) -> None: + headers = headers_for(ADMIN) + response = app_client.post( + "/admin/webhooks", + json={ + "url": "https://example.com/mail-events", + "events": ["mail.delivered"], + "secret": "shhh", + }, + headers=headers, + ) + webhook_id = response.json()["webhook"]["webhook_id"] + + response = app_client.patch( + f"/admin/webhooks/{webhook_id}", + json={"url": "https://example.com/elsewhere", "secret": "new-secret"}, + headers=headers, + ) + assert response.status_code == 200 + body = response.json()["webhook"] + assert body["webhook_id"] == webhook_id # id preserved across URL move + assert body["url"] == "https://example.com/elsewhere" + + # Refetch by id reflects the new URL. + refetched = app_client.get(f"/admin/webhooks/{webhook_id}", headers=headers) + assert refetched.json()["webhook"]["url"] == "https://example.com/elsewhere" diff --git a/tests/integration/test_lists.py b/tests/integration/test_lists.py index 02d77ff..2a8d13d 100644 --- a/tests/integration/test_lists.py +++ b/tests/integration/test_lists.py @@ -8,11 +8,11 @@ minimal app with monkeypatched auth; the assertions are unchanged. """ +from collections.abc import Callable from datetime import UTC, datetime from fastapi.testclient import TestClient from mail_protocol.core.lists import MAILListInBackend, MAILListPolicy -from mail_server.backends.memory.api import MemoryBackend ADMIN_ADDRESS = "admin:ryan@localhost" USER_ADDRESS = "user:alice@localhost" @@ -35,7 +35,11 @@ def _make_post_body( return body -def _seed_list(backend: MemoryBackend, *, members: list[str] | None = None) -> str: +def _seed_list( + seed_list: Callable[[MAILListInBackend], str], + *, + members: list[str] | None = None, +) -> str: now = datetime(2026, 6, 4, 0, 0, tzinfo=UTC) record = MAILListInBackend( name="welfare-discourse", @@ -48,8 +52,7 @@ def _seed_list(backend: MemoryBackend, *, members: list[str] | None = None) -> s created_at=now, updated_at=now, ) - backend.lists[record.get_address()] = record - return record.get_address() + return seed_list(record) # ─── Admin endpoints ─────────────────────────────────────────────── @@ -115,18 +118,22 @@ def test_admin_post_list_duplicate_returns_409( def test_admin_get_lists_returns_all( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - _seed_list(backend) + _seed_list(seed_list) response = app_client.get("/admin/lists", headers=headers_for(ADMIN_ADDRESS)) assert response.status_code == 200 assert len(response.json()["lists"]) == 1 def test_admin_get_list_returns_specific( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.get( f"/admin/lists/{address}", headers=headers_for(ADMIN_ADDRESS) ) @@ -145,9 +152,11 @@ def test_admin_get_list_missing_returns_404( def test_admin_patch_list_updates_policy_no_op_for_open( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.patch( f"/admin/lists/{address}", json={"policy": MAILListPolicy().model_dump()}, @@ -157,9 +166,11 @@ def test_admin_patch_list_updates_policy_no_op_for_open( def test_admin_patch_list_rejects_closed_policy( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.patch( f"/admin/lists/{address}", json={"policy": MAILListPolicy(visibility="private").model_dump()}, @@ -169,14 +180,22 @@ def test_admin_patch_list_rejects_closed_policy( def test_admin_delete_list_removes( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.delete( f"/admin/lists/{address}", headers=headers_for(ADMIN_ADDRESS) ) assert response.status_code == 200 - assert address not in backend.lists + # The list is gone: a follow-up read 404s. + assert ( + app_client.get( + f"/lists/{address}", headers=headers_for(ADMIN_ADDRESS) + ).status_code + == 404 + ) def test_admin_delete_list_missing_returns_404( @@ -190,9 +209,11 @@ def test_admin_delete_list_missing_returns_404( def test_admin_add_member( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.post( f"/admin/lists/{address}/members", json={"member_address": "philosopher@chorus@localhost"}, @@ -203,21 +224,26 @@ def test_admin_add_member( def test_admin_remove_member( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: - address = _seed_list(backend, members=["philosopher@chorus@localhost"]) + address = _seed_list(seed_list, members=["philosopher@chorus@localhost"]) response = app_client.delete( f"/admin/lists/{address}/members/philosopher@chorus@localhost", headers=headers_for(ADMIN_ADDRESS), ) assert response.status_code == 200 - assert backend.lists[address].members == [] + assert list_members(address) == [] def test_admin_lists_reject_non_admin( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - _seed_list(backend) + _seed_list(seed_list) response = app_client.get( "/admin/lists", headers=headers_for(USER_ADDRESS) ) @@ -233,18 +259,22 @@ def test_get_lists_requires_auth(app_client: TestClient) -> None: def test_get_lists_returns_visible_lists( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - _seed_list(backend) + _seed_list(seed_list) response = app_client.get("/lists", headers=headers_for(USER_ADDRESS)) assert response.status_code == 200 assert len(response.json()["lists"]) == 1 def test_get_list_specific( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.get( f"/lists/{address}", headers=headers_for(USER_ADDRESS) ) @@ -263,18 +293,24 @@ def test_get_list_missing_returns_404( def test_subscribe_self( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.post( f"/lists/{address}/subscribe", headers=headers_for(USER_ADDRESS) ) assert response.status_code == 200 - assert USER_ADDRESS in backend.lists[address].members + assert USER_ADDRESS in list_members(address) def test_subscribe_ignores_supplied_member_address( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: """ Subscribe is body-less: only the authenticated caller is ever @@ -282,15 +318,16 @@ def test_subscribe_ignores_supplied_member_address( no effect on the member list. """ - address = _seed_list(backend) + address = _seed_list(seed_list) response = app_client.post( f"/lists/{address}/subscribe", json={"member_address": OTHER_USER_ADDRESS}, headers=headers_for(USER_ADDRESS), ) assert response.status_code == 200 - assert USER_ADDRESS in backend.lists[address].members - assert OTHER_USER_ADDRESS not in backend.lists[address].members + members = list_members(address) + assert USER_ADDRESS in members + assert OTHER_USER_ADDRESS not in members def test_subscribe_missing_list_returns_404( @@ -304,24 +341,32 @@ def test_subscribe_missing_list_returns_404( def test_unsubscribe_self( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: - address = _seed_list(backend, members=[USER_ADDRESS, OTHER_USER_ADDRESS]) + address = _seed_list(seed_list, members=[USER_ADDRESS, OTHER_USER_ADDRESS]) response = app_client.post( f"/lists/{address}/unsubscribe", headers=headers_for(USER_ADDRESS) ) assert response.status_code == 200 - assert USER_ADDRESS not in backend.lists[address].members - assert OTHER_USER_ADDRESS in backend.lists[address].members + members = list_members(address) + assert USER_ADDRESS not in members + assert OTHER_USER_ADDRESS in members def test_unsubscribe_uses_authenticated_user( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, + headers_for, + seed_list: Callable[[MAILListInBackend], str], + list_members: Callable[..., list[str]], ) -> None: - address = _seed_list(backend, members=[USER_ADDRESS, OTHER_USER_ADDRESS]) + address = _seed_list(seed_list, members=[USER_ADDRESS, OTHER_USER_ADDRESS]) response = app_client.post( f"/lists/{address}/unsubscribe", headers=headers_for(OTHER_USER_ADDRESS) ) assert response.status_code == 200 - assert USER_ADDRESS in backend.lists[address].members - assert OTHER_USER_ADDRESS not in backend.lists[address].members + members = list_members(address) + assert USER_ADDRESS in members + assert OTHER_USER_ADDRESS not in members diff --git a/tests/integration/test_mailboxes.py b/tests/integration/test_mailboxes.py index 24964a0..301cf68 100644 --- a/tests/integration/test_mailboxes.py +++ b/tests/integration/test_mailboxes.py @@ -1,17 +1,17 @@ # SPDX-License-Identifier: Apache-2.0 # Copyright (c) 2026 Charon Labs (contribution PR) +from collections.abc import Callable from datetime import UTC, datetime import pytest from fastapi.testclient import TestClient from mail_protocol.core.constants import MESSAGE_SUBJECT_LEN_MAX from mail_protocol.core.messages import MAILMessage -from mail_protocol.core.trash import MAILTrashEntry -from mail_server.backends.memory.api import MemoryBackend USER = "user:alice@localhost" OTHER_USER = "user:bob@localhost" +DAEMON = "daemon:dummy@localhost" # ─── Inbox ───────────────────────────────────────────────────────── @@ -199,7 +199,7 @@ def test_drafts_isolated_between_users(app_client: TestClient, headers_for) -> N def test_send_draft_creates_message_and_buffers_it( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for ) -> None: response = app_client.post( "/drafts", @@ -219,7 +219,13 @@ def test_send_draft_creates_message_and_buffers_it( assert message["recipients"] == [OTHER_USER] assert message["subject"] == "Outgoing" assert message["message_id"] != draft_id - assert message["message_id"] in backend.message_buffer + + # The message is queued for delivery: the daemon's buffer-clear returns it. + buffer = app_client.post( + "/daemon/message-buffer/clear", headers=headers_for(DAEMON) + ) + assert buffer.status_code == 200 + assert message["message_id"] in buffer.json()["message_ids"] def test_send_draft_rejects_empty_recipients( @@ -413,7 +419,7 @@ def test_patch_draft_then_send_uses_new_content( # ─── Trash ───────────────────────────────────────────────────────── -def _seed_trash(backend: MemoryBackend, owner: str) -> str: +def _seed_trash(seed_trash: Callable[..., str], owner: str) -> str: message = MAILMessage( mail_version="2.0", message_id="22222222-2222-4222-8222-222222222222", @@ -425,12 +431,11 @@ def _seed_trash(backend: MemoryBackend, owner: str) -> str: sent_at=datetime(2026, 6, 11, tzinfo=UTC), metadata={}, ) - backend.trash_entries[message.message_id] = MAILTrashEntry( + return seed_trash( + owner, message=message, trashed_at=datetime(2026, 6, 11, 12, 0, tzinfo=UTC), ) - backend.trashes[owner].append(message.message_id) - return message.message_id def test_trash_starts_empty(app_client: TestClient, headers_for) -> None: @@ -440,9 +445,9 @@ def test_trash_starts_empty(app_client: TestClient, headers_for) -> None: def test_trash_lists_seeded_entry( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - message_id = _seed_trash(backend, USER) + message_id = _seed_trash(seed_trash, USER) response = app_client.get("/trash", headers=headers_for(USER)) assert response.status_code == 200 entries = response.json()["entries"] @@ -451,9 +456,9 @@ def test_trash_lists_seeded_entry( def test_trash_open_returns_entry( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - message_id = _seed_trash(backend, USER) + message_id = _seed_trash(seed_trash, USER) response = app_client.get(f"/trash/{message_id}", headers=headers_for(USER)) assert response.status_code == 200 assert response.json()["entry"]["message"]["subject"] == "Trashed" @@ -468,9 +473,9 @@ def test_trash_open_unknown_id_returns_404(app_client: TestClient, headers_for) def test_trash_isolated_between_users( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - message_id = _seed_trash(backend, USER) + message_id = _seed_trash(seed_trash, USER) response = app_client.get(f"/trash/{message_id}", headers=headers_for(OTHER_USER)) assert response.status_code == 404 @@ -478,7 +483,7 @@ def test_trash_isolated_between_users( # ─── Box query parameters ────────────────────────────────────────── -def _seed_trash_n(backend: MemoryBackend, owner: str, n: int) -> list[str]: +def _seed_trash_n(seed_trash: Callable[..., str], owner: str, n: int) -> list[str]: """ Seed ``n`` trash entries for ``owner``. ``trashed_at`` *increases* with insertion order while the underlying message's ``sent_at`` *decreases*, so @@ -486,9 +491,9 @@ def _seed_trash_n(backend: MemoryBackend, owner: str, n: int) -> list[str]: orders — letting a single seed exercise both. Returns the message IDs in insertion order (oldest-trashed first), so ``ids[-1]`` is the newest. - The message is also registered in ``backend.messages`` because the - ``sent_at`` sort resolves send time via that store (as the real local - delivery path populates it). + The message is registered in the canonical store too (the ``sent_at`` sort + resolves send time via that store, as the real local delivery path does); + the ``seed_trash`` fixture handles that for whichever backend is active. """ ids: list[str] = [] @@ -505,12 +510,11 @@ def _seed_trash_n(backend: MemoryBackend, owner: str, n: int) -> list[str]: sent_at=datetime(2026, 6, 1, 12, n - i, tzinfo=UTC), # decreasing metadata={}, ) - backend.messages[message_id] = message - backend.trash_entries[message_id] = MAILTrashEntry( + seed_trash( + owner, message=message, trashed_at=datetime(2026, 6, 11, 12, i, tzinfo=UTC), # increasing ) - backend.trashes[owner].append(message_id) ids.append(message_id) return ids @@ -558,9 +562,9 @@ def test_box_rejects_invalid_query_params( def test_box_pagination_slices_and_counts( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - ids = _seed_trash_n(backend, USER, 5) # oldest → newest + ids = _seed_trash_n(seed_trash, USER, 5) # oldest → newest response = app_client.get("/trash?limit=2&offset=0", headers=headers_for(USER)) assert response.status_code == 200 @@ -575,9 +579,9 @@ def test_box_pagination_slices_and_counts( def test_box_sort_order_ascending( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - ids = _seed_trash_n(backend, USER, 3) + ids = _seed_trash_n(seed_trash, USER, 3) response = app_client.get("/trash?order=asc", headers=headers_for(USER)) assert response.status_code == 200 @@ -585,9 +589,9 @@ def test_box_sort_order_ascending( def test_box_offset_past_end_returns_empty_page( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: - _seed_trash_n(backend, USER, 3) + _seed_trash_n(seed_trash, USER, 3) response = app_client.get("/trash?offset=10", headers=headers_for(USER)) assert response.status_code == 200 @@ -598,14 +602,14 @@ def test_box_offset_past_end_returns_empty_page( def test_box_sort_by_sent_at_uses_message_send_time( - app_client: TestClient, headers_for, backend: MemoryBackend + app_client: TestClient, headers_for, seed_trash: Callable[..., str] ) -> None: """ `sort_by=sent_at` orders by the underlying message's send time, which the seed makes the exact reverse of the default `entered_at` (trashed_at) order. """ - ids = _seed_trash_n(backend, USER, 3) + ids = _seed_trash_n(seed_trash, USER, 3) default = app_client.get("/trash", headers=headers_for(USER)) assert [e["message_id"] for e in default.json()["entries"]] == [ diff --git a/tests/integration/test_stubs.py b/tests/integration/test_stubs.py index 3b44b01..9d8ddd8 100644 --- a/tests/integration/test_stubs.py +++ b/tests/integration/test_stubs.py @@ -24,6 +24,20 @@ ) +@pytest.fixture +def backend_kind() -> str: + """ + Pin this suite to the memory backend. + + These endpoints are ``NotImplementedError`` stubs on the *memory* backend + only; the sqlite backend implements them all (its routes are exercised in + ``test_gap_fill.py``). Overriding the parametrized fixture from + ``conftest.py`` keeps the strict-xfail assertions valid. + """ + + return "memory" + + @stub def test_delete_inbox_message_moves_to_trash( app_client: TestClient, headers_for, deliver_message diff --git a/tests/unit/test_sqlite_backend.py b/tests/unit/test_sqlite_backend.py new file mode 100644 index 0000000..ee6b371 --- /dev/null +++ b/tests/unit/test_sqlite_backend.py @@ -0,0 +1,339 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Direct end-to-end coverage for ``SQLiteBackend``, exercised through the public +protocol methods against a real temp-file database. + +Emphasis on the gap-fill methods the memory backend leaves as +``NotImplementedError`` (``delete_inbox_message``, ``delete_draft``, +``delete_trash_message``, ``clear_trash``, ``admin_webhook_patch``, +``daemon_deliver_remote``), since the shared integration suite cannot reach +them — their HTTP routes are still router-level stubs. +""" + +from collections.abc import AsyncIterator +from datetime import UTC, datetime +from pathlib import Path + +import pytest +from mail_protocol.core.lists import MAILListPolicy +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILAgent, + MAILDaemon, + MAILUser, + MAILUserAgent, +) +from mail_protocol.network.requests import ( + AdminAgentPostRequest, + AdminDaemonPostRequest, + AdminListPatchRequest, + AdminListPostRequest, + AdminSwarmPostRequest, + AdminUserPostRequest, + AdminWebhooksPatchRequest, + AdminWebhooksPostRequest, + AuthPasswordResetRequest, + BoxFilterParams, + DaemonDeliverLocalRequest, + DaemonDeliverRemoteRequest, + DraftPatchRequest, + DraftPostRequest, + DraftSendPostRequest, +) +from mail_server.backends.sqlite.api import SQLiteBackend + +ADMIN = MAILAdmin(ua_type="admin", admin_id="ryan", host="localhost") +DAEMON = MAILDaemon(ua_type="daemon", worker_name="dummy", host="localhost") +ALICE = MAILUserAgent( + user_agent=MAILUser(ua_type="user", user_id="alice", host="localhost") +) +SAGE = MAILUserAgent( + user_agent=MAILAgent( + ua_type="agent", name="sage", swarm="chorus", host="localhost" + ) +) +SAGE_ADDR = "sage@chorus@localhost" + + +@pytest.fixture +async def backend(tmp_path: Path) -> AsyncIterator[SQLiteBackend]: + be = SQLiteBackend(f"sqlite:///{tmp_path / 'mail.db'}") + await be.on_server_startup(host="localhost") + # A cast of one user (sender) and one agent (recipient). + await be.admin_post_user( + ADMIN, AdminUserPostRequest(user_id="alice", user_password="pw") + ) + await be.admin_post_agent( + ADMIN, + AdminAgentPostRequest( + agent_name="sage", swarm_name="chorus", agent_password="pw" + ), + ) + await be.admin_post_daemon( + ADMIN, AdminDaemonPostRequest(worker_name="dummy", daemon_password="pw") + ) + yield be + await be.on_server_shutdown() + + +async def _send(backend: SQLiteBackend, recipients: list[str]) -> str: + """Alice drafts and sends a message; return its message id.""" + + entry = await backend.post_draft( + ALICE, DraftPostRequest(subject="Hi", body="hello there") + ) + message = await backend.send_draft( + ALICE, + entry.draft.draft_id, + DraftSendPostRequest(recipients=recipients), + ) + return message.message_id + + +# --------------------------------------------------------------------------- # +# Admin CRUD + lifecycle +# --------------------------------------------------------------------------- # + + +async def test_admin_agent_crud_and_duplicates(backend: SQLiteBackend) -> None: + assert await backend.admin_get_agents(ADMIN) == ["sage@chorus"] + assert (await backend.admin_get_agent(ADMIN, "sage@chorus")).name == "sage" + + with pytest.raises(ValueError, match="already taken"): + await backend.admin_post_agent( + ADMIN, + AdminAgentPostRequest( + agent_name="sage", swarm_name="chorus", agent_password="pw" + ), + ) + + deleted = await backend.admin_delete_agent(ADMIN, "sage@chorus") + assert deleted.name == "sage" + assert await backend.admin_get_agents(ADMIN) == [] + + +async def test_reset_password(backend: SQLiteBackend) -> None: + assert await backend.user_agent_exists("user:alice@localhost") + result = await backend.reset_password( + ALICE.user_agent, + AuthPasswordResetRequest(current_password="pw", new_password="pw2"), + ) + assert result == "success" + with pytest.raises(ValueError, match="incorrect password"): + await backend.reset_password( + ALICE.user_agent, + AuthPasswordResetRequest(current_password="wrong", new_password="pw3"), + ) + + +# --------------------------------------------------------------------------- # +# Draft -> send -> deliver -> inbox lifecycle +# --------------------------------------------------------------------------- # + + +async def test_full_local_delivery_lifecycle(backend: SQLiteBackend) -> None: + message_id = await _send(backend, [SAGE_ADDR]) + + # The send lands in alice's outbox and the delivery buffer. + outbox, total = await backend.get_outbox(ALICE, BoxFilterParams()) + assert total == 1 and outbox[0].message_id == message_id + + buffered = await backend.daemon_clear_message_buffer(DAEMON) + assert buffered == [message_id] + # Buffer is drained; a second clear is empty. + assert await backend.daemon_clear_message_buffer(DAEMON) == [] + + delivered = await backend.daemon_deliver_local( + DAEMON, DaemonDeliverLocalRequest(message_ids=[message_id]) + ) + assert [m.message_id for m in delivered] == [message_id] + + # The agent recipient now has it in their inbox. + inbox, total = await backend.get_inbox(SAGE, BoxFilterParams()) + assert total == 1 and inbox[0].message_id == message_id + full = await backend.get_inbox_message(SAGE, message_id) + assert full.message.body == "hello there" + assert full.delivered_by == DAEMON.get_address() + + # The sender's outbox entry is marked delivered. + out_msg = await backend.get_outbox_message(ALICE, message_id) + assert out_msg.delivered_at is not None + + +async def test_draft_patch_and_delete(backend: SQLiteBackend) -> None: + entry = await backend.post_draft( + ALICE, DraftPostRequest(subject="Draft", body="body") + ) + draft_id = entry.draft.draft_id + + patched = await backend.patch_draft( + ALICE, draft_id, DraftPatchRequest(subject="Edited") + ) + assert patched.draft.subject == "Edited" + assert patched.draft.updated_at is not None + + deleted = await backend.delete_draft(ALICE, draft_id) + assert deleted.draft.draft_id == draft_id + with pytest.raises(ValueError, match="not found in draft box"): + await backend.get_draft(ALICE, draft_id) + + +# --------------------------------------------------------------------------- # +# Gap-fill: inbox -> trash move, trash delete, clear +# --------------------------------------------------------------------------- # + + +async def test_delete_inbox_message_moves_to_trash(backend: SQLiteBackend) -> None: + message_id = await _send(backend, [SAGE_ADDR]) + await backend.daemon_deliver_local( + DAEMON, DaemonDeliverLocalRequest(message_ids=[message_id]) + ) + + moved = await backend.delete_inbox_message(SAGE, message_id) + assert moved.message.message_id == message_id + + # Gone from inbox, present in trash. + inbox, inbox_total = await backend.get_inbox(SAGE, BoxFilterParams()) + assert inbox_total == 0 and inbox == [] + trash, trash_total = await backend.get_trash(SAGE, BoxFilterParams()) + assert trash_total == 1 and trash[0].message_id == message_id + + fetched = await backend.get_trash_message(SAGE, message_id) + assert fetched.message.message_id == message_id + + +async def test_delete_trash_message_and_clear(backend: SQLiteBackend) -> None: + first = await _send(backend, [SAGE_ADDR]) + second = await _send(backend, [SAGE_ADDR]) + await backend.daemon_deliver_local( + DAEMON, DaemonDeliverLocalRequest(message_ids=[first, second]) + ) + await backend.delete_inbox_message(SAGE, first) + await backend.delete_inbox_message(SAGE, second) + + removed = await backend.delete_trash_message(SAGE, first) + assert removed.message.message_id == first + _, total = await backend.get_trash(SAGE, BoxFilterParams()) + assert total == 1 + + cleared = await backend.clear_trash(SAGE) + assert [s.message_id for s in cleared] == [second] + _, total_after = await backend.get_trash(SAGE, BoxFilterParams()) + assert total_after == 0 + + +# --------------------------------------------------------------------------- # +# Gap-fill: webhook patch + remote delivery +# --------------------------------------------------------------------------- # + + +async def test_webhook_crud_and_patch(backend: SQLiteBackend) -> None: + created = await backend.admin_webhook_post( + ADMIN, + AdminWebhooksPostRequest( + url="https://hooks.example.com/a", + events=["mail.delivered"], + secret="s1", + ), + ) + # Idempotent on URL. + again = await backend.admin_webhook_post( + ADMIN, + AdminWebhooksPostRequest( + url="https://hooks.example.com/a", + events=["mail.delivered"], + secret="ignored", + ), + ) + assert again.webhook_id == created.webhook_id + + patched = await backend.admin_webhook_patch( + ADMIN, + created.webhook_id, + AdminWebhooksPatchRequest(url="https://hooks.example.com/b", secret="s2"), + ) + assert patched.webhook_id == created.webhook_id # id preserved across URL move + assert patched.url == "https://hooks.example.com/b" + assert patched.secret == "s2" + + # Refetch by id reflects the move; the old URL is gone. + refetched = await backend.admin_webhook_get(ADMIN, created.webhook_id) + assert refetched.url == "https://hooks.example.com/b" + assert await backend.admin_webhooks_get(ADMIN) == [created.webhook_id] + + with pytest.raises(ValueError, match="not found"): + await backend.admin_webhook_patch( + ADMIN, + "wh_missing", + AdminWebhooksPatchRequest(url="https://x.example.com", secret="s"), + ) + + +async def test_daemon_deliver_remote(backend: SQLiteBackend) -> None: + remote = MAILMessage( + mail_version="2.0", + message_id="99999999-9999-4999-8999-999999999999", + sender="echo@otherswarm@remote.example.com", + recipients=[SAGE_ADDR], + subject="Remote", + body="from afar", + tags=[], + sent_at=datetime.now(UTC), + metadata={}, + ) + delivered = await backend.daemon_deliver_remote( + DAEMON, DaemonDeliverRemoteRequest(messages=[remote]) + ) + assert [m.message_id for m in delivered] == [remote.message_id] + + inbox, total = await backend.get_inbox(SAGE, BoxFilterParams()) + assert total == 1 and inbox[0].message_id == remote.message_id + # The remote message was persisted into the canonical store. + assert (await backend.get_message(remote.message_id)).body == "from afar" + + +# --------------------------------------------------------------------------- # +# Swarms + lists +# --------------------------------------------------------------------------- # + + +async def test_swarm_and_list_crud(backend: SQLiteBackend) -> None: + await backend.admin_post_swarm( + ADMIN, + AdminSwarmPostRequest(name="newswarm", description="d", keywords=["k"]), + ) + assert (await backend.get_swarm("newswarm")).name == "newswarm" + assert await backend.get_swarm_health("newswarm") == "ok" + + created = await backend.admin_post_list( + ADMIN, + AdminListPostRequest( + name="team", + swarm_name="chorus", + owner="user:alice@localhost", + members=[], + ), + ) + address = created.get_address() + + with_member = await backend.add_list_member(address, SAGE_ADDR) + assert with_member.members == [SAGE_ADDR] + # Idempotent re-add. + assert (await backend.add_list_member(address, SAGE_ADDR)).members == [SAGE_ADDR] + + without = await backend.remove_list_member(address, SAGE_ADDR) + assert without.members == [] + + patched = await backend.admin_patch_list( + ADMIN, + address, + AdminListPatchRequest(policy=MAILListPolicy(visibility="private")), + ) + assert patched.policy.visibility == "private" + + await backend.admin_delete_list(ADMIN, address) + with pytest.raises(ValueError, match="list not found"): + await backend.get_list(address) diff --git a/tests/unit/test_sqlite_cli_wiring.py b/tests/unit/test_sqlite_cli_wiring.py new file mode 100644 index 0000000..0c22db0 --- /dev/null +++ b/tests/unit/test_sqlite_cli_wiring.py @@ -0,0 +1,83 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +CLI / server wiring for the sqlite backend: the ``--backend sqlite`` flag, its +``--sqlite-path`` / ``--database-url`` options (with env fallbacks), the URL +resolution precedence, and that ``run_server`` actually constructs a +``SQLiteBackend``. +""" + +import os +from argparse import Namespace +from pathlib import Path + +import pytest + +# server.py reads MAIL_HOST and routers.auth reads MAIL_JWT_EXPIRE_MINUTES at +# import time. +os.environ.setdefault("MAIL_HOST", "localhost") +os.environ.setdefault("MAIL_JWT_EXPIRE_MINUTES", "15") + +from mail_server import server as server_module # noqa: E402 +from mail_server.backends.sqlite.api import SQLiteBackend # noqa: E402 +from mail_server.cli import build_parser # noqa: E402 + + +def test_parser_accepts_sqlite_backend_and_options() -> None: + parser = build_parser() + args = parser.parse_args( + ["--backend", "sqlite", "--sqlite-path", "/tmp/mail.db"] + ) + assert args.backend == "sqlite" + assert args.sqlite_path == "/tmp/mail.db" + assert args.database_url is None + + +def test_parser_reads_env_fallbacks(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("MAIL_SQLITE_PATH", "/env/path.db") + monkeypatch.setenv("MAIL_DATABASE_URL", "sqlite:////env/url.db") + args = build_parser().parse_args([]) + assert args.sqlite_path == "/env/path.db" + assert args.database_url == "sqlite:////env/url.db" + + +def test_resolve_url_prefers_database_url() -> None: + args = Namespace( + database_url="sqlite:////abs/custom.db", sqlite_path="/ignored.db" + ) + assert server_module._resolve_sqlite_url(args) == "sqlite:////abs/custom.db" + + +def test_resolve_url_uses_sqlite_path() -> None: + args = Namespace(database_url=None, sqlite_path="/var/lib/mail/mail.db") + assert ( + server_module._resolve_sqlite_url(args) + == "sqlite:////var/lib/mail/mail.db" + ) + + +def test_resolve_url_falls_back_to_default() -> None: + args = Namespace(database_url=None, sqlite_path=None) + url = server_module._resolve_sqlite_url(args) + assert url.startswith("sqlite:///") + assert url.endswith("deployments/default/mail.db") + + +def test_run_server_constructs_sqlite_backend( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # Stub out the blocking calls so run_server just performs backend selection. + monkeypatch.setattr(server_module, "init_logger", lambda: None) + monkeypatch.setattr(server_module.uvicorn, "run", lambda *a, **k: None) + + args = Namespace( + backend="sqlite", + host="localhost", + port=8000, + sqlite_path=str(tmp_path / "mail.db"), + database_url=None, + ) + server_module.run_server(args) + + assert isinstance(server_module._backend, SQLiteBackend) diff --git a/tests/unit/test_sqlite_database.py b/tests/unit/test_sqlite_database.py new file mode 100644 index 0000000..d106f4e --- /dev/null +++ b/tests/unit/test_sqlite_database.py @@ -0,0 +1,123 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Database-level guarantees for the sqlite backend: the connection pragmas, the +all-or-nothing transaction boundary around a multi-write operation, and that +concurrent writers don't trip ``database is locked`` (WAL + busy_timeout). +""" + +import asyncio +from collections.abc import AsyncIterator +from pathlib import Path + +import pytest +from mail_protocol.core.user_agents import ( + MAILAdmin, + MAILDaemon, + MAILUser, + MAILUserAgent, +) +from mail_protocol.network.requests import ( + AdminUserPostRequest, + BoxFilterParams, + DraftPostRequest, + DraftSendPostRequest, +) +from mail_server.backends.sqlite.api import SQLiteBackend +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.repositories import ( + MailStore, + MessageBufferRepository, +) +from sqlalchemy import text + +ADMIN = MAILAdmin(ua_type="admin", admin_id="ryan", host="localhost") +ALICE = MAILUserAgent( + user_agent=MAILUser(ua_type="user", user_id="alice", host="localhost") +) + + +@pytest.fixture +async def backend(tmp_path: Path) -> AsyncIterator[SQLiteBackend]: + be = SQLiteBackend(f"sqlite:///{tmp_path / 'mail.db'}") + await be.on_server_startup(host="localhost") + await be.admin_post_user( + ADMIN, AdminUserPostRequest(user_id="alice", user_password="pw") + ) + yield be + await be.on_server_shutdown() + + +async def test_connection_pragmas_applied(tmp_path: Path) -> None: + db = Database(f"sqlite:///{tmp_path / 'mail.db'}") + await db.create_schema() + try: + async with db.session() as session: + journal_mode = (await session.execute(text("PRAGMA journal_mode"))).scalar() + foreign_keys = (await session.execute(text("PRAGMA foreign_keys"))).scalar() + busy_timeout = (await session.execute(text("PRAGMA busy_timeout"))).scalar() + finally: + await db.dispose() + + assert journal_mode == "wal" + assert foreign_keys == 1 + assert busy_timeout == 5000 + + +async def test_send_draft_rolls_back_on_failure( + backend: SQLiteBackend, monkeypatch: pytest.MonkeyPatch +) -> None: + """A failure on the final write of ``send_draft`` leaves no partial rows.""" + + entry = await backend.post_draft( + ALICE, DraftPostRequest(subject="Tx", body="atomic") + ) + draft_id = entry.draft.draft_id + + async def _boom(self: MessageBufferRepository, message_id: str) -> None: + raise RuntimeError("buffer write failed") + + # Fail on the last step (buffer enqueue), after message/outbox/membership. + monkeypatch.setattr(MessageBufferRepository, "enqueue", _boom) + + with pytest.raises(RuntimeError, match="buffer write failed"): + await backend.send_draft( + ALICE, draft_id, DraftSendPostRequest(recipients=["user:alice@localhost"]) + ) + + # Nothing from the aborted send survived: no outbox entry, empty buffer. + _, outbox_total = await backend.get_outbox(ALICE, BoxFilterParams()) + assert outbox_total == 0 + async with backend._db.session() as session: + assert await MailStore(session).buffer.list_ids() == [] + # ...and the draft is untouched, so the send can be retried. + assert (await backend.get_draft(ALICE, draft_id)).draft.draft_id == draft_id + + +async def test_concurrent_sends_do_not_lock(backend: SQLiteBackend) -> None: + """WAL + busy_timeout: concurrent committed sends don't raise locked.""" + + drafts = [ + await backend.post_draft( + ALICE, DraftPostRequest(subject=f"D{i}", body="body") + ) + for i in range(8) + ] + + messages = await asyncio.gather( + *( + backend.send_draft( + ALICE, + entry.draft.draft_id, + DraftSendPostRequest(recipients=["user:alice@localhost"]), + ) + for entry in drafts + ) + ) + + # Every send committed a distinct message into the delivery buffer. + assert len({m.message_id for m in messages}) == 8 + daemon = MAILDaemon(ua_type="daemon", worker_name="dummy", host="localhost") + buffered = await backend.daemon_clear_message_buffer(daemon) + assert sorted(buffered) == sorted(m.message_id for m in messages) diff --git a/tests/unit/test_sqlite_init.py b/tests/unit/test_sqlite_init.py new file mode 100644 index 0000000..df84d97 --- /dev/null +++ b/tests/unit/test_sqlite_init.py @@ -0,0 +1,91 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Coverage for ``init_sqlite_backend`` — the ``backend-init --type sqlite`` path. + +Verifies it seeds the swarm + every user-agent, writes plaintext secrets whose +hash verifies, and is safely idempotent on re-run (no duplicates, secrets +preserved). +""" + +from pathlib import Path + +from mail_protocol.core.user_agents import MAILAdmin +from mail_server.auth import verify_password +from mail_server.backends.sqlite.api import SQLiteBackend +from mail_server.backends.sqlite.init import default_sqlite_path, init_sqlite_backend + +_ADMIN = MAILAdmin(ua_type="admin", admin_id="ryan", host="localhost") + + +def test_default_sqlite_path_layout() -> None: + path = default_sqlite_path("mydep") + assert path.name == "mail.db" + assert path.parent.name == "mydep" + assert path.parent.parent.name == "deployments" + + +async def test_init_seeds_swarm_agents_and_secrets(tmp_path: Path) -> None: + db_path = tmp_path / "mail.db" + await init_sqlite_backend( + swarm="chorus", + agents=["supervisor"], + daemons=["dummy"], + users=["alice"], + admins=["ryan"], + host="localhost", + db_path=db_path, + ) + + assert db_path.exists() + secrets_dir = tmp_path / ".secrets" + addresses = { + "supervisor@chorus@localhost", + "daemon:dummy@localhost", + "user:alice@localhost", + "admin:ryan@localhost", + } + assert {p.name for p in secrets_dir.iterdir()} == addresses + + backend = SQLiteBackend(f"sqlite:///{db_path}") + await backend.on_server_startup(host="localhost") + try: + assert (await backend.get_swarm("chorus")).agents == ["supervisor"] + for address in addresses: + ua = await backend.get_user_agent(address) + secret = (secrets_dir / address).read_text(encoding="utf-8") + # The plaintext written to disk verifies against the stored hash. + assert verify_password( + plain_password=secret, hashed_password=ua.hashed_password + ) + finally: + await backend.on_server_shutdown() + + +async def test_init_is_idempotent(tmp_path: Path) -> None: + db_path = tmp_path / "mail.db" + kwargs = dict( + swarm="chorus", + agents=["supervisor"], + daemons=["dummy"], + users=["alice"], + admins=["ryan"], + host="localhost", + db_path=db_path, + ) + await init_sqlite_backend(**kwargs) # type: ignore[arg-type] + secret_before = (tmp_path / ".secrets" / "user:alice@localhost").read_text() + + # Re-running must not duplicate rows or rotate the existing secret. + await init_sqlite_backend(**kwargs) # type: ignore[arg-type] + secret_after = (tmp_path / ".secrets" / "user:alice@localhost").read_text() + assert secret_before == secret_after + + backend = SQLiteBackend(f"sqlite:///{db_path}") + await backend.on_server_startup(host="localhost") + try: + assert await backend.admin_get_users(_ADMIN) == ["alice"] + assert await backend.admin_get_agents(_ADMIN) == ["supervisor@chorus"] + finally: + await backend.on_server_shutdown() diff --git a/tests/unit/test_sqlite_migrate.py b/tests/unit/test_sqlite_migrate.py new file mode 100644 index 0000000..804184c --- /dev/null +++ b/tests/unit/test_sqlite_migrate.py @@ -0,0 +1,187 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +One-time filesystem (memory) -> sqlite import. + +Writes a small deployment with the memory backend's own ``save_*`` loaders, +imports it, and verifies every collection — and the reconstructed per-owner box +membership — is readable through ``SQLiteBackend``. +""" + +from datetime import UTC, datetime +from pathlib import Path + +import pytest +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILList, MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.outbox import MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.user_agents import ( + MAILAgent, + MAILDaemon, + MAILUser, + MAILUserAgent, + MAILUserAgentInBackend, +) +from mail_protocol.core.webhooks import MAILWebhook +from mail_protocol.network.requests import BoxFilterParams +from mail_server.backends.memory import fs as memory_fs +from mail_server.backends.sqlite.api import SQLiteBackend +from mail_server.backends.sqlite.migrate import import_memory_deployment + +NOW = datetime(2026, 6, 12, 9, 0, tzinfo=UTC) +MID = "55555555-5555-4555-8555-555555555555" +UALICE = "user:alice@localhost" +SAGE = "sage@chorus@localhost" +DAEMON = "daemon:dummy@localhost" +LIST_ADDR = "list:team@chorus@localhost" + +ALICE_UA = MAILUserAgent( + user_agent=MAILUser(ua_type="user", user_id="alice", host="localhost") +) +SAGE_UA = MAILUserAgent( + user_agent=MAILAgent(ua_type="agent", name="sage", swarm="chorus", host="localhost") +) + + +async def _write_fs_deployment() -> None: + """Populate the (monkeypatched) memory deployment tree.""" + + await memory_fs.save_user_agents( + { + UALICE: MAILUserAgentInBackend( + user_agent=ALICE_UA.user_agent, hashed_password="h1" + ), + SAGE: MAILUserAgentInBackend( + user_agent=SAGE_UA.user_agent, hashed_password="h2" + ), + } + ) + await memory_fs.save_swarms( + { + "chorus": MAILSwarm( + name="chorus", + description="d", + keywords=["k"], + agents=["sage"], + metadata={}, + ) + } + ) + await memory_fs.save_messages( + { + MID: MAILMessage( + mail_version="2.0", + message_id=MID, + sender=UALICE, + recipients=[SAGE], + subject="Imported", + body="survives migration", + tags=[], + sent_at=NOW, + metadata={}, + ) + } + ) + await memory_fs.save_inbox_entries( + { + MID: MAILInboxEntrySummary( + message_id=MID, + sender=UALICE, + subject="Imported", + body_size=18, + received_at=NOW, + delivered_by=DAEMON, + ) + } + ) + await memory_fs.save_inboxes({SAGE: [MID], UALICE: []}) + await memory_fs.save_outbox_entries( + { + MID: MAILOutboxEntrySummary( + message_id=MID, + recipients=[SAGE], + subject="Imported", + body_size=18, + sent_at=NOW, + delivered_at=NOW, + delivered_by=DAEMON, + ) + } + ) + await memory_fs.save_outboxes({UALICE: [MID]}) + await memory_fs.save_message_buffer([MID]) + await memory_fs.save_webhooks( + { + "https://hooks.example.com/mail": MAILWebhook( + webhook_id=f"wh_{MID}", + url="https://hooks.example.com/mail", + events=["mail.delivered"], + secret="shh", + ) + } + ) + await memory_fs.save_lists( + { + LIST_ADDR: MAILListInBackend( + **MAILList( + name="team", swarm="chorus", host="localhost", owner=UALICE + ).model_dump(), + list_id=MID, + created_at=NOW, + updated_at=NOW, + ) + } + ) + + +async def test_import_filesystem_deployment(deployment_dir: Path) -> None: + await _write_fs_deployment() + db_path = deployment_dir / "mail.db" + + counts = await import_memory_deployment(source_dir=deployment_dir, db_path=db_path) + assert counts["user_agents"] == 2 + assert counts["messages"] == 1 + assert counts["buffered"] == 1 + + backend = SQLiteBackend(f"sqlite:///{db_path}") + await backend.on_server_startup(host="localhost") + try: + # User-agents + swarm imported. + assert (await backend.get_user_agent(SAGE)).hashed_password == "h2" + assert (await backend.get_swarm("chorus")).agents == ["sage"] + + # The recipient's inbox membership + entry survived... + inbox, total = await backend.get_inbox(SAGE_UA, BoxFilterParams()) + assert total == 1 and inbox[0].message_id == MID + opened = await backend.get_inbox_message(SAGE_UA, MID) + assert opened.message.body == "survives migration" + assert opened.delivered_by == DAEMON + + # ...as did the sender's outbox, the buffer, the webhook, and the list. + _, outbox_total = await backend.get_outbox(ALICE_UA, BoxFilterParams()) + assert outbox_total == 1 + daemon = MAILDaemon(ua_type="daemon", worker_name="dummy", host="localhost") + assert await backend.daemon_clear_message_buffer(daemon) == [MID] + assert (await backend.get_lists())[0].get_address() == LIST_ADDR + finally: + await backend.on_server_shutdown() + + +async def test_import_refuses_nonempty_database(deployment_dir: Path) -> None: + await _write_fs_deployment() + db_path = deployment_dir / "mail.db" + await import_memory_deployment(source_dir=deployment_dir, db_path=db_path) + + # A second import would clobber existing data; it must refuse. + with pytest.raises(ValueError, match="not empty"): + await import_memory_deployment(source_dir=deployment_dir, db_path=db_path) + + +async def test_import_missing_source_raises(tmp_path: Path) -> None: + with pytest.raises(FileNotFoundError, match="no filesystem deployment"): + await import_memory_deployment( + source_dir=tmp_path / "nope", db_path=tmp_path / "mail.db" + ) diff --git a/tests/unit/test_sqlite_repositories.py b/tests/unit/test_sqlite_repositories.py new file mode 100644 index 0000000..d29ff23 --- /dev/null +++ b/tests/unit/test_sqlite_repositories.py @@ -0,0 +1,345 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Repository-layer tests for the SQLite backend, run against a real temp-file +database so foreign keys, datetime columns, and ordering behave as in prod. + +The bulk pins ``MailboxRepository`` pagination: ``entered_at`` vs ``sent_at`` +ordering, ``asc`` / ``desc``, ``limit`` / ``offset``, ``total`` correctness, and +the always-ascending ``id`` tiebreaker that reproduces the memory backend's +stable insertion order. The rest cover CRUD round-trips and the FIFO buffer. +""" + +import uuid +from collections.abc import AsyncIterator +from datetime import UTC, datetime +from pathlib import Path + +import pytest +from mail_protocol.core.drafts import MAILDraft, MAILDraftsEntry +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILList, MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.trash import MAILTrashEntry +from mail_protocol.core.user_agents import MAILAgent, MAILUserAgentInBackend +from mail_protocol.core.webhooks import MAILWebhook +from mail_protocol.network.requests import BoxFilterParams +from mail_server.backends.sqlite.database import Database +from mail_server.backends.sqlite.repositories import ( + BOX_INBOX, + BOX_TRASH, + MailStore, +) + +OWNER = "sage@chorus@localhost" +SENDER = "user:alice@localhost" +DAEMON = "daemon:dummy@localhost" + +T1 = datetime(2026, 6, 1, 12, 0, tzinfo=UTC) +T2 = datetime(2026, 6, 2, 12, 0, tzinfo=UTC) +T3 = datetime(2026, 6, 3, 12, 0, tzinfo=UTC) + + +@pytest.fixture +async def db(tmp_path: Path) -> AsyncIterator[Database]: + database = Database(f"sqlite:///{tmp_path / 'mail.db'}") + await database.create_schema() + yield database + await database.dispose() + + +def _uuid() -> str: + return str(uuid.uuid4()) + + +def _agent(owner: str = OWNER) -> MAILUserAgentInBackend: + name, swarm, host = owner.split("@") + return MAILUserAgentInBackend( + user_agent=MAILAgent(ua_type="agent", name=name, swarm=swarm, host=host), + hashed_password="hash", + ) + + +def _message(message_id: str, sent_at: datetime) -> MAILMessage: + return MAILMessage( + mail_version="2.0", + message_id=message_id, + sender=SENDER, + recipients=[OWNER], + subject="subject", + body="body", + tags=[], + sent_at=sent_at, + metadata={}, + ) + + +async def _seed_inbox( + store: MailStore, *, sent_at: datetime, entered_at: datetime +) -> str: + """Create message + shared inbox entry + membership for OWNER; return id.""" + + message_id = _uuid() + await store.messages.add(_message(message_id, sent_at)) + await store.boxes.upsert_inbox_entry( + MAILInboxEntrySummary( + message_id=message_id, + sender=SENDER, + subject="subject", + body_size=4, + received_at=entered_at, + delivered_by=DAEMON, + ) + ) + await store.boxes.add_membership(OWNER, BOX_INBOX, message_id, entered_at) + return message_id + + +# --------------------------------------------------------------------------- # +# MailboxRepository pagination / ordering +# --------------------------------------------------------------------------- # + + +async def test_inbox_default_orders_entered_at_desc(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + first = await _seed_inbox(store, sent_at=T1, entered_at=T1) + second = await _seed_inbox(store, sent_at=T2, entered_at=T2) + third = await _seed_inbox(store, sent_at=T3, entered_at=T3) + + page, total = await store.boxes.list_inbox(OWNER, BoxFilterParams()) + + assert total == 3 + assert [e.message_id for e in page] == [third, second, first] + + +async def test_inbox_entered_at_asc(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + first = await _seed_inbox(store, sent_at=T1, entered_at=T1) + second = await _seed_inbox(store, sent_at=T2, entered_at=T2) + + page, _ = await store.boxes.list_inbox( + OWNER, BoxFilterParams(order="asc") + ) + + assert [e.message_id for e in page] == [first, second] + + +async def test_inbox_sort_by_sent_at_differs_from_entered_at(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + # Arrival order is the inverse of send order. + late_arrival_old_send = await _seed_inbox(store, sent_at=T1, entered_at=T3) + early_arrival_new_send = await _seed_inbox(store, sent_at=T3, entered_at=T1) + + by_entered, _ = await store.boxes.list_inbox(OWNER, BoxFilterParams()) + by_sent, _ = await store.boxes.list_inbox( + OWNER, BoxFilterParams(sort_by="sent_at") + ) + + # entered_at desc -> the late arrival is first. + assert [e.message_id for e in by_entered] == [ + late_arrival_old_send, + early_arrival_new_send, + ] + # sent_at desc -> the newer send is first, flipping the order. + assert [e.message_id for e in by_sent] == [ + early_arrival_new_send, + late_arrival_old_send, + ] + + +async def test_inbox_limit_offset_and_total(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + ids = [ + await _seed_inbox(store, sent_at=t, entered_at=t) + for t in (T1, T2, T3) + ] + + page, total = await store.boxes.list_inbox( + OWNER, BoxFilterParams(limit=1, offset=1, order="asc") + ) + + assert total == 3 # count is the whole box, not the page + assert [e.message_id for e in page] == [ids[1]] + + +async def test_inbox_tiebreak_is_insertion_order(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + # Identical entered_at; desc must still fall back to ascending id + # (insertion order), matching the memory backend's stable sort. + first = await _seed_inbox(store, sent_at=T1, entered_at=T1) + second = await _seed_inbox(store, sent_at=T1, entered_at=T1) + + page, _ = await store.boxes.list_inbox(OWNER, BoxFilterParams()) + + assert [e.message_id for e in page] == [first, second] + + +async def test_empty_box_returns_empty_page(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + + page, total = await store.boxes.list_inbox(OWNER, BoxFilterParams()) + + assert page == [] + assert total == 0 + + +# --------------------------------------------------------------------------- # +# Membership / orphan accounting +# --------------------------------------------------------------------------- # + + +async def test_membership_add_remove_and_orphan_count(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + other = "echo@chorus@localhost" + await store.user_agents.add(_agent(other)) + + message_id = await _seed_inbox(store, sent_at=T1, entered_at=T1) + # Fan the same shared entry out to a second owner. + await store.boxes.add_membership(other, BOX_INBOX, message_id, T1) + + assert await store.boxes.count_item_members(BOX_INBOX, message_id) == 2 + assert await store.boxes.is_member(OWNER, BOX_INBOX, message_id) + + assert await store.boxes.remove_membership(OWNER, BOX_INBOX, message_id) + assert not await store.boxes.is_member(OWNER, BOX_INBOX, message_id) + assert await store.boxes.count_item_members(BOX_INBOX, message_id) == 1 + # Removing a non-member is a no-op returning False. + assert not await store.boxes.remove_membership(OWNER, BOX_INBOX, message_id) + + +# --------------------------------------------------------------------------- # +# CRUD round-trips through real SQLite +# --------------------------------------------------------------------------- # + + +async def test_user_agent_crud(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + agent = _agent() + await store.user_agents.add(agent) + + assert await store.user_agents.exists(OWNER) + assert await store.user_agents.get(OWNER) == agent + assert await store.user_agents.list_by_type("agent") == [agent] + + updated = await store.user_agents.set_password(OWNER, "new-hash") + assert updated is not None and updated.hashed_password == "new-hash" + reloaded = await store.user_agents.get(OWNER) + assert reloaded is not None and reloaded.hashed_password == "new-hash" + + deleted = await store.user_agents.delete(OWNER) + assert deleted is not None + assert not await store.user_agents.exists(OWNER) + + +async def test_swarm_and_webhook_and_list_crud(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + + swarm = MAILSwarm( + name="chorus", + description="d", + keywords=["k"], + agents=[], + metadata={}, + ) + await store.swarms.add(swarm) + assert await store.swarms.get("chorus") == swarm + assert await store.swarms.list_all() == [swarm] + + webhook = MAILWebhook( + webhook_id=f"wh_{_uuid()}", + url="https://hooks.example.com/mail", + events=["mail.delivered"], + secret="shh", + ) + await store.webhooks.add(webhook) + assert await store.webhooks.get_by_id(webhook.webhook_id) == webhook + assert await store.webhooks.get_by_url(webhook.url) == webhook + + mail_list = MAILListInBackend( + **MAILList( + name="team", swarm="chorus", host="localhost", owner=SENDER + ).model_dump(), + list_id=_uuid(), + created_at=T1, + updated_at=T1, + ) + await store.lists.add(mail_list) + # Member edit rewrites the JSON body wholesale, mirroring memory. + mutated = mail_list.model_copy( + update={"members": [OWNER], "updated_at": T2} + ) + await store.lists.update(mutated) + reloaded = await store.lists.get_by_address(mail_list.get_address()) + assert reloaded is not None and reloaded.members == [OWNER] + + +async def test_message_buffer_is_fifo_and_drains(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + ids = [_uuid() for _ in range(3)] + for message_id in ids: + await store.buffer.enqueue(message_id) + + assert await store.buffer.list_ids() == ids + assert await store.buffer.drain() == ids + # Buffer is empty after draining. + assert await store.buffer.list_ids() == [] + assert await store.buffer.drain() == [] + + +async def test_draft_entry_crud(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + draft_id = _uuid() + entry = MAILDraftsEntry( + draft=MAILDraft( + draft_id=draft_id, + subject="s", + body="b", + created_at=T1, + updated_at=None, + ), + sent_at=None, + sent_by=None, + ) + await store.boxes.upsert_draft_entry(entry) + assert await store.boxes.get_draft_entry(draft_id) == entry + + await store.boxes.delete_draft_entry(draft_id) + assert await store.boxes.get_draft_entry(draft_id) is None + + +async def test_trash_summary_listing(db: Database) -> None: + async with db.session() as session: + store = MailStore(session) + await store.user_agents.add(_agent()) + message_id = _uuid() + await store.messages.add(_message(message_id, T1)) + await store.boxes.upsert_trash_entry( + MAILTrashEntry(message=_message(message_id, T1), trashed_at=T2) + ) + await store.boxes.add_membership(OWNER, BOX_TRASH, message_id, T2) + + page, total = await store.boxes.list_trash(OWNER, BoxFilterParams()) + + assert total == 1 + assert page[0].message_id == message_id + assert page[0].trashed_at == T2 diff --git a/tests/unit/test_sqlite_serializers.py b/tests/unit/test_sqlite_serializers.py new file mode 100644 index 0000000..d62c0c4 --- /dev/null +++ b/tests/unit/test_sqlite_serializers.py @@ -0,0 +1,226 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 Addison Kline + +""" +Round-trips for every model-bearing SQLite row serializer: +``model -> to_columns -> Row(**columns) -> from_row`` must be identity. + +This pins the hybrid invariant — ``from_row`` rehydrates solely from the +``body`` JSON column, so a serialization change can't silently corrupt a +collection. The two body-less tables (``mailbox_items``, ``message_buffer``) +have no serializer and are covered by repository tests instead. + +The typed columns are asserted separately, since those (not ``body``) are what +``WHERE`` / ``ORDER BY`` rely on. +""" + +from datetime import UTC, datetime + +from mail_protocol.core.drafts import MAILDraft, MAILDraftsEntry +from mail_protocol.core.inbox import MAILInboxEntrySummary +from mail_protocol.core.lists import MAILList, MAILListInBackend +from mail_protocol.core.messages import MAILMessage +from mail_protocol.core.outbox import MAILOutboxEntrySummary +from mail_protocol.core.swarms import MAILSwarm +from mail_protocol.core.trash import MAILTrashEntry +from mail_protocol.core.user_agents import ( + MAILAgent, + MAILUser, + MAILUserAgentInBackend, +) +from mail_protocol.core.webhooks import MAILWebhook +from mail_server.backends.sqlite import serializers as s +from mail_server.backends.sqlite.schema import ( + DraftEntryRow, + InboxEntryRow, + ListRow, + MessageRow, + OutboxEntryRow, + SwarmRow, + TrashEntryRow, + UserAgentRow, + WebhookRow, +) + +NOW = datetime(2026, 6, 12, 9, 0, tzinfo=UTC) +UUID = "55555555-5555-4555-8555-555555555555" +USER = "user:alice@localhost" +AGENT = "sage@chorus@localhost" +DAEMON = "daemon:dummy@localhost" + + +def _message() -> MAILMessage: + return MAILMessage( + mail_version="2.0", + message_id=UUID, + sender=USER, + recipients=[AGENT], + subject="Persisted", + body="Survives a round-trip.", + tags=[], + sent_at=NOW, + metadata={}, + ) + + +def test_user_agent_roundtrip() -> None: + model = MAILUserAgentInBackend( + user_agent=MAILUser(ua_type="user", user_id="alice", host="localhost"), + hashed_password="a-stored-hash", + ) + cols = s.user_agent_to_columns(model) + assert cols["address"] == USER + assert cols["ua_type"] == "user" + assert cols["swarm"] is None + assert cols["host"] == "localhost" + assert cols["hashed_password"] == "a-stored-hash" + assert s.user_agent_from_row(UserAgentRow(**cols)) == model + + +def test_user_agent_agent_carries_swarm_column() -> None: + model = MAILUserAgentInBackend( + user_agent=MAILAgent( + ua_type="agent", name="sage", swarm="chorus", host="localhost" + ), + hashed_password="hash", + ) + cols = s.user_agent_to_columns(model) + assert cols["address"] == AGENT + assert cols["swarm"] == "chorus" + assert s.user_agent_from_row(UserAgentRow(**cols)) == model + + +def test_swarm_roundtrip() -> None: + model = MAILSwarm( + name="chorus", + description="A test swarm.", + keywords=["testing"], + agents=["sage"], + metadata={}, + ) + cols = s.swarm_to_columns(model) + assert cols["name"] == "chorus" + assert s.swarm_from_row(SwarmRow(**cols)) == model + + +def test_message_roundtrip() -> None: + model = _message() + cols = s.message_to_columns(model) + assert cols["message_id"] == UUID + assert cols["sender"] == USER + assert cols["subject"] == "Persisted" + assert cols["reply_to"] is None + assert cols["sent_at"] == NOW + assert s.message_from_row(MessageRow(**cols)) == model + + +def test_inbox_entry_roundtrip() -> None: + model = MAILInboxEntrySummary( + message_id=UUID, + sender=USER, + subject="Persisted", + body_size=10, + received_at=NOW, + delivered_by=DAEMON, + ) + cols = s.inbox_entry_to_columns(model) + assert cols["message_id"] == UUID + assert cols["body_size"] == 10 + assert cols["received_at"] == NOW + assert cols["delivered_by"] == DAEMON + assert s.inbox_entry_from_row(InboxEntryRow(**cols)) == model + + +def test_outbox_entry_roundtrip() -> None: + model = MAILOutboxEntrySummary( + message_id=UUID, + recipients=[AGENT], + subject="Persisted", + body_size=10, + sent_at=NOW, + delivered_at=NOW, + delivered_by=DAEMON, + ) + cols = s.outbox_entry_to_columns(model) + assert cols["message_id"] == UUID + assert cols["sent_at"] == NOW + assert cols["delivered_at"] == NOW + assert cols["delivered_by"] == DAEMON + assert s.outbox_entry_from_row(OutboxEntryRow(**cols)) == model + + +def test_outbox_entry_undelivered_roundtrip() -> None: + model = MAILOutboxEntrySummary( + message_id=UUID, + recipients=[AGENT], + subject="Persisted", + body_size=10, + sent_at=NOW, + ) + cols = s.outbox_entry_to_columns(model) + assert cols["delivered_at"] is None + assert cols["delivered_by"] is None + assert s.outbox_entry_from_row(OutboxEntryRow(**cols)) == model + + +def test_draft_entry_roundtrip() -> None: + model = MAILDraftsEntry( + draft=MAILDraft( + draft_id=UUID, + subject="Persisted", + body="A draft body.", + created_at=NOW, + updated_at=None, + ), + sent_at=None, + sent_by=None, + ) + cols = s.draft_entry_to_columns(model) + assert cols["draft_id"] == UUID + assert cols["created_at"] == NOW + assert cols["updated_at"] is None + assert s.draft_entry_from_row(DraftEntryRow(**cols)) == model + + +def test_trash_entry_roundtrip() -> None: + model = MAILTrashEntry(message=_message(), trashed_at=NOW) + cols = s.trash_entry_to_columns(model) + assert cols["message_id"] == UUID + assert cols["trashed_at"] == NOW + assert s.trash_entry_from_row(TrashEntryRow(**cols)) == model + + +def test_webhook_roundtrip() -> None: + model = MAILWebhook( + webhook_id=f"wh_{UUID}", + url="https://hooks.example.com/mail", + events=["mail.delivered"], + secret="shhh", + ) + cols = s.webhook_to_columns(model) + assert cols["url"] == "https://hooks.example.com/mail" + assert cols["webhook_id"] == f"wh_{UUID}" + assert s.webhook_from_row(WebhookRow(**cols)) == model + + +def test_list_roundtrip() -> None: + model = MAILListInBackend( + **MAILList( + name="team", + swarm="chorus", + host="localhost", + owner=USER, + members=[AGENT], + ).model_dump(), + list_id=UUID, + created_at=NOW, + updated_at=NOW, + ) + cols = s.list_to_columns(model) + assert cols["address"] == "list:team@chorus@localhost" + assert cols["list_id"] == UUID + assert cols["swarm"] == "chorus" + assert cols["host"] == "localhost" + assert cols["created_at"] == NOW + assert cols["updated_at"] == NOW + assert s.list_from_row(ListRow(**cols)) == model diff --git a/uv.lock b/uv.lock index 97b401f..80f26b6 100644 --- a/uv.lock +++ b/uv.lock @@ -88,6 +88,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" }, ] +[[package]] +name = "aiosqlite" +version = "0.22.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/4e/8a/64761f4005f17809769d23e518d915db74e6310474e733e3593cfc854ef1/aiosqlite-0.22.1.tar.gz", hash = "sha256:043e0bd78d32888c0a9ca90fc788b38796843360c855a7262a532813133a0650", size = 14821, upload-time = "2025-12-23T19:25:43.997Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/00/b7/e3bf5133d697a08128598c8d0abc5e16377b51465a33756de24fa7dee953/aiosqlite-0.22.1-py3-none-any.whl", hash = "sha256:21c002eb13823fad740196c5a2e9d8e62f6243bd9e7e4a1f87fb5e44ecb4fceb", size = 17405, upload-time = "2025-12-23T19:25:42.139Z" }, +] + [[package]] name = "alabaster" version = "1.0.0" @@ -747,6 +756,7 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/44/69/9b804adb5fd0671f367781560eb5eb586c4d495277c93bde4307b9e28068/greenlet-3.2.4-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:3b67ca49f54cede0186854a008109d6ee71f66bd57bb36abd6d0a0267b540cdd", size = 274079, upload-time = "2025-08-07T13:15:45.033Z" }, { url = "https://files.pythonhosted.org/packages/46/e9/d2a80c99f19a153eff70bc451ab78615583b8dac0754cfb942223d2c1a0d/greenlet-3.2.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ddf9164e7a5b08e9d22511526865780a576f19ddd00d62f8a665949327fde8bb", size = 640997, upload-time = "2025-08-07T13:42:56.234Z" }, { url = "https://files.pythonhosted.org/packages/3b/16/035dcfcc48715ccd345f3a93183267167cdd162ad123cd93067d86f27ce4/greenlet-3.2.4-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f28588772bb5fb869a8eb331374ec06f24a83a9c25bfa1f38b6993afe9c1e968", size = 655185, upload-time = "2025-08-07T13:45:27.624Z" }, + { url = "https://files.pythonhosted.org/packages/31/da/0386695eef69ffae1ad726881571dfe28b41970173947e7c558d9998de0f/greenlet-3.2.4-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:5c9320971821a7cb77cfab8d956fa8e39cd07ca44b6070db358ceb7f8797c8c9", size = 649926, upload-time = "2025-08-07T13:53:15.251Z" }, { url = "https://files.pythonhosted.org/packages/68/88/69bf19fd4dc19981928ceacbc5fd4bb6bc2215d53199e367832e98d1d8fe/greenlet-3.2.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c60a6d84229b271d44b70fb6e5fa23781abb5d742af7b808ae3f6efd7c9c60f6", size = 651839, upload-time = "2025-08-07T13:18:30.281Z" }, { url = "https://files.pythonhosted.org/packages/19/0d/6660d55f7373b2ff8152401a83e02084956da23ae58cddbfb0b330978fe9/greenlet-3.2.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3b3812d8d0c9579967815af437d96623f45c0f2ae5f04e366de62a12d83a8fb0", size = 607586, upload-time = "2025-08-07T13:18:28.544Z" }, { url = "https://files.pythonhosted.org/packages/8e/1a/c953fdedd22d81ee4629afbb38d2f9d71e37d23caace44775a3a969147d4/greenlet-3.2.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:abbf57b5a870d30c4675928c37278493044d7c14378350b3aa5d484fa65575f0", size = 1123281, upload-time = "2025-08-07T13:42:39.858Z" }, @@ -757,6 +767,7 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/49/e8/58c7f85958bda41dafea50497cbd59738c5c43dbbea5ee83d651234398f4/greenlet-3.2.4-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:1a921e542453fe531144e91e1feedf12e07351b1cf6c9e8a3325ea600a715a31", size = 272814, upload-time = "2025-08-07T13:15:50.011Z" }, { url = "https://files.pythonhosted.org/packages/62/dd/b9f59862e9e257a16e4e610480cfffd29e3fae018a68c2332090b53aac3d/greenlet-3.2.4-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cd3c8e693bff0fff6ba55f140bf390fa92c994083f838fece0f63be121334945", size = 641073, upload-time = "2025-08-07T13:42:57.23Z" }, { url = "https://files.pythonhosted.org/packages/f7/0b/bc13f787394920b23073ca3b6c4a7a21396301ed75a655bcb47196b50e6e/greenlet-3.2.4-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:710638eb93b1fa52823aa91bf75326f9ecdfd5e0466f00789246a5280f4ba0fc", size = 655191, upload-time = "2025-08-07T13:45:29.752Z" }, + { url = "https://files.pythonhosted.org/packages/f2/d6/6adde57d1345a8d0f14d31e4ab9c23cfe8e2cd39c3baf7674b4b0338d266/greenlet-3.2.4-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:c5111ccdc9c88f423426df3fd1811bfc40ed66264d35aa373420a34377efc98a", size = 649516, upload-time = "2025-08-07T13:53:16.314Z" }, { url = "https://files.pythonhosted.org/packages/7f/3b/3a3328a788d4a473889a2d403199932be55b1b0060f4ddd96ee7cdfcad10/greenlet-3.2.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d76383238584e9711e20ebe14db6c88ddcedc1829a9ad31a584389463b5aa504", size = 652169, upload-time = "2025-08-07T13:18:32.861Z" }, { url = "https://files.pythonhosted.org/packages/ee/43/3cecdc0349359e1a527cbf2e3e28e5f8f06d3343aaf82ca13437a9aa290f/greenlet-3.2.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:23768528f2911bcd7e475210822ffb5254ed10d71f4028387e5a99b4c6699671", size = 610497, upload-time = "2025-08-07T13:18:31.636Z" }, { url = "https://files.pythonhosted.org/packages/b8/19/06b6cf5d604e2c382a6f31cafafd6f33d5dea706f4db7bdab184bad2b21d/greenlet-3.2.4-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:00fadb3fedccc447f517ee0d3fd8fe49eae949e1cd0f6a611818f4f6fb7dc83b", size = 1121662, upload-time = "2025-08-07T13:42:41.117Z" }, @@ -767,6 +778,7 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/22/5c/85273fd7cc388285632b0498dbbab97596e04b154933dfe0f3e68156c68c/greenlet-3.2.4-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:49a30d5fda2507ae77be16479bdb62a660fa51b1eb4928b524975b3bde77b3c0", size = 273586, upload-time = "2025-08-07T13:16:08.004Z" }, { url = "https://files.pythonhosted.org/packages/d1/75/10aeeaa3da9332c2e761e4c50d4c3556c21113ee3f0afa2cf5769946f7a3/greenlet-3.2.4-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:299fd615cd8fc86267b47597123e3f43ad79c9d8a22bebdce535e53550763e2f", size = 686346, upload-time = "2025-08-07T13:42:59.944Z" }, { url = "https://files.pythonhosted.org/packages/c0/aa/687d6b12ffb505a4447567d1f3abea23bd20e73a5bed63871178e0831b7a/greenlet-3.2.4-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:c17b6b34111ea72fc5a4e4beec9711d2226285f0386ea83477cbb97c30a3f3a5", size = 699218, upload-time = "2025-08-07T13:45:30.969Z" }, + { url = "https://files.pythonhosted.org/packages/dc/8b/29aae55436521f1d6f8ff4e12fb676f3400de7fcf27fccd1d4d17fd8fecd/greenlet-3.2.4-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:b4a1870c51720687af7fa3e7cda6d08d801dae660f75a76f3845b642b4da6ee1", size = 694659, upload-time = "2025-08-07T13:53:17.759Z" }, { url = "https://files.pythonhosted.org/packages/92/2e/ea25914b1ebfde93b6fc4ff46d6864564fba59024e928bdc7de475affc25/greenlet-3.2.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:061dc4cf2c34852b052a8620d40f36324554bc192be474b9e9770e8c042fd735", size = 695355, upload-time = "2025-08-07T13:18:34.517Z" }, { url = "https://files.pythonhosted.org/packages/72/60/fc56c62046ec17f6b0d3060564562c64c862948c9d4bc8aa807cf5bd74f4/greenlet-3.2.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:44358b9bf66c8576a9f57a590d5f5d6e72fa4228b763d0e43fee6d3b06d3a337", size = 657512, upload-time = "2025-08-07T13:18:33.969Z" }, { url = "https://files.pythonhosted.org/packages/23/6e/74407aed965a4ab6ddd93a7ded3180b730d281c77b765788419484cdfeef/greenlet-3.2.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:2917bdf657f5859fbf3386b12d68ede4cf1f04c90c3a6bc1f013dd68a22e2269", size = 1612508, upload-time = "2025-11-04T12:42:23.427Z" }, @@ -1424,6 +1436,7 @@ version = "2.0.1" source = { editable = "src/mail/server" } dependencies = [ { name = "aiohttp" }, + { name = "aiosqlite" }, { name = "fastapi" }, { name = "mail-swarms-protocol" }, { name = "pwdlib", extra = ["argon2"] }, @@ -1431,12 +1444,14 @@ dependencies = [ { name = "pyjwt" }, { name = "python-dotenv" }, { name = "python-multipart" }, + { name = "sqlalchemy", extra = ["asyncio"] }, { name = "uvicorn" }, ] [package.metadata] requires-dist = [ { name = "aiohttp", specifier = ">=3.12.15" }, + { name = "aiosqlite", specifier = ">=0.20" }, { name = "fastapi", specifier = ">=0.116.1" }, { name = "mail-swarms-protocol", editable = "src/mail/protocol" }, { name = "pwdlib", extras = ["argon2"], specifier = ">=0.3.0" }, @@ -1444,6 +1459,7 @@ requires-dist = [ { name = "pyjwt", specifier = ">=2.10.1" }, { name = "python-dotenv", specifier = ">=1.1.1" }, { name = "python-multipart", specifier = ">=0.0.20" }, + { name = "sqlalchemy", extras = ["asyncio"], specifier = ">=2.0" }, { name = "uvicorn", specifier = ">=0.35.0" }, ] @@ -2540,6 +2556,11 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b8/d9/13bdde6521f322861fab67473cec4b1cc8999f3871953531cf61945fad92/sqlalchemy-2.0.43-py3-none-any.whl", hash = "sha256:1681c21dd2ccee222c2fe0bef671d1aef7c504087c9c4e800371cfcc8ac966fc", size = 1924759, upload-time = "2025-08-11T15:39:53.024Z" }, ] +[package.optional-dependencies] +asyncio = [ + { name = "greenlet" }, +] + [[package]] name = "sse-starlette" version = "3.0.2"