Local single-process self-host + non-destructive data migration - #1
Merged
Conversation
…ost/port Foundation for the single-process self-host path: - db.py: file-backed SQLite now runs in WAL with PRAGMA foreign_keys=ON and a 5s busy_timeout (FK-off-by-default would break ON DELETE CASCADE locally). - config.py: add field_notes_auth_disabled + field_notes_data_dir (default off, so networked/Heroku deploys stay locked). - __main__.py: FIELD_NOTES_HOST + $PORT/FIELD_NOTES_PORT instead of hardcoded 0.0.0.0:8000. All additive/backward-compatible. 168 passed.
Both require_api_key (header) and require_api_key_query (SSE token) short-circuit to a no-op when field_notes_auth_disabled is set, so the loopback serve path doesn't lock the owner out and the MCP connects with no key. Default stays enforced. 172 passed.
New cli.py with argparse subcommands. serve resolves a data dir (default ~/.field-notes), sets SQLite/media/static env (honouring power-user overrides), disables auth on loopback-without-key (refuses a public bind without --key), runs alembic upgrade head in-process, then serves the existing app. - env.py: collapse sqlite+aiosqlite -> sqlite so in-process migrations work. - pyproject: register `field-notes` script; move aiosqlite to RUNTIME deps (local serve needs the driver; was dev-only -> fresh installs would crash). Verified e2e: serve boots on a temp dir, 5 migrations stamp 0005, WAL on, keyless /projects 200 + POST 201, data persists to disk. 184 passed.
media.py: copy files into the managed media root (media/<task>/) with byte-hash dedup (skip identical, refuse silent overwrite of different content); scan cell text for /media/ refs and report any missing from the root. CLI wires both; verify-media reads the local SQLite cells table directly. 191 passed.
transfer.py: import_db copies every table source->dest in FK-dependency order, preserving PKs/FKs/JSON/timestamps. Refuses on Alembic revision mismatch (column drift would drop data) and on a non-empty dest unless overwrite=True; source is never written. Verified SQLite->SQLite (same path runs asyncpg->aiosqlite). fetch_media downloads the Dockerfile-declared tarballs into the media root (tarfile data filter blocks path traversal). CLI: import-db + fetch-media. alembic.ini: add path_separator=os (silences in-process migration warning). 196 passed.
hatchling force-include ships apps/api/alembic -> field_notes_api/_alembic so `field-notes serve`/import-db can migrate from an installed wheel. The built SPA at field_notes_api/_web (produced by `make build-web`, committed so uvx-from-git serves the UI with no Node) is picked up by default package inclusion. cli: _package_dir() indirection for testability. Verified: wheel bundles _alembic (ini + 5 migrations) and _web; builds cleanly with _web absent (API-only fallback); run_migrations works from the bundled layout. 201 passed.
- LICENSE: MIT (holder: Mikul Rai — adjust if desired). - README: 'Run it yourself' uvx-from-git quickstart, MCP config (throwaway FIELD_NOTES_KEY=local since the loopback API ignores it), and a no-data-loss migration recipe (import-db -> fetch-media -> verify-media). - Makefile build-web: bake VITE_DEFAULT_KEY=local so the bundled SPA skips the key gate locally. - .env.example: note the local path needs none of these vars.
Built with VITE_DEFAULT_KEY=local so the loopback UI skips the key gate. With this committed, `field-notes serve` (from source or wheel) serves the full UI at / — no Node needed at run time. Rebuild via `make build-web` after frontend changes. Verified: GET / -> 200, title 'Field Notes', JS asset 200.
deploy/local-keepalive.sh: relaunches a downed local server (crash/reboot/never -started), flock-guarded, crash output preserved in server.log. Wire via cron @reboot + every-2-min. Resolves the binary from FIELD_NOTES_BIN/PATH so it works for any self-hoster, not just this machine.
_engine_for() adds asyncpg ssl=require for non-loopback Postgres hosts so import-db can pull from a hosted DB (Heroku won't accept a non-TLS connection); loopback Postgres + SQLite stay plaintext. Verified against live Heroku: pulled 7 projects / 170 cells / 69 verdicts / 1751 events.
cron's getpwuid HOME can differ from interactive HOME (AFS) and be unreadable without a token, which breaks the lock/data-dir. Document setting HOME + FIELD_NOTES_DATA_DIR to an absolute path in the crontab.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a single-process local self-host path so anyone can run field-notes on their
own machine with one command, and a non-destructive way to migrate an existing
(Heroku/Postgres) instance into it.
What's new:
field-notes serve— runs migrations on a file-backed SQLite DB and serves thebundled SPA + API in one process (loopback, auth disabled by default).
import-db/fetch-media/verify-media— copy a remote DB (over TLS) intolocal SQLite preserving IDs/links, pull media, and report gaps.
add-media— copy files into a managed media root.migrations shipped as package data.
deploy/local-keepalive.sh— cron supervisor for an always-on local server.All additive and backward-compatible; the existing networked/Heroku deploy is
unchanged. ~200 tests pass.