release: promote main to production (77 commits)

.github/copilot-instructions.md

@@ -33,21 +33,23 @@ suggestions and reviews match the architecture instead of fighting it. Deeper op
     (name the bad instance), not the tolerance.
 
 - **Every caught exception is persisted, once (ADR 0019 + 0001).** Errors live in the `AppError` table,
-  not just stdout. Use the two-arm dedup pattern so a single failure is logged once as it unwinds:
+  not just stdout. `persist_app_error` is idempotent — it marks the exception and returns the existing
+  row on any later call — so a single failure is one row as it unwinds, no wrapper needed:
 
   ```python
-  from apps.workflow.exceptions import AlreadyLoggedException
   from apps.workflow.services.error_persistence import persist_app_error
 
   try:
       operation()
-  except AlreadyLoggedException:
-      raise  # already persisted upstream — pass through unchanged
   except Exception as exc:
-      err = persist_app_error(exc)  # MANDATORY
-      raise AlreadyLoggedException(exc, err.id) from exc
+      persist_app_error(exc)  # idempotent — one AppError row per failure
+      raise
   ```
 
+  A handler that *converts* the exception must chain the cause (`raise ValueError(...) from exc`), or the
+  converted failure earns a second row (pylint `W0707` enforces this). At the HTTP boundary, read the id
+  with `app_error_for(exc)` for `error_id` (ADR 0013) and map the status from the exception's real type.
+
 - **Backend owns data; frontend owns presentation (ADR 0020).** Anything involving the DB, business
   rules, or external systems is backend. Static UI constants, layout, and ergonomics are frontend. The
   boundary is the *kind of value*, not the layer of code.

.github/workflows/ci.yml

@@ -21,7 +21,7 @@ jobs:
           cache-dependency-path: poetry.lock
 
       - name: Set up Node
-        uses: actions/setup-node@v6
+        uses: actions/setup-node@v7
         with:
           node-version: 22
           cache: 'npm'
@@ -162,7 +162,7 @@ jobs:
         working-directory: frontend
     steps:
       - uses: actions/checkout@v7
-      - uses: actions/setup-node@v6
+      - uses: actions/setup-node@v7
         with:
           node-version: 22
           cache: 'npm'
@@ -179,7 +179,7 @@ jobs:
         working-directory: frontend
     steps:
       - uses: actions/checkout@v7
-      - uses: actions/setup-node@v6
+      - uses: actions/setup-node@v7
         with:
           node-version: 22
           cache: 'npm'
@@ -195,7 +195,7 @@ jobs:
         working-directory: frontend
     steps:
       - uses: actions/checkout@v7
-      - uses: actions/setup-node@v6
+      - uses: actions/setup-node@v7
         with:
           node-version: 22
           cache: 'npm'
@@ -215,7 +215,7 @@ jobs:
       VITE_API_BASE_URL: http://localhost:8000
     steps:
       - uses: actions/checkout@v7
-      - uses: actions/setup-node@v6
+      - uses: actions/setup-node@v7
         with:
           node-version: 22
           cache: 'npm'

.gitignore

@@ -57,8 +57,11 @@ instance/
 # Sphinx documentation
 docs/_build/
 
-# Planning artefacts (Claude Code plan-mode files; local-only)
-docs/plans/
+# Planning artefacts (Claude Code plan-mode files; local-only).
+# The pattern must match the contents, not the directory: git never descends
+# into an excluded directory, which would leave the negation unreachable.
+docs/plans/*
+!docs/plans/_template.md
 
 # PyBuilder
 target/
@@ -275,3 +278,6 @@ session-replays/
 docs/.codesight/
 frontend/.codesight/
 *.tsv
+
+# Local state for scripts/write_google_doc.py (per-instance runtime data)
+scripts/google_doc_manifest.json

.pre-commit-config.yaml

@@ -105,17 +105,34 @@ repos:
         types: [python]
         exclude: '.*/migrations/.*\.py$'
 
-      # pylint - check for common bugs and merge issues
+      # pylint - check for common bugs and merge issues.
+      #
+      # Deliberately an allowlist, not full pylint. Full pylint reports ~4300
+      # findings here, ~72% of which are noise or actively fight this codebase:
+      # line-too-long duplicates Black, abstract-method is a DRF false positive,
+      # duplicate-code is cross-module (so it cannot baseline stably), and
+      # broad-exception-caught fires on the handler pattern ADR 0019 mandates.
+      # Rules below earn their place; each was verified against real findings.
+      # Rejected after testing, all high false-positive on Django/SDK dynamism:
+      # E1111/E1121 (SDK property monkeypatching), E0203 (Django FK _id
+      # descriptors), E0606, E1120, C0325.
+      #
       # E0102: function-redefined (duplicate methods)
       # E0108: duplicate-argument-name
       # E0118: used-prior-to-assignment
+      # E0701: bad-except-order (subclass handler shadowed - found a live bug)
+      # E1123: unexpected-keyword-arg (found a live TypeError)
+      # W0101: unreachable
       # W0104: pointless-statement (no effect)
       # W0199: assert-on-tuple (always true)
       # W0221: arguments-differ (override signature mismatch)
+      # W0612: unused-variable
       # W0622: redefined-builtin (redefining id, type, etc)
+      # W0707: raise-missing-from (exception chaining, per ADR 0001)
+      # R1710: inconsistent-return-statements
       - id: pylint-bugs
         name: Check for common bugs with pylint
-        entry: poetry run python -m pylint --disable=all --enable=E0102,E0108,E0118,W0104,W0199,W0221,W0622 --ignore-patterns='migrations/.*\.py'
+        entry: poetry run python -m pylint --disable=all --enable=E0102,E0108,E0118,E0701,E1123,W0101,W0104,W0199,W0221,W0612,W0622,W0707,R1710 --ignore-patterns='migrations/.*\.py'
         language: system
         files: '^apps/.+\.py$'
         types: [python]

.vscode/tasks.json

@@ -28,24 +28,24 @@
       }
     },
     {
-      "label": "Frontend Manual Dev Server",
+      "label": "Frontend Preview (build)",
       "type": "process",
       "command": "${env:HOME}/.nvm/nvm-exec",
-      "args": ["npm", "run", "manual:dev"],
+      "args": ["npm", "run", "preview:e2e"],
       "hide": true,
       "options": {
         "cwd": "${workspaceFolder}/frontend"
       },
       "isBackground": true,
       "problemMatcher": {
-        "owner": "vitepress",
+        "owner": "vite-preview",
         "pattern": {
           "regexp": "^$"
         },
         "background": {
           "activeOnStart": true,
-          "beginsPattern": "vitepress",
-          "endsPattern": "http://localhost"
+          "beginsPattern": ".",
+          "endsPattern": "Local:"
         }
       },
       "presentation": {
@@ -75,6 +75,34 @@
         "panel": "dedicated"
       }
     },
+    {
+      "label": "Django (runserver)",
+      "type": "process",
+      "command": "${workspaceFolder}/.venv/bin/python",
+      "args": ["${workspaceFolder}/manage.py", "runserver", "--noreload"],
+      "options": {
+        "env": {
+          "PYTHONPATH": "${workspaceFolder}"
+        }
+      },
+      "hide": true,
+      "isBackground": true,
+      "problemMatcher": {
+        "owner": "django",
+        "pattern": {
+          "regexp": "^$"
+        },
+        "background": {
+          "activeOnStart": true,
+          "beginsPattern": ".",
+          "endsPattern": "Starting development server"
+        }
+      },
+      "presentation": {
+        "reveal": "always",
+        "panel": "dedicated"
+      }
+    },
     {
       "label": "Celery Worker",
       "type": "shell",
@@ -123,7 +151,19 @@
       "label": "Start Dev Environment",
       "dependsOn": [
         "Frontend Dev Server",
-        "Frontend Manual Dev Server",
+        "Ngrok Tunnels",
+        "Celery Worker",
+        "Celery Beat"
+      ],
+      "dependsOrder": "parallel",
+      "problemMatcher": []
+    },
+    {
+      "label": "Start E2E Environment",
+      "detail": "Serves the production BUILD on :5173 + plain Django on :8000 (no debugger) for E2E runs, fast over ngrok/LAN. Stop 'Start Dev Environment' AND the Run>Debug 'Django' session first — both bind :5173/:8000.",
+      "dependsOn": [
+        "Frontend Preview (build)",
+        "Django (runserver)",
         "Ngrok Tunnels",
         "Celery Worker",
         "Celery Beat"

CLAUDE.md

@@ -7,10 +7,11 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 Major architectural decisions are recorded in [`docs/adr/`](docs/adr/README.md). Read the ADR index before non-trivial work — the codebase deviates from typical Django/Vue defaults in deliberate ways that aren't reconstructable from the code alone. The most operationally consequential ADRs:
 
 - **0017** — Zero backwards compatibility: when a name, URL, or shape changes, every caller changes in the same PR. No deprecation aliases, no `getattr` shims, no "for safety" columns.
-- **0019 + 0001** — Every exception is persisted to `AppError` (0019); nested handlers re-raise via the `AlreadyLoggedException` two-arm dedup pattern (0001).
+- **0019 + 0001** — Every exception is persisted to `AppError` (0019); `persist_app_error` is idempotent, so handlers just persist and re-raise — one row per failure by construction, no wrapper (0001).
 - **0015** — When a consumer finds malformed data, fix the data (migration). Consumers stay strict; never add a read-side fallback.
 - **0020** — Backend owns data, calculations, and external systems; frontend owns presentation. The boundary is the kind of value, not the layer of code.
 - **0021** — Frontend reads/writes the API only via the generated client; raw `fetch`/`axios` is forbidden.
+- **0032** — Less code is better: prefer a maintained library over a homegrown implementation. Writing your own for something a library provides needs an explicit, recorded reason it isn't a library.
 
 CLAUDE.md is the operational layer (session behaviour, code-style gotchas, architecture facts). ADRs explain *why*.
 
@@ -120,11 +121,17 @@ ADJUSTMENT entries (kind='adjust'):
   fixes when that is the pragmatic path, but direct-to-main commits are banned.
 - Before committing, check the current branch. If it is `main`, create or switch
   to a branch first.
-- Do not leave uncommitted changes behind at the end of a task. If the change is
-  complete and scoped, commit it on the current branch. If the scope is unclear,
-  mixed with unrelated work, or the user may not want it committed, ask before
-  committing.
+- Commits are all-or-nothing for the worktree: either do not commit at all, or
+  commit every tracked change together. Never use a path-limited commit.
+- Do not leave uncommitted changes behind at the end of a task. Finish and
+  verify incomplete work before committing the whole worktree.
+- `docs/plans/` is ephemeral scratch — one plan per piece of work, gitignored
+  (except `_template.md`). Delete a plan when its PR is opened, having first
+  migrated anything durable to its real home: open work → the Jira ticket, tools
+  → `scripts/`, decisions → an ADR. Never leave a non-plan artifact (script, data
+  file) sitting in `docs/plans/` — it goes through that same migrate-or-delete gate.
 - Run focused tests for touched code when useful. Do not manually run expensive hook commands like `bash scripts/check_mypy.sh`, `npm run test:unit`, `npm run lint`, `npm run type-check`, or frontend builds unless diagnosing a hook failure; they run automatically during `git commit`/`git push`.
+- Tests must protect enduring behaviour, invariants, or algorithms. Never assert the implementation's own text — `assertIn` on source code, a CLI flag or log string, or source line ordering — which mirrors the code, breaks on every refactor, and catches no bug. Execute the code path and assert the observable outcome: return value, exit code, output, or resulting state.
 
 ### Code Style and Quality
 
@@ -159,24 +166,33 @@ ADR 0015 (fix data, not fallback) and ADR 0017 (zero backwards compatibility) ar
 
 ### Mandatory error persistence
 
-Every exception handler persists once via `persist_app_error(exc)` (ADR 0019) and re-raises through the two-arm dedup pattern (ADR 0001).
+A `try` needs a strong reason: you are going to **handle** the failure — reshape it (domain error, or an HTTP status at the boundary), or persist it from the layer that understands it well enough to add business context. Otherwise let it raise.
+
+Every handler you do write persists via `persist_app_error(exc)` (ADR 0019) and re-raises. `persist_app_error` is idempotent — it marks the exception and returns the existing row on any later call — so one failure is one `AppError` row no matter how many layers catch it (ADR 0001). No wrapper type, no pass-through arm.
 
 ```python
-from apps.workflow.exceptions import AlreadyLoggedException
 from apps.workflow.services.error_persistence import persist_app_error
 
 try:
     operation()
-except AlreadyLoggedException:
-    raise  # already persisted upstream — pass through unchanged
 except Exception as exc:
-    err = persist_app_error(exc)  # MANDATORY
-    raise AlreadyLoggedException(exc, err.id) from exc
+    persist_app_error(exc, job_id=job.id)  # the context is why this handler exists
+    raise
 ```
 
+A handler that *converts* the exception must chain the cause, or the converted failure earns a second row (pylint `W0707` enforces this):
+
+```python
+except Job.DoesNotExist as exc:
+    persist_app_error(exc)
+    raise ValueError(f"Job {job_id} not found") from exc
+```
+
+At the HTTP boundary, read the persisted id with `app_error_for(exc)` to include `error_id` in the response (ADR 0013), and map the status from the exception's real type.
+
 ## Environment Configuration
 
-See `.env.example` for required environment variables. Key integrations: Xero API, Dropbox, PostgreSQL. Frontend tooling reads `APP_DOMAIN` from the backend `.env` at `../.env` and derives URLs from it (see ADR 0008's Consequences). Deploy uses `scripts/server/deploy.sh` (per-instance `<client>-<env>`); it also runs on boot via systemd so a cold machine catches up to `production`. Servers only ever run the `production` branch — `main` is the integration branch and is never deployed (ADR 0029).
+See `.env.example` for required environment variables. Key integrations: Xero API, Dropbox, PostgreSQL. Frontend tooling reads `APP_DOMAIN` from the backend `.env` at `../.env` and derives URLs from it (see ADR 0008's Consequences). Deploy uses `scripts/server/deploy.sh` (per-instance `<client>-<env>`); it also runs on boot via systemd so a cold machine catches up to `production`. Servers run the `production` branch by default; `main` is the integration branch — never deployed to production, but deployed to UAT as a release candidate via `deploy.sh --ref` / `instance.sh create --ref` (ADR 0029).
 
 ## Migration Management
 
@@ -197,8 +213,6 @@ See ADR 0020. Backend owns data, calculations, and external systems; frontend ow
 
 `npm run test:e2e` (from `frontend/`) runs Playwright tests against live HTTP endpoints. The suite takes ~20–25 min.
 
-**CRITICAL: The backend serving `APP_DOMAIN` must run with `XERO_READONLY=True`.** The flag swaps in a provider that suppresses all Xero writes (contacts, invoices, quotes, attachments, history notes) while reads and token refresh stay live; without it the suite writes real entities into the connected Xero tenant. Global-setup enforces this via `/api/xero/ping/` (`xero_readonly` field) and aborts otherwise. The flag is process-scoped: any celery worker/beat sharing the DB must also run with `XERO_READONLY=True`, or the hourly `xero_regular_sync_task` will push local `[TEST]` stock to Xero — global-setup cannot verify a worker's environment.
-
 **CRITICAL: The global teardown (`global-teardown.ts`) MUST always run to completion.** It restores the database from backup, saves/reinjects Xero tokens, removes the lock file, and runs integrity checks. If the bash process is killed (timeout, SIGTERM, etc.) the teardown never executes and the database is left polluted with `[TEST]` data.
 
 - **Never set a bash timeout on the E2E command.** A timeout (or SIGTERM) kills the node process before Playwright calls `globalTeardown`, leaving the DB polluted with `[TEST]` data and a stale lock file. The teardown is NOT a signal handler — it only fires on normal exit.