diff --git a/pna-build-eval-contrib/SKILL.md b/pna-build-eval-contrib/SKILL.md index bf26c88..82b9a77 100644 --- a/pna-build-eval-contrib/SKILL.md +++ b/pna-build-eval-contrib/SKILL.md @@ -39,14 +39,20 @@ Inputs: a candidate PNA's source tree (or a description sufficient to read its b - If the candidate has an Architecture document with an AC attestation table, check that the declared verification mechanism actually runs and passes. 2. **For each flavor-derived AC in `spec/axes.md`** triggered by the candidate's axis picks, do the same. 3. **For each typed contract relevant to the candidate's axis picks**, check that the candidate implements the contract correctly. Contract headers (`Realizes: AC-X, AC-Y`) tell you which ACs the contract serves. -4. **Produce a structured report keyed by AC ID.** The canonical form is the typed artifact at `tools/evaluate-report.schema.json` (JSON Schema). Emit an instance of that schema as the source of truth, then render the human-readable report as a *view* over it — don't hand-write the prose report and skip the artifact. Emitting the typed form is what makes two runs on the same candidate diffable (which ACs changed status). Per-AC status is one of: +4. **Detect and verify Exceptions** (see `spec/exceptions.md`). For each Exception the candidate can raise — declared in its Architecture document's exception attestation, or inferred from the source where undeclared: + - **Caught & handled?** Confirm consent is obtained *before* the raise (EX-H2), a persistent non-PNA-mode signal is shown while active (EX-H3), and a runtime active-set explainer exists (EX-H4). Cite code/UX for each. + - **Reversibility?** Read the `Reversible:` declaration; if `yes`, trace the `Reversal:` mechanism and decide whether the code/UX delivers a practical path back to PNA mode. Mode only — do not credit a handler that implies returning to PNA mode undoes prior disclosure (EX-H5). + - **Consent reaches the human?** Where an agent/proxy can drive the app, check the handler makes a best-effort attempt to propagate consent to the ultimate human and does not let an intermediary self-consent (EX-H7). + - **Strength profile accurate?** Check each dimension's class (EX-H8) against the code/UX; the lint already confirmed the classes are valid vocabulary — you judge whether they're truthful (e.g. nothing about the provider's behavior is claimed above `provider-asserted`). + - **Undeclared deviations.** You are the backstop: if the candidate departs from an AC or the PNA definition WITHOUT declaring an Exception, that is a silent (uncaught) deviation — a conformance failure. Flag it and name the `EX-*` it should have raised. +5. **Produce a structured report keyed by AC or EX ID.** The canonical form is the typed artifact at `tools/evaluate-report.schema.json` (JSON Schema). Emit an instance of that schema as the source of truth, then render the human-readable report as a *view* over it — don't hand-write the prose report and skip the artifact. Emitting the typed form is what makes two runs on the same candidate diffable (which ACs changed status). Per-AC status is one of: - `conformant` — with cited code locations. - `non-conformant` — with cited code locations showing the violation and the AC's stated requirement. - `not-applicable` — with reason (typically: the candidate's flavor doesn't trigger this AC). - `unable-to-determine` — with explanation; defaults to flagging for human review. Each finding may also carry `evidence` entries tagged by `source` (`deterministic` / `llm` / `human`). When a deterministic check in `tools/` (e.g. the egress lint) has run against the candidate, fold its output in as a `source: deterministic` evidence entry on the AC it bears on, so the deterministic and LLM layers land on one finding. -5. **Summarize at the top** (the artifact's `summary` object): overall posture and the most concerning non-conformances. Goals 1–5 are the load-bearing user-facing concerns — anything compromising private-data sovereignty (Goal 1), source-mirroring honesty (Goal 2), transport security (Goal 3), durability (Goal 4), or local diagnosability (Goal 5) leads the summary. +6. **Summarize at the top** (the artifact's `summary` object): overall posture and the most concerning non-conformances. Goals 1–5 are the load-bearing user-facing concerns — anything compromising private-data sovereignty (Goal 1), source-mirroring honesty (Goal 2), transport security (Goal 3), durability (Goal 4), or local diagnosability (Goal 5) leads the summary. Callers may ask you to emphasize specific Goals or axes at runtime (e.g., "focus on private-data sovereignty"). Treat that as a hint for the summary, not a structural variation. diff --git a/reference_designs/fellows_local_db/Architecture.md b/reference_designs/fellows_local_db/Architecture.md new file mode 100644 index 0000000..b646184 --- /dev/null +++ b/reference_designs/fellows_local_db/Architecture.md @@ -0,0 +1,382 @@ +# Architecture (fellows_local_db) + +This document is fellows_local_db's **specialization-and-conformance layer**: it declares which version of the PNA Spec this repo conforms to, names the axis picks fellows has made, and catalogs the fellows-specific values that the spec leaves to each implementation (HTTP routes, schema, worker constants, debug placeholders, distribution tunables). + +Universal PNA architecture — vocabulary, goals, the two-store ownership split, the worker-owned-OPFS rule, the version-handshake contract, the universal ACs — lives in the [PNA Spec](https://github.com/richbodo/personal_network_toolkit/blob/main/PNA_Spec.md) at the [personal_network_toolkit](https://github.com/richbodo/personal_network_toolkit) repo. This file does not restate it. + +--- + +## Spec conformance + +**Spec-Version:** [0.1 (draft)](https://github.com/richbodo/personal_network_toolkit/blob/main/CHANGELOG.md) +**Use case:** [Directory Archive](https://github.com/richbodo/personal_network_toolkit/blob/main/use_cases.md#directory-archive) + +### Flavor — fellows's six axis picks + +| Axis | Pick | Why | +|---|---|---| +| [Distribution](https://github.com/richbodo/personal_network_toolkit/blob/main/axes.md#distribution) | `web-bundle-with-magic-link` | EHF-allowlisted PWA; multiple fellows install from one origin behind a magic-link gate. | +| [Storage substrate](https://github.com/richbodo/personal_network_toolkit/blob/main/axes.md#storage-substrate) | `opfs-sqlite-wasm` | Browser-only deployment; sqlite3.wasm in a dedicated worker with OPFS-SAH-Pool VFS. | +| [Ingestion shape](https://github.com/richbodo/personal_network_toolkit/blob/main/axes.md#ingestion-shape) | `single-source-static-mirror` | One source (Knack JSON dump); no dedup; opt-in user-driven re-import. | +| [Workspace shell](https://github.com/richbodo/personal_network_toolkit/blob/main/axes.md#workspace-shell) | `vanilla-js-spa` | Single-IIFE `app/static/app.js`; hash routing; no framework, no bundler. | +| [Comms transport set](https://github.com/richbodo/personal_network_toolkit/blob/main/axes.md#comms-transport-set) | `mailto-only` | `mailto:` (+ `tel:`) today; Signal planned. | +| [MCP-exposure](https://github.com/richbodo/personal_network_toolkit/blob/main/axes.md#mcp-exposure) | `shared+private+comms` | `mcp_servers/` ships three stdio MCP servers for Claude Desktop and similar clients. | + +This section is fellows's **AC attestation table** — the Security Target role from the toolkit's +[`ARCHITECTURE_TEMPLATE.md`](https://github.com/richbodo/personal_network_toolkit/blob/main/reference_designs/templates/ARCHITECTURE_TEMPLATE.md). +Every applicable AC carries a **Realization** (how the code satisfies it), a **Verification** (the +test, rubric, or human-review note that proves it), and a **Status**. Verification refs are +`file::test_function` where a deterministic test exists; otherwise an LLM rubric or a human-review +note is named (both acceptable per the template). Status is `conformant` / `partial-conformance` / +`not-applicable`. Partial rows state honestly what is and isn't covered. + +### Universal ACs + +| AC | Realization | Verification | Status | +|---|---|---|---| +| AC-1 (two-store ownership split) | Two SQLite DBs: `fellows.db` (read-only contact data) + `relationships.db` (read-write, user-owned), separate OPFS files. `app/relationships.py:open_db()` ATTACHes fellows as `f` with `?mode=ro`; the worker owns the pair. | `tests/test_relationships.py::test_attach_fellows_readonly_allows_select`, `::test_attach_fellows_readonly_denies_write`; `tests/test_database.py` | conformant | +| AC-4 (versioned cross-boundary handshake) | `WORKER_RPC_VERSION`/`RELATIONSHIPS_SCHEMA_VERSION` in `vendor/sqlite-worker.js`; `EXPECTED_WORKER_RPC_VERSION` in `app.js`; `refuseIfVersionSkew()` gates mutating RPCs on mismatch, reads still pass; build label is not the gate. | `tests/e2e/test_version_handshake.py::test_version_skew_refuses_mutations_but_allows_reads`; `tests/e2e/test_worker_rpc.py` | conformant | +| AC-6 (always-reachable diagnostic escape) | `?gate=1` forces the email gate regardless of stuck state; Reset Everything + Clear App Cache `POST /api/logout` and reload. | `tests/e2e/test_email_gate.py`; `tests/e2e/test_reset_everything.py`; `tests/e2e/test_clear_app_cache.py` | conformant | +| AC-7 (self-service field-debug substrate) | Build label (AC-15), `?diag=1` state-dump, sanitized error capture (`deploy/client_error_sanitizer.py`), bug-report flow, `?gate=1` escape, boot watchdog with named phase marks, slow-boot persistence. | `tests/e2e/test_diagnostics_panel.py`; `test_boot_watchdog.py`; `test_boot_error_panel.py`; `test_bug_report.py`; `test_boot_beacon.py` | conformant | +| AC-9 (auto-backup of private data) | `vendor/sqlite-worker.js:maybeBackupRelationshipsDb()` — per-boot debounced (`BACKUP_DEBOUNCE_MS`), 5-slot rotation ring; folder mode writes the ring into the user folder. | `tests/e2e/test_user_folder_storage.py::test_snapshot_lands_in_folder_when_folder_mode_active`, `::test_opfs_to_folder_backup_migration_on_folder_boot`; restore via `tests/e2e/test_settings.py`. Debounce cadence by code inspection (LLM rubric). | conformant | +| AC-10 (opt-in non-destructive re-imports) | About-page *Update directory data*; `previewFellowsDbSwap()`/`applyFellowsDbSwap()` preview `group_members` orphaned by the swap before commit; one-shot soft scan. | `tests/e2e/test_directory_data_update_flow.py::test_apply_with_group_impact_shows_dialog_and_can_cancel`, `::test_apply_with_group_impact_confirm_completes_swap`; `test_orphan_soft_scan.py`; `test_versioned_fellows_db.py` | conformant | +| AC-11 (concurrent-access detection) | Worker `isOwnershipConflictError()` → `OWNERSHIP_CONFLICT` with a specific "another tab/window of this app is already open" message; Web Lock `fellows-relationships-folder-write` guards folder writes. | `tests/e2e/test_user_folder_storage.py::TestPhase2WriteLock::test_lock_held_during_write_surfaces_failure_then_recovers`; `test_worker_spawn_failure.py` | conformant | +| AC-15 (build label tied to source revision) | `build/build_pwa.py:compute_build_label()` → `-`, stamped into `app.js`/`sw.js`/`vendor/sqlite-worker.js` at build time; `app/server.py` substitutes the same at serve time. | `tests/test_build_pwa.py`; `tests/e2e/test_update_check.py`; `test_bug_report.py` (asserts `app: -`); `test_boot_beacon.py` | conformant | +| AC-16 (user-driven transport selection) | Group/fellow export surfaces `mailto:` (+ `tel:`); the user picks per outreach; no transport is hardcoded as the sole option. Axis pick is `comms-transport-set: mailto-only` (Signal planned). | `tests/e2e/test_groups_export.py`; `tests/test_comms.py` | partial-conformance (conformant to the `mailto-only` axis pick; richer transports planned) | +| AC-17 (mirrored data is sourced) | `build/restore_from_knack_scrapefile.py` maps every column to a Knack `field_*` (raw_dump fallback); no contact data introduced beyond the configured Knack source. | `tests/test_database.py`; `build/diff_fellows_db.py` (bytewise vs `fellows.db.backup.2026-04-08`, via `just db-verify`); [`./data_provenance.md`](./data_provenance.md) (human review) | conformant | +| AC-18 (transports cannot read message contents) | Only `mailto:` / `tel:` offered — no centralized SaaS message broker (Slack/Discord). `mailto:` hands to the user's client; MCP comms only stages a `mailto:` URL. | Architectural / human-review; `tests/test_comms.py` (stage-only, returns `mailto:` URL); `tests/test_private_data_ops.py` (`mode=ro`) | conformant | +| AC-19 (user-visible payload before send) | Group export panel shows recipients + subject + body + merged data before launch, editable/cancelable; bulk shows recipient count + warning. | `tests/e2e/test_groups_export.py`; `tests/e2e/test_groups_compose.py` | conformant | +| AC-PRM-A (LLM calls over user data are transports) | Cloud-LLM use is opt-in via the `EX-CLOUD-LLM` exception (consent gate → non-PNA mode); a local model is the default green path. Per-call prompt + merged-data visibility lives in the cloud client's own UI (the user drives Claude Desktop). | `tests/e2e/test_pna_exception_mode.py`; `test_mcpb_settings.py`; see Exception attestation below | partial-conformance (cloud opt-in via per-install consent; per-call prompt visibility is the cloud client's UI, not the workspace's) | +| AC-PRM-D (re-ingestion is user-initiated) | Directory-data refresh is an explicit About-page button only; boot is install-only and never background-polls. | `tests/e2e/test_directory_data_update_flow.py`; `test_versioned_fellows_db.py::test_install_only_does_not_refetch_on_sha_mismatch` | conformant | +| AC-MCP-A (cloud AI clients require consent for Private DB) | Realized as the `EX-CLOUD-LLM` exception: a workspace consent gate before the user wires up a cloud client + a persistent non-PNA-mode signal; `mcp_servers/private_data_ops.py` opens both DBs `mode=ro`. The stdio servers are not per-call gated by design (out-of-band from the workspace — see [`../plans/pna_toolkit_exceptions_contribution.md`](../plans/pna_toolkit_exceptions_contribution.md) open question). | `tests/e2e/test_pna_exception_mode.py`; `tests/test_private_data_ops.py` (`mode=ro`) | partial-conformance (per-session/per-install opt-in via `EX-CLOUD-LLM`; not per-call server-side gating) | +| AC-MCP-B (MCP Communications stages; workspace launches) | `mcp_servers/comms.py:stage_email()` returns a `mailto:` URL + payload preview and never fires a transport; the user's mail client launches it. | `tests/test_comms.py::test_stage_email_basic_to`, `::test_stage_email_bcc_group_send` | conformant | + +### Flavor-derived ACs triggered by fellows's picks + +Cross-referenced to the toolkit's [axes.md](https://github.com/richbodo/personal_network_toolkit/blob/main/axes.md): + +| AC | Triggered by | Realization | Verification | Status | +|---|---|---|---|---| +| AC-2 (no SaaS surface) | `dist:web-bundle-with-magic-link` | `deploy/server.py` ships no per-user RW endpoints; the dev server's retired `/api/groups` and `/api/settings` were the only ones that ever existed (Phase 1 cutover). | `tests/test_deploy_auth_round_trip.py::test_directory_api_is_403_without_session`; `test_deploy_sqlite_api.py`; `test_deploy_mcpb_routes.py` | conformant | +| AC-3 (single OPFS owner) | `storage:opfs-sqlite-wasm` | `app/static/vendor/sqlite-worker.js` is the sole context that calls `navigator.storage.getDirectory` or opens a `FileSystemSyncAccessHandle`. | `tests/e2e/test_worker_rpc.py`; `test_worker_cold_start.py`; `test_local_first_boot.py` | conformant | +| AC-5 (stale session never locks users out of cache) | `dist:web-bundle-with-magic-link` (auth-gated) | Three-tier `window.__dataProvider` hot-swaps `worker` → `api+idb` on 401/403 mid-boot; the cached directory stays readable. | `tests/e2e/test_offline_only_mode.py::test_returning_visit_renders_from_local_opfs_when_network_down`; `test_search_offline_fallback.py`; `test_local_first_boot.py` | conformant | +| AC-8 (anti-enumeration + abuse-bounded analytics) | `dist:web-bundle-with-magic-link` + `debug:has-error-sink` | `deploy/server.py` auth endpoints return neutral payloads with per-IP rate limits (`deploy/magic_link_auth.py:check_rate_limit`); the `/api/client-errors` sink is sanitized (`deploy/client_error_sanitizer.py`). See [`./email_gate.md`](./email_gate.md). | `tests/test_magic_link_auth.py`; `test_deploy_auth_round_trip.py`; `test_deploy_client_errors.py`; `test_client_error_sanitizer.py` | conformant | +| AC-12 (capability detection inside worker) | `storage:opfs-sqlite-wasm` | Worker `init` reports `opfsCapable`; the main thread reads the field and renders the unsupported-browser panel rather than UA-sniffing. | `tests/e2e/test_unsupported_browser.py::test_no_sah_falls_back_to_api_idb_provider`; `test_worker_cold_start.py` | conformant | +| AC-13 (COOP/COEP required) | `storage:opfs-sqlite-wasm` + `dist:web-served` | Both dev (`app/server.py:Handler.end_headers`) and prod (`deploy/server.py`) send `Cross-Origin-Opener-Policy: same-origin` + `Cross-Origin-Embedder-Policy: require-corp` and a strict CSP. Caddy preserves them at the edge. | `tests/test_api.py::TestSecurityHeaders::test_coop_coep_present`, `::test_strict_csp_present`, `::test_other_hardening_headers_present` | conformant | +| AC-14 (SW never owns SQLite) | `dist:web-bundle-with-magic-link` (PWA) | `app/static/sw.js` is app-shell + update/signature only; `/fellows.db` is explicitly bypassed in the fetch handler. | `tests/e2e/test_sw_post_caching.py`; `test_image_cache_no_bust.py` | conformant | + +### ACs that are not applicable in fellows's flavor + +| AC | Reason | +|---|---| +| AC-PRM-B | Applies to `ingestion:multi-source-merge-with-dedup`; fellows is single-source (`single-source-static-mirror`). | +| AC-PRM-C | Applies to `storage:native-sqlite-via-filesystem`; fellows uses `opfs-sqlite-wasm`. | + +Picks fellows did not take on other axes carry their own flavor-derived ACs in [axes.md](https://github.com/richbodo/personal_network_toolkit/blob/main/axes.md); none fire here. + +### Exception attestation (non-PNA mode) + +fellows raises one PNA **Exception** — `EX-CLOUD-LLM` — when the user wires the directory to a +cloud LLM (Claude Desktop MCP). See [`./architectural_findings.md`](./architectural_findings.md) for +the discovery and [`../plans/pna_toolkit_exceptions_contribution.md`](../plans/pna_toolkit_exceptions_contribution.md) +for the upstream-contribution plan (the handler contract `EX-H1..EX-H8`). + +| EX | Relaxes | Handled? | Realization | Verification | Status | +|---|---|---|---|---|---| +| EX-CLOUD-LLM | PNA-DEFINITION (local-only / never-SaaS), AC-MCP-A; stresses Goal 1 | yes; reversible (mode only) | Workspace-side handler: consent gate in Settings before the user wires up the cloud client (`recordMcpbConsent()`); persistent dismissable "Going rogue — not a PNA" banner naming the exception (`syncNotAPnaBanner()`, `index.html`); in-app explainer `#/exception/EX-CLOUD-LLM` rendering the **per-dimension strength profile** (`PNA_EXCEPTION_STRENGTH` → `renderExceptionPage()`, EX-H8); "Return to PNA mode" control (`returnToPnaMode()`); `` machine-readable marker. **EX-H7** consent-to-human propagation is surfaced best-effort via the MCP `instructions` handshake on the data-returning servers (`CLOUD_LLM_PROPAGATION_NOTICE` in `mcp_servers/private_data_ops.py` + `mcp_servers/shared_data_ops.py`); servers stay `mode=ro`, not per-call gated. Code: `app/static/app.js`, `app/static/index.html`, `mcp_servers/`. | `tests/e2e/test_pna_exception_mode.py` (raise/dismiss/persist, banner→explainer, return-to-PNA from explainer + Settings, explainer active/inactive/unknown, `test_explainer_shows_per_dimension_strength_profile`); `tests/e2e/test_mcpb_settings.py`; `tests/test_private_data_ops.py::test_instructions_carry_cloud_llm_propagation_notice`; `tests/test_shared_data_ops.py::test_instructions_carry_cloud_llm_propagation_notice` | conformant for EX-H1–H6 and EX-H8; EX-H7 (consent-to-human) surfaced best-effort via MCP `instructions` | + +--- + +## HTTP API + +Read-only fellow data (served from `app/fellows.db`): + +| Method | Path | Purpose | +|---|---|---| +| GET | `/api/fellows` | Minimal list (`record_id`, `slug`, `name`, `has_contact_email`) for instant directory render. | +| GET | `/api/fellows?full=1` | Full fellow rows (phase 2 of the two-phase load). | +| GET | `/api/fellows/` | One fellow by `slug` or `record_id`. | +| GET | `/api/search?q=…` | FTS5 search across name / bio / cohort / fellow_type / search_tags / key_links. | +| GET | `/api/stats` | Aggregates for the About page: total, breakdowns by fellow_type / cohort / region, field completeness. | +| GET | `/fellows.db` | Raw SQLite snapshot for the PWA's OPFS bootstrap. | +| GET | `/images/.{jpg,png}` | Profile image; alphanumeric-fuzzy filename fallback. | +| GET | `/` and other static paths | App shell from `app/static/`. | + +Production-only routes (added by `deploy/server.py`; conform to the Distribution slot's auth contract [`distribution-auth.openapi.yaml`](https://github.com/richbodo/personal_network_toolkit/blob/main/spec/contracts/distribution-auth.openapi.yaml)): + +| Method | Path | Purpose | +|---|---|---| +| GET | `/api/auth/status` | Never gated; returns `{authEnabled, authenticated, hasSessionCookie, installRecentlyAllowed, build, buildGitSha}`. | +| POST | `/api/send-unlock` | Anti-enum, always 200; rate-limited per email-hash. | +| POST | `/api/verify-token` | 200 + Set-Cookie on success; 401 with distinct `expired`/`invalid` strings otherwise. | +| POST | `/api/logout` | Idempotent, always 200. | +| POST | `/api/client-errors` | Unauthenticated client-error sink. Always 204. Sanitized + rate-limited; logs `event=client_error` to journald. Dev stub mirrors prod for round-trip. Schema: [`client-errors-payload.schema.json`](https://github.com/richbodo/personal_network_toolkit/blob/main/spec/contracts/client-errors-payload.schema.json); privacy boundary detailed in [`./email_gate.md` § Client error reporting](./email_gate.md#client-error-reporting). | +| GET | `/healthz` | Liveness probe. | +| GET | `/build-meta.json` | Build label + git SHA + `fellows_db_sha` for SW drift-check. | +| GET | `/api/debug/diagnostics` | Operator diagnostics blob. | + +The server opens a new SQLite connection per request (no pool; unnecessary at local scale). `/api/stats` is heavier than a simple row fetch (region split + field-completeness pass over `extra_json` via `json_extract`); still fine at local scale. + +The Two-Phase Load pattern (`/api/fellows` then `/api/fellows?full=1` in the background, falling back to `/api/fellows/` if the user clicks before phase 2 completes) is a Workspace concern; the route names are fellows specifics. + +--- + +## Cross-origin and CSP headers + +Both servers send (must be preserved by Caddy at the edge): + +- `Cross-Origin-Opener-Policy: same-origin` + `Cross-Origin-Embedder-Policy: require-corp` — AC-13 prerequisite for OPFS-SAH-Pool. +- `Cross-Origin-Resource-Policy: same-origin`, `Referrer-Policy: strict-origin-when-cross-origin`, `X-Content-Type-Options: nosniff`. +- A strict Content-Security-Policy: `default-src 'self'; script-src 'self' 'wasm-unsafe-eval'; worker-src 'self'; connect-src 'self'; img-src 'self' data:; style-src 'self'; font-src 'self'; object-src 'none'; base-uri 'self'; frame-ancestors 'none';` +- A locked-down `Permissions-Policy` (geolocation / camera / microphone / payment / sensors / USB / etc. all `=()`). + +HSTS is added by Caddy (`ansible/roles/caddy/templates/Caddyfile.j2`), not the Python server. + +**Subresource Integrity (SRI):** `index.html` carries SHA-384 `integrity=` attributes on both `