Add a scoped station journal companion API

[ryan-mahoney]

Add a scoped station journal companion API

Summary

This branch implements the station journal backend from the current main branch while retaining the companion-facing API shape introduced in PR #641. It adds journal entry synchronization, photo upload and public delivery, journal data in station bundles, and pin-coordinate refresh during station alignment.

PR #641 is used here as the compatibility reference for endpoint paths, request envelopes, response nesting, and the normal sync/upload/bundle workflow. The implementation is otherwise independent and introduces more explicit tenant scope, authorization, retry, storage, and database boundaries.

API shape retained from PR #641

Surface	Retained contract
Journal sync	POST /api/v1/versions/:version_id/stations/:station_id/sync, required pathways list, optional journal_entries list, client-generated UUIDs, and HTTP 200 partial-success responses with synced_count, conditional journal_synced_count, synced_at, and per-item errors.
Station bundle	Station entries remain in data.journal_entries; node and pathway entries remain nested under their corresponding bundle objects; pin entries remain under data.levels[].journal_entries; levels expose nullable stop_level_id.
Entry representation	Entry ID, target type/identifier, body, author, captured time, closure fields, and nested photos remain represented. Pin entries retain stop_level_id, diagram_coordinate, and nullable derived latitude/longitude.
Photo upload	POST /api/v1/versions/:version_id/stations/:station_id/journal-photos remains multipart with file and JSON metadata; successful creation and identical retry return HTTP 201 under data.photo.
Photo representation	Photo ID, owning journal entry ID, absolute public URL, media type, optional dimensions, and captured time remain represented.
Public delivery	Returned photo URLs remain readable without API authentication.
Pin alignment	Diagram coordinates remain canonical; latitude/longitude remain optional values derived when level alignment is applied.

The companion repository was not available while this branch was prepared, so the provider-side mobile workflow test uses PR #641's shapes as the executable compatibility baseline.

Observable incompatibilities and contract differences

These differences are called out for rollout and consumer review; they are not judgments about the earlier implementation.

Area	Behavior in this branch	Compatibility effect
Write authorization	Security: sync and journal-photo upload require pathways_studio_editor on the exact selected organization membership. Reads remain available to any active selected membership.	A user who previously wrote with membership alone now receives 403 unless that selected membership has the editor role. No migration grants roles automatically.
Pathway URL scope	Security / correctness: pathway IDs are loaded from the URL station and version once and outside-station IDs return per-item not_found.	PR #641 looked pathways up by organization; requests that relied on a station/version URL unrelated to the pathway no longer succeed.
URL station validation	Security / correctness: the version/station scope is resolved before processing either collection; malformed IDs return 400 and missing or cross-scope resources return top-level 404.	PR #641 could still process organization-scoped pathway items and report journal station failures in the HTTP 200 partial-success envelope.
Journal target membership	Security / correctness: node, pathway, and stop-level UUIDs must belong to the resolved station scope, not only satisfy the target's field shape.	Previously accepted references outside the station are now rejected as invalid_target.
Journal batch size	Performance / resource control: journal_entries is limited to 100 items.	Larger batches now return top-level HTTP 400 and must be split by the client.
Client audit fields	Security / correctness: organization, version, station, author, derived coordinates, closed_at, and closed_by are accepted as unknown compatibility fields but ignored. Closure remains response-only in this API slice.	A client attempting to change closure or audit ownership through sync will no longer persist those changes.
Entry UUID ownership	Security / correctness: a UUID is bound to its original organization/version/station. A competing scope receives non-disclosing id_conflict; same-scope replay may update only mutable body/target fields.	Global last-write behavior is replaced with scoped ownership and immutable author/capture/closure/derived fields.
Photo UUID retry	Correctness: an existing photo UUID succeeds only for the same scoped entry and identical detected bytes. Missing/corrupt materialization can be repaired from matching retry bytes; different ownership or digest returns 409.	Reusing a photo UUID with changed bytes no longer replaces the public file or metadata.
Photo validation	Security: JPEG/PNG type is selected from byte markers, the optional content-type hint must agree, dimensions must be positive when present, and the exact file maximum is 25 MiB.	Files previously accepted from declared MIME alone may now receive 422; a mismatched retry receives 409 rather than replacing content. Marker validation is not a full image decode.
Multipart request budget	Performance / security: only the exact photo POST route receives a 26 MiB request budget; all other multipart routes retain 8,000,000 bytes.	Valid photos between the default parser limit and 25 MiB can reach storage; oversized exact-route requests now receive a CORS-enabled JSON 413.
Empty journal list response	"journal_entries": [] produces "journal_synced_count": 0; omitting journal_entries omits the count.	PR #641 omitted the count for an explicitly empty list.
Error details	Journal sync distinguishes invalid_id, invalid_target, id_conflict, and validation_error; photo upload adds 409 id_conflict, 413 payload_too_large, and 500 storage_error. Some new 403/409/413/422 responses contain error.code without error.message.	Consumers matching exact error codes, status codes, or message presence should be reviewed.
Bundle photo fields	The shared serializer includes journal_entry_id in both upload and bundle photo objects.	This is additive for bundle photos; strict decoders that reject unknown fields should be checked.
Public photo headers	Security / correctness: UUID-named field captures receive server-selected Content-Type, X-Content-Type-Options: nosniff, and one-year immutable public caching.	Header behavior is more specific; replacement under an existing photo UUID is intentionally disallowed so immutable caching remains valid.
Ordering	Entries and photos are ordered by captured_at, inserted_at, then UUID.	The UUID tie-breaker makes equal-timestamp ordering deterministic.

Implementation inventory and differences

Persistence and domain boundaries

• Adds separate JournalEntry, JournalPhoto, StationJournal.Scope, StationJournal, and PhotoStorage modules rather than placing journal persistence in the general GTFS context.
• Uses a trusted scope struct containing organization, version, top-level station, GTFS station identifier, and authenticated actor. Controllers cannot supply loose tenant/audit fields to journal operations.
• Splits entry create, replay, and derived-coordinate changesets so mutable client fields and server-owned fields have distinct write paths.
• Normalizes photo tenant ownership through the required journal-entry foreign key instead of duplicating organization and version columns on every photo row.
• Adds database constraints for target shape, paired closure fields, allowed media types, positive byte size/dimensions, and an exact 32-byte SHA-256 digest.
• Retains historical entry rows when a node, pathway, stop-level, author, or closer is no longer resolvable; entries with deleted targets remain in station history but do not gain a new bundle location in this slice.

Synchronization correctness

• Loads allowed node, pathway, and stop-level UUID sets once per journal batch.
• Processes journal items in independent transactions so an invalid item does not roll back valid siblings.
• Uses insert-on-conflict, locked reload, and full-scope ownership comparison for client-generated UUIDs.
• Locks an existing entry before deriving and validating its effective replay target, avoiding validation against a stale snapshot.
• Carries each successfully updated pathway struct forward when duplicate pathway IDs appear in one request, preserving request-order last-write behavior.
• Accumulates pathway and journal errors by prepend/reverse rather than repeated list append.

Photo storage and recovery

• Streams files in 64 KiB chunks while enforcing size, retaining only small signature/tail windows, and computing SHA-256.
• Stages files to a unique path in the destination directory, then uses same-filesystem rename for publication.
• Uses one lock per scoped photo UUID so concurrent attempts cannot remove one another's active staging file.
• Treats the photo row as authoritative for owner, digest, media type, canonical filename, and original metadata; the deterministic public file is repairable materialization.
• Handles temporary-file/no-row, final-file/no-row, row/no-file, row/correct-file, and row/wrong-file retry states. Matching retries adopt or repair; mismatches return non-destructive conflicts and emit structured storage logs.
• Checks both canonical .jpg and .png candidates before adopting an orphan, preventing an alternate-extension file from being left reachable under the same UUID.
• Keeps public storage paths relative in the domain layer; the web serializer alone constructs absolute URLs.

Web composition

• Retains existing authenticated read routes and adds a selected-membership editor pipeline for sync/upload writes.
• Extends organization assignment to retain the exact membership used by authorization, including multi-organization selection through X-Organization-Id.
• Adds a path-aware multipart parser that delegates to Plug's parser and changes only the exact journal-photo route's request budget.
• Uses one JournalJSON serializer for upload and bundle photo representations.
• Adds fixed headers only for strict /uploads/field-captures/<org>/<station>/<uuid>.jpg|png paths; other uploaded-file behavior and traversal checks remain in place.

Pin alignment

• Refreshes matching pin coordinates inside the existing stop-level alignment transaction after alignment and child-stop writes.
• Reuses the existing alignment transform and coordinate normalization modules.
• Filters pin refresh by organization, version, station, target type, and stop-level ID, while preserving diagram coordinates and the existing alignment response shape.

Correctness, security, and performance notes

Correctness

• API tests cover station/node/pathway/pin placement, exact photo representations, duplicate request ordering, immutable audit fields, partial success, identity conflicts, crash-state repair, and public-file retrieval.
• Concurrent same-scope and competing-scope entry UUID tests exercise ownership and last-committed-write behavior.
• PostgreSQL and the uploads filesystem still cannot commit atomically. The retry state model repairs known user-visible crash states, but an orphan that is never retried requires operational cleanup.
• Same-scope mutable journal edits remain last-committed-write-wins because the compatibility surface has no revision token.
• Marker-valid JPEG/PNG checking does not establish that every image is fully decodable.

Security

• Tenant, station, author, closure, and derived-coordinate ownership are server-selected.
• Cross-organization/version/station lookups use 404 or non-disclosing item conflicts.
• Expanded multipart capacity is restricted to one method/path and storage applies a separate exact limit.
• Original filenames and declared MIME types never select storage paths or public response types.
• Public photo access remains intentional. Anyone with a returned URL can fetch the image, and the immutable cache policy permits long-lived intermediary/browser storage.
• Release coordination must confirm intended field users have pathways_studio_editor before enabling this branch.

Performance

• Bundle loading uses one scoped entry query with ordered photo preload, followed by in-memory grouping; it does not query once per target or photo.
• Journal target membership is indexed once per batch, file processing is streamed, request/error collections are bounded or accumulated linearly, and photo URLs remain static-file responses.
• The bundle still returns complete journal history without pagination. Response size should be measured on stations with production-like journal/photo counts.
• Existing-entry sync performs a pre-read and then a locked read per item. A 100-item replay therefore has avoidable sequential database round trips.
• Pin realignment currently updates each historical pin row individually while holding the alignment transaction.
• Stale-photo cleanup uses a directory wildcard for the scoped UUID; directory scan cost can grow with retained station photos.

Review findings and known follow-ups

The latest recorded branch review (reviews/branch-2-review.md) has a needs-fix verdict. Its matching fix record (reviews/branch-2-fix.md) resolves all actionable findings from that review, including response-envelope drift, retry metadata handling, duplicate pathway updates, linear error accumulation, orphan-extension checks, locked target validation, concurrent temp-file cleanup, parser tests, collision coverage, controller status coverage, and exact consumer assertions.

Deferred advisory items remain visible for follow-up:

• Performance: remove the existing-entry pre-read duplicated by the locked transaction query.
• Performance: consider bounded/bulk pin-coordinate updates and a temp layout that avoids directory wildcard scans.
• Correctness: handle output write failures without a MatchError, including guaranteed descriptor and temp-file cleanup.
• Correctness / observability: decide whether persisted pins with missing diagram coordinates should fail alignment or emit a structured warning instead of being skipped.
• Security test coverage: add explicit existing cross-organization and cross-version station fixtures for the top-level 404 path.
• Contract consistency: decide whether the new editor 403 and parser 413 responses should add stable human-readable messages.
• Persistence test coverage: add direct database writes for every named constraint in addition to changeset-level coverage.

There is no blockers.md, and the prepared spec records no open implementation blocker.

Verification

• A database-enabled development machine ran the full suite: 2,002 tests, 2 failures, 6 skipped. Both failures were in newly added test assumptions rather than production execution: the sparse-image helper did not create its temporary directory, and the invalid-field-capture assertion did not account for Phoenix's default private revalidation cache header. Commit 4db6c8f corrects both and removes an unused optional-argument warning.
• Follow-up targeted verification after that commit: upload plug suite, 13 tests and 0 failures; exact 25 MiB/25 MiB+1 storage-boundary smoke check passed; editor test module compiled without the warning; multipart parser suite, 4 tests and 0 failures; test-environment compile with warnings as errors passed; whitespace checks passed.
• Complete database-backed rerun after the final changes: 2,002 tests, 0 failures, 6 skipped.
• The same-ID storage contention test passed 100 consecutive repetitions with the lock-aware assertion timeout.
• mix compile --warnings-as-errors, dependency cleanup, formatting, and whitespace checks pass. The mix precommit alias still stops at the repository-wide Credo advisory backlog before tests; the journal-specific Credo warnings are cleared, and the full test gate was run independently.

Deployment and review considerations

• Apply the two journal migrations before serving journal routes.
• Verify field-companion memberships carry pathways_studio_editor; no automatic role migration is included.
• Confirm the companion tolerates the error/status differences, the 100-entry cap, ignored closure fields, additive journal_entry_id in bundle photos, and journal_synced_count: 0 for an explicit empty list.
• Confirm the configured uploads volume is shared and durable across application instances. Per-photo locks coordinate BEAM nodes through :global, while PostgreSQL row locks remain authoritative for durable identity.
• Monitor structured storage conflict/repair logs and complete-history bundle size after rollout.
• Public photo URLs and immutable caching are deliberate API behavior.

Spec folder: .specs/station-journal-api/

Closes #656

[themightychris]

We ran the pending verification item — the complete database-backed suite on 4db6c8f against a fresh Postgres — and reviewed the branch against the companion client. Recommend merging, with the items below.

Result: 2002 tests, 1 failure, 6 skipped. The two failures addressed in 4db6c8f are fixed; the remaining failure is new and flaky, not a product bug:

• Flaky test — photo_storage_test.exs:339 ("same-id staging serializes cleanup…") fails ~2 of 3 isolated runs here. PhotoStorage.acquire_lock/2 uses :global.set_lock, whose contention retry sleeps a random backoff that can exceed 100 ms, while the test's final assert_receive waits only the default 100 ms for the blocked second attempt to finish after the first releases. Product behavior is correct (acquisition succeeds, cleanup is serialized); the assertion timeout is just tighter than the lock's worst-case backoff. A longer assert_receive timeout resolves it. Related awareness item: that same backoff means a genuinely contended photo UUID can add up to ~1 s of upload latency — acceptable, but worth knowing.
• Rollout gate: role audit. No migration grants pathways_studio_editor, so any field collector whose selected membership lacks it loses sync/upload the moment this deploys. Please confirm intended field users carry the role before release. (The companion now degrades gracefully — read-only banner + blocked inputs — rather than failing confusingly.)
• Closure write path. closed_at/closed_by are schema + response only in this slice — there is no close/reopen operation. The Studio desk-review surface (feat(studio): station journal review surface in the diagram view #643) is the closing workflow, so it needs those context functions to exist somewhere. Fine to keep out of this PR; flagging so we agree where it lands when feat(studio): station journal review surface in the diagram view #643 is rebased onto this backend.

Client-side compatibility: we verified the retained contract against the shipped companion parsers (bundle shapes, per-item error message, journal_synced_count handling, data.photo envelope — all compatible) and merged client updates for the new constraints: request chunking for the 100-entry cap (JarvusInnovations/pathways-field-companion#94), byte-marker photo format validation (#95), and read-only mode for non-editors (#96). Also opened #656 requesting membership roles in the login response so the client can stop inferring writability via a probe.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HoGkr6FT6SKZCYDBkNqKBj