feat(#114): the outbound validate_startup hook, with its ledger banner (train)

docs/BACKLOG.md

@@ -1683,12 +1683,11 @@ lane; demand-gated on a first enterprise Windows/AD deployment.
 
 ## 114. Directory validation toggle (perform vs suppress startup validation)
 
-> 🔢 **Re-scored 2026-08-03 → DEMAND-GATE.** Value **6/10** · Difficulty **3/10** · _quick win_. The old score's "clean workaround via the on-demand test probe" does not exist in the direction that remains — both destinations' `test_connection` *create* the target directory, so the probe cannot answer the question the toggle asks, which is what lifts this off the parity-with-a-workaround band; the silent-ignore half is closed (PR #162 raises `WiringError` on `File(validate_directory=True)` for an outbound), leaving only the validation hook — a `validate_startup` on the `DestinationConnector` contract plus a runner outbound start-path call, mirroring the source seam already at `transports/base.py:436`. _(was 5/10 · 2/10.)_
-> **On-trigger / demand-gate.** Numbered for tracking only — build when the trigger below fires (“demand-gate, don’t schedule”).
+> ✅ **SHIPPED 2026-08-11 — the outbound half is built; #114 is complete.** `DestinationConnector.validate_startup()` is a **default no-op**, so the other eleven destination connectors are untouched and this is not a protocol change that ripples. `FileDestination` and `RemoteFileDestination` override it, and the runner awaits it in `_start_outbound` and `_ensure_destination_built` **inside the existing [ADR 0031](adr/0031-startup-connection-fault-isolation.md) isolation `try`** — so a refusal is a `failed` lane with no connector, whose delivery worker **still spawns**: routed rows are retained, retried and buildup-alerted rather than dropped, and the count-and-log invariant holds. **The defect, stated in the conditional:** `FileDestination` previously `mkdir`-ed its target on write, so a deploying operator who typo'd `remote_dir` would get a directory silently **created** and messages delivered into it — the feed reading healthy while writing to the wrong place, with nothing reporting it. `validate_directory` becomes a **both-directions** option and the 2026-08-03 outbound `WiringError` is **removed** — it existed only because no destination read the setting. Under the toggle nothing is ever created: not at start, not on write, and not by `POST /connections/{name}/test`. **One deliberate change reaches every existing outbound, toggled or not:** a target directory the engine actually had to **create** now logs a `WARNING`, so a delivery into an invented path is no longer indistinguishable from a normal one. ADR 0031 is amended, with its 2026-08-03 follow-on marked superseded. **This item's own 6/10 rationale was re-measured and is FALSE in this direction.** It claimed a *"clean workaround via the on-demand test probe"* — but both destinations' `test_connection` **create the directory**, so the act of asking changes the answer. An item's scoring rationale falsified by execution, the same class as #1011's refuted premise.
 
-> **AMENDED 2026-08-03 — the INBOUND half and the outbound WIRING REJECTION are BUILT; only the outbound validation HOOK remains.** Adversarial verification refuted a full close. **BUILT 2026-07-28:** `validate_directory` on the File/RemoteFile source (`messagefoundry/transports/file.py:311`, `remotefile.py:735`) with its opt-in at-start check (`file.py:389-398`) — a no-mkdir probe that reports the connection `failed` at start rather than deferring to first poll (`file.py:170`). **BUILT 2026-08-03:** the option on an **outbound** is now a **`WiringError` at bind** (`build_outbound_connection`, `messagefoundry/config/wiring.py`) instead of being accepted and silently ignored. That is the single choke point both code-first `outbound()` and the `connections.toml` loader (ADR 0007) pass through, so one guard covers both authoring surfaces; it is truthy-only, so the `False` the factories always write into settings is unaffected and every outbound authored today builds byte-identically.
+> **AMENDED 2026-08-03 — SUPERSEDED 2026-08-11 by the banner above, which closed the remaining outbound half. Kept as the record of how the item was built in stages.** At the time it read *"only the outbound validation HOOK remains"*, which was true then and is not now. **One statement below was also REVERSED by the close:** the 2026-08-03 outbound `WiringError` described as BUILT was subsequently **removed**, because it existed only to reject a setting no destination read -- and a destination now reads it. Adversarial verification refuted a full close. **BUILT 2026-07-28:** `validate_directory` on the File/RemoteFile source (`messagefoundry/transports/file.py:311`, `remotefile.py:735`) with its opt-in at-start check (`file.py:389-398`) — a no-mkdir probe that reports the connection `failed` at start rather than deferring to first poll (`file.py:170`). **BUILT 2026-08-03:** the option on an **outbound** is now a **`WiringError` at bind** (`build_outbound_connection`, `messagefoundry/config/wiring.py`) instead of being accepted and silently ignored. That is the single choke point both code-first `outbound()` and the `connections.toml` loader (ADR 0007) pass through, so one guard covers both authoring surfaces; it is truthy-only, so the `False` the factories always write into settings is unaffected and every outbound authored today builds byte-identically.
 >
-> ⚠️ **REMAINDER: the outbound validation HOOK — and this item's scoring rationale is WRONG for that direction.** `DestinationConnector` still has no `validate_startup` hook and `FileDestination` still `mkdir`s on write. The "clean workaround via the on-demand test probe" cited in the score above **does not exist on an outbound**: both destinations' `test_connection` *create* the target directory (see ADR 0031's 2026-08-03 follow-on for the call chain), so nothing shipped can tell "the directory exists" from "I just made it" — a typo'd target path is fabricated and every message reports delivered. **Re-score against that.** And if the hook is built, build it **together with** suppressing the mkdir-on-write under the flag: a start-time-only check leaves the run-time fabrication intact under a setting name that promises otherwise.
+> ⚠️ **WHAT THE REMAINDER WAS, and the scoring rationale it refuted — BUILT 2026-08-11, so read the two claims below in the PAST TENSE.** `DestinationConnector` **had** no `validate_startup` hook and `FileDestination` **mkdir'd** on write. The "clean workaround via the on-demand test probe" cited in the score above **does not exist on an outbound**: both destinations' `test_connection` *create* the target directory (see ADR 0031's 2026-08-03 follow-on for the call chain), so nothing shipped can tell "the directory exists" from "I just made it" — a typo'd target path is fabricated and every message reports delivered. **That instruction was a build gate, and it was HONOURED:** it required that the hook be built **together with** suppressing the mkdir-on-write, because a start-time-only check would leave the run-time fabrication intact under a setting name that promises otherwise. Under the shipped toggle nothing is created at start, on write, or by `POST /connections/{name}/test`.
 
 **Cluster:** Connections & Transports. **Priority:** P3. **Verdict:** demand-gate. **Severity (vs Corepoint):** minor.
 

docs/CONNECTIONS.md

@@ -648,7 +648,7 @@ def route(msg):
 | `sort` | in | `name` | process order: `name` or `mtime` |
 | `recursive` | in | `false` | also scan subdirectories |
 | `max_file_bytes` | in | `16 MiB` | route files larger than this to the error dir instead of reading them into memory (OOM guard). `None`/`0` = unlimited. |
-| `validate_directory` | in | `false` | validate the poll directory **at startup** (#114): a missing/unusable dir reports the connection **`failed`** (ADR 0031) instead of the default deferral to run time. No mkdir — a merely-missing dir fails. A `leave` source validates read-only (a read-only share passes); `move`/`delete` also require write. **Inbound only** — on an outbound it is a `WiringError` at bind (an outbound target directory is never validated at startup; it is `mkdir`ed on write). |
+| `validate_directory` | both | `false` | validate the directory **at startup** (#114): a missing/unusable dir reports the connection **`failed`** (ADR 0031) instead of the default deferral to run time. **No mkdir** — a merely-missing dir fails. **In:** a `leave` source validates read-only (a read-only share passes); `move`/`delete` also require write. **Out:** the target must already exist and accept a write, and is then **never created** — not at start, not on write (a delivery into a vanished dir fails retryably instead), and not by `POST /connections/{name}/test`. Left off (the default) the outbound target is still created on first write, but the creation is now logged as a `WARNING`. |
 | `processed_subdir` / `error_subdir` | in | `.processed` / `.error` | where read/failed files go |
 | `filename` | out | `{MSH-10}.hl7` | output name (supports `{HL7-path}` placeholders). Resolved values are sanitized to a **single safe filename** — path separators/unsafe chars stripped, leading dots removed, and `.`/`..`/reserved device names fall back — so a message field can never write outside the directory. |
 | `overwrite` | out | `false` | overwrite vs. uniquify a name collision (collisions are resolved by an **atomic** exclusive create, so concurrent writes never clobber) |
@@ -871,7 +871,7 @@ poll/write shape against a remote server, selected by an internal `protocol` set
 | `min_age_seconds` | in | `0.0` | **accepted but not honoured on a remote source today** — the connector never reads it (a remote directory listing carries no reliable mtime). Only `File(...)` implements it; use `after_read`/the partner's own write-then-rename to avoid partial reads. |
 | `after_read` | in | `move` | `move` (→ `processed_subdir`), `delete`, or `leave` (process **in place**, #142 — a durable dedup ledger keyed on a hash of the **full remote path** + size ensures a left file is ingested once) |
 | `max_file_bytes` | in | `16 MiB` | move a file larger than this to `error_subdir` instead of retrieving it (OOM guard). `None`/`0` = unlimited. |
-| `validate_directory` | in | `false` | validate `remote_dir` **at startup** (#114): unreachable/unusable reports the connection **`failed`** (ADR 0031) instead of deferring to run time. **Inbound only** — on an outbound it is a `WiringError` at bind (the upload dir is `ensure_dir`ed on write, never validated at startup). |
+| `validate_directory` | both | `false` | validate `remote_dir` **at startup** (#114): unreachable/unusable reports the connection **`failed`** (ADR 0031) instead of deferring to run time. The probe is a **listing** — it never creates. **Out:** the upload dir is then never `ensure_dir`ed either, on send or by `POST /connections/{name}/test`; an upload into a vanished dir fails **retryably** rather than dead-lettering on the partner's permanent no-such-dir. Left off (the default) the upload dir is still created on first send, but the creation is now logged as a `WARNING`. |
 | `processed_subdir` / `error_subdir` | in | `.processed` / `.error` | where read / failed files go |
 | `filename` | out | `{MSH-10}.hl7` | upload name (supports `{HL7-path}` placeholders, sanitized to a **single safe filename** exactly as `File(...)`) |
 | `overwrite` | out | `false` | overwrite vs. uniquify a name collision (never a silent clobber) |

docs/adr/0031-startup-connection-fault-isolation.md

@@ -198,13 +198,77 @@ The equivalent outbound (FileDestination) is out of scope here — it already `m
 the on-demand `POST /connections/{name}/test` probe.
 
 **Follow-on (2026-08-03, BACKLOG #114) — the outbound rejects the option rather than ignoring it.**
-Because `File()`/`Sftp()`/`Ftp()` are single factories serving both directions, the option above could
-be *written* onto an outbound, where nothing reads it — accepted and silently ignored. That is now a
-**`WiringError` at bind** in `build_outbound_connection`, the one choke point both the code-first
-`outbound()` and the `connections.toml` loader (ADR 0007) pass through. Truthy-only, so the `False`
-the factories always write is unaffected and every outbound authored today builds byte-identically.
-The outbound *validation hook* itself (`DestinationConnector.validate_startup`) remains **out of
-scope** and deferred — note that the "on-demand test probe" workaround cited above is **inbound-only
-in effect**: `FileDestination.test_connection` → `_probe_dir_writable` and
-`RemoteFileDestination.test_connection` → `ensure_dir` both **create** the target directory, so on an
-outbound no shipped mechanism can distinguish "the directory exists" from "I just made it."
+**Superseded by the 2026-08-10 amendment below, which builds the hook and removes this `WiringError`;
+kept for the reasoning, which still holds.** Because `File()`/`Sftp()`/`Ftp()` are single factories
+serving both directions, the option above could be *written* onto an outbound, where nothing read it —
+accepted and silently ignored. That was made a **`WiringError` at bind** in
+`build_outbound_connection`, the one choke point both the code-first `outbound()` and the
+`connections.toml` loader (ADR 0007) pass through. The outbound *validation hook* itself
+(`DestinationConnector.validate_startup`) was out of scope here — note that the "on-demand test probe"
+workaround cited above is **inbound-only in effect**: `FileDestination.test_connection` →
+`_probe_dir_writable` and `RemoteFileDestination.test_connection` → `ensure_dir` both **create** the
+target directory, so on an outbound no shipped mechanism could distinguish "the directory exists" from
+"I just made it."
+
+## Amendment (2026-08-10, BACKLOG #114) — the outbound half: `DestinationConnector.validate_startup`
+
+**Status:** Accepted (owner go — build the remainder). Built in the same change.
+
+**Context.** The 2026-07-17 amendment deferred the outbound hook on two grounds: the destination
+"already `mkdir`s on write", and it "has the on-demand `POST /connections/{name}/test` probe". The
+2026-08-03 follow-on already withdrew the second (both destinations' `test_connection` **create** the
+directory — re-measured against the shipped code before this amendment was written, on a missing
+directory, for both FILE and REMOTEFILE). This withdraws the first: `mkdir`-on-write is not a weaker
+form of validation, it is the defect. A typo'd `directory`/`remote_dir` does not fail — it is
+**created**, and every message delivered into it is counted and logged as delivered, because it was.
+On a first deployment that is a feed landing in a path nobody is watching with no error anywhere.
+(Nothing is misdelivering today; there are zero deployments — see CLAUDE.md §0.)
+
+**Decision.** Three parts.
+
+1. **`DestinationConnector.validate_startup()`**, defaulting to a **no-op** — the exact shape of the
+   `SourceConnector` hook above, so the other eleven destination connectors are untouched and this is
+   not a protocol change that ripples. `FileDestination` and `RemoteFileDestination` override it. The
+   runner awaits it in `_start_outbound` immediately after the connector is built, **inside the
+   existing ADR-0031 isolation `try`**: a `DestinationStartupError` therefore takes the same path as a
+   build failure — the lane is recorded `failed` with **no live connector**, its delivery worker is
+   **still spawned**, and rows routed to it are retried + buildup-alerted, never dropped. On an
+   outbound, "invalid means not-started" *is* that degraded-lane state, so §1's reliability and
+   count-and-log invariants are preserved rather than re-argued. The same call is made on the operator
+   start path (`_ensure_destination_built`, which already isolates rather than raises). It is **not**
+   made on the reload path, whose stated invariant is that a connector build there cannot fail (intake
+   is quiesced at that point, so a raise would strand the swap).
+
+2. **`validate_directory` becomes a both-directions option** on `File`/`Sftp`/`Ftp`, and the
+   2026-08-03 outbound `WiringError` is **removed**. That guard existed for exactly one reason — no
+   destination read the setting — and that reason is now gone; keeping it would mean shipping the hook
+   behind a second, differently-named knob. Default stays `false`, so every outbound authored today
+   builds and runs byte-identically.
+
+3. **A created directory is loud.** Under the default (defer) arm the target is still created on
+   write and the delivery still succeeds — but a create that actually happened now logs a `WARNING`
+   naming the path. This is the half that applies to every existing outbound, because it is the
+   default arm: the failure mode being closed is silence, not the creation itself.
+
+Three deliberate details, each the mirror of a source-side one:
+
+- **No-create at every asking point.** `validate_startup` uses `_probe_dir_startup` (FILE) or a
+  `list_dir` (REMOTEFILE) — never `_probe_dir_writable`/`ensure_dir`, both of which create. Under
+  `validate_directory=true` `test_connection` switches to the same no-create probes, because
+  otherwise the operator's own `POST /connections/{name}/test` would silently repair the typo the
+  toggle exists to catch and the next restart would then validate clean.
+- **No-create at delivery time too, under the toggle.** "This directory must exist" has to keep
+  meaning that after start, so `_write`/`_upload` do not create it either: a share that vanished
+  mid-run fails the send **retryably** and the lane backs off and self-heals. The REMOTEFILE arm
+  pre-checks with a `list_dir` specifically to reclassify — an SFTP/FTP no-such-dir is a **permanent**
+  error, so letting the upload fail naturally would dead-letter live traffic over a merely-unmounted
+  share. It costs one extra round trip per delivery, on the opt-in path only.
+- **The default arm still serves the item's own trigger.** An intermittently-available directory must
+  **not** fail startup — which is why the toggle is opt-in and defaults to defer in both directions.
+
+**Consequences.** Additive; a graph that never sets `validate_directory` on an outbound behaves as
+before apart from the create-on-write WARNING. The FILE default path's syscall count is unchanged
+(`mkdir(parents=True, exist_ok=True)` already probed `is_dir()` on its `FileExistsError` branch, which
+is the common one). `_RemoteClient.ensure_dir` now reports whether it created — a module-private
+contract with two implementations. No new schema and no new dependency; a refusal rides the existing
+`_failed`/`failed` surfacing and the `connection_stopped` alert.