Skip to content
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,17 @@ All notable changes to MessageFoundry are documented here. The format follows
documenting the weakness without changing the gate.

### Fixed
- **A CR/LF inside an exception message could forge a whole log line on the text sink.**
`ControlCharScrubFilter` escaped only the rendered message, and `logging.Formatter` appends a record's
traceback (`exc_text`) and stack dump (`stack_info`) **verbatim** — so a newline-bearing exception
string landed at column 0 on its own physical line, where a payload padded to the record layout was
byte-indistinguishable from a real entry to an operator or a line-oriented SIEM parser. Both fields
are now scrubbed too (ASVS 16.4.1; the residual ADR 0034 §1 disclosed, BACKLOG #335). **Visible
change:** a traceback is *not* collapsed onto one line — its line breaks are kept and every line is
indented with ` | `, so it stays readable while no line of it can start at column 0. A log parser
keyed on `Traceback (most recent call last):` at the start of a line needs that prefix added. The
JSON sink is unchanged in substance (`json.dumps` already escaped these fields); its `exception`
and `stack` values now carry the same indent.
- **The DICOM C-STORE SCP's fail-closed refusal named a settings key that does not exist.** It told
the operator to set `[inbound].source_ip_allowlist`; `InboundSettings` has no such field and section
models ignore unknown keys, so an operator following the engine's **own error message** wrote a key
Expand Down
41 changes: 41 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,43 @@ stops matching the code, fix the doc.

---

## 0. Deployment status — read this before writing any severity claim

> ⛔ **MessageFoundry is a NOT-DEPLOYED beta. There are ZERO production instances. Nobody is
> running it.** **Published to PyPI is *not* deployed** — a release artifact on an index is not a
> running instance, and the two get conflated constantly. Distinguish **shipped** (on `main`, on
> PyPI), **deployable**, and **deployed**: only the first two are true today.

This is load-bearing because the wrong premise silently corrupts severity, urgency, and prose
across the repo. **Two consequences, and they pull in opposite directions — apply both:**

1. **Present-tense impact claims are factually false.** *"PHI is exposed"*, *"customers are
affected"*, *"operators rely on this today"*, *"live feeds are shipping X"*, *"this needs an
incident response"* — none of these are true of anything here. Write beta defects in the
conditional: **"would expose X on first deployment"**, *"a deploying site would hit Y"*, *"is
wrong in the shipped code"*. False present tense does not stay local; it propagates into
security scorecards, review registers, BACKLOG banners and public docs, and a security record
asserting a live exposure that does not exist is exactly the *"compensating control resting on
a false premise"* defect §11 forbids.
2. **Hypothetical migration costs are vacuous.** *"breaks a running deployment on upgrade"*,
*"operators need notice / a migration window / a deprecation period"*, *"backward compatibility
with what sites have configured"* — there is nothing to break and nobody to notify, so the cost
of a breaking change is currently **zero**. Prefer the simple, correct end state over a staged
migration or compatibility shim; those are real costs paid to protect users who do not exist.

⛔ **It cuts one way only — never cite "not deployed" to relax a rule.** It removes false urgency
and vacuous costs. It does **not** downgrade a fix, justify skipping a gate, weaken a control, or
make a finding unimportant. The security, PHI (§9) and leak-gate rules exist so the **first**
deployment is safe; zero deployments is why there is still time to get them right, not permission
to lower the bar. Note that §9's *"this engine carries PHI"* is a statement about the design and
intended use — **not** evidence of a live PHI-carrying instance.

This is an **owner-stated fact**, repeatedly. Do not re-derive it, do not go looking for
deployments to confirm it, and do not soften it to "as far as I can tell". If an adopter ever goes
live, this section must be revised first — check with the owner before assuming it still holds.

---

## 1. Project Overview

MessageFoundry routes, transforms, and validates HL7 v2.x messages between **connections**,
Expand Down Expand Up @@ -370,6 +407,10 @@ python samples/send_mllp.py samples/messages/adt_a01.hl7
This engine carries PHI. The full PHI map — threat model, data-at-rest inventory, redaction rules,
and the retention/encryption roadmap + secure-ops checklist — is [`docs/PHI.md`](docs/PHI.md). Treat
these as hard rules:

> "Carries PHI" describes the **design and intended use** — it is not a claim that a live instance is
> holding PHI today (§0: zero deployments). That changes how you word a *finding*, never whether these
> rules apply: they are what make the first deployment safe, so none of them relax.
- **Never log full message bodies at INFO or above.** Full payloads go only to the secured
store, never to the general log. (Logging is stdlib today; structlog + redaction is planned —
until then, don't raise the service to `DEBUG` in production.)
Expand Down
4 changes: 2 additions & 2 deletions docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2822,7 +2822,7 @@ No test covers it. `tests/test_scan_tokens_source.py:559-583` (`test_absolute_ho

4. Add the regression case to `tests/test_scan_tokens_source.py:559`, alongside the existing canonical fixtures — a lowercased and an upper-cased Windows path must both produce a hit, and the POSIX `/users/…` non-match should be asserted deliberately so the next person does not "fix" it into the 47-false-positive form.

5. **Same fix site, sibling defect:** `_WORKTREE_SLUG` at `scripts/security/scan_forbidden.py:92` is case-blind the same way (`[a-z0-9]+`); `claude/Some-Task-a1b2c3` is MISSED. `scripts/worktree/new.ps1:43,86` passes `-Name` through verbatim with no lowercasing, so an upper-cased worktree name is reachable. Narrower than the home-path case (agent-created slugs are lowercase by convention), but it is a two-character edit in the same block — take it in the same change or say why not.
5. **Same fix site, sibling defect:** `_WORKTREE_SLUG` at `scripts/security/scan_forbidden.py:92` is case-blind the same way (`[a-z0-9]+`); an upper-cased slug — `claude/` followed by `Some-Task-a1b2c3` is MISSED. (Written split on purpose, for the reason in the note above: once the fix lands, the joined literal trips the very detector it documents, and unlike `_HOME_PATH` the slug pattern has no `<…>` exemption to write it into.) `scripts/worktree/new.ps1:43,86` passes `-Name` through verbatim with no lowercasing, so an upper-cased worktree name is reachable. Narrower than the home-path case (agent-created slugs are lowercase by convention), but it is a two-character edit in the same block — take it in the same change or say why not.

**Related:** `scripts/security/scan_forbidden.py` (`_HOME_PATH` :99-106, `_WORKTREE_SLUG` :92, call site :758-759), `tests/test_scan_tokens_source.py:559-583`, `.github/workflows/security.yml:446-493`, `.github/required-contexts.txt`, `scripts/worktree/new.ps1`. Sibling **#321** — same gate, same "green gate that cannot see the class" root cause, but the **opposite mechanism**: #321 is an incomplete *token source* (data, fixed by the owner updating a private secret) and explicitly scopes itself away from scanner defects at `docs/BACKLOG.md:7356`; this is a *structural detector* defect (code, fixed by a regex edit) that is live even with no token source. Also **#322**, and the anonymizer's structural-detector item from this same audit. Note #321's **Related:** line at `docs/BACKLOG.md:7363` cites `tests/test_scan_forbidden.py` for regression tests, but the home-path test actually lives in `tests/test_scan_tokens_source.py` — worth correcting when someone next touches #321.

Expand Down Expand Up @@ -3290,7 +3290,7 @@ What is *not* covered is the thing that will grow: `.semgrep/messagefoundry.yml`

## 335. Control-char scrub misses `exc_text`/`stack_info`

> 🔢 **Filed 2026-08-01 — not started.** Value **4/10** · Difficulty **3/10** · _fill-in_. `ControlCharScrubFilter.filter` still translates only `record.getMessage()` while `RedactionFilter` is the sole toucher of `exc_text`/`stack_info` (`logging_setup.py:124-131`), so a CR/LF traceback can forge a record on the text sink — but `JsonFormatter` escapes C0 regardless, the off-box forwarder defaults to json, and the message-path `exc_info` sites are a handful of non-peer-derived guards, so it is log-record integrity on one sink; the filter already runs last, so the cost is the readability call ADR 0034:146 defers plus tests and an ADR amendment.
> **DONE (2026-08-04).** `ControlCharScrubFilter.filter` now applies `_CTRL_TRANSLATION` to `record.exc_text` and `record.stack_info` as well as the rendered message, so a CR/LF-bearing traceback can no longer forge a record on the text sink. The readability call ADR 0034 §1 deferred was taken explicitly and amended there in the same commit: the traceback is **not** collapsed to one line — its line breaks are kept and every line is indented with `_CONTINUATION_PREFIX` (`" | "`), so no traceback line starts at column 0 and none can impersonate `_LOG_FORMAT`. Pinned by the `test_control_char_*` tests in `tests/test_logging.py`. One residual stays open and is recorded in ADR 0034 §1: a handler carrying this filter *without* `RedactionFilter` would still hand the formatter an unrendered `exc_info` — no shipped handler is in that state. Scored **4/10** value · **3/10** difficulty when filed 2026-08-01.

**Cluster:** Security / Logging. **Priority:** P3. **Verdict:** build (small). **Severity:** low.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -141,10 +141,18 @@ the class rationale **must not be inherited** by a future finding on a `log.exce
`exc_info=True` site — the engine has many (the delivery/router/transform catches, the `_on_*_worker_done`
callbacks, the pollers). `JsonFormatter` escapes `exc_text` through `json.dumps`, so the off-box
forwarder (JSON by default) is unaffected; the residual is the human-readable stdout/NSSM text log.
**Open hardening (not done):** apply `_CTRL_TRANSLATION` to `exc_text`/`stack_info` in
`ControlCharScrubFilter` — it runs after `RedactionFilter`, so `exc_text` is already populated. It is a
few lines, but it collapses every traceback to one physical line, which is an operator-facing
readability change and wants an explicit decision rather than a drive-by edit.
**Open hardening — CLOSED 2026-08-04 (BACKLOG #335).** `ControlCharScrubFilter` now scrubs
`record.exc_text` and `record.stack_info` as well as the rendered message. The readability decision
deferred above was taken explicitly, and it is **not** the collapse-to-one-line this paragraph feared:
the traceback keeps its line breaks and every line is indented with `_CONTINUATION_PREFIX`, so no
traceback line begins at column 0 and none can impersonate the `_LOG_FORMAT` record prefix. Why the
block's *first* line is indented too, and why re-application is idempotent (every handler carries its
own filter chain, so one record is scrubbed once per sink), is recorded at `_scrub_block`; the property
is pinned by the `test_control_char_*` tests in `tests/test_logging.py`. **One residual survives, so the
register line at `:40` still reads wider than the code:** a handler carrying `ControlCharScrubFilter`
*without* `RedactionFilter` hands the formatter an unrendered `exc_info` that no filter has touched. No
shipped handler is in that state — `_install_phi_filters` installs both — but that is a construction
guarantee, not a scrub. The paragraph above stands as the record of what was true before.

**2. `PinnedDependenciesID` — the blanket rationale was applied too widely.** "CI installs editably
(`pip install -e .[extras]`), which cannot use `--require-hashes`" is true of the editable installs and
Expand Down
40 changes: 39 additions & 1 deletion messagefoundry/logging_setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,30 @@
_CTRL_TRANSLATION[_i] = f"\\x{_i:02x}"
_CTRL_TRANSLATION[0x7F] = "\\x7f"

#: Stamped on every physical line of a record's ``exc_text``/``stack_info`` (BACKLOG #335). A traceback
#: is multi-line by nature, so collapsing it the way the rendered message is collapsed would cost the
#: operator the readability an incident depends on. Its line breaks are kept and every line is indented
#: instead, so no traceback line starts at column 0 and none can impersonate the ``_LOG_FORMAT`` record
#: prefix (ASVS 16.4.1 — the readability call ADR 0034 §1 deferred).
_CONTINUATION_PREFIX = " | "


def _scrub_block(text: str) -> str:
"""Escape control characters in a multi-line block (``exc_text``/``stack_info``) while KEEPING its
line breaks, indenting every line with :data:`_CONTINUATION_PREFIX`.

The **first** line is indented too, so the guarantee does not rest on it being the stdlib
``Traceback (most recent call last):`` header: ``Formatter.formatException`` emits no header at all
when the exception carries no ``__traceback__``, and that first line is then peer-derived text.

Idempotent — the prefix is stripped before it is re-applied — because every handler carries its own
filter chain, so a record dispatched to stdout *and* the off-box forwarder is scrubbed twice and the
two sinks must not disagree."""
return "\n".join(
_CONTINUATION_PREFIX + line.removeprefix(_CONTINUATION_PREFIX).translate(_CTRL_TRANSLATION)
for line in text.split("\n")
)


class ControlCharScrubFilter(logging.Filter):
"""Neutralize CR/LF and other control characters in the rendered log message to prevent log
Expand All @@ -76,14 +100,28 @@ class ControlCharScrubFilter(logging.Filter):
Untrusted MLLP peer data and HL7-derived exception text reach the general log; without this a
crafted value containing a newline could inject a forged log line into NSSM's captured stdout.
We render the message (applying ``%`` args) once, escape any control characters, and only then
replace ``record.msg`` — clean messages keep their lazy ``msg``/``args`` untouched."""
replace ``record.msg`` — clean messages keep their lazy ``msg``/``args`` untouched.

``record.exc_text`` and ``record.stack_info`` are covered too (BACKLOG #335, ADR 0034 §1), via
:func:`_scrub_block`. This filter is installed **last** (see :func:`_install_phi_filters`), so
:class:`RedactionFilter` has already rendered ``exc_info`` into ``exc_text`` and cleared it; a
handler carrying this filter *without* that one would leave an unrendered ``exc_info`` for the
formatter to expand unscrubbed."""

def filter(self, record: logging.LogRecord) -> bool:
message = record.getMessage()
scrubbed = message.translate(_CTRL_TRANSLATION)
if scrubbed != message:
record.msg = scrubbed
record.args = ()
# The rendered message is only half the record: ``Formatter.format`` appends ``exc_text`` and
# ``stack_info`` VERBATIM, so a CR/LF inside an exception message forged a whole line on the
# text sink (BACKLOG #335). ``RedactionFilter`` is installed first and renders ``exc_info``
# into ``exc_text``, so both fields are already populated when this filter runs.
if record.exc_text:
record.exc_text = _scrub_block(record.exc_text)
if record.stack_info:
record.stack_info = _scrub_block(record.stack_info)
return True


Expand Down
20 changes: 17 additions & 3 deletions scripts/security/scan_forbidden.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,16 +88,30 @@
#
# A worktree/branch slug is whatever the task happened to be CALLED, so it can name a prospect segment,
# a customer engagement, or a competitor study. That is unbounded: the leak is the project name itself,
# and there is no list to add it to. Matching the shape is the only control that scales.
_WORKTREE_SLUG = re.compile(r"(?:claude/|worktrees/)[a-z0-9]+(?:-[a-z0-9]+)*-[0-9a-f]{6}")
# and there is no list to add it to. Matching the shape is the only control that scales. It is
# case-folded whole: agent slugs are lowercase by convention, but scripts/worktree/new.ps1
# validates -Name as ^[A-Za-z0-9._-]+$ and hands it to `git worktree add -b` verbatim, so an
# upper-cased slug is reachable -- and unlike _HOME_PATH no common URL shape collides here.
_WORKTREE_SLUG = re.compile(r"(?i:(?:claude/|worktrees/)[a-z0-9]+(?:-[a-z0-9]+)*-[0-9a-f]{6})")
# An absolute user-home path carries the OS account name, and inside a worktree path the slug as well.
# Exempt: bracket/env placeholders (<you>, $HOME, %USERPROFILE%, {home}), the well-known shared and CI
# accounts, and the DOCUMENTATION placeholder names this repo already uses in examples (me, svc, you,
# user, username, example). Everything else looks like a real account and fires. That list is the whole
# judgement call here: "is this a real person's login" is not decidable by shape, so the pattern trusts
# a small, explicit set of conventional stand-ins and treats anything else as a disclosure.
#
# The drive-letter arm folds case INLINE. Windows paths are case-INSENSITIVE, so `C:\Users\<name>`,
# `c:\users\<name>` and `C:\USERS\<name>` are the SAME directory naming the SAME account, and a
# literal `Users` caught only one of those four spellings. (The examples use the `<name>`
# placeholder the lookahead below exempts: a real account segment written here would trip this
# very detector.) Keep the fold SCOPED to that arm -- do NOT lift it to a whole-pattern
# re.IGNORECASE. That also lower-cases the POSIX /Users arm, and `/users/` is an extremely
# common URL segment: measured, it then matches the web console's /ui/users/... routes in 47
# places on the tracked tree and reds this required context on its first run. The exemption
# list below stays case-SENSITIVE for the inverse reason -- on POSIX `Public` and `public` are
# DIFFERENT accounts, and widening an exemption is the under-detection direction.
_HOME_PATH = re.compile(
r"(?:[A-Za-z]:[\\/]Users|/home|/Users)[\\/]"
r"(?:(?i:[A-Za-z]:[\\/]users)|/home|/Users)[\\/]"
r"(?!<|\$|%|\{"
r"|(?:Public|Default|All|ContainerAdministrator|runner|vsts"
r"|me|svc|you|user|username|example)[\\/\s\"'`]"
Expand Down
Loading
Loading