Skip to content

Commit 605a21d

Browse files
dmealingclaude
andcommitted
docs(spec): FR-011 — configurable @normalize modes + guarded fuzzy (research-driven)
Refine the enum coercion design after researching normalization + fuzzy matching: - Normalization is now CONFIGURABLE via a @normalize closed-enum attr (none|collapse|strip|unicode), per-field with an object.value-level default; global default = strip. collapse/strip are pure-ASCII → Tier-1 byte-identical cross-port; unicode is best-effort (excluded from the byte-identical conformance assertion — per-port Unicode facilities diverge / ICU-class differences). - Fuzzy INCLUDED as guarded integer Levenshtein (≤1 + clear margin + normalized-length ≥5 guard), gated by tolerance=loose, off by default — chosen over Jaro-Winkler to avoid float cross-port nondeterminism (short closed enum vocabularies are exactly where fuzzy is risky, hence the guards). - ASCII-only case fold (a-z↔A-Z) sidesteps the locale/Turkish-İ + Unicode-version traps for the guaranteed modes since enum members are ASCII by definition. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 54fb5b4 commit 605a21d

1 file changed

Lines changed: 68 additions & 24 deletions

File tree

docs/superpowers/specs/2026-05-30-fr-011-tolerant-ingestion-engine-design.md

Lines changed: 68 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -107,17 +107,19 @@ partially support it, Java/Kotlin defer) with one corpus-pinned behavior.
107107

108108
### Enum coercion pipeline
109109

110-
Resolution order for an enum value during ingest (Tier-1 invariant — identical across ports):
110+
Resolution order for an enum value during ingest, given the field's resolved **normalization mode
111+
`M`** (see below) and the runtime **tolerance `T`**:
111112

112113
1. **exact** match against `@values``RECOVERED`.
113-
2. **normalize** then match: `normalize(s) = uppercase(trim(s))` with `[\s_\-]+` collapsed to a
114-
single `_`. So `"in progress"`, `"In-Progress"`, `"in__progress"` all normalize to `IN_PROGRESS`.
115-
A normalized hit → `RECOVERED` (a `normalize` coercion is logged).
116-
3. **`@enumAlias`** (explicit synonyms, properties map) — runtime aliases win over schema; a hit →
117-
`RECOVERED` (alias coercion logged). Alias keys are themselves normalized before comparison.
118-
4. **fuzzy (opt-in)** — only at `tolerance = loose`: Levenshtein distance on the normalized forms,
119-
accept the unique nearest member within a conservative threshold (distance ≤ 1 for members ≤ 4
120-
chars, ≤ 2 otherwise; reject on ties). Deterministic → cross-port byte-identical. Off by default.
114+
2. **normalized** match using mode `M` (both input and members normalized by `M`; skipped when
115+
`M = none`) → `RECOVERED` (a `normalize` coercion logged).
116+
3. **`@enumAlias`** (explicit synonyms, properties map) — runtime aliases win over schema; alias
117+
keys are normalized by `M` before comparison; a hit → `RECOVERED` (alias coercion logged).
118+
4. **fuzzy** — enabled only when `T = loose`. **Guarded integer Levenshtein** on the `M`-normalized
119+
forms: accept the **uniquely nearest** member with edit distance ≤ 1 **and** a clear margin (the
120+
runner-up's distance ≥ best + 2), **and** only for members whose normalized length ≥ 5 (short
121+
members never fuzzy-match). Integer distance → deterministic, byte-identical cross-port. A hit →
122+
`RECOVERED` (fuzzy coercion logged). Off by default (`T ≠ loose`).
121123
5. **`@coerceDefault`** (new attr) — a present-but-uncoercible value falls back to this member →
122124
`DEFAULTED` (coercion logged). If absent, fall through.
123125
6. **MALFORMED**.
@@ -128,34 +130,69 @@ supplied"; `@coerceDefault` is "value when supplied-but-garbage" — distinct be
128130
a field vs hallucinating one may warrant different fallbacks. A present-uncoercible value does **not**
129131
borrow `@default` (only `@coerceDefault`), keeping the two semantics independent.
130132

133+
### Normalization modes (configurable via `@normalize`)
134+
135+
Normalization is selectable, not fixed, via a closed-enum **`@normalize`** attr. The three modes:
136+
137+
- **`none`** — exact only; skip step 2 (strictest).
138+
- **`collapse`** — ASCII case-fold (`a–z``A–Z` only) + trim + collapse runs of `[\s_\-]+` to a
139+
single `_`. Preserves separator boundaries. `"in progress"`/`"In-Progress"``IN_PROGRESS`;
140+
`"inprogress"` stays distinct.
141+
- **`strip`** *(default)* — ASCII case-fold + strip everything except `[A-Z0-9]`. Separator/
142+
punctuation/whitespace-insensitive: `"in progress"`/`"In-Progress"`/`"inprogress"`/`"in progress!"`
143+
`INPROGRESS`. Most forgiving. (Load-time warning if two members collide under `strip`.)
144+
- **`unicode`** — Unicode caseless match (NFKC_Casefold-style) for non-ASCII input/diacritics.
145+
146+
`collapse` and `strip` are **pure ASCII transforms → Tier-1 byte-identical** across all 5 ports.
147+
`unicode` is **best-effort** per each port's standard Unicode facilities; because implementations
148+
diverge (ICU-class differences), `unicode`-mode results are **NOT part of the byte-identical
149+
cross-port conformance assertion** — they're tested per-port for reasonableness only. Pick `unicode`
150+
only when a field genuinely has Unicode-domain input; ASCII enum members rarely need it.
151+
152+
**Resolution of the mode for a field** (the codegen schema-emitter bakes the resolved `M` into the
153+
`FieldSpec`): field-level `@normalize` → owning `object.value`-level `@normalize` (the
154+
**object-level default** for its enum fields) → global default **`strip`**.
155+
131156
### Metamodel attrs
132157

133158
- **`@coerceDefault`** (NEW) — on `field.enum` (extensible to other constrained kinds later). String
134-
member symbol; must be one of `@values` (loader-validated). Drives step 5 above. Generic name
135-
(not enum/LLM-specific) so it reads as an input-coercion concern.
159+
member symbol; must be one of `@values` (loader-validated → `ERR_BAD_ATTR_VALUE`). Drives step 5.
160+
Generic name (not enum/LLM-specific) so it reads as an input-coercion concern.
161+
- **`@normalize`** (NEW) — closed enum `none | collapse | strip | unicode` (loader-validated). On
162+
**`field.enum`** (per-field) and on **`object.value`** (the default for the object's enum fields).
163+
Resolution: field → owning object → global default `strip`. The codegen schema-emitter bakes the
164+
resolved mode into the `FieldSpec`. Generic name; honored for enums in FR-011, extensible later.
136165
- **`@default`** (EXISTING) — its enum value now also serves as the *absent*-fill during ingest. No
137166
new attr; documented dual role (DB/codegen default + ingest absent-fill).
138-
- **`@enumAlias`** (EXISTING, FR-010) — unchanged; now normalized-key matched (step 3).
139-
- Registered identically across ports; `@coerceDefault` bad value (not a member) → `ERR_BAD_ATTR_VALUE`.
167+
- **`@enumAlias`** (EXISTING, FR-010) — unchanged; alias keys normalized by the field's mode `M` (step 3).
168+
- Fuzzy matching has no attr — it's gated by the runtime `tolerance = loose` (RecoverOptions), off by default.
169+
- All registered identically across ports.
140170

141171
## Tiers (cross-language-porting)
142172

143-
- **Tier 1 (INVARIANT — corpus-pinned):** per-field classification + `empty`; canonical normalized
144-
value (numeric ±1e-9); the **resolution order** and the **exact normalization rule**; the **fuzzy
145-
algorithm + threshold** (deterministic); `@coerceDefault`/`@default`/`@enumAlias` vocabulary +
146-
semantics; nested dotted-path classification; the strip/locate/read/coerce observable behavior;
147-
MALFORMED/TRUNCATED sentinel effects.
148-
- **Tier 2 (idiomatic):** the `ingest` package internals + module layout; the `recover()``ingest()`
149-
naming; enums/dataclasses/records per language; opt-in-fuzzy plumbing.
173+
- **Tier 1 (INVARIANT — corpus-pinned):** per-field classification + `empty`; canonical value
174+
(numeric ±1e-9); the **resolution order**; the **`none`/`collapse`/`strip` normalization rules**
175+
(pure ASCII → byte-identical); the **guarded integer-Levenshtein fuzzy** (algorithm + ≤1 + clear-
176+
margin + min-length-5 guard, deterministic); `@coerceDefault`/`@default`/`@enumAlias`/`@normalize`
177+
vocabulary + the field→object→global mode resolution; nested dotted-path classification; the
178+
strip/locate/read/coerce observable behavior; MALFORMED/TRUNCATED sentinel effects.
179+
- **Tier 2 (idiomatic / best-effort):** the `ingest` package internals + module layout; the
180+
`recover()``ingest()` naming; enums/dataclasses/records per language; opt-in-fuzzy plumbing.
181+
**The `unicode` normalization mode is explicitly best-effort and NOT in the byte-identical
182+
conformance assertion** (per-port Unicode facilities diverge); `collapse`/`strip` are the
183+
cross-port-guaranteed modes.
150184
- **Tier 3:** conformance-runner plumbing; the executable-proof harness.
151185

152186
## Conformance corpus expansion
153187

154188
Add to the shared `fixtures/recover-conformance/` (renamed/aliased to ingest-conformance if the API
155189
rename lands; the existing 10 cases stay green):
156190

157-
- **Enum pipeline:** normalized-variant (`"in progress"`), `@enumAlias` synonym, fuzzy-typo
158-
(loose-only), `@coerceDefault` fallback (→ DEFAULTED), `@default` absent-fill (→ DEFAULTED).
191+
- **Enum pipeline:** normalized-variant per mode (`strip`: `"in progress"`/`"inprogress"`; `collapse`:
192+
`"in progress"` but NOT `"inprogress"`; `none`: exact only); field-level vs object-level `@normalize`
193+
default resolution; `@enumAlias` synonym; fuzzy-typo (loose-only, incl. a short-member NON-match and
194+
a close-members tie NON-match to pin the guards); `@coerceDefault` fallback (→ DEFAULTED); `@default`
195+
absent-fill (→ DEFAULTED). (`unicode`-mode cases are per-port-only, not in the cross-port assertion.)
159196
- **XML breadth:** nested element, repeated→array, attribute-vs-element, mixed-content, more
160197
malformations (currently only 1 XML case of 10).
161198
- **Nested objects:** clean nested, partial-nested (some sub-fields MALFORMED), array-of-objects.
@@ -189,8 +226,15 @@ each merge; forward-only merges; push for durability. (Established FR-010 workfl
189226

190227
## Resolved design decisions (from brainstorming 2026-05-30)
191228

192-
- Pipeline = exact → normalize → alias → fuzzy(opt-in, `loose`) → `@coerceDefault` → MALFORMED.
193-
- New attr named **`@coerceDefault`** (general, not `@enumDefault`); `@default` handles absent.
229+
- Pipeline = exact → normalize(mode `M`) → `@enumAlias` → fuzzy(guarded Levenshtein, `tolerance=loose`)
230+
`@coerceDefault` → MALFORMED; `@default` fills absent.
231+
- New attr **`@coerceDefault`** (general, not `@enumDefault`); `@default` handles absent.
232+
- Normalization is **configurable** via **`@normalize`** (`none|collapse|strip|unicode`), per-field
233+
with an `object.value`-level default; global default `strip`. `collapse`/`strip` are ASCII →
234+
Tier-1 byte-identical; `unicode` is best-effort (excluded from the cross-port byte assertion).
235+
- Fuzzy **included** as guarded integer Levenshtein (≤1, clear margin, normalized-length ≥5),
236+
gated by `tolerance=loose`, off by default — chosen over Jaro-Winkler to avoid float cross-port
237+
nondeterminism. Researched 2026-05-30.
194238
- **Full decouple** into the **`ingest`** package; render depends on ingest.
195239
- Nested-object recovery is in-scope and uniform.
196240
- TS pilot.

0 commit comments

Comments
 (0)