E14: Custom Structured Data

[DavidCozens]

Per-message structured data — caller-built SD-ELEMENTs attached to individual
Log calls — plus the safe authoring API that makes caller-supplied SD content
injection-proof. Custom/enterprise SD-IDs (name@enterprise-number format).

Deferred until E07 (Standard Structured Data) was complete and the SD
serialisation machinery was proven.

Agreed design (2026-06-05)

Authoring is moved off the raw SolidSyslogFormatter and onto two opaque,
stack-transient writer types (no pool — like the formatter, deviation D.002):

• SolidSyslogSdElement — handed to the SD vtable Format(base, element).
  Owns grammar + names: _Begin(name, enterpriseNumber) (validates SD-NAME;
  IANA name when no enterprise, else name@num), _Param(name) → returns a
  SolidSyslogSdValue*, _End.
• SolidSyslogSdValue — the per-param value sink. Escaped UTF-8 + convenience
  primitives (_String, _Uint32 now; _Timestamp later — the menu is additive).
  Streaming, bounded by the message buffer (cannot overflow). A third-party value
  producer only ever sees the SdValue — it physically cannot open a param or
  break the framing.

The SD vtable changes Format(base, formatter) → Format(base, element), which
removes the raw-formatter footgun; SolidSyslogFormatter then leaves the
public interface entirely. Header-field callbacks (hostname/appName/processId)
move to a small US-ASCII writer (SolidSyslogToken), retiring
SolidSyslogStringFunction. The three existing SDs are refactored onto the new
API (dogfood), which also eliminates the OriginSd preformat scratch blob (see
Constraints).

Key decisions: two types (not one with a state machine); dedicated header
writer (not const char* return — that reintroduces materialisation/ownership
cost); value-primitive menu additive forever (only the types and callback
signatures freeze at beta); no per-value cap (buffer-bounded only); param
presence gated by the element author (NULL callback → no _Param), emptiness
controlled by the producer, producer cannot signal "skip me"; the value writer
carries ≤4-byte UTF-8 continuation state so a multi-byte codepoint streamed
across two _String calls reassembles correctly (Option B) — single-shot
_EscapedString never had this; streaming introduces it.

This supersedes the S12.32 (#533) formatter escape-toggle proposal: the
escaping fix falls out of routing values through the escaped SdValue and lands
across S14.04–S14.05.

Scope

• API for attaching zero or more SD-ELEMENTs per Log call via SolidSyslogMessage
• Custom SD-ID format: name@enterprise-number (RFC 5424 §7.1.5)
• Combining per-instance SD (from E07) with per-message SD in the correct order
• Enterprise number registration guidance
• SD-ID and PARAM-NAME syntax validation per RFC 5424 §6.3.2 (1–32
  PRINTUSASCII excluding = ] " and space). Caller-supplied names are
  validated at SD Create time; invalid names cause the custom SD's
  Create to return NULL, which in turn causes SolidSyslog_Create to
  fail. Fail-loud-at-boundary only — no format-time re-validation.
• Oracle round-trip BDD for PARAM-VALUE escape rules — parameterised
  example fixture driving software/swVersion (and custom-SD values)
  containing each of ", \, ] through syslog-ng and asserting the
  decoded value matches.

Decomposition

Authoring-API foundation (this wave) — each story its own red→green→refactor:

1. S14.01 — SolidSyslogSdValue: escaped streaming value sink with Option-B
   continuation state. Tested standalone.
2. S14.02 — SolidSyslogSdElement: framing + SD-NAME/enterprise validation +
   internal _FromFormatter constructor. Tested standalone.
3. S14.03 — Migrate TimeQualitySd onto the writer. Walking skeleton: vtable
   unchanged (SD wraps the handed formatter via _FromFormatter); no string
   callback so the cleanest first cut; existing tests stand (output unchanged).
4. S14.04 — Migrate MetaSd (GetLanguage → SolidSyslogSdValue* + void* context). Closes the language injection vector (part of S12.32: Escape integrator-supplied SD param values (scoped formatter escape toggle) #533).
5. S14.05 — Migrate OriginSd (GetIpAt → SolidSyslogSdValue* + void* context). Closes the ip vector → S12.32: Escape integrator-supplied SD param values (scoped formatter escape toggle) #533 fully done; eliminates the
   preformat scratch blob.
6. S14.06 — Flip the SD vtable to Format(base, element); move the wrap into
   MessageFormatter. Footgun closed — authors can no longer reach a raw
   formatter. (Small/mechanical; may merge into S14.05 tail or S14.08 head.)
7. S14.07 — Header fields off the formatter: SolidSyslogToken writer +
   hostname/appName/processId callbacks + Posix/Windows helpers.
   SolidSyslogStringFunction retired.
8. S14.08 — Internalise the formatter: header → Core/Source/, drop public
   forward-decls, update the CLAUDE.md public-header table; author
   docs/structured-data.md.
9. S14.09 — Custom-SD integrator guide + worked example / BDD (JSON-in-value).

Later wave (per-message attachment): attaching caller-built SD-ELEMENTs to
individual Log calls via SolidSyslogMessage; max-elements / allocation strategy
on constrained targets. To be decomposed once the foundation lands.

Deferred from

• S07.04 (S07.04: SD escaping and validation #68) — SD-NAME syntax validation. The infrastructure was
  originally scoped into S07.04 but deferred here because all three
  standard SDs (meta/timeQuality/origin) have compile-time-constant
  names, so validation protects nothing until callers supply names.
• S07.04 (S07.04: SD escaping and validation #68) — oracle round-trip BDD for PARAM-VALUE escape rules.
  Requires parameterised example fixture; natural fit here since custom
  SDs introduce caller-supplied values.

Constraints from related work

• OriginSd preformat tail audit (from S07.08: OriginSd.Format is non-reentrant — shared scratch corrupts concurrent Log output #346). The current SolidSyslogOriginSd
  pre-formats a [origin software="…" swVersion="…" enterpriseId="…" prefix into a
  static blob at Create and reads it on every Format. Concurrent-Log reentrancy is
  preserved today only because the blob's last byte is always ASCII (", or n from
  [origin in the all-NULL case), which keeps TrimTruncatedMultiByteTail inside
  SolidSyslogFormatter_AsFormattedBuffer from entering its write loop. If the SD-helper
  restructure changes what trails any shared SD scratch buffer, either audit that the
  final byte stays ASCII or eliminate the shared scratch entirely by emitting straight
  into the caller's formatter (the shape MetaSd/TimeQualitySd already use). Eliminating
  the shared scratch is the cleaner outcome and would also retire the latent
  formattedLength concern noted against E12. → S14.05 takes the eliminate-the-scratch
  path.

Open questions

• C API design for caller-built SD — builder, pre-formatted string, or struct?
  Answered: two-type writer (SolidSyslogSdElement + SolidSyslogSdValue),
  streaming. See Agreed design.
• Maximum number of SD-ELEMENTs per message — fixed limit or dynamic? (per-message wave)
• Memory allocation strategy for per-message SD on constrained targets (per-message wave)

Research context

Industrial use cases for per-message custom SD include access control events,
process safety alarms, configuration change tracking, and device diagnostics.
See E07 for full standards research.