refactor: Phase 3 — JSON serializer, IO alignment, sanitization, operator>> decomposition

[OldCrow]

Summary

Completes Tiers 4 and 5 of the code quality refactoring roadmap, plus a comprehensive fix of pre-existing IO correctness bugs discovered during implementation.

---

Tier 5 — Custom JSON serializer/deserializer

No external dependencies. Full round-trip at max_digits10 precision.

• json_utils (include/libhmm/io/json_utils.h + src/io/json_utils.cpp): schema-aware Reader tokenizer + write helpers (write_double, write_array, write_matrix, write_distribution). read_double_array(N) and read_double_matrix(N,N) accept optional size caps.
• All 15 distributions: to_json() const override + static from_json(json::Reader&) factory method.
• hmm_json (include/libhmm/io/hmm_json.h + src/io/hmm_json.cpp): to_json(Hmm), from_json(string_view), save_json(Hmm, path), load_json(path) free functions. Anonymous-namespace factory map gives CC 2 dispatch regardless of distribution count.
• Dead code deleted: serialization.h, distribution_io_utils.h (zero callers).
• XML classes deprecated: dead boost TODO comment blocks removed, isValidXMLFile renamed canParseAsHmm, format documented, deprecation notices added.
• libhmm.h updated: hmm_json.h added as the recommended IO entry point; XML headers marked legacy.

Input sanitization

Guard	Limit	Rationale
kMaxHmmStates	4096	N×N matrix at N=4096 ≈ 128 MB
kMaxJsonInputBytes	10 MB	Well above any realistic HMM file
kMaxDiscreteSymbols	65536	512 KB per distribution
read_double_array(N)	N elements	Prevents heap growth before dimension check
read_double_matrix(N,N)	N rows × N cols	Same
isfinite on states value	—	Guards UB static_cast<size_t>(inf)

---

Tier 4 — operator>>(istream, Hmm) decomposition

operator>> was 258 lines, CC 18 (lizard warning). Decomposed by extracting each distribution parser lambda into a named static function in the anonymous namespace:

• CC: 18 → 7 on operator>>; zero lizard warnings in hmm.cpp.
• Replaced std::function<> map with plain StreamParserFn function-pointer map (eliminates heap-allocated closures).
• Added parse_binomial and parse_rayleigh (previously absent from dispatch table).
• NegativeBinomial two-word type limitation documented and confirmed by test.

---

Pre-existing IO correctness bugs (discovered during implementation)

All 15 distributions had inconsistent operator<<, operator>>, and toString() implementations — many parsers were reading formats that no operator<< ever wrote. Fixed systematically:

• 9 distributions: operator<< rewrote a different format from toString() (e.g. Gaussian wrote "Normal Distribution: Mean = V" while toString() writes "Gaussian Distribution: μ (mean) = V"). All now delegate to toString().
• All 15 standalone operator>>: rewritten to token-scan the toString() format, including skipping derived-value lines (Mean, Variance, etc.) so the HMM stream parser advances correctly to the next state.
• 6 parse_* functions in hmm.cpp: wrong token counts corrected (e.g. parse_poisson expected λ = but toString() writes λ (rate parameter) =).
• Discrete::operator>>: replaced legacy bare-integer format with current toString() format.
• StudentT::operator>>: CC 22 over-engineered key=value parser replaced with 3-line token scan (CC 2).
• All GTEST_SKIP() removed from IO tests — they were hiding these bugs.

---

Tests added

tests/io/test_hmm_json.cpp (new):

• TypeFieldPrefix: all 15 distribution to_json() type fields
• AllDistributionsRoundTrip: 15-state HMM, exact (bit-identical) parameter recovery
• FileSaveLoadRoundTrip: save_json / load_json file round-trip
• Error cases: unknown type, malformed JSON, zero states
• 8 sanitization boundary tests: states cap, non-finite states, 11 MB input, overlong pi/trans/probs arrays, Discrete n cap

tests/io/test_hmm_stream_io.cpp (extended):

• AllDistributionsStreamRoundTrip: 14-state HMM via operator<</operator>> (all distributions except NegativeBinomial)
• NegativeBinomialStreamLimitation: documented limitation confirmed

tests/io/test_xml_file_io.cpp (updated):

• Removed all GTEST_SKIP() — round-trip tests now pass with correct parsers
• Added distribution parameter verification in XMLRoundTripConsistency

---

Other

• segmental_kmeans_trainer.cpp: implementation-level documentation added (algorithm overview, per-function comments explaining partitioning strategy, probability floor rationale, convergence indicator, etc.)
• hmm_validator: accepts .json (JSON) and any other extension (legacy XML); format auto-detected from file extension.
• basic_hmm_example: section 4 replaced raw ofstream/ifstream round-trip with save_json/load_json demonstration.
• samples/: new directory with two_state_gaussian.{json,xml} and casino.{json,xml} — validated through hmm_validator, tracked in the repo.

---

Metrics

Metric	Before	After
Lizard warnings (hmm.cpp)	1 (CC 18)	0
operator>>(Hmm) CC	18	7
StudentT::operator>> CC	22	2
Tests passing	37/37	38/38
GTEST_SKIP instances	4	0

Pre-existing lizard warnings retained as acknowledged: BetaDistribution::getProbability (CC 23), BetaDistribution::fit (CC 16), NegativeBinomialDistribution::fit (CC 19), ParetoDistribution::fit (CC 17) — all algorithmic complexity in numerical methods, not structural.

---

Conversation: https://app.warp.dev/conversation/ca60ff01-fa5e-45cf-be1a-5dd234ddd87c
Plan: https://app.warp.dev/drive/notebook/WeK7cgQicMiiFNphwUmzXT