From 6f26d4c28eea56922946e09800896253993c1854 Mon Sep 17 00:00:00 2001 From: mdheller <21163552+mdheller@users.noreply.github.com> Date: Fri, 8 May 2026 03:30:17 -0400 Subject: [PATCH 1/4] Add NLP component alignment for Holmes --- docs/NLP_COMPONENT_ALIGNMENT.md | 95 +++++++++++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 docs/NLP_COMPONENT_ALIGNMENT.md diff --git a/docs/NLP_COMPONENT_ALIGNMENT.md b/docs/NLP_COMPONENT_ALIGNMENT.md new file mode 100644 index 0000000..f842952 --- /dev/null +++ b/docs/NLP_COMPONENT_ALIGNMENT.md @@ -0,0 +1,95 @@ +# NLP Component Alignment + +## Purpose + +Holmes needs a disciplined NLP component map so it can support primitive analysis, task models, retrieval, evidence, semantic graph conversion, policy, and agentic investigation without becoming a loose model zoo. + +This document records the lower-level NLP families Holmes must cover and how those families map across Holmes, nlplab, Sherlock Search, and prophet-platform. + +## Component map + +| Family | Holmes surface | Lab/runtime owner | Sherlock boundary | +| --- | --- | --- | --- | +| Basic primitives | `language.primitive.v1/Analyze` | `SociOS-Linux/nlplab` prototypes adapters; `prophet-platform` hosts stable services | Index primitive outputs only as pointer-backed evidence | +| Advanced primitives | dependency parsing, semantic role labeling, coreference, morphology extensions | `nlplab` evaluates parser, SRL, and coreference adapters | Search over parse, entity, and relation evidence without canonical-truth claims | +| Rule techniques | rule packs, gazetteers, dictionaries, regular expressions, table/header rules | `nlplab` keeps rule DSL experiments; Holmes promotes validated rule packs | Preserve rule version, policy decision, source, handling tags, and evidence refs | +| Classical ML | CRF, SVM/logistic/maxent, clustering, topic modeling, similarity baselines | `nlplab` benchmarks and calibrates classical models | Retrieve model outputs with corpus, model, and eval refs | +| Neural NLP | sequence/text models and embedding pipelines | `nlplab` handles PyTorch/ONNX experiments and benchmarks | Index spans, classes, and embedding metadata under evidence controls | +| Transformers | token classification, text classification, relation extraction, embeddings, reranking, translation, summarization, RAG | `nlplab` evaluates candidate models; Mycroft routes by cost, quality, privacy, and latency | Search and rerank evidence packets under policy ceilings | +| Task models | entities, numeric entities, PII, sentiment, target sentiment, categories, concepts, keywords, relations, emotion, tone | Holmes exposes stable contract families after eval and promotion | Sherlock indexes outputs with provenance and confidence | + +## Architectural claim + +A component NLP library can extract spans, tags, classes, relations, and task predictions. Holmes must do more: + +- bind every output to corpus, model, policy, eval, and evidence references; +- route among rule, classical, neural, transformer, and foundation-language paths using explicit cost, latency, quality, and privacy constraints; +- preserve source provenance and rollback metadata; +- convert selected outputs into semantic graph candidates; +- support contradiction detection, claim extraction, and casefile assembly; +- keep retrieval and indexing separate from truth promotion; +- require promotion evidence before a pipeline becomes stable. + +The target position is: + +> Component NLP annotates. Holmes investigates, governs, retrieves, graphs, reasons, and promotes with evidence. + +## Algorithm selection doctrine + +Holmes should not default every task to transformers. + +Use rules when the variation space is bounded, latency requirements are strict, labels are unavailable, or policy-sensitive patterns need deterministic inspection. + +Use classical ML when training must be fast, features are strong, labels exist, and the workload is CPU-bound or latency-sensitive. + +Use neural non-transformer models when higher quality is required but transformer runtime cost is unacceptable. + +Use transformers and foundation-language services when multilinguality, semantic abstraction, long-context synthesis, or task quality justifies compute cost and governance overhead. + +Use hybrid pipelines when deterministic guards, statistical extraction, retrieval grounding, and foundation-language synthesis must be composed under one evidence contract. + +## Required executable proof + +Holmes should not claim runtime superiority until `nlplab` produces benchmark receipts for: + +1. primitive quality and speed; +2. entity, relation, and classification metrics; +3. PII and sensitive-context precision/recall; +4. retrieval impact through Sherlock evidence packets; +5. semantic graph conversion fidelity; +6. policy propagation and rollback coverage; +7. cost, latency, and memory profiles across CPU and GPU lanes. + +## Required records + +The next standards and runtime work should define or import these records: + +- `LanguageAnalysisRecord`; +- `PrimitiveSpan`; +- `EntityMention`; +- `RelationMention`; +- `ClassificationDecision`; +- `TopicAssignment`; +- `SentimentDecision`; +- `KeywordCandidate`; +- `ClaimRecord`; +- `SemanticGraphCandidate`; +- `LanguagePipelineReceipt`; +- `HolmesEvidencePacket`. + +## Promotion rule + +A Holmes NLP component graduates only when it has: + +1. corpus reference; +2. pipeline or model reference; +3. algorithm family declaration; +4. task contract; +5. quality evaluation; +6. latency and footprint measurement; +7. guardrail policy result; +8. evidence receipt; +9. promotion record; +10. rollback reference. + +This keeps local labs, governed platform services, SourceOS clients, and Sherlock retrieval connected without collapsing those layers into one monolith. From ab7875ac071942772cb05931c27c78cd4d0662ef Mon Sep 17 00:00:00 2001 From: mdheller <21163552+mdheller@users.noreply.github.com> Date: Fri, 8 May 2026 03:30:45 -0400 Subject: [PATCH 2/4] Expand Holmes NLP component coverage --- examples/holmes-surface.json | 47 +++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/examples/holmes-surface.json b/examples/holmes-surface.json index 7f9508f..a3cbb8e 100644 --- a/examples/holmes-surface.json +++ b/examples/holmes-surface.json @@ -7,7 +7,7 @@ }, "spec": { "product": "Holmes", - "tagline": "Watson-style systems answer. Holmes investigates.", + "tagline": "Component NLP annotates. Holmes investigates.", "components": [ "sherlock-search", "221b", @@ -17,6 +17,48 @@ "the-canon", "deduction-engine" ], + "componentFamilies": [ + "basic-primitives", + "advanced-primitives", + "rule-techniques", + "classical-ml", + "neural-nlp", + "transformers", + "foundation-language-services", + "retrieval-and-knowledge", + "guardrails-and-governance", + "agent-and-tool-orchestration" + ], + "nlpTasks": [ + "language-identification", + "sentence-segmentation", + "tokenization", + "lemmatization", + "part-of-speech-tagging", + "morphological-features", + "dependency-parsing", + "semantic-role-labeling", + "entity-extraction", + "numeric-entity-extraction", + "pii-extraction", + "coreference-resolution", + "relation-extraction", + "text-classification", + "zero-shot-classification", + "sentiment-classification", + "target-sentiment-extraction", + "keyword-extraction", + "category-classification", + "concept-linking", + "topic-modeling", + "topical-clustering", + "text-similarity", + "table-header-identification", + "claim-extraction", + "contradiction-detection", + "semantic-graph-conversion", + "evidence-governance" + ], "methodFamilies": [ "language.primitive.v1/Analyze", "language.entity.v1/Extract", @@ -33,7 +75,10 @@ "requiredPromotionEvidence": [ "corpusRef", "pipelineOrModelRef", + "algorithmFamily", + "taskContract", "evalRecord", + "latencyFootprintRecord", "guardrailPolicy", "evidenceReceipt", "promotionRecord", From 3df4a5637e964f381b34191cfc0e62fabd8775ef Mon Sep 17 00:00:00 2001 From: mdheller <21163552+mdheller@users.noreply.github.com> Date: Fri, 8 May 2026 03:36:33 -0400 Subject: [PATCH 3/4] Validate Holmes NLP component families --- tools/validate_holmes.py | 70 +++++++++++++++++++++++++++++++++++----- 1 file changed, 62 insertions(+), 8 deletions(-) diff --git a/tools/validate_holmes.py b/tools/validate_holmes.py index f433588..7d73b8a 100644 --- a/tools/validate_holmes.py +++ b/tools/validate_holmes.py @@ -16,10 +16,55 @@ "the-canon", "deduction-engine", } +REQUIRED_COMPONENT_FAMILIES = { + "basic-primitives", + "advanced-primitives", + "rule-techniques", + "classical-ml", + "neural-nlp", + "transformers", + "foundation-language-services", + "retrieval-and-knowledge", + "guardrails-and-governance", + "agent-and-tool-orchestration", +} +REQUIRED_NLP_TASKS = { + "language-identification", + "sentence-segmentation", + "tokenization", + "lemmatization", + "part-of-speech-tagging", + "morphological-features", + "dependency-parsing", + "semantic-role-labeling", + "entity-extraction", + "numeric-entity-extraction", + "pii-extraction", + "coreference-resolution", + "relation-extraction", + "text-classification", + "zero-shot-classification", + "sentiment-classification", + "target-sentiment-extraction", + "keyword-extraction", + "category-classification", + "concept-linking", + "topic-modeling", + "topical-clustering", + "text-similarity", + "table-header-identification", + "claim-extraction", + "contradiction-detection", + "semantic-graph-conversion", + "evidence-governance", +} REQUIRED_EVIDENCE = { "corpusRef", "pipelineOrModelRef", + "algorithmFamily", + "taskContract", "evalRecord", + "latencyFootprintRecord", "guardrailPolicy", "evidenceReceipt", "promotionRecord", @@ -32,6 +77,14 @@ def fail(message: str) -> int: return 1 +def require_set(spec: dict, field: str, required: set[str]) -> int | None: + observed = set(spec.get(field, [])) + missing = required - observed + if missing: + return fail(f"missing {field}: {sorted(missing)}") + return None + + def main() -> int: if not EXAMPLE.exists(): return fail("missing examples/holmes-surface.json") @@ -41,14 +94,15 @@ def main() -> int: if data.get("kind") != "HolmesSurface": return fail("wrong kind") spec = data.get("spec", {}) - components = set(spec.get("components", [])) - missing_components = REQUIRED_COMPONENTS - components - if missing_components: - return fail(f"missing components: {sorted(missing_components)}") - evidence = set(spec.get("requiredPromotionEvidence", [])) - missing_evidence = REQUIRED_EVIDENCE - evidence - if missing_evidence: - return fail(f"missing promotion evidence: {sorted(missing_evidence)}") + for field, required in [ + ("components", REQUIRED_COMPONENTS), + ("componentFamilies", REQUIRED_COMPONENT_FAMILIES), + ("nlpTasks", REQUIRED_NLP_TASKS), + ("requiredPromotionEvidence", REQUIRED_EVIDENCE), + ]: + result = require_set(spec, field, required) + if result is not None: + return result integrations = spec.get("integrations", {}) for key in ["standards", "platform", "search", "lab", "sourceosCarry"]: if key not in integrations: From 6fadcf187f96131be965f2a3522fb869773c5618 Mon Sep 17 00:00:00 2001 From: mdheller <21163552+mdheller@users.noreply.github.com> Date: Fri, 8 May 2026 03:37:01 -0400 Subject: [PATCH 4/4] Document Holmes NLP alignment contract --- README.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c3e0eb0..c4695db 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ It is built to outgrow assistant-grade discovery: classical NLP, neural NLP, sem ## Product thesis -Watson-style systems answer. Holmes investigates. +Component NLP annotates. Holmes investigates. Holmes is not a chatbot wrapper, a loose model zoo, or a domain NLP repo. It is the governed language layer above search, evidence, retrieval, casefiles, semantic graphs, tools, models, evals, and agents. @@ -33,6 +33,23 @@ Holmes is not a chatbot wrapper, a loose model zoo, or a domain NLP repo. It is 8. Guardrails and governance: PII checks, source provenance, prompt-injection checks, policy gates, eval gates, factsheets, promotion records. 9. Agent and tool orchestration: tool contracts, agent identity, sessions, memory, MCP/A2A, execution traces, model routing. +## NLP component alignment + +Holmes explicitly covers these component families: + +- basic primitives; +- advanced primitives; +- rule techniques; +- classical ML; +- neural NLP; +- transformers; +- foundation-language services; +- retrieval and knowledge; +- guardrails and governance; +- agent and tool orchestration. + +The alignment contract is documented in [`docs/NLP_COMPONENT_ALIGNMENT.md`](docs/NLP_COMPONENT_ALIGNMENT.md). That document is the lower-layer NLP map for Holmes, nlplab, Sherlock Search, and the platform runtime. + ## Repo role This repo is the Holmes product surface and integration spine.