Skip to content

Add WisdomAI domain converter (both directions)#239

Open
mwu-wisdom wants to merge 5 commits into
apache:mainfrom
mwu-wisdom:wisdom-converter
Open

Add WisdomAI domain converter (both directions)#239
mwu-wisdom wants to merge 5 commits into
apache:mainfrom
mwu-wisdom:wisdom-converter

Conversation

@mwu-wisdom

Copy link
Copy Markdown

Supersedes #192, which was closed during the repository migration (the old repo is now apache/ossie-temp, and its head points at a stale mirror branch). This is the same converter rebased onto the current main, with two adaptations to changes that landed since:

  • BigQuery connections now map to the new BIGQUERY dialect (with backtick identifier quoting) instead of falling back to ANSI_SQL with a warning.
  • The converter pyproject.toml follows the repo's UV conventions (dependency-groups, tool.uv.sources for the in-repo apache-ossie).

What

Adds converters/wisdom, a bidirectional converter between WisdomAI domain exports (format 1.0, the JSON produced by wisdom's exportDomain API and consumed by importDomain) and Ossie semantic model YAML. It follows the layout and conventions of the dbt converter (apache-ossie dependency, ConverterResult/ConverterIssue loss reporting, argparse CLI). Also registers WISDOM as a well-known custom-extension vendor in the spec documents and OSIVendor enum.

ossie-wisdom wisdom-to-osi -i domain-export.json -o semantic_model.yaml
ossie-wisdom osi-to-wisdom -i semantic_model.yaml -o domain-export.json

Import mapping (wisdom → Ossie)

Ossie Wisdom
semantic_model[].name / description domain ref.name / description
ai_context (model level) domainSystemInstructions + domain knowledge contents as a bulleted list
datasets[] tables (location.database.schema.dbTablesource; primaryKey.columns or isPrimaryKey flags → primary_key)
datasets[].fields[] columns (bare, dialect-quoted names) and formulas (verbatim expressions); displayNamelabel; DATE/DATETIME/TIMESTAMP → dimension.is_time
relationships[] relationship graph edges — cardinality folded into edge direction (from = many side); compound AND-joins flattened to composite key arrays; OR/non-equi joins dropped with a warning
metrics[] per-table measures hoisted to the model level, cross-table name collisions resolved by table-name prefixing

Expressions are emitted verbatim under the Ossie dialect mapped from the table's connection (snowflake → SNOWFLAKE, databricks → DATABRICKS, bigquery → BIGQUERY); other dialects fall back to ANSI_SQL verbatim with an UNSUPPORTED_DIALECT warning.

Export mapping (Ossie → wisdom)

The exact inverse, so round-trips are stable in both directions:

  • Model ai_context splits back into domainSystemInstructions (leading text) + one knowledge item per - bullet.
  • A field whose expression is just its own (possibly quoted) name becomes a column; anything else becomes a formula. dimension.is_time maps to a TIMESTAMP data type (wisdom re-derives exact types from the warehouse).
  • Relationships become MANY_TO_ONE edges; the ai_context notes written by the import direction restore ONE_TO_ONE/MANY_TO_MANY, and composite keys become compound AND join conditions.
  • Metrics attach to the table their expression references (fallback to the first dataset with a warning).
  • IDs are derived deterministically from element names so re-runs are reproducible; connections are per-dialect placeholders expected to be remapped at import.
  • Not representable in wisdom — dropped with warnings: semantic models beyond the first, unique_keys, custom_extensions, and ai_context on fields/metrics (plus synonyms/examples anywhere).

Testing

  • 27 tests against a synthetic fixture covering all four cardinalities, compound AND-join flattening, OR-join dropping, dialect fallback, hidden columns, stale measures, duplicate fields, metric name collisions, full round-trip equality (OSI → wisdom → OSI yields an identical OSIDocument), and determinism of the emitted export.
  • Verified end-to-end against two real wisdom domain exports (Snowflake and BigQuery connections): both outputs pass validation/validate.py fully (schema, uniqueness, reference integrity, sqlglot SQL parsing), and OSI → wisdom → OSI round-trips are byte-identical YAML for both.

🤖 Generated with Claude Code

mwu-wisdom and others added 4 commits July 21, 2026 07:39
Adds converters/wisdom, a one-way converter from a WisdomAI domain
export JSON (format 1.0) to an Ossie semantic model YAML, following
the layout of the dbt converter.

Mapping: domain -> semantic model; domain knowledge and system
instructions -> model-level ai_context; tables/columns/formulas ->
datasets/fields; relationship graph -> relationships (cardinality
folded into edge direction, compound AND-joins flattened to composite
keys); per-table measures -> model-level metrics.

Expressions are emitted verbatim under the dialect mapped from the
table's connection (snowflake, databricks), falling back to ANSI_SQL
with a warning for unsupported dialects. Information loss is reported
through the same ConverterResult/ConverterIssue pattern as ossie-dbt.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds WISDOM to the OSIVendor enum, the vendor examples in the spec
documents and JSON schema, and the supported-vendors table in the
converter guide.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds osi-to-wisdom, the reverse of the wisdom converter: an Ossie
document becomes a wisdom domain export (format 1.0) consumable by
wisdom's importDomain API.

The mapping inverts wisdom-to-osi so round-trips are stable in both
directions: model ai_context splits back into system instructions and
knowledge items, fields split into columns and formulas, relationship
direction is read as many-to-one with ai_context notes restoring
one-to-one/many-to-many and composite keys becoming compound AND join
conditions, and metrics attach to the table their expression
references. IDs are deterministic (derived from names) and connections
are per-dialect placeholders remapped at import time.

Verified: Ossie -> wisdom -> Ossie round-trips of two real domain
exports are byte-identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BigQuery connections now map to the BIGQUERY dialect added to the spec
instead of falling back to ANSI_SQL with a warning, with backtick
identifier quoting; both directions round-trip it. The pyproject
follows the repo's UV layout (dependency-groups, uv sources for the
in-repo apache-ossie).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
from ossie_wisdom.osi_to_wisdom import OSIToWisdomConverter
from ossie_wisdom.wisdom_to_osi import WisdomToOSIConverter

_ISSUE_REASON: dict[ConverterIssueType, str] = {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dict is manually kept in sync with ConverterIssueType. Any new type added to the enum without a corresponding entry will crash the CLI at runtime with a KeyError.

I suggest to use .get() with a fallback:

reason = _ISSUE_REASON.get(issue.issue_type, issue.issue_type.value)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in 63f125b using .get() with the enum value as the fallback reason, so a missing entry degrades to a terse-but-correct warning instead of a KeyError.

Comment thread converters/wisdom/src/ossie_wisdom/osi_to_wisdom.py


def _stable_id(prefix: str, value: str) -> str:
return f"{prefix}_{hashlib.md5(value.encode('utf-8')).hexdigest()}"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On FIPS-compliant systems, MD5 raises ValueError by default. Since this is for ID generation (not security), adding usedForSecurity=False makes the intent clear and avoids the crash:

Suggested change
return f"{prefix}_{hashlib.md5(value.encode('utf-8')).hexdigest()}"
return f"{prefix}_{hashlib.md5(value.encode('utf-8'), usedForSecurity=False).hexdigest()}"

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed and fixed in 63f125b. One note: the Python keyword is lowercase usedforsecurity (the camelCase usedForSecurity from the suggestion raises TypeError), so I applied it with the correct spelling plus a comment explaining it's for FIPS-enabled builds.

Use .get() with the enum value as fallback when printing converter
warnings so a ConverterIssueType without a reason entry cannot crash
the CLI, and pass usedforsecurity=False to hashlib.md5 so deterministic
ID generation works on FIPS-enabled builds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mwu-wisdom
mwu-wisdom requested a review from jbonofre July 21, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants