Add WisdomAI domain converter (both directions)#13
Open
mwu-wisdom wants to merge 109 commits into
Open
Conversation
… ontology spec that refs the core spec so that both may evolve independently.
Implements OSI → Honeydew and Honeydew → OSI conversion with full round-trip fidelity. Relationship names are stored natively on the relation, and relations are always placed on the many side. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix empty-string expression bypassing None guard - Restore calculated_attribute routing via HONEYDEW type hint - Preserve bool datatype through OSI round-trip - Store string metric ai_context in osi metadata for recovery - Warn on duplicate metric names instead of silently overwriting - Restore connection_expr from HONEYDEW custom_extension on OSI→Honeydew - Warn on malformed JSON in _read_osi_metadata instead of silent drop - Remove filter limitation from README (not applicable) - Update Honeydew docs link to honeydew.ai/docs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ssion guards - Restore Honeydew-specific fields (display_name, hidden, format_string, timegrain, owner, folder) in OSI→Honeydew direction by reading them back from the HONEYDEW custom_extension on both entity and attribute objects - Add path traversal guard in main() using os.path.normpath + startswith check - Guard against whitespace-only expressions (not expr or not expr.strip()) - Warn when field expression is a non-dict value instead of silently dropping it - Simplify elif not from_cols: to else: in _osi_relation_to_honeydew - Strengthen test_ai_context_string_preserved to assert the string value is recoverable in description - Add 5 new tests: display_name/format round-trip, calc attr Honeydew fields, entity owner, whitespace expression skipped, non-dict expression warns Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rewrite entire test file from classes to module-level pytest functions; parametrize repeated cases (is_simple_identifier, parse_osi_source, field datatypes, entity honeydew fields, dataset/calc attr fields, empty/whitespace expressions, path traversal guard, check_safe_path) - Extract _check_safe_path into a named helper in the converter (was inlined in main()) so path traversal logic is independently testable - Add parametrized test_check_safe_path covering ../evil.yml and ../../etc/passwd rejection and legitimate nested paths - Add test_empty_or_whitespace_metric_expression_skipped to cover the OSI→Honeydew whitespace guard on metrics (was previously untested) - Parametrize entity/attr/calc Honeydew field round-trip tests (owner, display_name, hidden, folder each verified independently) - Update _write_workspace helper to pass through entity-level fields (owner, display_name, hidden, folder) so round-trip tests use the standard _honeydew_roundtrip() helper instead of manual workspace setup - Promote _HD_ATTR_KEYS tuple to module-level constant (was defined inside the field loop on every iteration) - Drop from __future__ import annotations — requires Python 3.12+ - Add pyproject.toml with requires-python = ">=3.12" - Add Python 3.12+ requirement section to README Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove requirements.txt (superseded by pyproject.toml) - Extract _fields_to_honeydew() from _dataset_to_files(): field classification is now a named, independently testable function; four direct unit tests added - Warn when a relationship has neither from_columns nor connection_expr so callers discover the incomplete join before it reaches Honeydew - Round-trip the vendors list: non-HONEYDEW vendors are stored in the workspace osi metadata on OSI→Honeydew and merged back on the return trip (HONEYDEW always appears first) - Add main() CLI smoke tests via subprocess: osi-to-honeydew writes the expected workspace.yml, honeydew-to-osi writes a parseable OSI YAML, and path traversal in an entity name is rejected with exit code 1 - Update README setup instructions to use pip install . / pip install -e . Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collapse 13 individual OSI→Honeydew tests, 7 Honeydew→OSI tests, 10 OSI round-trip tests, and 13 Honeydew round-trip tests into four @pytest.mark.parametrize blocks. Every assertion now compares the entire output dict rather than cherry-picked fields. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Docstring: relationship name is mapped directly to Honeydew's relation name field (not osi metadata); ai_context is mapped natively to description/labels/AI metadata, not treated as having no equivalent - README mapping table: rename rows to use Honeydew's canonical terms (Source Attribute, Calculated Attribute) and add missing ai_context row - README limitations: rewrite the confusing "One dataset per entity" bullet to clarify that OSI dataset = one table/query, Honeydew supports multiple dataset files per entity but the converter generates exactly one Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rror changes to the core semantic model spec.
…ead constants Addresses review feedback on this PR: 1. Stop vendoring a copy of the OSI spec. validation._osi_schema_path resolves the OSI core schema from a search path: a copy vendored beside the package if present, otherwise core-spec/osi-schema.json from the enclosing checkout. The vendored osi-schema.json is deleted, so OSI-core validation now links the single canonical core-spec copy with nothing to keep in sync. 2. Omit ontology-schema validation. ossie ships no OSI ontology schema and the converter emits ontology documents but never consumes them, so the vendored osi-ontology-schema.json is removed and validate_osi_ontology runs semantic checks only (unique concepts, reference integrity). 3. Remove the unused _OSI_KNOWN_VENDORS and _OSI_KNOWN_DIALECTS constants (only re-exported; passthrough uses _INTERNAL_VENDORS, dialect parsing uses _SQL_PARSEABLE_DIALECTS). Only the OBML schema (OrionBelt's own format) stays vendored. mypy override added for the optional orionbelt engine import. 145 tests pass; ruff clean; mypy clean with the dev extra.
…spec The vendored osi-schema.json copy was removed so the source tree carries no duplicate of the spec. But a pip-installed wheel has no core-spec/ alongside it, so validate_osi would silently skip schema validation. force-include the single canonical core-spec/osi-schema.json into the built wheel: no tracked duplicate, yet a pip install still validates OSI documents self-contained. In-tree runs still resolve it from core-spec/ via validation._osi_schema_path.
Add OrionBelt (OBML) converter
python: accept free-form custom extension vendors
Add branch protection on main
Co-authored-by: Emil Sadek <esadek@users.noreply.github.com>
Co-authored-by: Emil Sadek <esadek@users.noreply.github.com>
* Test commit * Add the expression language spec * Extract embedded base64 image to img folder in core-spec Replace inline base64 PNG in expression_language.md with a relative file reference to core-spec/img/osi_layers.png, which renders correctly both locally and in the GitHub UI. * Update expression language status to Proposed Final * Remove links to old doc * Address reviewer comments * Shrink casting to supported core. * Addressed some feedback * Move to Ossie name * Update time --------- Co-authored-by: Will Pugh <willpugh@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
* Add Omni bidirectional converter
Adds converters/omni: an offline OSI <-> Omni semantic model converter,
filling the Omni spoke in the hub-and-spoke architecture.
- Export (osi-omni export): OSI -> Omni model files (views/*.view.yaml,
relationships.yaml, a generated topic, model.yaml when stashed).
- Import (osi-omni import): Omni -> OSI. Omni-only features are preserved
in custom_extensions[OMNI], so Omni -> OSI -> Omni is lossless.
- Unified osi-omni CLI plus a string-in / files-dict-out Python API.
- Example-based unit tests, fixture round-trips (incl. TPC-DS), and
Hypothesis property-based round-trip tests with a seeded-random fallback.
- Adds the OMNI vendor row to converters/index.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Handle real-world Omni model layouts in the OSI converter
Testing against production Omni instances surfaced model shapes the
fixtures did not cover; importing all of them now round-trips losslessly
(verified on a 3,498-file and a 427-file production model plus the
Ecommerce demo model, all validating against the OSI spec):
- Resolve schema-qualified view names (schema__table) from the
"# Reference this view as" header, accept per-schema folder layouts,
and preserve original file paths through a round trip.
- Stash-and-restore instead of erroring: extends-only views, non-equi/
range joins, joins touching query/extends views, and fields or
measures using Omni template ({{...}}) syntax.
- Preserve explicit Omni defaults verbatim (join_type, relationship_type,
redundant table_name, self-named sql), non-canonical on_sql (aliases,
casing, spacing), ${view.field} compound keys, empty-string metadata,
and timeframes: [].
- Accept real Omni identifiers (_fivetran_id, trailing underscores,
camelCase); collision checks stay case-insensitive.
- Quote source parts that need it ("Omni Views".upload), tag date values
in the JSON stash, and suffix duplicate generated relationship names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Catch sqlglot TokenError in the SQL validator
An expression that cannot be tokenized (e.g. templated SQL) crashed the
validator instead of being reported as a finding.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…(#206) Mirrors orionbelt-semantic-layer#201 (canonical source) into the ossie converter: - Metric round-trip: OSItoOBML resolves metric references by BOTH the OSI dataset/field name and the physical code (source-table code and field expression, quoted or bare), so metrics emitted against physical codes (e.g. SUM(fact_orders.amount)) round-trip instead of dropping when a data object's code differs from its display name. Resolved refs are bracket-quoted so display names with spaces stay intact through the dataset.column parsers. - Dimension collision: _extract_dimensions qualifies a field name that occurs in more than one dataset (Orders.date / Invoices.date) with its data object and warns, instead of silently overwriting the earlier dimension. - Validation robustness: validate_osi guards its semantic loops against malformed structures (datasets/fields that are not lists of dicts) and returns schema errors instead of raising AttributeError. Adds tests/test_osi_converter_roundtrip_robustness.py. 154 converter tests pass; ruff clean; mypy clean with the dev extra.
* feat(cli): scaffold go module with cobra command tree
The OSI converters currently require manual environment setup and
per-converter invocation. This lays the foundation for a unified CLI
(osi) that will discover and invoke converters as plugins via a
stdin/stdout JSON protocol.
Creates cli/ at the repo root as a self-contained Go module
(github.com/open-semantic-interchange/osi/cli). All commands are
stubbed — no logic is wired in this commit.
Design decisions:
- Go chosen for static binary distribution; no runtime dependency
for end users (brew/apt installable)
- internal/osidir owns ~/.osi/plugins/ initialization and respects
$OSI_PLUGIN_DIR for override; uses os.UserHomeDir() rather than
$HOME for Windows portability
- PersistentPreRunE on the root command ensures dir init runs before
every subcommand; commented caveat that Cobra does not chain this
automatically if a subcommand defines its own
- MarkFlagsMutuallyExclusive("from", "to") handles the both-set case
on osi convert; the neither-set case is validated manually in RunE
since Cobra only guards against both being provided
Build pipeline (Makefile, .goreleaser.yaml) and CI follow in
separate commits.
* chore(cli): add makefile and goreleaser build pipeline
Enables local builds and cross-platform release artifacts for the
OSI CLI.
Makefile provides standard targets for day-to-day development:
build, test, lint, release-dry-run, and clean. All targets are
designed to run from within cli/.
GoReleaser config targets linux/amd64, linux/arm64, darwin/amd64,
darwin/arm64, and windows/amd64. windows/arm64 is excluded — no
widely available CI runner and negligible current demand.
CGO_ENABLED=0 is set for fully static binaries, enabling
cross-compilation from any host without a C toolchain. Version,
commit, and date are injected at build time via ldflags from the
vars declared in main.go.
* ci(cli): add github actions workflow for build and test
Runs go build, go vet, and go test on every push and pull request
that touches cli/ or the workflow file itself.
The paths filter prevents CLI changes from triggering unrelated
workflows in this polyglot repo and vice versa.
Go version is derived from go-version-file: cli/go.mod so the
workflow automatically picks up any future toolchain bumps without
a separate workflow edit. defaults.run.working-directory avoids
repeating cd cli/ on every step.
Cross-platform build testing is not included — CGO_ENABLED=0
means the linux/amd64 build is representative of all targets.
GoReleaser snapshot builds are deferred to a future release workflow.
* test(cli): add unit tests for internal/osidir
Covers the only logic in F1 that warrants testing: $OSI_PLUGIN_DIR
env var override, default path construction via os.UserHomeDir(),
directory creation, and idempotent re-invocation of EnsurePluginDir.
t.Setenv is used throughout so env var mutations are automatically
restored after each test. t.TempDir is used for filesystem tests so
no cleanup is needed and tests are safe to run in parallel.
* chore(cli): rename osi to ossie throughout cli scaffold
Project branding has changed from OSI to OSSIE. Updates all
user-facing and internal references in the CLI:
- Binary name: osi → ossie
- Go module path: .../osi/cli → .../ossie/cli
- Default plugin directory: ~/.osi → ~/.ossie
- Environment variable: OSI_PLUGIN_DIR → OSSIE_PLUGIN_DIR
- GoReleaser project name and archive ids
- All command descriptions and flag help text
- Output directory default: ./osi-output → ./ossie-output
* chore(cli): rename internal osidir package to ossiedir
Completes the OSI → OSSIE rename by updating the internal package
directory, package declaration, and import reference in cmd/root.go.
* fix(ossiedir): rename defaultOSIDir constant to defaultOssieDir
The osi → ossie rename missed this unexported constant. The value
(.ossie) was already correct; only the identifier name was stale.
* chore(cli): fix .gitignore to ignore cli/ossie build artifact
The osi → ossie rename missed the gitignore entry, so the local
build artifact cli/ossie would no longer be ignored by git.
* chore(ossiedir): rename osidir.go and osidir_test.go to ossiedir
File names were inconsistent with the package directory name (ossiedir/).
Pure rename — no code changes.
* Swap references to Open Semantic Interchange to Apache
Ossie has moved from the Open Semantic Interchange to incubation with
the Apache Software Foundation. As such, the references in this PR
needed to be updated accordingly.
* chore(cli): align go.mod version with .tool-versions pin
cli/go.mod declared `go 1.22` while cli/.tool-versions pins
golang 1.26.2, giving contributors two conflicting sources of
truth for which Go version this module targets. Flagged in
review by khush-bhatia on PR #151.
There's no existing compatibility requirement forcing a lower
minimum, so align go.mod to the same version .tool-versions
already pins rather than introduce a toolchain directive or
lower the asdf pin.
* refactor(cli): let cobra enforce --from/--to via MarkFlagsOneRequired
convert.go hand-rolled a check for the "neither --from nor --to
set" case alongside MarkFlagsMutuallyExclusive, which already
covers the "both set" case. Suggested by khush-bhatia on PR #151
that cobra's MarkFlagsOneRequired covers this directly.
Combining MarkFlagsMutuallyExclusive with MarkFlagsOneRequired on
the same flag set gives "exactly one of" semantics entirely
through cobra's flag annotations, so the manual check and its
now-unneeded fmt.Errorf branch in runConvert are removed.
* Apply suggestions from code review
Co-authored-by: Khushboo Bhatia <khushboo.kanjani@gmail.com>
* chore(cli): stop tracking .tool-versions, gitignore for local use
cli/go.mod and cli/.tool-versions each pinned the Go version
independently — even after b834d75 aligned their values, nothing
forced the two to move together, so they could drift apart again
on the next bump.
Removed cli/.tool-versions from the repo so go.mod is the only
in-repo source of truth. Verified this doesn't break tooling:
CI already resolves its Go version from go.mod via actions/setup-go's
`go-version-file: cli/go.mod` (.github/workflows/cli-ci.yml), and
asdf's golang plugin (~/.asdf/plugins/golang/bin/parse-legacy-file)
reads the version straight out of go.mod's `go` directive when no
.tool-versions is present.
Rather than deleting the file with no path back, gitignored
cli/.tool-versions so contributors whose version manager still
wants its own pin file can keep one locally without risk of it
getting committed and re-introducing a second tracked source of
truth. Treats the toolchain-manager pin as personal environment
config, not project config — the same rationale as gitignoring
editor-specific config instead of committing it.
Caveats:
- Relies on asdf's legacy-version-file lookup, which is off by
default and must be enabled per-user via `legacy_version_file =
yes` in ~/.asdfrc (or ASDF_LEGACY_VERSION_FILE=yes) for `asdf
install`/`asdf current` to auto-detect the version from go.mod.
Not adding a repo-level workaround since asdf has no per-project
way to set this.
- No automated enforcement that a contributor's local pin matches
go.mod; a stale local .tool-versions can still silently drift, as
demonstrated on this machine when the global asdf fallback (golang
1.20.1) was too old to parse a 3-component go directive. Accepted
as the tradeoff for not owning tooling-manager config in the repo.
---------
Co-authored-by: Khushboo Bhatia <khushboo.kanjani@gmail.com>
…efile (#225) Co-authored-by: Emil Sadek <esadek@users.noreply.github.com>
* Checkin snapshot files for consistent testing * Add header to ambr files * Keep syrupy snapshots command
…icks) (#224) Bidirectional, offline converter between Apache Ossie semantic models and Databricks Unity Catalog Metric Views (YAML v1.1), filling the DATABRICKS spoke already listed in converters/README.md. Packaged like the sibling spokes: pyproject.toml (apache-ossie-databricks), an ossie_databricks package under src/, ASF license headers, and a tests/ suite (example-based + Hypothesis property-based round-trip; 74 tests). PyYAML is the only runtime dependency. Co-authored-by: jackstein21 <82542300+jackstein21@users.noreply.github.com>
Co-authored-by: Emil Sadek <esadek@users.noreply.github.com>
* Switch to UV * Switch to UV * Switch to UV * Restored ASF header
Adds converters/wisdom, a one-way converter from a WisdomAI domain export JSON (format 1.0) to an Ossie semantic model YAML, following the layout of the dbt converter. Mapping: domain -> semantic model; domain knowledge and system instructions -> model-level ai_context; tables/columns/formulas -> datasets/fields; relationship graph -> relationships (cardinality folded into edge direction, compound AND-joins flattened to composite keys); per-table measures -> model-level metrics. Expressions are emitted verbatim under the dialect mapped from the table's connection (snowflake, databricks), falling back to ANSI_SQL with a warning for unsupported dialects. Information loss is reported through the same ConverterResult/ConverterIssue pattern as ossie-dbt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds WISDOM to the OSIVendor enum, the vendor examples in the spec documents and JSON schema, and the supported-vendors table in the converter guide. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds osi-to-wisdom, the reverse of the wisdom converter: an Ossie document becomes a wisdom domain export (format 1.0) consumable by wisdom's importDomain API. The mapping inverts wisdom-to-osi so round-trips are stable in both directions: model ai_context splits back into system instructions and knowledge items, fields split into columns and formulas, relationship direction is read as many-to-one with ai_context notes restoring one-to-one/many-to-many and composite keys becoming compound AND join conditions, and metrics attach to the table their expression references. IDs are deterministic (derived from names) and connections are per-dialect placeholders remapped at import time. Verified: Ossie -> wisdom -> Ossie round-trips of two real domain exports are byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BigQuery connections now map to the BIGQUERY dialect added to the spec instead of falling back to ANSI_SQL with a warning, with backtick identifier quoting; both directions round-trip it. The pyproject follows the repo's UV layout (dependency-groups, uv sources for the in-repo apache-ossie). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mwu-wisdom
force-pushed
the
wisdom-converter
branch
from
July 21, 2026 14:42
ba8672b to
cd803b4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
converters/wisdom, a bidirectional converter between WisdomAI domain exports (format1.0, the JSON produced by wisdom'sexportDomainAPI and consumed byimportDomain) and Ossie semantic model YAML. It follows the layout and conventions of the dbt converter (apache-ossiedependency,ConverterResult/ConverterIssueloss reporting, argparse CLI). Also registersWISDOMas a well-known custom-extension vendor in the spec documents andOSIVendorenum.Import mapping (wisdom → Ossie)
semantic_model[].name/descriptionref.name/descriptionai_context(model level)domainSystemInstructions+ domain knowledge contents as a bulleted listdatasets[]location.database.schema.dbTable→source;primaryKey.columnsorisPrimaryKeyflags →primary_key)datasets[].fields[]displayName→label; DATE/DATETIME/TIMESTAMP →dimension.is_timerelationships[]from= many side); compound AND-joins flattened to composite key arrays; OR/non-equi joins dropped with a warningmetrics[]Expressions are emitted verbatim under the Ossie dialect mapped from the table's connection (
snowflake → SNOWFLAKE,databricks → DATABRICKS); other dialects fall back toANSI_SQLverbatim with anUNSUPPORTED_DIALECTwarning.Export mapping (Ossie → wisdom)
The exact inverse, so round-trips are stable in both directions:
ai_contextsplits back intodomainSystemInstructions(leading text) + one knowledge item per-bullet.dimension.is_timemaps to aTIMESTAMPdata type (wisdom re-derives exact types from the warehouse).MANY_TO_ONEedges; theai_contextnotes written by the import direction restoreONE_TO_ONE/MANY_TO_MANY, and composite keys become compoundANDjoin conditions.unique_keys,custom_extensions, andai_contexton fields/metrics (plus synonyms/examples anywhere).Testing
OSI → wisdom → OSIyields an identicalOSIDocument), and determinism of the emitted export.validation/validate.pyfully (schema, uniqueness, reference integrity, sqlglot SQL parsing), andOSI → wisdom → OSIround-trips are byte-identical YAML for both.🤖 Generated with Claude Code