ACE-072: cross-datasource bridge as a validated model type - #156
ACE-072: cross-datasource bridge as a validated model type#156ashwin-agami wants to merge 4 commits into
Conversation
Add CrossDatasourceRelationship (deployment-level bridge) to the semantic model, merge it into the OrgRecord from inline/sidecar/legacy sources on load (deduped, idempotent), and validate its endpoints in a new validate_deployment pass wired into sm validate. Legacy cross_profile_relationships.yaml is migrated in-memory.
…et schema match (ACE-072)
There was a problem hiding this comment.
Pull request overview
This PR implements the ACE-072 foundation for F16 cross-datasource metrics by promoting the “bridge” concept (shared-key link across two datasources) into a first-class, validated deployment-level model on OrgRecord, plus a deployment-wide validation pass and CLI wiring.
Changes:
- Adds a new
CrossDatasourceRelationshipmodel type and stores a list of them onOrgRecord. - Adds deployment-level validation (
validate_deployment) to resolve bridge endpoints across all attached datasources and reject invalid bridges. - Updates
sm validateto run the deployment-level pass when invoked within a deployment layout, and adds comprehensive tests for model/merge/migration/validation behavior.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_cross_datasource_bridge.py | New tests covering the bridge model, deployment validation, merge/migration behavior, and CLI integration. |
| packages/agami-core/src/semantic_model/validator.py | Adds validate_deployment() and supporting helpers to validate cross-datasource bridges across all datasource models. |
| packages/agami-core/src/semantic_model/org_record.py | Loads/merges bridges from inline + sidecar + legacy sources and dedupes them by endpoint identity. |
| packages/agami-core/src/semantic_model/models.py | Introduces CrossDatasourceRelationship and adds cross_datasource_relationships to OrgRecord. |
| packages/agami-core/src/semantic_model/cli.py | Extends sm validate to also run the deployment-level validation pass when applicable. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if sidecar.exists(): | ||
| doc = yaml.safe_load(sidecar.read_text(encoding="utf-8")) or {} | ||
| for r in doc.get("relationships", doc if isinstance(doc, list) else []): | ||
| try: | ||
| out.append(CrossDatasourceRelationship(**r)) | ||
| except Exception: | ||
| # load_org_record is lenient by contract (never raises — it's on runtime paths). A | ||
| # malformed entry is skipped so its valid siblings in the same file still load. | ||
| continue |
| if legacy.exists(): | ||
| doc = yaml.safe_load(legacy.read_text(encoding="utf-8")) or {} | ||
| for r in doc.get("relationships", doc if isinstance(doc, list) else []): | ||
| try: | ||
| out.append(_migrate_legacy_bridge(r, str(legacy))) | ||
| except Exception: | ||
| # Same leniency: a legacy entry missing a required key is skipped, not raised, so a | ||
| # single malformed row can't take down a runtime load_org_record call. | ||
| continue |
…ecord (ACE-072) The per-entry parse was already lenient; wrap the sidecar/legacy yaml.safe_load too so a corrupt bridge FILE degrades to no bridges rather than raising out of the runtime-path load_org_record. Adds test_corrupt_bridge_yaml_file_does_not_raise.
_load_bridges claimed to support a mapping OR a bare YAML list, but doc.get(...) raises AttributeError on a bare list before the default is used. Extract entries via an explicit shape check (_bridge_entries); add a bare-list test.
|
Thanks @copilot — both findings were valid and are fixed in 2ade01a.
Both are locked by new tests: |
Confirmed — both issues are addressed in the current code. |
Spec: ACE-072
Summary
First slice of F16 — cross-datasource metrics. Promotes the cross-datasource "bridge" (the shared-key link that says a customer in one datasource is the same entity in another, e.g.
accounts.account_key=customers.account_key) from a skill-side file into a first-class, validated semantic-model type on the deployment-levelOrgRecord. This is the foundation the cross-datasource metric (ACE-073) will reconcile through. Previously the bridge lived only in<artifacts_dir>/local/cross_profile_relationships.yaml— unvalidated, and lost on a machine re-onboard (the confirmed Northpeak v2 regression).Scope is the foundation only. The idempotent "link your databases" offer + the governed
sm bridgesCLI are deliberately out of scope → ACE-078; the metric type/execution/browse are ACE-073/074/076.Changes
models.py— new standaloneCrossDatasourceRelationship(datasource+schema.table+column-list endpoints;executablerestricted tosplit/informational, rejectingsame_engine; optional cardinality; trust block;endpoint_keyfor dedup). Adds thecross_datasource_relationshipsbucket toOrgRecord.org_record.py—load_org_recordnow merges bridges from three sources (inline · sidecarcross_datasource_relationships.yaml· legacycross_profile_relationships.yamlmigration), deduped by endpoint tuple, in-memory / read-only, and lenient (a malformed optional file is skipped per-entry, never raised — preserving the runtime-path contract).validator.py— new deployment-levelvalidate_deployment()pass: loads every attached datasource model and rejects a bridge that issame_engineor whose datasource/dataset/column endpoint doesn't resolve (cross_datasource_executable_mismatch,cross_datasource_endpoint_unresolved), fail-closed.cli.py—sm validatealso runs the deployment pass when anOrgRecordis present (no new subcommand; a no-op on pre-F16 layouts).tests/test_cross_datasource_bridge.py(model, validation reject/accept, idempotent + precedence-correct migration, malformed-file leniency, distinct unreadable-profile message, CLI wiring).Test plan
uv run dev.py checkgreen (1576 passed / 1 skipped; ruff + gitleaks clean).uv run dev.py cover= 100% patch coverage.Checklist
Spec: ACE-072), scope-contained (no ACE-073/074/078 work)acme/demo/account_keyfixtures only — no customer or internal naming