Skip to content

feat: add --mode flag to create_external_models command#5879

Open
christophmeier90 wants to merge 1 commit into
SQLMesh:mainfrom
christophmeier90:feat/create-external-models-mode
Open

feat: add --mode flag to create_external_models command#5879
christophmeier90 wants to merge 1 commit into
SQLMesh:mainfrom
christophmeier90:feat/create-external-models-mode

Conversation

@christophmeier90

@christophmeier90 christophmeier90 commented Jul 5, 2026

Copy link
Copy Markdown

Problem

Metadata is destroyed on every run

The current create_external_models command has no way to preserve hand-edited metadata in external_models.yaml. On every run, it fully replaces same-gateway entries with only name, columns, and gateway — losing any description, audits, dialect, start, tags, owner, stamp, signals, etc. that users may have configured by hand.

Stale entries accumulate silently

External models that are no longer referenced by any SQL model stay in the YAML file indefinitely. There's no way to detect or remove them. This caused #4906, where a stale table entry with a mismatched nesting level persisted in the YAML file and produced confusing errors that only resolved after manually deleting and recreating the file.

Changes

Adds a --mode flag with three values:

Mode Metadata Stale entries
overwrite (default, current behavior) Destroyed Replaced
sync Preserved Warned + kept
sync_prune Preserved Removed

How each mode works

overwrite (unchanged): same-gateway entries are removed, freshly fetched {name, columns, gateway} entries are appended. Other-gateway entries are preserved.

sync: matches entries by name, replaces only the columns field with DB-fetched types, and keeps all other fields. Entries in the YAML that are no longer referenced by any model get a warning but are preserved.

sync_prune: same as sync but removes unreferenced entries from the file with a warning.

@christophmeier90 christophmeier90 force-pushed the feat/create-external-models-mode branch 2 times, most recently from 6fe62a5 to afc7d2b Compare July 5, 2026 04:28
Adds a --mode option to create_external_models with three values:
- overwrite (default): current behavior, replaces same-gateway entries
- sync: preserves hand-edited metadata, only syncs columns from DB,
  warns on stale entries but keeps them
- sync_prune: same as sync but removes stale entries

Introduces CreateExternalModelsMode enum following the codebase's
(str, Enum) convention with is_* properties and a default classproperty.

Fixes a bug where external models loaded from the existing YAML file
were always re-fetched from the database, preventing stale detection.
The new logic only fetches FQNs referenced by non-external model
depends_on, excluding known internal models to prevent conversion.

Signed-off-by: Christoph Meier <chrism216@gmail.com>
@christophmeier90 christophmeier90 force-pushed the feat/create-external-models-mode branch from afc7d2b to 5ddff66 Compare July 5, 2026 04:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant