External models default yaml (#2717)

Author: dbrtly

docs/concepts/models/external_models.md

@@ -10,15 +10,15 @@ SQLMesh stores external tables' column information as `EXTERNAL` models.
 
 `EXTERNAL` models consist solely of an external table's column information, so there is no query for SQLMesh to run.
 
-SQLMesh has no information about the data contained in the table represented by an `EXTERNAL` model. The table could be altered or have all its data deleted, and SQLMesh will not detect it. All SQLMesh knows about the table is that it contains the columns specified in the `EXTERNAL` model's `schema.yaml` file (more information below).
+SQLMesh has no information about the data contained in the table represented by an `EXTERNAL` model. The table could be altered or have all its data deleted, and SQLMesh will not detect it. All SQLMesh knows about the table is that it contains the columns specified in the `EXTERNAL` model's file (more information below).
 
 SQLMesh will not take any actions based on an `EXTERNAL` model - its actions are solely determined by the model whose query selects from the `EXTERNAL` model.
 
 The querying model's [`kind`](./model_kinds.md), [`cron`](./overview.md#cron), and previously loaded time intervals determine when SQLMesh will query the `EXTERNAL` model.
 
 ## Generating an external models schema file
 
-External models can be defined in the `schema.yaml` file in the SQLMesh project's root folder.
+External models can be defined in the `external_models.yaml` file in the SQLMesh project's root folder. The alternative name for this file is `schema.yaml`.
 
 You can create this file by either writing the YAML by hand or allowing SQLMesh to fetch information about external tables with the `create_external_models` CLI command.
 
@@ -38,13 +38,13 @@ FROM
 
 The following sections demonstrate how to create an external model containing `external_db.external_table`'s column information.
 
-All of a SQLMesh project's external models are defined in a single `schema.yaml` file, so the files created below might also include column information for other external models.
+All of a SQLMesh project's external models are defined in a single `external_models.yaml` file, so the files created below might also include column information for other external models.
 
 Alternatively, additional external models can also be defined in the [external_models/](#using-the-external_models-directory) folder.
 
 ### Writing YAML by hand
 
-This example demonstrates the structure of a `schema.yaml` file:
+This example demonstrates the structure of a `external_models.yaml` file:
 
 ```yaml
 - name: external_db.external_table
@@ -65,9 +65,9 @@ The file can be constructed by hand using a standard text editor or IDE.
 
 ### Using CLI
 
-Instead of creating the `schema.yaml` file manually, SQLMesh can generate it for you with the [create_external_models](../../reference/cli.md#create_external_models) CLI command.
+Instead of creating the `external_models.yaml` file manually, SQLMesh can generate it for you with the [create_external_models](../../reference/cli.md#create_external_models) CLI command.
 
-The command identifies all external tables referenced in your SQLMesh project, fetches their column information from the SQL engine's metadata, and then stores the information in the `schema.yaml` file.
+The command identifies all external tables referenced in your SQLMesh project, fetches their column information from the SQL engine's metadata, and then stores the information in the `external_models.yaml` file.
 
 If SQLMesh does not have access to an external table's metadata, the table will be omitted from the file and SQLMesh will issue a warning.
 
@@ -77,18 +77,18 @@ If SQLMesh does not have access to an external table's metadata, the table will
 
 Sometimes, SQLMesh cannot infer the structure of a model and you need to add it manually.
 
-However, since `sqlmesh create_external_models` replaces the `schema.yaml` file, any manual changes you made to that file will be overwritten.
+However, since `sqlmesh create_external_models` replaces the `external_models.yaml` file, any manual changes you made to that file will be overwritten.
 
 The solution is to create the manual model definition files in the `external_models/` directory, like so:
 
 ```
-schema.yaml
-external_models/another_schema.yaml
-external_models/yet_another_schema.yaml
+external_models.yaml
+external_models/more_external_models.yaml
+external_models/even_more_external_models.yaml
 ```
 
-Files in the `external_models` directory must be `.yaml` files that follow the same structure as the `schema.yaml` file.
+Files in the `external_models` directory must be `.yaml` files that follow the same structure as the `external_models.yaml` file.
 
-When SQLMesh loads the definitions, it will first load the models defined in `schema.yaml` followed by any models it can find in `external_models/*.yaml`.
+When SQLMesh loads the definitions, it will first load the models defined in `external_models.yaml` (or `schema.yaml`) and  any models found in `external_models/*.yaml`.
 
-Therefore, you can use `sqlmesh create_external_models` to manage the `schema.yaml` file and then put any models that need to be defined manually inside the `external_models/` directory.
+Therefore, you can use `sqlmesh create_external_models` to manage the `external_models.yaml` file and then put any models that need to be defined manually inside the `external_models/` directory.

docs/concepts/tests.md

@@ -474,7 +474,7 @@ It's not always possible to correctly interpret certain values in a unit test wi
 
 To avoid this ambiguity, SQLMesh needs to know the columns' types. By default, it will try to infer these types based on the model definitions, but they can also be explicitly specified:
 
-- in the [`schema.yaml`](models/external_models.md#generating-an-external-models-schema-file) file (for external models)
+- in the [`external_models.yaml`](models/external_models.md#generating-an-external-models-schema-file) file (for external models)
 - using the [`columns`](models/overview.md#columns) model property
 - using the [`columns`](#creating_tests) attribute of the unit test
 

docs/guides/table_migration.md

@@ -39,7 +39,7 @@ Consider an existing table named `my_schema.existing_table`. Migrating this tabl
 
 1. Ensure `my_schema.existing_table` is up to date (has ingested all available source data)
 2. Rename `my_schema.existing_table` to any other name, such as `my_schema.existing_table_historical`
-    - Optionally, enable column-level lineage for the table by making it an [`EXTERNAL` model](../concepts/models/model_kinds.md#external) and adding it to the project's `schema.yaml` file
+    - Optionally, enable column-level lineage for the table by making it an [`EXTERNAL` model](../concepts/models/model_kinds.md#external) and adding it to the project's `external_models.yaml` file
 3. Create a new incremental staging model named `my_schema.existing_table_staging` (see below for code)
 4. Create a new [`VIEW` model](../concepts/models/model_kinds.md#view) named `my_schema.existing_table` (see below for code)
 5. Run `sqlmesh plan` to create and backfill the models

examples/sushi/schema.yaml examples/sushi/external_models.yamlexamples/sushi/schema.yaml renamed to examples/sushi/external_models.yaml

@@ -40,15 +40,16 @@
 """The default directory for log files."""
 
 AUDITS = "audits"
+CACHE = ".cache"
+EXTERNAL_MODELS = "external_models"
 MACROS = "macros"
 METRICS = "metrics"
 MODELS = "models"
-EXTERNAL_MODELS = "external_models"
 SEEDS = "seeds"
 TESTS = "tests"
-CACHE = ".cache"
-SCHEMA_YAML = "schema.yaml"
 
+EXTERNAL_MODELS_YAML = "external_models.yaml"
+EXTERNAL_MODELS_DEPRECATED_YAML = "schema.yaml"
 
 DEFAULT_SCHEMA = "default"
 

sqlmesh/core/constants.py

@@ -81,7 +81,7 @@
 from sqlmesh.core.plan import Plan, PlanBuilder
 from sqlmesh.core.reference import ReferenceGraph
 from sqlmesh.core.scheduler import Scheduler
-from sqlmesh.core.schema_loader import create_schema_file
+from sqlmesh.core.schema_loader import create_external_models_file
 from sqlmesh.core.selector import Selector
 from sqlmesh.core.snapshot import (
     DeployabilityIndex,
@@ -1598,17 +1598,22 @@ def rollback(self) -> None:
 
     @python_api_analytics
     def create_external_models(self) -> None:
-        """Create a schema file with all external models.
+        """Create a file to document the schema of external models.
 
-        The schema file contains all columns and types of external models, allowing for more robust
-        lineage, validation, and optimizations.
+        The external models file contains all columns and types of external models, allowing for more
+        robust lineage, validation, and optimizations.
         """
         if not self._models:
             self.load(update_schemas=False)
 
         for path, config in self.configs.items():
-            create_schema_file(
-                path=path / c.SCHEMA_YAML,
+            deprecated_yaml = path / c.EXTERNAL_MODELS_DEPRECATED_YAML
+
+            external_models_yaml = (
+                path / c.EXTERNAL_MODELS_YAML if not deprecated_yaml.exists() else deprecated_yaml
+            )
+            create_external_models_file(
+                path=external_models_yaml,
                 models=UniqueKeyDict(
                     "models",
                     {

sqlmesh/core/context.py

@@ -182,12 +182,15 @@ def _load_metrics(self) -> UniqueKeyDict[str, MetricMeta]:
     def _load_external_models(self) -> UniqueKeyDict[str, Model]:
         models: UniqueKeyDict[str, Model] = UniqueKeyDict("models")
         for context_path, config in self._context.configs.items():
-            schema_path = Path(context_path / c.SCHEMA_YAML)
+            external_models_yaml = Path(context_path / c.EXTERNAL_MODELS_YAML)
+            deprecated_yaml = Path(context_path / c.EXTERNAL_MODELS_DEPRECATED_YAML)
             external_models_path = context_path / c.EXTERNAL_MODELS
 
             paths_to_load = []
-            if schema_path.exists():
-                paths_to_load.append(schema_path)
+            if external_models_yaml.exists():
+                paths_to_load.append(external_models_yaml)
+            elif deprecated_yaml.exists():
+                paths_to_load.append(deprecated_yaml)
 
             if external_models_path.exists() and external_models_path.is_dir():
                 paths_to_load.extend(external_models_path.glob("*.yaml"))

sqlmesh/core/loader.py

@@ -16,15 +16,15 @@
 logger = logging.getLogger(__name__)
 
 
-def create_schema_file(
+def create_external_models_file(
     path: Path,
     models: UniqueKeyDict[str, Model],
     adapter: EngineAdapter,
     state_reader: StateReader,
     dialect: DialectType,
     max_workers: int = 1,
 ) -> None:
-    """Create or replace a YAML file with model schemas.
+    """Create or replace a YAML file with column and types of all columns in all external models.
 
     Args:
         path: The path to store the YAML file.

sqlmesh/core/schema_loader.py

@@ -142,7 +142,7 @@ def setUp(self) -> None:
                             _raise_error(
                                 f"Failed to infer the data type of column '{col}' for '{name}'. This issue can be "
                                 "mitigated by casting the column in the model definition, setting its type in "
-                                "schema.yaml if it's an external model, setting the model's 'columns' property, "
+                                "external_models.yaml if it's an external model, setting the model's 'columns' property, "
                                 "or setting its 'columns' mapping in the test itself",
                                 self.path,
                             )

sqlmesh/core/test/definition.py

@@ -668,7 +668,7 @@ def test_load_external_models(copy_to_temp_path):
 
     assert len(external_model_names) > 0
 
-    # from schema.yaml in root dir
+    # from default external_models.yaml in root dir
     assert "raw.demographics" in external_model_names
 
     # from external_models/model1.yaml