Skip to content

Commit 9c37b9e

Browse files
fresioASfresioAS
authored
Merge branch 'main' into add_fabric_warehouse
2 parents 145b69b + fcfbe8f commit 9c37b9e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+741
-214
lines changed

‎docs/guides/configuration.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -288,6 +288,34 @@ Conceptually, we can group the root level parameters into the following types. E
288288

289289
The rest of this page provides additional detail for some of the configuration options and provides brief examples. Comprehensive lists of configuration options are at the [configuration reference page](../reference/configuration.md).
290290

291+
### Cache directory
292+
293+
By default, the SQLMesh cache is stored in a `.cache` directory within your project folder. You can customize the cache location using the `cache_dir` configuration option:
294+
295+
=== "YAML"
296+
297+
```yaml linenums="1"
298+
# Relative path to project directory
299+
cache_dir: my_custom_cache
300+
301+
# Absolute path
302+
cache_dir: /tmp/sqlmesh_cache
303+
304+
```
305+
306+
=== "Python"
307+
308+
```python linenums="1"
309+
from sqlmesh.core.config import Config, ModelDefaultsConfig
310+
311+
config = Config(
312+
model_defaults=ModelDefaultsConfig(dialect="duckdb"),
313+
cache_dir="/tmp/sqlmesh_cache",
314+
)
315+
```
316+
317+
The cache directory is automatically created if it doesn't exist. You can clear the cache using the `sqlmesh clean` command.
318+
291319
### Table/view storage locations
292320

293321
SQLMesh creates schemas, physical tables, and views in the data warehouse/engine. Learn more about why and how SQLMesh creates schema in the ["Why does SQLMesh create schemas?" FAQ](../faq/faq.md#schema-question).

‎docs/integrations/engines/mssql.md‎

Lines changed: 53 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,65 @@
11
# MSSQL
22

3-
## Local/Built-in Scheduler
4-
**Engine Adapter Type**: `mssql`
3+
## Installation
54

6-
### Installation
7-
#### User / Password Authentication:
5+
### User / Password Authentication:
86
```
97
pip install "sqlmesh[mssql]"
108
```
11-
#### Microsoft Entra ID / Azure Active Directory Authentication:
9+
### Microsoft Entra ID / Azure Active Directory Authentication:
1210
```
1311
pip install "sqlmesh[mssql-odbc]"
1412
```
1513

14+
## Incremental by unique key `MERGE`
15+
16+
SQLMesh executes a `MERGE` statement to insert rows for [incremental by unique key](../../concepts/models/model_kinds.md#incremental_by_unique_key) model kinds.
17+
18+
By default, the `MERGE` statement updates all non-key columns of an existing row when a new row with the same key values is inserted. If all column values match between the two rows, those updates are unnecessary.
19+
20+
SQLMesh provides an optional performance optimization that skips unnecessary updates by comparing column values with the `EXISTS` and `EXCEPT` operators.
21+
22+
Enable the optimization by setting the `mssql_merge_exists` key to `true` in the [`physical_properties`](../../concepts/models/overview.md#physical_properties) section of the `MODEL` statement.
23+
24+
For example:
25+
26+
```sql linenums="1" hl_lines="7-9"
27+
MODEL (
28+
name sqlmesh_example.unique_key,
29+
kind INCREMENTAL_BY_UNIQUE_KEY (
30+
unique_key id
31+
),
32+
cron '@daily',
33+
physical_properties (
34+
mssql_merge_exists = true
35+
)
36+
);
37+
```
38+
39+
!!! warning "Not all column types supported"
40+
The `mssql_merge_exists` optimization is not supported for all column types, including `GEOMETRY`, `XML`, `TEXT`, `NTEXT`, `IMAGE`, and most user-defined types.
41+
42+
Learn more in the [MSSQL `EXCEPT` statement documentation](https://learn.microsoft.com/en-us/sql/t-sql/language-elements/set-operators-except-and-intersect-transact-sql?view=sql-server-ver17#arguments).
43+
44+
## Local/Built-in Scheduler
45+
**Engine Adapter Type**: `mssql`
46+
1647
### Connection options
1748

18-
| Option | Description | Type | Required |
19-
| ----------------- | ------------------------------------------------------------ | :----------: | :------: |
20-
| `type` | Engine type name - must be `mssql` | string | Y |
21-
| `host` | The hostname of the MSSQL server | string | Y |
22-
| `user` | The username / client id to use for authentication with the MSSQL server | string | N |
23-
| `password` | The password / client secret to use for authentication with the MSSQL server | string | N |
24-
| `port` | The port number of the MSSQL server | int | N |
25-
| `database` | The target database | string | N |
26-
| `charset` | The character set used for the connection | string | N |
27-
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
28-
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
29-
| `appname` | The application name to use for the connection | string | N |
30-
| `conn_properties` | The list of connection properties | list[string] | N |
31-
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
32-
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
33-
| `driver_name` | The driver name to use for the connection. E.g., *ODBC Driver 18 for SQL Server* | string | N |
34-
| `odbc_properties` | The dict of ODBC connection properties. E.g., authentication: ActiveDirectoryServicePrincipal. See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16). | dict | N |
49+
| Option | Description | Type | Required |
50+
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------: | :------: |
51+
| `type` | Engine type name - must be `mssql` | string | Y |
52+
| `host` | The hostname of the MSSQL server | string | Y |
53+
| `user` | The username / client id to use for authentication with the MSSQL server | string | N |
54+
| `password` | The password / client secret to use for authentication with the MSSQL server | string | N |
55+
| `port` | The port number of the MSSQL server | int | N |
56+
| `database` | The target database | string | N |
57+
| `charset` | The character set used for the connection | string | N |
58+
| `timeout` | The query timeout in seconds. Default: no timeout | int | N |
59+
| `login_timeout` | The timeout for connection and login in seconds. Default: 60 | int | N |
60+
| `appname` | The application name to use for the connection | string | N |
61+
| `conn_properties` | The list of connection properties | list[string] | N |
62+
| `autocommit` | Is autocommit mode enabled. Default: false | bool | N |
63+
| `driver` | The driver to use for the connection. Default: pymssql | string | N |
64+
| `driver_name` | The driver name to use for the connection (e.g., *ODBC Driver 18 for SQL Server*). | string | N |
65+
| `odbc_properties` | ODBC connection properties (e.g., *authentication: ActiveDirectoryServicePrincipal*). See more [here](https://learn.microsoft.com/en-us/sql/connect/odbc/dsn-connection-string-attribute?view=sql-server-ver16). | dict | N |

‎docs/reference/configuration.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ Configuration options for SQLMesh project directories.
2020
| ------------------ | ------------------------------------------------------------------------------------------------------------------ | :----------: | :------: |
2121
| `ignore_patterns` | Files that match glob patterns specified in this list are ignored when scanning the project folder (Default: `[]`) | list[string] | N |
2222
| `project` | The project name of this config. Used for [multi-repo setups](../guides/multi_repo.md). | string | N |
23+
| `cache_dir` | The directory to store the SQLMesh cache. Can be an absolute path or relative to the project directory. (Default: `.cache`) | string | N |
2324

2425
### Environments
2526

‎sqlmesh/core/config/root.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,7 @@ class Config(BaseConfig):
120120
disable_anonymized_analytics: Whether to disable the anonymized analytics collection.
121121
before_all: SQL statements or macros to be executed at the start of the `sqlmesh plan` and `sqlmesh run` commands.
122122
after_all: SQL statements or macros to be executed at the end of the `sqlmesh plan` and `sqlmesh run` commands.
123+
cache_dir: The directory to store the SQLMesh cache. Defaults to .cache in the project folder.
123124
"""
124125

125126
gateways: GatewayDict = {"": GatewayConfig()}
@@ -165,6 +166,7 @@ class Config(BaseConfig):
165166
after_all: t.Optional[t.List[str]] = None
166167
linter: LinterConfig = LinterConfig()
167168
janitor: JanitorConfig = JanitorConfig()
169+
cache_dir: t.Optional[str] = None
168170

169171
_FIELD_UPDATE_STRATEGY: t.ClassVar[t.Dict[str, UpdateStrategy]] = {
170172
"gateways": UpdateStrategy.NESTED_UPDATE,

‎sqlmesh/core/config/scheduler.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ def create_state_sync(self, context: GenericContext) -> StateSync:
105105

106106
schema = context.config.get_state_schema(context.gateway)
107107
return EngineAdapterStateSync(
108-
engine_adapter, schema=schema, context_path=context.path, console=context.console
108+
engine_adapter, schema=schema, cache_dir=context.cache_dir, console=context.console
109109
)
110110

111111
def state_sync_fingerprint(self, context: GenericContext) -> str:
@@ -130,6 +130,7 @@ class BuiltInSchedulerConfig(_EngineAdapterStateSyncSchedulerConfig, BaseConfig)
130130
def create_plan_evaluator(self, context: GenericContext) -> PlanEvaluator:
131131
return BuiltInPlanEvaluator(
132132
state_sync=context.state_sync,
133+
snapshot_evaluator=context.snapshot_evaluator,
133134
create_scheduler=context.create_scheduler,
134135
default_catalog=context.default_catalog,
135136
console=context.console,

0 commit comments

Comments
 (0)