You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/dbt.md
+27-16Lines changed: 27 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,7 +17,7 @@ SQLMesh will use the data warehouse connection target in your dbt project `profi
17
17
18
18
### Setting model backfill start dates
19
19
20
-
Models **require** a start date for backfilling data through use of the `start` configuration parameter. `start` can be defined individually for each model, or globally in the `dbt_project.yml` file as follows:
20
+
Models **require** a start date for backfilling data through use of the `start` configuration parameter. `start` can be defined individually for each model in its `config` block or globally in the `dbt_project.yml` file as follows:
21
21
22
22
```
23
23
> models:
@@ -43,9 +43,9 @@ Consider the following when using a dbt project:
43
43
44
44
## How to use SQLMesh incremental models with dbt projects
45
45
46
-
Incremental loading is a powerful technique when datasets are large and recomputing tables is expensive. SQLMesh offers first-class support for incremental models, and its approach differs from dbt's in that SQLMesh will always run incrementally, even the first time, detecting and backfilling each missing time interval.
46
+
Incremental loading is a powerful technique when datasets are large and recomputing tables is expensive. SQLMesh offers first-class support for incremental models, and its approach differs from dbt's.
47
47
48
-
This section describes how to implement SQLMesh incremental models in a dbt-formatted project.
48
+
This section describes how to adapt dbt's incremental models to run on sqlmesh and maintain backwards compatibility with dbt.
49
49
50
50
### Incremental types
51
51
@@ -56,27 +56,30 @@ SQLMesh supports two approaches to implement [idempotent](../concepts/glossary.m
56
56
57
57
#### Incremental by unique key
58
58
59
-
To enable incremental_by_unique_key incrementality, make sure the model configuration contains the following:
59
+
To enable incremental_by_unique_key incrementality, the model configuration should contain:
60
60
61
-
*`unique_key`
62
-
*`materialized` type of `incremental`
63
-
* Either no `incremental_strategy` or `incremental_strategy` of `merge`
61
+
* The `unique_key` key with the model's unique key field name or names as the value
62
+
* The `materialized` key with value `'incremental'`
63
+
* Either:
64
+
* No `incremental_strategy` key or
65
+
* The `incremental_strategy` key with value `'merge'`
64
66
65
67
#### Incremental by time range
66
68
67
-
To enable incremental_by_time_range incrementality, make sure the model configuration contains the following:
69
+
To enable incremental_by_time_range incrementality, the model configuration should contain:
68
70
69
-
*`time_column` (see [`time column`](../concepts/models/model_kinds.md#time-column) for details)
70
-
*`materialized` type of `incremental`
71
-
*`incremental_strategy` of either `insert_ovewrite` or `delete+insert`
72
-
73
-
Note: SQLMesh will use the [`best incremental strategy`](../concepts/models/model_kinds.md#materialization-strategy) for the target engine.
71
+
* The `time_column` key with the model's time column field name as the value (see [`time column`](../concepts/models/model_kinds.md#time-column) for details)
72
+
* The `materialized` key with value `'incremental'`
73
+
* Either:
74
+
* The `incremental_strategy` key with value `'insert_overwrite'` or
75
+
* The `incremental_strategy` key with value `'delete+insert'`
76
+
* Note: in this context, these two strategies are synonyms. Regardless of which one is specified SQLMesh will use the [`best incremental strategy`](../concepts/models/model_kinds.md#materialization-strategy) for the target engine.
74
77
75
78
### Incremental logic
76
79
77
-
SQLMesh's incremental logic will ignore dbt's incremental jinja block `{% if is_incremental() %}`, to maintain backwards compatibility, and instead use a new jinja block gated by `{% if sqlmesh is defined %}`. The new block will contain the where clause selecting the time interval.
80
+
SQLMeshwill ignore dbt's incremental jinja block `{% if is_incremental() %}`and requires a new jinja block gated by `{% if sqlmesh is defined %}`. The new block should contain the `WHERE` clause selecting the time interval.
78
81
79
-
For example, the SQL `WHERE` clause with the "ds" column goes in a jinja block gated by `{% if sqlmesh is defined %}` as follows:
82
+
For example, the SQL `WHERE` clause with the "ds" column goes in a new jinja block gated by `{% if sqlmesh is defined %}` as follows:
80
83
81
84
```bash
82
85
> {% if sqlmesh is defined %}
@@ -85,7 +88,15 @@ For example, the SQL `WHERE` clause with the "ds" column goes in a jinja block g
85
88
> {% endif %}
86
89
```
87
90
88
-
`{{ start_ds }}` and `{{ end_ds }}` are the jinja equivalent of SQLMesh's `@start_ds` and `@end_ds` predefined time variables. See all [`predefined time variables`](../concepts/macros.md#predefined-variables) available in jinja.
91
+
`{{ start_ds }}` and `{{ end_ds }}` are the jinja equivalents of SQLMesh's `@start_ds` and `@end_ds` predefined time macro variables. See all [predefined time variables](../concepts/macros.md#predefined-variables) available in jinja.
92
+
93
+
### Incremental model configuration
94
+
95
+
SQLMesh provides configuration parameters that enable control over how incremental computations occur. These parameters are set in the model's `config` block.
96
+
97
+
The [`batch_size` parameter](../concepts/models/overview.md#batch_size) determines the maximum number of time intervals to run in a single job.
98
+
99
+
The [`lookback` parameter](../concepts/models/overview.md#lookback) is used to capture late arriving data. It sets the number of units of late arriving data the model should expect and must be a positive integer.
89
100
90
101
## Tests
91
102
SQLMesh uses dbt tests to perform SQLMesh [audits](../concepts/audits.md) (coming soon).
0 commit comments