Skip to content

Commit ce2098c

Browse files
authored
Docs: Update dbt incremental docs (#790)
* Update dbt incremental docs * Note sqlmesh batch_size parameter * PR feedback
1 parent 6635d93 commit ce2098c

1 file changed

Lines changed: 27 additions & 16 deletions

File tree

docs/integrations/dbt.md

Lines changed: 27 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ SQLMesh will use the data warehouse connection target in your dbt project `profi
1717

1818
### Setting model backfill start dates
1919

20-
Models **require** a start date for backfilling data through use of the `start` configuration parameter. `start` can be defined individually for each model, or globally in the `dbt_project.yml` file as follows:
20+
Models **require** a start date for backfilling data through use of the `start` configuration parameter. `start` can be defined individually for each model in its `config` block or globally in the `dbt_project.yml` file as follows:
2121

2222
```
2323
> models:
@@ -43,9 +43,9 @@ Consider the following when using a dbt project:
4343

4444
## How to use SQLMesh incremental models with dbt projects
4545

46-
Incremental loading is a powerful technique when datasets are large and recomputing tables is expensive. SQLMesh offers first-class support for incremental models, and its approach differs from dbt's in that SQLMesh will always run incrementally, even the first time, detecting and backfilling each missing time interval.
46+
Incremental loading is a powerful technique when datasets are large and recomputing tables is expensive. SQLMesh offers first-class support for incremental models, and its approach differs from dbt's.
4747

48-
This section describes how to implement SQLMesh incremental models in a dbt-formatted project.
48+
This section describes how to adapt dbt's incremental models to run on sqlmesh and maintain backwards compatibility with dbt.
4949

5050
### Incremental types
5151

@@ -56,27 +56,30 @@ SQLMesh supports two approaches to implement [idempotent](../concepts/glossary.m
5656

5757
#### Incremental by unique key
5858

59-
To enable incremental_by_unique_key incrementality, make sure the model configuration contains the following:
59+
To enable incremental_by_unique_key incrementality, the model configuration should contain:
6060

61-
* `unique_key`
62-
* `materialized` type of `incremental`
63-
* Either no `incremental_strategy` or `incremental_strategy` of `merge`
61+
* The `unique_key` key with the model's unique key field name or names as the value
62+
* The `materialized` key with value `'incremental'`
63+
* Either:
64+
* No `incremental_strategy` key or
65+
* The `incremental_strategy` key with value `'merge'`
6466

6567
#### Incremental by time range
6668

67-
To enable incremental_by_time_range incrementality, make sure the model configuration contains the following:
69+
To enable incremental_by_time_range incrementality, the model configuration should contain:
6870

69-
* `time_column` (see [`time column`](../concepts/models/model_kinds.md#time-column) for details)
70-
* `materialized` type of `incremental`
71-
* `incremental_strategy` of either `insert_ovewrite` or `delete+insert`
72-
73-
Note: SQLMesh will use the [`best incremental strategy`](../concepts/models/model_kinds.md#materialization-strategy) for the target engine.
71+
* The `time_column` key with the model's time column field name as the value (see [`time column`](../concepts/models/model_kinds.md#time-column) for details)
72+
* The `materialized` key with value `'incremental'`
73+
* Either:
74+
* The `incremental_strategy` key with value `'insert_overwrite'` or
75+
* The `incremental_strategy` key with value `'delete+insert'`
76+
* Note: in this context, these two strategies are synonyms. Regardless of which one is specified SQLMesh will use the [`best incremental strategy`](../concepts/models/model_kinds.md#materialization-strategy) for the target engine.
7477

7578
### Incremental logic
7679

77-
SQLMesh's incremental logic will ignore dbt's incremental jinja block `{% if is_incremental() %}`, to maintain backwards compatibility, and instead use a new jinja block gated by `{% if sqlmesh is defined %}`. The new block will contain the where clause selecting the time interval.
80+
SQLMesh will ignore dbt's incremental jinja block `{% if is_incremental() %}` and requires a new jinja block gated by `{% if sqlmesh is defined %}`. The new block should contain the `WHERE` clause selecting the time interval.
7881

79-
For example, the SQL `WHERE` clause with the "ds" column goes in a jinja block gated by `{% if sqlmesh is defined %}` as follows:
82+
For example, the SQL `WHERE` clause with the "ds" column goes in a new jinja block gated by `{% if sqlmesh is defined %}` as follows:
8083

8184
```bash
8285
> {% if sqlmesh is defined %}
@@ -85,7 +88,15 @@ For example, the SQL `WHERE` clause with the "ds" column goes in a jinja block g
8588
> {% endif %}
8689
```
8790
88-
`{{ start_ds }}` and `{{ end_ds }}` are the jinja equivalent of SQLMesh's `@start_ds` and `@end_ds` predefined time variables. See all [`predefined time variables`](../concepts/macros.md#predefined-variables) available in jinja.
91+
`{{ start_ds }}` and `{{ end_ds }}` are the jinja equivalents of SQLMesh's `@start_ds` and `@end_ds` predefined time macro variables. See all [predefined time variables](../concepts/macros.md#predefined-variables) available in jinja.
92+
93+
### Incremental model configuration
94+
95+
SQLMesh provides configuration parameters that enable control over how incremental computations occur. These parameters are set in the model's `config` block.
96+
97+
The [`batch_size` parameter](../concepts/models/overview.md#batch_size) determines the maximum number of time intervals to run in a single job.
98+
99+
The [`lookback` parameter](../concepts/models/overview.md#lookback) is used to capture late arriving data. It sets the number of units of late arriving data the model should expect and must be a positive integer.
89100
90101
## Tests
91102
SQLMesh uses dbt tests to perform SQLMesh [audits](../concepts/audits.md) (coming soon).

0 commit comments

Comments
 (0)