Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ optional-dependencies.develop = [
"poethepoet<1",
"pyproject-fmt<3",
"pytest-repeat<1",
"ruff<0.16",
"ruff==0.16.*",
"ty==0.0.63",
"validate-pyproject<1",
]
Expand Down Expand Up @@ -206,6 +206,9 @@ lint.select = [
"C4",
# Pycodestyle
"E",
"E4",
"E7",
"E9",
# eradicate
"ERA",
# Pyflakes
Expand Down
19 changes: 14 additions & 5 deletions src/omniload/source/google_sheets/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,26 +69,35 @@ In case of Airflow, the source is created and executed separately. In typical co
**Moreover, you should not use `scc` decomposition in our Airflow helper**. It will create an instance of the source for each requested range in order to run a task that corresponds to it! Following our [Airflow deployment guide](https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deploy-with-airflow-composer#2-modify-dag-file), this is how you should use `tasks.add_run` on `PipelineTasksGroup`:
```python
@dag(
schedule_interval='@daily',
schedule_interval="@daily",
start_date=pendulum.datetime(2023, 2, 1),
catchup=False,
max_active_runs=1,
default_args=default_task_args
default_args=default_task_args,
)
def get_named_ranges():
tasks = PipelineTasksGroup("get_named_ranges", use_data_folder=False, wipe_local_data=True)
tasks = PipelineTasksGroup(
"get_named_ranges", use_data_folder=False, wipe_local_data=True
)

# import your source from pipeline script
from google_sheets import google_spreadsheet

pipeline = dlt.pipeline(
pipeline_name="get_named_ranges",
dataset_name="named_ranges_data",
destination='bigquery',
destination="bigquery",
)

# do not use decompose to run `google_spreadsheet` in single task
tasks.add_run(pipeline, google_spreadsheet("1HhWHjqouQnnCIZAFa2rL6vT91YRN8aIhts22SUUR580"), decompose="none", trigger_rule="all_done", retries=0, provide_context=True)
tasks.add_run(
pipeline,
google_spreadsheet("1HhWHjqouQnnCIZAFa2rL6vT91YRN8aIhts22SUUR580"),
decompose="none",
trigger_rule="all_done",
retries=0,
provide_context=True,
)
```

## Setup credentials
Expand Down
Loading