Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions build_backends/build_backends/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -440,6 +440,9 @@ def main():
# legs; imported lazily (sqlalchemy dialect adapter / is_available
# probe), so list it explicitly like the other DB drivers.
"pymssql",
# Snowflake driver: imported lazily inside the snowflake dialect, so
# PyInstaller's static scan misses it (hooks-contrib bundles its data).
"snowflake.connector",
"alembic",
# certifi ships cacert.pem; ssl uses it via certifi.where(). The
# data_downloader builds its SSL context against this so urllib calls
Expand Down
40 changes: 40 additions & 0 deletions docs/examples/integrations/database_read_snowflake.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
"""Read from Snowflake through a stored database connection.

Runs only when FLOWFILE_TEST_SNOWFLAKE_* credentials are configured (see
flowfile_core/tests/docs_examples/test_docs_examples.py); the account,
warehouse, and role travel in the connection's extra_params.
"""

import os

import flowfile as ff

ff.create_database_connection_if_not_exists(
"analytics-snowflake",
database_type="snowflake",
database=os.environ.get("FLOWFILE_TEST_SNOWFLAKE_DATABASE", "SNOWFLAKE_SAMPLE_DATA"),
username=os.environ["FLOWFILE_TEST_SNOWFLAKE_USER"],
password=os.environ["FLOWFILE_TEST_SNOWFLAKE_PASSWORD"],
extra_params={
"account": os.environ["FLOWFILE_TEST_SNOWFLAKE_ACCOUNT"],
"warehouse": os.environ["FLOWFILE_TEST_SNOWFLAKE_WAREHOUSE"],
},
)

# --8<-- [start:example]
suppliers = ff.read_database(
"analytics-snowflake",
query="""
SELECT s_name, s_acctbal
FROM tpch_sf1.supplier
WHERE s_acctbal > 9900
""",
).collect()

nations = ff.read_database(
"analytics-snowflake", schema_name="TPCH_SF1", table_name="NATION"
).collect()
# --8<-- [end:example]

assert nations.height == 25
assert suppliers.height > 0
2 changes: 1 addition & 1 deletion docs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -637,7 +637,7 @@ <h3 class="feature-title">Scheduling &amp; Triggers</h3>
<h3 class="feature-title">Kafka, Databases &amp; Cloud Storage</h3>
<p class="feature-description">
Ingest from Kafka/Redpanda as a canvas node. Read and write S3, Azure Data Lake, and GCS.
Connect PostgreSQL, MySQL, SQL Server, SQLite, and DuckDB.
Connect PostgreSQL, MySQL, SQL Server, Snowflake, SQLite, and DuckDB.
</p>
<span class="feature-arrow">→</span>
</a>
Expand Down
4 changes: 2 additions & 2 deletions docs/users/connect/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Each connector below is read-only, write-only, or both. "Where configured" names

| Connector | Reads | Writes | Where configured | Setup |
|---|---|---|---|---|
| PostgreSQL / MySQL / SQLite / DuckDB / SQL Server | yes | yes | Connections → Database | [Databases](../visual-editor/tutorials/database-connectivity.md) |
| PostgreSQL / MySQL / SQLite / DuckDB / SQL Server / Snowflake | yes | yes | Connections → Database | [Databases](../visual-editor/tutorials/database-connectivity.md) |
| Cloud storage (S3 / ADLS / GCS) | yes | yes | Connections → Cloud Storage | [Cloud storage](../visual-editor/tutorials/cloud-connections.md) |
| Kafka / Redpanda | yes | no | Connections → Kafka | [Kafka](kafka.md) |
| REST API | yes | no | REST API Reader node (inline) | [REST APIs](apis.md#rest-apis) |
Expand All @@ -23,7 +23,7 @@ Each connector below is read-only, write-only, or both. "Where configured" names

## Databases

Typed connections to **PostgreSQL**, **MySQL**, **SQLite**, **DuckDB**, and **SQL Server**. Both directions are supported: the Database Reader node (`ff.read_database`) runs a query or reads a whole table, and the Database Writer node (`ff.write_database`) writes a frame back. Credentials are stored encrypted and referenced by name.
Typed connections to **PostgreSQL**, **MySQL**, **SQLite**, **DuckDB**, **SQL Server**, and **Snowflake**. Both directions are supported: the Database Reader node (`ff.read_database`) runs a query or reads a whole table, and the Database Writer node (`ff.write_database`) writes a frame back. Credentials are stored encrypted and referenced by name.

See [Databases](../visual-editor/tutorials/database-connectivity.md) for the connection form and worked reader/writer examples.

Expand Down
4 changes: 2 additions & 2 deletions docs/users/data-elsewhere.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@ None of this route requires code: every source below is a node in the visual edi

![One encrypted connection store in the middle holds named connections — warehouse, data-lake, events — while flows on the canvas and Python scripts around it reference those connections by name only; rotating a credential once updates every reference.](../assets/images/concepts/connection-store.svg)

Supported out of the box: databases (PostgreSQL, MySQL, SQLite, DuckDB, SQL Server), cloud storage (S3, Azure Data Lake, Google Cloud Storage — with auth methods from stored keys to ambient `aws-cli`/environment credentials), Kafka/Redpanda brokers, and Google Analytics 4 properties. [Connections](visual-editor/connections.md) covers every form field.
Supported out of the box: databases (PostgreSQL, MySQL, SQLite, DuckDB, SQL Server, Snowflake), cloud storage (S3, Azure Data Lake, Google Cloud Storage — with auth methods from stored keys to ambient `aws-cli`/environment credentials), Kafka/Redpanda brokers, and Google Analytics 4 properties. [Connections](visual-editor/connections.md) covers every form field.

## 2. Read from the source, not from copies

Each source is a drag-and-drop node on the canvas — with a matching `ff.*` call for code workflows:

| Your data is in… | On the canvas | In Python |
|---|---|---|
| PostgreSQL / MySQL / SQLite / DuckDB / SQL Server | Database Reader | `ff.read_database` |
| PostgreSQL / MySQL / SQLite / DuckDB / SQL Server / Snowflake | Database Reader | `ff.read_database` |
| S3 / ADLS / GCS (CSV, Parquet, JSON, Delta, Iceberg) | Cloud Storage Reader | `ff.scan_parquet_from_cloud_storage`, … |
| A Kafka / Redpanda topic | Kafka Source | `ff.read_kafka` |
| A REST endpoint | REST API Reader | `ff.read_api` |
Expand Down
48 changes: 48 additions & 0 deletions docs/users/python-api/reference/reading-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -397,6 +397,54 @@ The tested SQL Server example reads a table and a query through a stored connect
--8<-- "docs/examples/integrations/database_read_mssql.py:example"
```

### Snowflake

Snowflake connections use `database_type="snowflake"` with no host or port — the account
identifier (plus an optional warehouse and role) goes in `extra_params`:

```python
ff.create_database_connection(
connection_name="analytics-snowflake",
database_type="snowflake",
database="ANALYTICS",
username="user",
password="pass",
extra_params={
"account": "myorg-myaccount",
"warehouse": "COMPUTE_WH",
"role": "ANALYST",
},
)

df = ff.read_database("analytics-snowflake", schema_name="PUBLIC", table_name="EVENTS")
```

For key-pair (JWT) authentication — Snowflake's recommended method for programmatic
access — pass `auth_method="key_pair"` with the private key PEM *text* (never a file
path; read the file yourself). Add `private_key_passphrase` when the PEM is encrypted:

```python
ff.create_database_connection(
connection_name="analytics-snowflake-kp",
database_type="snowflake",
database="ANALYTICS",
username="svc_user",
auth_method="key_pair",
private_key=open("rsa_key.p8").read(),
extra_params={"account": "myorg-myaccount", "warehouse": "COMPUTE_WH"},
)
```

The key is stored as an encrypted secret, exactly like a password.

Semi-structured columns (`VARIANT`, `OBJECT`, `ARRAY`) are read as JSON text. The tested
Snowflake example reads a table and a query through a stored connection (it runs only when
Snowflake test credentials are configured):

```python
--8<-- "docs/examples/integrations/database_read_snowflake.py:example"
```

## Connection Management

Set up cloud and database connections once, then reference them by name. See [Cloud Connection Management](cloud-connections.md).
Expand Down
6 changes: 6 additions & 0 deletions docs/users/python-api/reference/writing-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,12 @@ Returns a new child `FlowFrame`.
natively rather than text-encoded. A DuckDB file accepts one writer at a time — close
other tools using the file while a flow writes to it.

!!! note "Snowflake"
Connections with `database_type="snowflake"` write through the native Snowflake
connector with the same `if_exists` modes. The table is created from the frame's
schema; column names are written case-exact, and nested columns are stored as JSON
text.

## Write Modes

### Overwrite vs Append
Expand Down
62 changes: 60 additions & 2 deletions docs/users/visual-editor/connections.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ and Cloud Storage Writer nodes without re-entering credentials each time.
| **SQLite** | `sqlite` |
| **DuckDB** | `duckdb` |
| **SQL Server** | `mssql` |
| **Snowflake** | `snowflake` |

!!! note "File-based connections (SQLite, DuckDB)"
SQLite and DuckDB connect to a local database **file path** (e.g. `/path/to/database.db`
Expand All @@ -40,6 +41,63 @@ and Cloud Storage Writer nodes without re-entering credentials each time.
`INTERVAL` columns are read as text — calendar intervals (months) have no fixed length,
so there is no matching Polars type.

!!! note "Snowflake connections"
Snowflake has no host or port: the form asks for the **Account** identifier
(e.g. `myorg-myaccount`) plus an optional **Warehouse** and **Role**, together with the
usual username, password, and database. Connections always use TLS, so there is no SSL
toggle. Semi-structured columns (`VARIANT`, `OBJECT`, `ARRAY`) are read as JSON text.

Snowflake also supports **key-pair (JWT) authentication** — Snowflake's recommended
method for programmatic access now that password-only logins are being phased out.
Pick *Key pair (JWT)* in the **Authentication Method** selector, then paste the
private key PEM text into the key field (plus its passphrase when the key is
encrypted). The key is stored as an encrypted secret, exactly like a password, and
is never written back to the form when editing — leave the field blank to keep the
existing key.

!!! note "Snowflake single sign-on (OAuth)"
Snowflake connections can also authenticate through your identity provider: pick
*Single sign-on (OAuth)* in the **Authentication Method** selector. You log in through
the browser **once**; Flowfile stores only the resulting refresh token (encrypted) and
silently exchanges it for short-lived access tokens whenever the connection is used —
including **scheduled runs**, which need no browser. When the refresh token expires or
is revoked (identity-provider policy, typically up to 90 days), runs fail with a
*"Reconnect your connection"* error and the connection form offers **Re-authenticate**.

Two flavors are supported through the same form:

- **Snowflake OAuth** (the default): a Snowflake admin creates a security integration
and hands you its client id/secret; the authorize/token endpoints are derived from
the account, so leave the endpoint fields blank.

```sql
CREATE SECURITY INTEGRATION flowfile_oauth
TYPE = OAUTH
ENABLED = TRUE
OAUTH_CLIENT = CUSTOM
OAUTH_CLIENT_TYPE = 'CONFIDENTIAL'
OAUTH_REDIRECT_URI = 'http://localhost:63578/db_connection_lib/oauth/callback'
OAUTH_ISSUE_REFRESH_TOKENS = TRUE
OAUTH_REFRESH_TOKEN_VALIDITY = 7776000; -- 90 days (the maximum)

-- client id / secret for the connection form:
SELECT SYSTEM$SHOW_OAUTH_CLIENT_SECRETS('FLOWFILE_OAUTH');
```

- **External OAuth** (Okta, Entra ID, PingFederate): create an OAuth app at your IdP
with the same redirect URI, configure Snowflake to trust it
(`CREATE SECURITY INTEGRATION ... TYPE = EXTERNAL_OAUTH`), and fill in the
**Authorize Endpoint** and **Token Endpoint** fields with the IdP's URLs.

The redirect URI defaults to
`http://localhost:63578/db_connection_lib/oauth/callback` — register exactly that URL
with the security integration / IdP app (override it in the form if your Flowfile
server runs elsewhere).

**Sharing note:** a group-shared OAuth connection always runs as the **owner's**
Snowflake identity — exactly like a shared password connection, and like Power BI's
dataset-owner refresh model. Share it only with people who may act as that identity.

### Creating a Database Connection

1. Open the **Connections** page from the left sidebar and select the **Database** tab
Expand All @@ -49,8 +107,8 @@ and Cloud Storage Writer nodes without re-entering credentials each time.
| Field | Description | Example |
|-------|-------------|---------|
| **Connection Name** | Unique identifier for this connection | `prod_postgres` |
| **Database Type** | PostgreSQL, MySQL, SQLite, DuckDB, or SQL Server | `postgresql` |
| **Host** | Database server hostname | `db.example.com` |
| **Database Type** | PostgreSQL, MySQL, SQLite, DuckDB, SQL Server, or Snowflake | `postgresql` |
| **Host** | Database server hostname (Snowflake asks for an account/warehouse/role instead) | `db.example.com` |
| **Port** | Database port | `5432` |
| **Database** | Database name | `analytics` |
| **Username** | Database user | `readonly_user` |
Expand Down
2 changes: 1 addition & 1 deletion docs/what-is-flowfile-technical.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Flowfile bundles a visual flow editor, a data catalog, a scheduler, and a Polars

**Python environments.** [Kernels](users/visual-editor/kernels.md) are Docker containers with CPU/memory limits, optional GPU passthrough, and a pinned package set. Everyone who runs the flow or opens the [notebook](users/visual-editor/catalog/notebooks.md) executes against the same environment. User code accesses data through `flowfile_ctx`; raw credentials are not exposed to it.

**Source connections.** [Named connections](users/data-elsewhere.md) cover PostgreSQL/MySQL/SQLite/DuckDB/SQL Server, S3/ADLS/GCS (CSV, Parquet, JSON, Delta, Iceberg), Kafka, REST endpoints, and GA4. The database reader accepts a query, so filtering and pre-aggregation can run at the source. [Kafka consumption](users/connect/kafka.md) is incremental by consumer group: offsets commit only after a successful run, so a scheduled flow reads exactly the messages that arrived since its last successful run.
**Source connections.** [Named connections](users/data-elsewhere.md) cover PostgreSQL/MySQL/SQLite/DuckDB/SQL Server/Snowflake, S3/ADLS/GCS (CSV, Parquet, JSON, Delta, Iceberg), Kafka, REST endpoints, and GA4. The database reader accepts a query, so filtering and pre-aggregation can run at the source. [Kafka consumption](users/connect/kafka.md) is incremental by consumer group: offsets commit only after a successful run, so a scheduled flow reads exactly the messages that arrived since its last successful run.

**Data freshness.** [Cron schedules and table triggers](users/visual-editor/catalog/schedules.md) are part of the catalog; the scheduler is embedded in the core service (a standalone mode exists). A table update triggers the flows watching it; set-triggers fire when all listed tables have updated. Dependent-pipeline execution is derived from trigger edges — there is no separate DAG definition to maintain.

Expand Down
2 changes: 1 addition & 1 deletion flowfile_core/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Central FastAPI backend and DAG execution engine for Flowfile: manages flows as

## Layout
- `flowfile_core/main.py` — FastAPI app, lifespan (scheduler/kernel/local-model shutdown), CORS (Tauri origin regex + explicit dev/Docker origins), all router mounts, `--run-flow` CLI.
- `flowfile_core/routes/` — REST routers: `routes.py` (editor/transform, JWT-gated), `flow_api.py` (`data_router` API-key data + `management_router` JWT), `auth.py`, `secrets.py`, `catalog.py`, `cloud_connections.py`, `storage_browser.py` (`GET /storage_browser/cloud` — object-storage browsing for the file-browser UI; connection resolved for the *calling* user via `get_cloud_connection_schema` with secrets still owner-encrypted, ambient credentials refused in docker mode, typed `error_code` payloads and never 401), `ga_connections.py`, `kafka.py`, `file_manager.py`, `api_consumers.py`, `user_defined_components.py` (all JWT-gated; save/preview/dry-run/rescan), `custom_node_mounts.py`, `community_nodes.py` (browse/install/publish + `publish-pr` at `/community_nodes`; JWT, install/uninstall additionally `require_admin`), `community_github.py` (GitHub device-flow/PAT token lifecycle at `/community_nodes/github`; JWT, per-user token in `app_settings` secrets), `logs.py`, `public.py`. (More routers live under `ai/`, `kernel/`, `artifacts/`, `ml/`.)
- `flowfile_core/routes/` — REST routers: `routes.py` (editor/transform, JWT-gated), `flow_api.py` (`data_router` API-key data + `management_router` JWT), `auth.py`, `secrets.py`, `catalog.py`, `cloud_connections.py`, `storage_browser.py` (`GET /storage_browser/cloud` — object-storage browsing for the file-browser UI; connection resolved for the *calling* user via `get_cloud_connection_schema` with secrets still owner-encrypted, ambient credentials refused in docker mode, typed `error_code` payloads and never 401), `ga_connections.py`, `db_oauth.py` (`GET /db_connection_lib/oauth/{start,callback}` — Snowflake SSO sign-in for `auth_method="oauth"` database connections, GA-style signed-state flow; callback unauthenticated by design, trust comes from the state JWT; token custody in `flowfile/database_connection_manager/db_oauth.py`, whose `ReconnectRequiredError` maps to 422 `RECONNECT_REQUIRED` via a `main.py` exception handler — never 401), `kafka.py`, `file_manager.py`, `api_consumers.py`, `user_defined_components.py` (all JWT-gated; save/preview/dry-run/rescan), `custom_node_mounts.py`, `community_nodes.py` (browse/install/publish + `publish-pr` at `/community_nodes`; JWT, install/uninstall additionally `require_admin`), `community_github.py` (GitHub device-flow/PAT token lifecycle at `/community_nodes/github`; JWT, per-user token in `app_settings` secrets), `logs.py`, `public.py`. (More routers live under `ai/`, `kernel/`, `artifacts/`, `ml/`.)
- `flowfile_core/flowfile/flow_graph.py` — DAG execution engine (`FlowGraph`, node add/run, worker offload). `flowfile/handler.py` — `FlowfileHandler` in-memory flow registry.
- `flowfile_core/flowfile/settings_validation.py` — conservative static check that node settings only reference existing input columns (per-node-type extractor registry + `validate_flow_settings`); served by `GET /flow/settings_validation`, gated per flow by `FlowSettings.validate_settings`. Flowfile formulas (formula node, advanced filter) are covered by walking polars_expr_transformer's parse tree for `pl.col` references. A second phase (`@_expression_probe` registry) asks whether the expression can run at all via `_extensions/real_time_interface.check_expression` — the runtime parser (`simple_function_to_expr`) applied to an **empty LazyFrame** built from the predicted schema (`filter()` for the advanced filter, mirroring `do_filter`, which is what enforces Boolean-ness), so a type error like `[n] + "a"` surfaces on the canvas without touching data; `${param}` refs are resolved first and skipped when undefined. **The registry's entry criterion is "columns whose absence makes the node *fail*", not "columns the node references"** — `select`, `dynamic_rename`, and the join-family select lists skip missing columns and keep running, so they deliberately have no extractor (`run_flow` too: its parameter columns arrive on keyed handle `input-0`, which `main_inputs` cannot address). `tests/flowfile/test_settings_validation.py::test_warning_matches_runtime_behaviour` pins this by running each node type with a renamed-away column and asserting warn ⇔ failure, with a control run proving attribution — add a case there before adding an extractor. Warns only when the input schema is confidently known: no extractor (custom nodes, raw polars/SQL/python code), blocked or failed prediction, and empty schemas all stay silent — never add an extractor or probe that can false-positive.
- `flowfile_core/flowfile/flow_data_engine/flow_data_engine.py` — per-node Polars compute wrapper (lazy frames, previews; `join/`, `fuzzy_matching/`, `subprocess_operations/` subdirs).
Expand Down
Loading
Loading