Ingest from Kafka/Redpanda as a canvas node. Read and write S3, Azure Data Lake, and GCS. - Connect PostgreSQL, MySQL, SQL Server, SQLite, and DuckDB. + Connect PostgreSQL, MySQL, SQL Server, Snowflake, SQLite, and DuckDB.
→ diff --git a/docs/users/connect/index.md b/docs/users/connect/index.md index 10954dbcc..1797413fc 100644 --- a/docs/users/connect/index.md +++ b/docs/users/connect/index.md @@ -10,7 +10,7 @@ Each connector below is read-only, write-only, or both. "Where configured" names | Connector | Reads | Writes | Where configured | Setup | |---|---|---|---|---| -| PostgreSQL / MySQL / SQLite / DuckDB / SQL Server | yes | yes | Connections → Database | [Databases](../visual-editor/tutorials/database-connectivity.md) | +| PostgreSQL / MySQL / SQLite / DuckDB / SQL Server / Snowflake | yes | yes | Connections → Database | [Databases](../visual-editor/tutorials/database-connectivity.md) | | Cloud storage (S3 / ADLS / GCS) | yes | yes | Connections → Cloud Storage | [Cloud storage](../visual-editor/tutorials/cloud-connections.md) | | Kafka / Redpanda | yes | no | Connections → Kafka | [Kafka](kafka.md) | | REST API | yes | no | REST API Reader node (inline) | [REST APIs](apis.md#rest-apis) | @@ -23,7 +23,7 @@ Each connector below is read-only, write-only, or both. "Where configured" names ## Databases -Typed connections to **PostgreSQL**, **MySQL**, **SQLite**, **DuckDB**, and **SQL Server**. Both directions are supported: the Database Reader node (`ff.read_database`) runs a query or reads a whole table, and the Database Writer node (`ff.write_database`) writes a frame back. Credentials are stored encrypted and referenced by name. +Typed connections to **PostgreSQL**, **MySQL**, **SQLite**, **DuckDB**, **SQL Server**, and **Snowflake**. Both directions are supported: the Database Reader node (`ff.read_database`) runs a query or reads a whole table, and the Database Writer node (`ff.write_database`) writes a frame back. Credentials are stored encrypted and referenced by name. See [Databases](../visual-editor/tutorials/database-connectivity.md) for the connection form and worked reader/writer examples. diff --git a/docs/users/data-elsewhere.md b/docs/users/data-elsewhere.md index 81f736ca8..ebb0280ab 100644 --- a/docs/users/data-elsewhere.md +++ b/docs/users/data-elsewhere.md @@ -10,7 +10,7 @@ None of this route requires code: every source below is a node in the visual edi  -Supported out of the box: databases (PostgreSQL, MySQL, SQLite, DuckDB, SQL Server), cloud storage (S3, Azure Data Lake, Google Cloud Storage — with auth methods from stored keys to ambient `aws-cli`/environment credentials), Kafka/Redpanda brokers, and Google Analytics 4 properties. [Connections](visual-editor/connections.md) covers every form field. +Supported out of the box: databases (PostgreSQL, MySQL, SQLite, DuckDB, SQL Server, Snowflake), cloud storage (S3, Azure Data Lake, Google Cloud Storage — with auth methods from stored keys to ambient `aws-cli`/environment credentials), Kafka/Redpanda brokers, and Google Analytics 4 properties. [Connections](visual-editor/connections.md) covers every form field. ## 2. Read from the source, not from copies @@ -18,7 +18,7 @@ Each source is a drag-and-drop node on the canvas — with a matching `ff.*` cal | Your data is in… | On the canvas | In Python | |---|---|---| -| PostgreSQL / MySQL / SQLite / DuckDB / SQL Server | Database Reader | `ff.read_database` | +| PostgreSQL / MySQL / SQLite / DuckDB / SQL Server / Snowflake | Database Reader | `ff.read_database` | | S3 / ADLS / GCS (CSV, Parquet, JSON, Delta, Iceberg) | Cloud Storage Reader | `ff.scan_parquet_from_cloud_storage`, … | | A Kafka / Redpanda topic | Kafka Source | `ff.read_kafka` | | A REST endpoint | REST API Reader | `ff.read_api` | diff --git a/docs/users/python-api/reference/reading-data.md b/docs/users/python-api/reference/reading-data.md index e65539f19..f366fb1a2 100644 --- a/docs/users/python-api/reference/reading-data.md +++ b/docs/users/python-api/reference/reading-data.md @@ -397,6 +397,54 @@ The tested SQL Server example reads a table and a query through a stored connect --8<-- "docs/examples/integrations/database_read_mssql.py:example" ``` +### Snowflake + +Snowflake connections use `database_type="snowflake"` with no host or port — the account +identifier (plus an optional warehouse and role) goes in `extra_params`: + +```python +ff.create_database_connection( + connection_name="analytics-snowflake", + database_type="snowflake", + database="ANALYTICS", + username="user", + password="pass", + extra_params={ + "account": "myorg-myaccount", + "warehouse": "COMPUTE_WH", + "role": "ANALYST", + }, +) + +df = ff.read_database("analytics-snowflake", schema_name="PUBLIC", table_name="EVENTS") +``` + +For key-pair (JWT) authentication — Snowflake's recommended method for programmatic +access — pass `auth_method="key_pair"` with the private key PEM *text* (never a file +path; read the file yourself). Add `private_key_passphrase` when the PEM is encrypted: + +```python +ff.create_database_connection( + connection_name="analytics-snowflake-kp", + database_type="snowflake", + database="ANALYTICS", + username="svc_user", + auth_method="key_pair", + private_key=open("rsa_key.p8").read(), + extra_params={"account": "myorg-myaccount", "warehouse": "COMPUTE_WH"}, +) +``` + +The key is stored as an encrypted secret, exactly like a password. + +Semi-structured columns (`VARIANT`, `OBJECT`, `ARRAY`) are read as JSON text. The tested +Snowflake example reads a table and a query through a stored connection (it runs only when +Snowflake test credentials are configured): + +```python +--8<-- "docs/examples/integrations/database_read_snowflake.py:example" +``` + ## Connection Management Set up cloud and database connections once, then reference them by name. See [Cloud Connection Management](cloud-connections.md). diff --git a/docs/users/python-api/reference/writing-data.md b/docs/users/python-api/reference/writing-data.md index bbb39e144..41dd9d3d3 100644 --- a/docs/users/python-api/reference/writing-data.md +++ b/docs/users/python-api/reference/writing-data.md @@ -286,6 +286,12 @@ Returns a new child `FlowFrame`. natively rather than text-encoded. A DuckDB file accepts one writer at a time — close other tools using the file while a flow writes to it. +!!! note "Snowflake" + Connections with `database_type="snowflake"` write through the native Snowflake + connector with the same `if_exists` modes. The table is created from the frame's + schema; column names are written case-exact, and nested columns are stored as JSON + text. + ## Write Modes ### Overwrite vs Append diff --git a/docs/users/visual-editor/connections.md b/docs/users/visual-editor/connections.md index b1f0515ab..36d31e605 100644 --- a/docs/users/visual-editor/connections.md +++ b/docs/users/visual-editor/connections.md @@ -26,6 +26,7 @@ and Cloud Storage Writer nodes without re-entering credentials each time. | **SQLite** | `sqlite` | | **DuckDB** | `duckdb` | | **SQL Server** | `mssql` | +| **Snowflake** | `snowflake` | !!! note "File-based connections (SQLite, DuckDB)" SQLite and DuckDB connect to a local database **file path** (e.g. `/path/to/database.db` @@ -40,6 +41,63 @@ and Cloud Storage Writer nodes without re-entering credentials each time. `INTERVAL` columns are read as text — calendar intervals (months) have no fixed length, so there is no matching Polars type. +!!! note "Snowflake connections" + Snowflake has no host or port: the form asks for the **Account** identifier + (e.g. `myorg-myaccount`) plus an optional **Warehouse** and **Role**, together with the + usual username, password, and database. Connections always use TLS, so there is no SSL + toggle. Semi-structured columns (`VARIANT`, `OBJECT`, `ARRAY`) are read as JSON text. + + Snowflake also supports **key-pair (JWT) authentication** — Snowflake's recommended + method for programmatic access now that password-only logins are being phased out. + Pick *Key pair (JWT)* in the **Authentication Method** selector, then paste the + private key PEM text into the key field (plus its passphrase when the key is + encrypted). The key is stored as an encrypted secret, exactly like a password, and + is never written back to the form when editing — leave the field blank to keep the + existing key. + +!!! note "Snowflake single sign-on (OAuth)" + Snowflake connections can also authenticate through your identity provider: pick + *Single sign-on (OAuth)* in the **Authentication Method** selector. You log in through + the browser **once**; Flowfile stores only the resulting refresh token (encrypted) and + silently exchanges it for short-lived access tokens whenever the connection is used — + including **scheduled runs**, which need no browser. When the refresh token expires or + is revoked (identity-provider policy, typically up to 90 days), runs fail with a + *"Reconnect your connection"* error and the connection form offers **Re-authenticate**. + + Two flavors are supported through the same form: + + - **Snowflake OAuth** (the default): a Snowflake admin creates a security integration + and hands you its client id/secret; the authorize/token endpoints are derived from + the account, so leave the endpoint fields blank. + + ```sql + CREATE SECURITY INTEGRATION flowfile_oauth + TYPE = OAUTH + ENABLED = TRUE + OAUTH_CLIENT = CUSTOM + OAUTH_CLIENT_TYPE = 'CONFIDENTIAL' + OAUTH_REDIRECT_URI = 'http://localhost:63578/db_connection_lib/oauth/callback' + OAUTH_ISSUE_REFRESH_TOKENS = TRUE + OAUTH_REFRESH_TOKEN_VALIDITY = 7776000; -- 90 days (the maximum) + + -- client id / secret for the connection form: + SELECT SYSTEM$SHOW_OAUTH_CLIENT_SECRETS('FLOWFILE_OAUTH'); + ``` + + - **External OAuth** (Okta, Entra ID, PingFederate): create an OAuth app at your IdP + with the same redirect URI, configure Snowflake to trust it + (`CREATE SECURITY INTEGRATION ... TYPE = EXTERNAL_OAUTH`), and fill in the + **Authorize Endpoint** and **Token Endpoint** fields with the IdP's URLs. + + The redirect URI defaults to + `http://localhost:63578/db_connection_lib/oauth/callback` — register exactly that URL + with the security integration / IdP app (override it in the form if your Flowfile + server runs elsewhere). + + **Sharing note:** a group-shared OAuth connection always runs as the **owner's** + Snowflake identity — exactly like a shared password connection, and like Power BI's + dataset-owner refresh model. Share it only with people who may act as that identity. + ### Creating a Database Connection 1. Open the **Connections** page from the left sidebar and select the **Database** tab @@ -49,8 +107,8 @@ and Cloud Storage Writer nodes without re-entering credentials each time. | Field | Description | Example | |-------|-------------|---------| | **Connection Name** | Unique identifier for this connection | `prod_postgres` | -| **Database Type** | PostgreSQL, MySQL, SQLite, DuckDB, or SQL Server | `postgresql` | -| **Host** | Database server hostname | `db.example.com` | +| **Database Type** | PostgreSQL, MySQL, SQLite, DuckDB, SQL Server, or Snowflake | `postgresql` | +| **Host** | Database server hostname (Snowflake asks for an account/warehouse/role instead) | `db.example.com` | | **Port** | Database port | `5432` | | **Database** | Database name | `analytics` | | **Username** | Database user | `readonly_user` | diff --git a/docs/what-is-flowfile-technical.md b/docs/what-is-flowfile-technical.md index ffaa58248..b1cee8ca0 100644 --- a/docs/what-is-flowfile-technical.md +++ b/docs/what-is-flowfile-technical.md @@ -12,7 +12,7 @@ Flowfile bundles a visual flow editor, a data catalog, a scheduler, and a Polars **Python environments.** [Kernels](users/visual-editor/kernels.md) are Docker containers with CPU/memory limits, optional GPU passthrough, and a pinned package set. Everyone who runs the flow or opens the [notebook](users/visual-editor/catalog/notebooks.md) executes against the same environment. User code accesses data through `flowfile_ctx`; raw credentials are not exposed to it. -**Source connections.** [Named connections](users/data-elsewhere.md) cover PostgreSQL/MySQL/SQLite/DuckDB/SQL Server, S3/ADLS/GCS (CSV, Parquet, JSON, Delta, Iceberg), Kafka, REST endpoints, and GA4. The database reader accepts a query, so filtering and pre-aggregation can run at the source. [Kafka consumption](users/connect/kafka.md) is incremental by consumer group: offsets commit only after a successful run, so a scheduled flow reads exactly the messages that arrived since its last successful run. +**Source connections.** [Named connections](users/data-elsewhere.md) cover PostgreSQL/MySQL/SQLite/DuckDB/SQL Server/Snowflake, S3/ADLS/GCS (CSV, Parquet, JSON, Delta, Iceberg), Kafka, REST endpoints, and GA4. The database reader accepts a query, so filtering and pre-aggregation can run at the source. [Kafka consumption](users/connect/kafka.md) is incremental by consumer group: offsets commit only after a successful run, so a scheduled flow reads exactly the messages that arrived since its last successful run. **Data freshness.** [Cron schedules and table triggers](users/visual-editor/catalog/schedules.md) are part of the catalog; the scheduler is embedded in the core service (a standalone mode exists). A table update triggers the flows watching it; set-triggers fire when all listed tables have updated. Dependent-pipeline execution is derived from trigger edges — there is no separate DAG definition to maintain. diff --git a/flowfile_core/CLAUDE.md b/flowfile_core/CLAUDE.md index 441407006..949953285 100644 --- a/flowfile_core/CLAUDE.md +++ b/flowfile_core/CLAUDE.md @@ -9,7 +9,7 @@ Central FastAPI backend and DAG execution engine for Flowfile: manages flows as ## Layout - `flowfile_core/main.py` — FastAPI app, lifespan (scheduler/kernel/local-model shutdown), CORS (Tauri origin regex + explicit dev/Docker origins), all router mounts, `--run-flow` CLI. -- `flowfile_core/routes/` — REST routers: `routes.py` (editor/transform, JWT-gated), `flow_api.py` (`data_router` API-key data + `management_router` JWT), `auth.py`, `secrets.py`, `catalog.py`, `cloud_connections.py`, `storage_browser.py` (`GET /storage_browser/cloud` — object-storage browsing for the file-browser UI; connection resolved for the *calling* user via `get_cloud_connection_schema` with secrets still owner-encrypted, ambient credentials refused in docker mode, typed `error_code` payloads and never 401), `ga_connections.py`, `kafka.py`, `file_manager.py`, `api_consumers.py`, `user_defined_components.py` (all JWT-gated; save/preview/dry-run/rescan), `custom_node_mounts.py`, `community_nodes.py` (browse/install/publish + `publish-pr` at `/community_nodes`; JWT, install/uninstall additionally `require_admin`), `community_github.py` (GitHub device-flow/PAT token lifecycle at `/community_nodes/github`; JWT, per-user token in `app_settings` secrets), `logs.py`, `public.py`. (More routers live under `ai/`, `kernel/`, `artifacts/`, `ml/`.) +- `flowfile_core/routes/` — REST routers: `routes.py` (editor/transform, JWT-gated), `flow_api.py` (`data_router` API-key data + `management_router` JWT), `auth.py`, `secrets.py`, `catalog.py`, `cloud_connections.py`, `storage_browser.py` (`GET /storage_browser/cloud` — object-storage browsing for the file-browser UI; connection resolved for the *calling* user via `get_cloud_connection_schema` with secrets still owner-encrypted, ambient credentials refused in docker mode, typed `error_code` payloads and never 401), `ga_connections.py`, `db_oauth.py` (`GET /db_connection_lib/oauth/{start,callback}` — Snowflake SSO sign-in for `auth_method="oauth"` database connections, GA-style signed-state flow; callback unauthenticated by design, trust comes from the state JWT; token custody in `flowfile/database_connection_manager/db_oauth.py`, whose `ReconnectRequiredError` maps to 422 `RECONNECT_REQUIRED` via a `main.py` exception handler — never 401), `kafka.py`, `file_manager.py`, `api_consumers.py`, `user_defined_components.py` (all JWT-gated; save/preview/dry-run/rescan), `custom_node_mounts.py`, `community_nodes.py` (browse/install/publish + `publish-pr` at `/community_nodes`; JWT, install/uninstall additionally `require_admin`), `community_github.py` (GitHub device-flow/PAT token lifecycle at `/community_nodes/github`; JWT, per-user token in `app_settings` secrets), `logs.py`, `public.py`. (More routers live under `ai/`, `kernel/`, `artifacts/`, `ml/`.) - `flowfile_core/flowfile/flow_graph.py` — DAG execution engine (`FlowGraph`, node add/run, worker offload). `flowfile/handler.py` — `FlowfileHandler` in-memory flow registry. - `flowfile_core/flowfile/settings_validation.py` — conservative static check that node settings only reference existing input columns (per-node-type extractor registry + `validate_flow_settings`); served by `GET /flow/settings_validation`, gated per flow by `FlowSettings.validate_settings`. Flowfile formulas (formula node, advanced filter) are covered by walking polars_expr_transformer's parse tree for `pl.col` references. A second phase (`@_expression_probe` registry) asks whether the expression can run at all via `_extensions/real_time_interface.check_expression` — the runtime parser (`simple_function_to_expr`) applied to an **empty LazyFrame** built from the predicted schema (`filter()` for the advanced filter, mirroring `do_filter`, which is what enforces Boolean-ness), so a type error like `[n] + "a"` surfaces on the canvas without touching data; `${param}` refs are resolved first and skipped when undefined. **The registry's entry criterion is "columns whose absence makes the node *fail*", not "columns the node references"** — `select`, `dynamic_rename`, and the join-family select lists skip missing columns and keep running, so they deliberately have no extractor (`run_flow` too: its parameter columns arrive on keyed handle `input-0`, which `main_inputs` cannot address). `tests/flowfile/test_settings_validation.py::test_warning_matches_runtime_behaviour` pins this by running each node type with a renamed-away column and asserting warn ⇔ failure, with a control run proving attribution — add a case there before adding an extractor. Warns only when the input schema is confidently known: no extractor (custom nodes, raw polars/SQL/python code), blocked or failed prediction, and empty schemas all stay silent — never add an extractor or probe that can false-positive. - `flowfile_core/flowfile/flow_data_engine/flow_data_engine.py` — per-node Polars compute wrapper (lazy frames, previews; `join/`, `fuzzy_matching/`, `subprocess_operations/` subdirs). diff --git a/flowfile_core/flowfile_core/alembic/versions/030_database_connection_extra_params.py b/flowfile_core/flowfile_core/alembic/versions/030_database_connection_extra_params.py new file mode 100644 index 000000000..abb69c0a2 --- /dev/null +++ b/flowfile_core/flowfile_core/alembic/versions/030_database_connection_extra_params.py @@ -0,0 +1,37 @@ +"""Add extra_params to database_connections. + +JSON dict of dialect-specific connection parameters that don't fit the +host/port shape (e.g. Snowflake's account/warehouse/role). Keys that could +override credentials are rejected at the API boundary and dropped again at +URI-build time (shared.db_dialects.base.is_blocked_extra_param). NULL means +the connection has no dialect-specific parameters. + +Revision ID: 030 +Revises: 029 +Create Date: 2026-08-05 +""" + +from collections.abc import Sequence + +import sqlalchemy as sa +from alembic import op +from sqlalchemy import inspect + +revision: str = "030" +down_revision: str | None = "029" +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None + + +def _has_column(table: str, column: str) -> bool: + return column in {c["name"] for c in inspect(op.get_bind()).get_columns(table)} + + +def upgrade() -> None: + if not _has_column("database_connections", "extra_params"): + op.add_column("database_connections", sa.Column("extra_params", sa.Text, nullable=True)) + + +def downgrade() -> None: + if _has_column("database_connections", "extra_params"): + op.drop_column("database_connections", "extra_params") diff --git a/flowfile_core/flowfile_core/alembic/versions/031_database_connection_key_pair.py b/flowfile_core/flowfile_core/alembic/versions/031_database_connection_key_pair.py new file mode 100644 index 000000000..20130368d --- /dev/null +++ b/flowfile_core/flowfile_core/alembic/versions/031_database_connection_key_pair.py @@ -0,0 +1,45 @@ +"""Add key-pair auth columns to database_connections. + +auth_method selects the connection's authentication method (NULL means +password, the historical default). private_key_id / private_key_passphrase_id +reference encrypted secrets holding the PEM text and its optional passphrase. +The id columns are plain integers here — the ForeignKey lives in the ORM only, +because SQLite cannot drop a column that carries a FK constraint on downgrade. + +Revision ID: 031 +Revises: 030 +Create Date: 2026-08-05 +""" + +from collections.abc import Sequence + +import sqlalchemy as sa +from alembic import op +from sqlalchemy import inspect + +revision: str = "031" +down_revision: str | None = "030" +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None + +_COLUMNS = ( + ("auth_method", sa.String), + ("private_key_id", sa.Integer), + ("private_key_passphrase_id", sa.Integer), +) + + +def _has_column(table: str, column: str) -> bool: + return column in {c["name"] for c in inspect(op.get_bind()).get_columns(table)} + + +def upgrade() -> None: + for name, column_type in _COLUMNS: + if not _has_column("database_connections", name): + op.add_column("database_connections", sa.Column(name, column_type, nullable=True)) + + +def downgrade() -> None: + for name, _ in _COLUMNS: + if _has_column("database_connections", name): + op.drop_column("database_connections", name) diff --git a/flowfile_core/flowfile_core/alembic/versions/032_database_connection_oauth.py b/flowfile_core/flowfile_core/alembic/versions/032_database_connection_oauth.py new file mode 100644 index 000000000..48ef1b8b2 --- /dev/null +++ b/flowfile_core/flowfile_core/alembic/versions/032_database_connection_oauth.py @@ -0,0 +1,50 @@ +"""Add OAuth (SSO) columns to database_connections. + +auth_method="oauth" connections store a per-connection OAuth client config +(client id, optional authorize/token endpoint overrides for External OAuth, +optional redirect-uri override) plus two secret FKs: the encrypted client +secret and the encrypted refresh token minted by the interactive sign-in. +The id columns are plain integers here — the ForeignKey lives in the ORM +only, because SQLite cannot drop a column that carries a FK constraint on +downgrade (the 031 precedent). + +Revision ID: 032 +Revises: 031 +Create Date: 2026-08-05 +""" + +from collections.abc import Sequence + +import sqlalchemy as sa +from alembic import op +from sqlalchemy import inspect + +revision: str = "032" +down_revision: str | None = "031" +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None + +_COLUMNS = ( + ("oauth_client_id", sa.String), + ("oauth_authorize_endpoint", sa.String), + ("oauth_token_endpoint", sa.String), + ("oauth_redirect_uri", sa.String), + ("oauth_client_secret_id", sa.Integer), + ("oauth_refresh_token_id", sa.Integer), +) + + +def _has_column(table: str, column: str) -> bool: + return column in {c["name"] for c in inspect(op.get_bind()).get_columns(table)} + + +def upgrade() -> None: + for name, column_type in _COLUMNS: + if not _has_column("database_connections", name): + op.add_column("database_connections", sa.Column(name, column_type, nullable=True)) + + +def downgrade() -> None: + for name, _ in _COLUMNS: + if _has_column("database_connections", name): + op.drop_column("database_connections", name) diff --git a/flowfile_core/flowfile_core/database/models.py b/flowfile_core/flowfile_core/database/models.py index 57dd8fd26..4b3dfa7bc 100644 --- a/flowfile_core/flowfile_core/database/models.py +++ b/flowfile_core/flowfile_core/database/models.py @@ -58,7 +58,18 @@ class DatabaseConnection(Base): port = Column(Integer) database = Column(String, default=None) ssl_enabled = Column(Boolean, default=False) + extra_params = Column(Text, nullable=True) # JSON dict of dialect-specific params (e.g. snowflake account) + auth_method = Column(String, nullable=True) # NULL == password (the historical default) password_id = Column(Integer, ForeignKey("secrets.id")) + private_key_id = Column(Integer, ForeignKey("secrets.id"), nullable=True) + private_key_passphrase_id = Column(Integer, ForeignKey("secrets.id"), nullable=True) + # OAuth (SSO) auth: per-connection client config; empty endpoints derive from the account. + oauth_client_id = Column(String, nullable=True) + oauth_authorize_endpoint = Column(String, nullable=True) + oauth_token_endpoint = Column(String, nullable=True) + oauth_redirect_uri = Column(String, nullable=True) + oauth_client_secret_id = Column(Integer, ForeignKey("secrets.id"), nullable=True) + oauth_refresh_token_id = Column(Integer, ForeignKey("secrets.id"), nullable=True) user_id = Column(Integer, ForeignKey("users.id")) diff --git a/flowfile_core/flowfile_core/flowfile/database_connection_manager/db_connections.py b/flowfile_core/flowfile_core/flowfile/database_connection_manager/db_connections.py index 9d0b6182e..908fa8539 100644 --- a/flowfile_core/flowfile_core/flowfile/database_connection_manager/db_connections.py +++ b/flowfile_core/flowfile_core/flowfile/database_connection_manager/db_connections.py @@ -1,3 +1,6 @@ +import json + +from pydantic import SecretStr from sqlalchemy.orm import Session from flowfile_core.auth import sharing @@ -13,6 +16,21 @@ _OWNER_ACCESS = AccessInfo(is_owner=True, access_level="owner") +def _dump_extra_params(extra_params: dict[str, str] | None) -> str | None: + return json.dumps(extra_params, sort_keys=True) if extra_params else None + + +def parse_extra_params(raw: str | None) -> dict[str, str] | None: + """Parse the JSON extra_params column; malformed or empty values resolve to None.""" + if not raw: + return None + try: + parsed = json.loads(raw) + except (TypeError, ValueError): + return None + return parsed or None + + def _project_sync_connection(kind: str, name: str, user_id: int, deleted: bool = False) -> None: """Mirror a connection change into the active project folder (no-op when none active).""" from flowfile_core.project import project_sync @@ -33,7 +51,35 @@ def store_database_connection(db: Session, connection: FullDatabaseConnection, u f" Please use a unique connection name or delete the existing connection first." ) + if connection.auth_method == "key_pair" and connection.private_key is None: + raise ValueError("key_pair authentication requires a private key when creating a connection.") + if connection.auth_method == "oauth" and (not connection.oauth_client_id or connection.oauth_client_secret is None): + raise ValueError("oauth authentication requires an OAuth client id and client secret.") + password_id = store_secret(db, SecretInput(name=connection.connection_name, value=connection.password), user_id).id + # `is not None` (not truthiness): a project import may pass an empty placeholder + # value that must still create a linked secret row for the refill UI. + private_key_id = None + if connection.private_key is not None: + private_key_id = store_secret( + db, SecretInput(name=f"{connection.connection_name}_private_key", value=connection.private_key), user_id + ).id + private_key_passphrase_id = None + if connection.private_key_passphrase is not None: + private_key_passphrase_id = store_secret( + db, + SecretInput( + name=f"{connection.connection_name}_private_key_passphrase", value=connection.private_key_passphrase + ), + user_id, + ).id + oauth_client_secret_id = None + if connection.oauth_client_secret is not None: + oauth_client_secret_id = store_secret( + db, + SecretInput(name=f"{connection.connection_name}_oauth_client_secret", value=connection.oauth_client_secret), + user_id, + ).id db_connection = DBConnectionModel( connection_name=connection.connection_name, @@ -43,7 +89,16 @@ def store_database_connection(db: Session, connection: FullDatabaseConnection, u database_type=connection.database_type, username=connection.username, password_id=password_id, + private_key_id=private_key_id, + private_key_passphrase_id=private_key_passphrase_id, + oauth_client_id=connection.oauth_client_id, + oauth_authorize_endpoint=connection.oauth_authorize_endpoint, + oauth_token_endpoint=connection.oauth_token_endpoint, + oauth_redirect_uri=connection.oauth_redirect_uri, + oauth_client_secret_id=oauth_client_secret_id, + auth_method=connection.auth_method, ssl_enabled=connection.ssl_enabled, + extra_params=_dump_extra_params(connection.extra_params), user_id=user_id, ) @@ -73,6 +128,8 @@ def update_database_connection(db: Session, connection: FullDatabaseConnection, db_connection.database_type = connection.database_type db_connection.username = connection.username db_connection.ssl_enabled = connection.ssl_enabled + db_connection.extra_params = _dump_extra_params(connection.extra_params) + db_connection.auth_method = connection.auth_method password_value = connection.password.get_secret_value() if password_value: @@ -84,6 +141,86 @@ def update_database_connection(db: Session, connection: FullDatabaseConnection, new_secret = store_secret(db, secret_input, user_id) db_connection.password_id = new_secret.id + incoming_key = connection.private_key.get_secret_value() if connection.private_key else "" + if connection.auth_method == "key_pair": + if not incoming_key and db_connection.private_key_id is None: + raise ValueError("key_pair authentication requires a private key.") + # Rotate key material only when a non-empty value arrived (empty == keep existing, + # mirroring password). _update_cloud_secret is generic despite the name. + db_connection.private_key_id = _update_cloud_secret( + db, + db_connection.private_key_id, + incoming_key, + f"{connection.connection_name}_private_key", + user_id, + ) + db_connection.private_key_passphrase_id = _update_cloud_secret( + db, + db_connection.private_key_passphrase_id, + connection.private_key_passphrase.get_secret_value() if connection.private_key_passphrase else "", + f"{connection.connection_name}_private_key_passphrase", + user_id, + ) + else: + # Switching away from key-pair auth: detach AND delete the key secrets — and + # ignore any stray incoming key — or a rotated-away (possibly compromised) + # key would keep authenticating silently (build_uri infers key-pair from key + # presence, and the reference resolver forwards whatever the row links). + stale_ids = [ + secret_id + for secret_id in (db_connection.private_key_id, db_connection.private_key_passphrase_id) + if secret_id is not None + ] + db_connection.private_key_id = None + db_connection.private_key_passphrase_id = None + if stale_ids: + db.query(Secret).filter(Secret.id.in_(stale_ids)).delete(synchronize_session=False) + + if connection.auth_method == "oauth": + if not (connection.oauth_client_id or db_connection.oauth_client_id): + raise ValueError("oauth authentication requires an OAuth client id.") + # A changed client or token endpoint invalidates the stored refresh token + # (it was minted against the old authorization server): drop it so the + # user re-authenticates — this is also what defuses endpoint repointing. + oauth_target_changed = ( + (connection.oauth_client_id or None) != (db_connection.oauth_client_id or None) + or (connection.oauth_authorize_endpoint or None) != (db_connection.oauth_authorize_endpoint or None) + or (connection.oauth_token_endpoint or None) != (db_connection.oauth_token_endpoint or None) + ) + db_connection.oauth_client_id = connection.oauth_client_id + db_connection.oauth_authorize_endpoint = connection.oauth_authorize_endpoint + db_connection.oauth_token_endpoint = connection.oauth_token_endpoint + db_connection.oauth_redirect_uri = connection.oauth_redirect_uri + db_connection.oauth_client_secret_id = _update_cloud_secret( + db, + db_connection.oauth_client_secret_id, + connection.oauth_client_secret.get_secret_value() if connection.oauth_client_secret else "", + f"{connection.connection_name}_oauth_client_secret", + user_id, + ) + if db_connection.oauth_client_secret_id is None: + raise ValueError("oauth authentication requires an OAuth client secret.") + if oauth_target_changed and db_connection.oauth_refresh_token_id is not None: + stale_token_id = db_connection.oauth_refresh_token_id + db_connection.oauth_refresh_token_id = None + db.query(Secret).filter(Secret.id == stale_token_id).delete(synchronize_session=False) + else: + # Switching away from OAuth: drop the client config and delete both OAuth + # secrets, mirroring the key-pair branch above. + stale_oauth_ids = [ + secret_id + for secret_id in (db_connection.oauth_client_secret_id, db_connection.oauth_refresh_token_id) + if secret_id is not None + ] + db_connection.oauth_client_id = None + db_connection.oauth_authorize_endpoint = None + db_connection.oauth_token_endpoint = None + db_connection.oauth_redirect_uri = None + db_connection.oauth_client_secret_id = None + db_connection.oauth_refresh_token_id = None + if stale_oauth_ids: + db.query(Secret).filter(Secret.id.in_(stale_oauth_ids)).delete(synchronize_session=False) + db.commit() db.refresh(db_connection) _project_sync_connection("database", connection.connection_name, user_id) @@ -162,6 +299,16 @@ def get_database_connection_schema(db: Session, connection_name: str, user_id: i if not password_secret: raise Exception("Password secret not found") + def _ciphertext(secret_id: int | None) -> str | None: + if secret_id is None: + return None + secret = db.query(Secret).filter(Secret.id == secret_id).first() + return secret.encrypted_value if secret else None + + private_key = _ciphertext(db_connection.private_key_id) + private_key_passphrase = _ciphertext(db_connection.private_key_passphrase_id) + oauth_client_secret = _ciphertext(db_connection.oauth_client_secret_id) + oauth_refresh_token = _ciphertext(db_connection.oauth_refresh_token_id) return FullDatabaseConnection( connection_name=db_connection.connection_name, host=db_connection.host, @@ -171,6 +318,16 @@ def get_database_connection_schema(db: Session, connection_name: str, user_id: i username=db_connection.username, password=password_secret.encrypted_value, ssl_enabled=db_connection.ssl_enabled, + extra_params=parse_extra_params(db_connection.extra_params), + auth_method=db_connection.auth_method, + private_key=SecretStr(private_key) if private_key is not None else None, + private_key_passphrase=SecretStr(private_key_passphrase) if private_key_passphrase is not None else None, + oauth_client_id=db_connection.oauth_client_id, + oauth_authorize_endpoint=db_connection.oauth_authorize_endpoint, + oauth_token_endpoint=db_connection.oauth_token_endpoint, + oauth_redirect_uri=db_connection.oauth_redirect_uri, + oauth_client_secret=SecretStr(oauth_client_secret) if oauth_client_secret is not None else None, + oauth_refresh_token=SecretStr(oauth_refresh_token) if oauth_refresh_token is not None else None, ) return None @@ -202,12 +359,23 @@ def delete_database_connection(db: Session, connection_name: str, user_id: int) db_connection = _get_own_database_connection(db, connection_name, user_id) if db_connection: + # Collect secret ids before the delete is staged (cloud-connection ordering). + secret_ids_to_delete = [ + secret_id + for secret_id in ( + db_connection.password_id, + db_connection.private_key_id, + db_connection.private_key_passphrase_id, + db_connection.oauth_client_secret_id, + db_connection.oauth_refresh_token_id, + ) + if secret_id is not None + ] + sharing.delete_grants_for_resource(db, "database_connection", db_connection.id) db.delete(db_connection) - - password_secret = db.query(Secret).filter(Secret.id == db_connection.password_id).first() - if password_secret: - db.delete(password_secret) + if secret_ids_to_delete: + db.query(Secret).filter(Secret.id.in_(secret_ids_to_delete)).delete(synchronize_session=False) db.commit() _project_sync_connection("database", connection_name, user_id, deleted=True) @@ -227,6 +395,13 @@ def database_connection_interface_from_db_connection( port=db_connection.port, database=db_connection.database, ssl_enabled=db_connection.ssl_enabled, + extra_params=parse_extra_params(db_connection.extra_params), + auth_method=db_connection.auth_method, + oauth_client_id=db_connection.oauth_client_id, + oauth_authorize_endpoint=db_connection.oauth_authorize_endpoint, + oauth_token_endpoint=db_connection.oauth_token_endpoint, + oauth_redirect_uri=db_connection.oauth_redirect_uri, + oauth_connected=db_connection.oauth_refresh_token_id is not None, id=db_connection.id, access=access, ) diff --git a/flowfile_core/flowfile_core/flowfile/database_connection_manager/db_oauth.py b/flowfile_core/flowfile_core/flowfile/database_connection_manager/db_oauth.py new file mode 100644 index 000000000..6267a5758 --- /dev/null +++ b/flowfile_core/flowfile_core/flowfile/database_connection_manager/db_oauth.py @@ -0,0 +1,137 @@ +"""OAuth credential custody for ``auth_method="oauth"`` database connections. + +Core owns the whole token lifecycle: the interactive callback stores the +refresh token (owner-keyed ``$ffsec$`` ciphertext), and at credential +resolution time :func:`resolve_oauth_access_token` silently exchanges it for +a short-lived access token — persisting a rotated refresh token when the IdP +returns one (Okta does under some policies). The worker never refreshes; it +only receives the encrypted access token and connects. This keeps rotation +races out of worker subprocesses and honors the worker-never-touches-the-DB +rule. + +An expired or revoked refresh token raises :class:`ReconnectRequiredError`, +which the API layer maps to HTTP 422 with ``error_code="RECONNECT_REQUIRED"`` +(never 401 — the frontend treats 401 as JWT expiry). +""" + +from __future__ import annotations + +import json +from typing import NamedTuple + +from pydantic import SecretStr +from sqlalchemy.orm import Session + +from flowfile_core.database.connection import get_db_context +from flowfile_core.database.models import DatabaseConnection as DBConnectionModel +from flowfile_core.database.models import Secret +from flowfile_core.flowfile.database_connection_manager.db_connections import get_database_connection +from flowfile_core.secret_manager.secret_manager import SecretInput, decrypt_secret, encrypt_secret, store_secret +from shared.snowflake_oauth import SnowflakeOAuthError, derive_snowflake_endpoints, refresh_access_token + + +class ReconnectRequiredError(Exception): + """The stored refresh token is expired/revoked; the user must sign in again.""" + + def __init__(self, connection_name: str, detail: str = ""): + self.connection_name = connection_name + suffix = f" ({detail})" if detail else "" + super().__init__( + f"Reconnect your '{connection_name}' connection: its sign-in has expired or was revoked{suffix}. " + "Open the connection settings and sign in again." + ) + + +class OAuthEndpoints(NamedTuple): + authorize_endpoint: str + token_endpoint: str + + +def _connection_account(db_connection: DBConnectionModel) -> str | None: + if not db_connection.extra_params: + return None + try: + parsed = json.loads(db_connection.extra_params) + except (TypeError, ValueError): + return None + account = (parsed or {}).get("account") or db_connection.host + return account or None + + +def resolve_oauth_endpoints(db_connection: DBConnectionModel) -> OAuthEndpoints: + """Explicit endpoint overrides (External OAuth) or Snowflake-derived defaults.""" + authorize = db_connection.oauth_authorize_endpoint + token = db_connection.oauth_token_endpoint + if authorize and token: + return OAuthEndpoints(authorize, token) + account = _connection_account(db_connection) + if not account: + raise ValueError( + "Cannot derive Snowflake OAuth endpoints: the connection has no 'account' extra param. " + "Set the account, or configure explicit authorize/token endpoints." + ) + derived_authorize, derived_token = derive_snowflake_endpoints(str(account)) + return OAuthEndpoints(authorize or derived_authorize, token or derived_token) + + +def _secret_ciphertext(db: Session, secret_id: int | None) -> str | None: + if secret_id is None: + return None + secret = db.query(Secret).filter(Secret.id == secret_id).first() + return secret.encrypted_value if secret else None + + +def store_refresh_token(db: Session, db_connection: DBConnectionModel, refresh_token: str) -> None: + """Persist a (new or rotated) refresh token encrypted under the OWNER's key.""" + owner_id = db_connection.user_id + if db_connection.oauth_refresh_token_id is not None: + secret = db.query(Secret).filter(Secret.id == db_connection.oauth_refresh_token_id).first() + if secret: + secret.encrypted_value = encrypt_secret(refresh_token, owner_id) + db.commit() + return + new_secret = store_secret( + db, + SecretInput(name=f"{db_connection.connection_name}_oauth_refresh_token", value=SecretStr(refresh_token)), + owner_id, + ) + db_connection.oauth_refresh_token_id = new_secret.id + db.commit() + + +def resolve_oauth_access_token(connection_name: str, user_id: int) -> str: + """Mint a short-lived access token for a stored OAuth connection. + + Returns the access token as owner-keyed ``$ffsec$`` ciphertext, ready for + the core→worker wire. Raises :class:`ReconnectRequiredError` when the + refresh token is missing, expired, or revoked. + """ + with get_db_context() as db: + db_connection = get_database_connection(db, connection_name, user_id) + if db_connection is None: + raise ValueError(f"Database connection '{connection_name}' not found or not accessible for this user") + if db_connection.oauth_refresh_token_id is None: + raise ReconnectRequiredError(connection_name, "no sign-in on record") + refresh_ciphertext = _secret_ciphertext(db, db_connection.oauth_refresh_token_id) + client_secret_ciphertext = _secret_ciphertext(db, db_connection.oauth_client_secret_id) + if refresh_ciphertext is None: + raise ReconnectRequiredError(connection_name, "no sign-in on record") + if not db_connection.oauth_client_id or client_secret_ciphertext is None: + raise ValueError(f"Connection '{connection_name}' has no OAuth client configured") + endpoints = resolve_oauth_endpoints(db_connection) + + try: + token_response = refresh_access_token( + endpoints.token_endpoint, + db_connection.oauth_client_id, + decrypt_secret(client_secret_ciphertext).get_secret_value(), + decrypt_secret(refresh_ciphertext).get_secret_value(), + ) + except SnowflakeOAuthError as e: + if e.requires_reauthentication: + raise ReconnectRequiredError(connection_name, str(e)) from e + raise + + if token_response.refresh_token: + store_refresh_token(db, db_connection, token_response.refresh_token) + return encrypt_secret(token_response.access_token, db_connection.user_id) diff --git a/flowfile_core/flowfile_core/flowfile/flow_graph.py b/flowfile_core/flowfile_core/flowfile/flow_graph.py index d1320bce2..35e434f2f 100644 --- a/flowfile_core/flowfile_core/flowfile/flow_graph.py +++ b/flowfile_core/flowfile_core/flowfile/flow_graph.py @@ -1316,15 +1316,26 @@ def _handle_physical_table_write( return df +class ResolvedDatabaseCredentials(NamedTuple): + """A resolved connection plus its credential ciphertexts (``$ffsec$``, decrypted at point of use).""" + + connection: Any + password: str | None + private_key: str | None + private_key_passphrase: str | None + reference_settings: input_schema.FullDatabaseConnection | None + # Short-lived access token minted by core at resolution time (oauth auth only). + oauth_token: str | None = None + + def _resolve_database_credentials( database_settings, user_id: int, -) -> tuple: - """Resolve database connection and encrypted password from settings. +) -> ResolvedDatabaseCredentials: + """Resolve database connection and encrypted credentials from settings. - Returns: - (database_connection, encrypted_password, database_reference_settings) - where database_reference_settings is the stored connection (or None for inline). + For key-pair auth the private key is the required credential and the password + becomes optional; ``reference_settings`` is the stored connection (None for inline). """ is_file_based = ( database_settings.connection_mode == "inline" @@ -1333,12 +1344,33 @@ def _resolve_database_credentials( ) if database_settings.connection_mode == "inline" and not is_file_based: database_connection = database_settings.database_connection - encrypted_password = get_encrypted_secret(current_user_id=user_id, secret_name=database_connection.password_ref) - if encrypted_password is None: + use_key_pair = database_connection.auth_method == "key_pair" + encrypted_password = None + if database_connection.password_ref: + encrypted_password = get_encrypted_secret( + current_user_id=user_id, secret_name=database_connection.password_ref + ) + if encrypted_password is None and not use_key_pair: raise HTTPException(status_code=400, detail="Password not found") - return database_connection, encrypted_password, None + encrypted_private_key = None + encrypted_passphrase = None + if use_key_pair: + encrypted_private_key = get_encrypted_secret( + current_user_id=user_id, secret_name=database_connection.private_key_ref + ) + if encrypted_private_key is None: + raise HTTPException(status_code=400, detail="Private key secret not found") + if database_connection.private_key_passphrase_ref: + encrypted_passphrase = get_encrypted_secret( + current_user_id=user_id, secret_name=database_connection.private_key_passphrase_ref + ) + if encrypted_passphrase is None: + raise HTTPException(status_code=400, detail="Private key passphrase secret not found") + return ResolvedDatabaseCredentials( + database_connection, encrypted_password, encrypted_private_key, encrypted_passphrase, None + ) elif is_file_based: - return database_settings.database_connection, None, None + return ResolvedDatabaseCredentials(database_settings.database_connection, None, None, None, None) else: ref_settings = get_local_database_connection(database_settings.database_connection_name, user_id) if ref_settings is None: @@ -1349,8 +1381,21 @@ def _resolve_database_credentials( "or not accessible for this user" ), ) - encrypted_password = ref_settings.password.get_secret_value() - return ref_settings, encrypted_password, ref_settings + oauth_token = None + if ref_settings.auth_method == "oauth": + # Core-side refresh; raises ReconnectRequiredError (422 RECONNECT_REQUIRED at + # the API surface via main.py's exception handler, a plain node error at run time). + from flowfile_core.flowfile.database_connection_manager.db_oauth import resolve_oauth_access_token + + oauth_token = resolve_oauth_access_token(database_settings.database_connection_name, user_id) + return ResolvedDatabaseCredentials( + ref_settings, + ref_settings.password.get_secret_value(), + ref_settings.private_key.get_secret_value() if ref_settings.private_key else None, + ref_settings.private_key_passphrase.get_secret_value() if ref_settings.private_key_passphrase else None, + ref_settings, + oauth_token, + ) class _FlowIdentity(NamedTuple): @@ -4570,9 +4615,8 @@ def add_database_writer(self, node_database_writer: input_schema.NodeDatabaseWri database_settings: input_schema.DatabaseWriteSettings = node_database_writer.database_write_settings def _func(df: FlowDataEngine): - database_connection, encrypted_password, database_reference_settings = _resolve_database_credentials( - database_settings, node_database_writer.user_id - ) + creds = _resolve_database_credentials(database_settings, node_database_writer.user_id) + database_connection = creds.connection df.lazy = True table_name = ( database_settings.schema_name + "." + database_settings.table_name @@ -4589,9 +4633,16 @@ def _func(df: FlowDataEngine): port=database_connection.port, database=database_connection.database, username=database_connection.username, - password=decrypt_secret(encrypted_password) if encrypted_password else None, + password=decrypt_secret(creds.password) if creds.password else None, ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)), connect_timeout=10, + auth_method=database_connection.auth_method, + private_key=decrypt_secret(creds.private_key) if creds.private_key else None, + private_key_passphrase=( + decrypt_secret(creds.private_key_passphrase) if creds.private_key_passphrase else None + ), + oauth_token=decrypt_secret(creds.oauth_token) if creds.oauth_token else None, + **(database_connection.extra_params or {}), ), table_name=table_name, if_exists=database_settings.if_exists or "append", @@ -4601,12 +4652,15 @@ def _func(df: FlowDataEngine): database_external_write_settings = ( sql_models.DatabaseExternalWriteSettings.create_from_from_node_database_writer( node_database_writer=node_database_writer, - password=encrypted_password, + password=creds.password, table_name=table_name, database_reference_settings=( - database_reference_settings if database_settings.connection_mode == "reference" else None + creds.reference_settings if database_settings.connection_mode == "reference" else None ), lf=df.data_frame, + private_key=creds.private_key, + private_key_passphrase=creds.private_key_passphrase, + oauth_token=creds.oauth_token, ) ) external_database_writer = ExternalDatabaseWriter( @@ -4659,7 +4713,8 @@ def _get_creds(): return _creds["v"] def _func(): - database_connection, encrypted_password, database_reference_settings = _get_creds() + creds = _get_creds() + database_connection = creds.connection sql_source = BaseSqlSource( query=None if database_settings.query_mode == "table" else database_settings.query, table_name=database_settings.table_name, @@ -4677,9 +4732,16 @@ def _func(): port=database_connection.port, database=database_connection.database, username=database_connection.username, - password=decrypt_secret(encrypted_password) if encrypted_password else None, + password=decrypt_secret(creds.password) if creds.password else None, ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)), connect_timeout=10, + auth_method=database_connection.auth_method, + private_key=decrypt_secret(creds.private_key) if creds.private_key else None, + private_key_passphrase=( + decrypt_secret(creds.private_key_passphrase) if creds.private_key_passphrase else None + ), + oauth_token=decrypt_secret(creds.oauth_token) if creds.oauth_token else None, + **(database_connection.extra_params or {}), ), query=None if database_settings.query_mode == "table" else database_settings.query, table_name=database_settings.table_name, @@ -4696,11 +4758,14 @@ def _func(): database_external_read_settings = ( sql_models.DatabaseExternalReadSettings.create_from_from_node_database_reader( node_database_reader=node_database_reader, - password=encrypted_password, + password=creds.password, query=sql_source.query, database_reference_settings=( - database_reference_settings if database_settings.connection_mode == "reference" else None + creds.reference_settings if database_settings.connection_mode == "reference" else None ), + private_key=creds.private_key, + private_key_passphrase=creds.private_key_passphrase, + oauth_token=creds.oauth_token, ) ) @@ -4718,7 +4783,8 @@ def schema_callback(): # when fields were never captured (failures here are caught per-node). if node_database_reader.fields: return [FlowfileColumn.from_input(f.name, f.data_type) for f in node_database_reader.fields] - database_connection, encrypted_password, _ = _get_creds() + creds = _get_creds() + database_connection = creds.connection sql_source = SqlSource( connection_string=sql_utils.construct_sql_uri( database_type=database_connection.database_type, @@ -4726,9 +4792,16 @@ def schema_callback(): port=database_connection.port, database=database_connection.database, username=database_connection.username, - password=decrypt_secret(encrypted_password) if encrypted_password else None, + password=decrypt_secret(creds.password) if creds.password else None, ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)), connect_timeout=10, + auth_method=database_connection.auth_method, + private_key=decrypt_secret(creds.private_key) if creds.private_key else None, + private_key_passphrase=( + decrypt_secret(creds.private_key_passphrase) if creds.private_key_passphrase else None + ), + oauth_token=decrypt_secret(creds.oauth_token) if creds.oauth_token else None, + **(database_connection.extra_params or {}), ), query=None if database_settings.query_mode == "table" else database_settings.query, table_name=database_settings.table_name, diff --git a/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/models.py b/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/models.py index fa3e0fb15..e0b2a202b 100644 --- a/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/models.py +++ b/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/models.py @@ -29,10 +29,21 @@ def _decode_bytes(v: Any) -> bytes: class ExtDatabaseConnection(DatabaseConnection): - """Database connection configuration with password handling.""" + """Database connection configuration with credential handling. + + password / private_key / private_key_passphrase carry $ffsec$ ciphertext + over the core -> worker wire; the worker decrypts them independently. + """ password: str | None = None ssl_enabled: bool | None = False + private_key: str | None = None + private_key_passphrase: str | None = None + # Short-lived OAuth access token ($ffsec$ ciphertext); minted core-side at resolve time. + oauth_token: str | None = None + + +_SECRET_FIELDS = ("password", "private_key", "private_key_passphrase", "oauth_client_secret", "oauth_refresh_token") class DatabaseExternalWriteSettings(BaseModel): @@ -53,24 +64,39 @@ def create_from_from_node_database_writer( table_name: str, lf: pl.LazyFrame, database_reference_settings: FullDatabaseConnection = None, + private_key: str | None = None, + private_key_passphrase: str | None = None, + oauth_token: str | None = None, ) -> "DatabaseExternalWriteSettings": """ Create DatabaseExternalWriteSettings from NodeDatabaseWriter. Args: node_database_writer (NodeDatabaseWriter): an instance of NodeDatabaseWriter - password (str): the password for the database connection + password (str): the encrypted password for the database connection table_name (str): the table name to be used for writing lf (pl.LazyFrame): the LazyFrame to be written to the database database_reference_settings (FullDatabaseConnection): optional database reference settings + private_key (str): the encrypted private key (key-pair auth) + private_key_passphrase (str): the encrypted private-key passphrase + oauth_token (str): the encrypted short-lived OAuth access token (oauth auth) Returns: DatabaseExternalReadSettings: an instance of DatabaseExternalReadSettings """ if node_database_writer.database_write_settings.connection_mode == "inline": database_connection = node_database_writer.database_write_settings.database_connection.model_dump() else: - database_connection = {k: v for k, v in database_reference_settings.model_dump().items() if k != "password"} - - ext_database_connection = ExtDatabaseConnection(**database_connection, password=password) + # Exclude the SecretStr fields from the splat; the ciphertexts are passed explicitly. + database_connection = { + k: v for k, v in database_reference_settings.model_dump().items() if k not in _SECRET_FIELDS + } + + ext_database_connection = ExtDatabaseConnection( + **database_connection, + password=password, + private_key=private_key, + private_key_passphrase=private_key_passphrase, + oauth_token=oauth_token, + ) return cls( connection=ext_database_connection, table_name=table_name, @@ -96,23 +122,38 @@ def create_from_from_node_database_reader( password: str, query: str, database_reference_settings: FullDatabaseConnection = None, + private_key: str | None = None, + private_key_passphrase: str | None = None, + oauth_token: str | None = None, ) -> "DatabaseExternalReadSettings": """ Create DatabaseExternalReadSettings from NodeDatabaseReader. Args: node_database_reader (NodeDatabaseReader): an instance of NodeDatabaseReader - password (str): the password for the database connection + password (str): the encrypted password for the database connection query (str): the SQL query to be executed database_reference_settings (FullDatabaseConnection): optional database reference settings + private_key (str): the encrypted private key (key-pair auth) + private_key_passphrase (str): the encrypted private-key passphrase + oauth_token (str): the encrypted short-lived OAuth access token (oauth auth) Returns: DatabaseExternalReadSettings: an instance of DatabaseExternalReadSettings """ if node_database_reader.database_settings.connection_mode == "inline": database_connection = node_database_reader.database_settings.database_connection.model_dump() else: - database_connection = {k: v for k, v in database_reference_settings.model_dump().items() if k != "password"} - - ext_database_connection = ExtDatabaseConnection(**database_connection, password=password) + # Exclude the SecretStr fields from the splat; the ciphertexts are passed explicitly. + database_connection = { + k: v for k, v in database_reference_settings.model_dump().items() if k not in _SECRET_FIELDS + } + + ext_database_connection = ExtDatabaseConnection( + **database_connection, + password=password, + private_key=private_key, + private_key_passphrase=private_key_passphrase, + oauth_token=oauth_token, + ) return cls( connection=ext_database_connection, query=query, diff --git a/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/sql_source.py b/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/sql_source.py index 3cbfbd880..8200a51a4 100644 --- a/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/sql_source.py +++ b/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/sql_source.py @@ -430,20 +430,43 @@ class ResolvedConnection(NamedTuple): def _resolve_connection(database_settings: DatabaseSettings, user_id: int) -> ResolvedConnection: """Resolve DatabaseSettings into a connection URI + dialect, handling inline/reference mode.""" database_connection = database_settings.database_connection + private_key = None + private_key_passphrase = None + oauth_token = None if database_settings.connection_mode == "inline": if database_connection is None: raise ValueError("Database connection is required in inline mode") is_file_based = get_dialect_or_generic(database_connection.database_type).file_based - if is_file_based: - password = None - else: + use_key_pair = database_connection.auth_method == "key_pair" + password = None + if not is_file_based and database_connection.password_ref: encrypted_secret = get_encrypted_secret( current_user_id=user_id, secret_name=database_connection.password_ref ) - if encrypted_secret is None: - raise ValueError(f"Secret with name {database_connection.password_ref} not found for user {user_id}") - password = decrypt_secret(encrypted_secret) + if encrypted_secret is not None: + password = decrypt_secret(encrypted_secret) + if password is None and not is_file_based and not use_key_pair: + raise ValueError(f"Secret with name {database_connection.password_ref} not found for user {user_id}") + if use_key_pair: + encrypted_key = get_encrypted_secret( + current_user_id=user_id, secret_name=database_connection.private_key_ref + ) + if encrypted_key is None: + raise ValueError( + f"Private key secret with name {database_connection.private_key_ref} not found for user {user_id}" + ) + private_key = decrypt_secret(encrypted_key) + if database_connection.private_key_passphrase_ref: + encrypted_passphrase = get_encrypted_secret( + current_user_id=user_id, secret_name=database_connection.private_key_passphrase_ref + ) + if encrypted_passphrase is None: + raise ValueError( + f"Private key passphrase secret with name " + f"{database_connection.private_key_passphrase_ref} not found for user {user_id}" + ) + private_key_passphrase = decrypt_secret(encrypted_passphrase) else: database_connection = get_local_database_connection(database_settings.database_connection_name, user_id) if database_connection is None: @@ -453,6 +476,18 @@ def _resolve_connection(database_settings: DatabaseSettings, user_id: int) -> Re ) encrypted_secret = database_connection.password.get_secret_value() password = decrypt_secret(encrypted_secret) + if database_connection.private_key: + private_key = decrypt_secret(database_connection.private_key.get_secret_value()) + if database_connection.private_key_passphrase: + private_key_passphrase = decrypt_secret(database_connection.private_key_passphrase.get_secret_value()) + if database_connection.auth_method == "oauth": + # Core-side refresh: mint a short-lived access token from the stored + # refresh token (raises ReconnectRequiredError when sign-in expired). + from flowfile_core.flowfile.database_connection_manager.db_oauth import resolve_oauth_access_token + + oauth_token = decrypt_secret( + resolve_oauth_access_token(database_settings.database_connection_name, user_id) + ) uri = construct_sql_uri( database_type=database_connection.database_type, @@ -463,6 +498,11 @@ def _resolve_connection(database_settings: DatabaseSettings, user_id: int) -> Re password=password, ssl_enabled=bool(getattr(database_connection, "ssl_enabled", False)), connect_timeout=10, + auth_method=database_connection.auth_method, + private_key=private_key, + private_key_passphrase=private_key_passphrase, + oauth_token=oauth_token, + **(database_connection.extra_params or {}), ) return ResolvedConnection(uri=uri, database_type=database_connection.database_type) diff --git a/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/utils.py b/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/utils.py index 567b0e84a..0a6a523d3 100644 --- a/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/utils.py +++ b/flowfile_core/flowfile_core/flowfile/sources/external_sources/sql_source/utils.py @@ -367,12 +367,16 @@ def construct_sql_uri( url: str | None = None, ssl_enabled: bool = False, connect_timeout: int | None = None, + auth_method: str | None = None, + private_key: SecretStr | None = None, + private_key_passphrase: SecretStr | None = None, + oauth_token: SecretStr | None = None, **kwargs, ) -> str: """ Constructs a SQL URI string from the provided parameters. - Thin wrapper around shared.sql_utils.construct_sql_uri that unwraps SecretStr passwords. + Thin wrapper around shared.sql_utils.construct_sql_uri that unwraps SecretStr credentials. Args: database_type: Database type (postgresql, mysql, sqlite, etc.) @@ -384,6 +388,10 @@ def construct_sql_uri( url: Complete database URL (overrides other parameters if provided) ssl_enabled: Adds sslmode=require for postgres-family databases connect_timeout: Connection timeout in seconds (postgres-family only) + auth_method: Authentication method ("password"/None, or "key_pair" where supported) + private_key: Private key PEM text as SecretStr (key-pair auth) + private_key_passphrase: Optional passphrase for an encrypted private key + oauth_token: OAuth access token as SecretStr (oauth auth; minted by core) **kwargs: Additional connection parameters Returns: @@ -403,6 +411,10 @@ def construct_sql_uri( url=url, ssl_enabled=ssl_enabled, connect_timeout=connect_timeout, + auth_method=auth_method, + private_key=private_key.get_secret_value() if private_key else None, + private_key_passphrase=private_key_passphrase.get_secret_value() if private_key_passphrase else None, + oauth_token=oauth_token.get_secret_value() if oauth_token else None, **kwargs, ) diff --git a/flowfile_core/flowfile_core/main.py b/flowfile_core/flowfile_core/main.py index 56532268e..e03aac4cb 100644 --- a/flowfile_core/flowfile_core/main.py +++ b/flowfile_core/flowfile_core/main.py @@ -12,6 +12,7 @@ import uvicorn from fastapi import BackgroundTasks, FastAPI from fastapi.middleware.cors import CORSMiddleware +from fastapi.responses import JSONResponse from flowfile_core.ai import router as ai_router from flowfile_core.ai.admin_routes import router as ai_admin_router @@ -24,6 +25,7 @@ WORKER_PORT, WORKER_URL, ) +from flowfile_core.flowfile.database_connection_manager.db_oauth import ReconnectRequiredError from flowfile_core.kernel import router as kernel_router from flowfile_core.lsp.admin_routes import router as lsp_admin_router from flowfile_core.lsp.routes import router as lsp_router @@ -35,6 +37,7 @@ from flowfile_core.routes.community_github import router as community_github_router from flowfile_core.routes.community_nodes import router as community_nodes_router from flowfile_core.routes.custom_node_mounts import router as custom_node_mounts_router +from flowfile_core.routes.db_oauth import router as db_oauth_router from flowfile_core.routes.file_manager import router as file_manager_router from flowfile_core.routes.flow_api import data_router as flow_api_data_router from flowfile_core.routes.flow_api import management_router as flow_api_management_router @@ -172,6 +175,14 @@ def _shutdown_local_model(): allow_headers=["*"], ) + +@app.exception_handler(ReconnectRequiredError) +async def reconnect_required_handler(request, exc: ReconnectRequiredError): + """An expired OAuth sign-in is user-actionable, not a server fault: 422 with a + typed error_code (never 401 — the frontend treats 401 as JWT expiry).""" + return JSONResponse(status_code=422, content={"detail": {"error_code": "RECONNECT_REQUIRED", "message": str(exc)}}) + + app.include_router(public_router) app.include_router(router) app.include_router(catalog_router) @@ -190,6 +201,7 @@ def _shutdown_local_model(): app.include_router(project_router, prefix="/project", tags=["project"]) app.include_router(cloud_connections_router, prefix="/cloud_connections", tags=["cloud_connections"]) app.include_router(storage_browser_router, prefix="/storage_browser", tags=["storage_browser"]) +app.include_router(db_oauth_router) app.include_router(ga_connections_router, prefix="/ga_connections", tags=["ga_connections"]) app.include_router(kafka_router) app.include_router(user_defined_components_router, prefix="/user_defined_components", tags=["user_defined_components"]) diff --git a/flowfile_core/flowfile_core/project/importer.py b/flowfile_core/flowfile_core/project/importer.py index f9eaf161c..b27dd5740 100644 --- a/flowfile_core/flowfile_core/project/importer.py +++ b/flowfile_core/flowfile_core/project/importer.py @@ -170,6 +170,15 @@ def _import_db_connection(data: dict, owner_id: int, dotenv: dict, result: Setup name = entry.connection_name secret_name = placeholder_name(entry.password) or name value = _resolve_secret_value(secret_name, owner_id, dotenv, result) + + def _resolve_optional(raw: str | None) -> SecretStr | None: + # Only a placeholder resolves (no `or name` fallback like password): a None + # field means the connection has no such secret at all. + key_name = placeholder_name(raw) + if not key_name: + return None + return SecretStr(_resolve_secret_value(key_name, owner_id, dotenv, result)) + conn = FullDatabaseConnection( connection_name=name, database_type=entry.database_type, @@ -179,12 +188,31 @@ def _import_db_connection(data: dict, owner_id: int, dotenv: dict, result: Setup port=entry.port, database=entry.database, ssl_enabled=entry.ssl_enabled, + extra_params=entry.extra_params, + auth_method=entry.auth_method, + private_key=_resolve_optional(entry.private_key), + private_key_passphrase=_resolve_optional(entry.private_key_passphrase), + oauth_client_id=entry.oauth_client_id, + oauth_authorize_endpoint=entry.oauth_authorize_endpoint, + oauth_token_endpoint=entry.oauth_token_endpoint, + oauth_redirect_uri=entry.oauth_redirect_uri, + oauth_client_secret=_resolve_optional(entry.oauth_client_secret), ) - with get_db_context() as db: - if _get_own_database_connection(db, name, owner_id): - update_database_connection(db, conn, owner_id) - else: - store_database_connection(db, conn, owner_id) + if entry.auth_method == "oauth": + # Refresh tokens are never projected (interactive, expiring credentials) — + # the connection lands without a sign-in and needs the user in a browser. + result.warnings.append( + f"Database connection '{name}' uses OAuth sign-in and requires interactive re-authentication." + ) + try: + with get_db_context() as db: + if _get_own_database_connection(db, name, owner_id): + update_database_connection(db, conn, owner_id) + else: + store_database_connection(db, conn, owner_id) + except ValueError: + logger.warning("Project import: skipping invalid database connection %r", name, exc_info=True) + return None result.imported_connections += 1 return name diff --git a/flowfile_core/flowfile_core/project/manifest_entries.py b/flowfile_core/flowfile_core/project/manifest_entries.py index 970221487..f6286d67a 100644 --- a/flowfile_core/flowfile_core/project/manifest_entries.py +++ b/flowfile_core/flowfile_core/project/manifest_entries.py @@ -23,7 +23,16 @@ class DatabaseConnectionEntry(_Entry): database: str | None = None username: str = "" ssl_enabled: bool = False + auth_method: str | None = None + extra_params: dict[str, str] | None = None password: str | None = None + private_key: str | None = None + private_key_passphrase: str | None = None + oauth_client_id: str | None = None + oauth_authorize_endpoint: str | None = None + oauth_token_endpoint: str | None = None + oauth_redirect_uri: str | None = None + oauth_client_secret: str | None = None class CloudConnectionEntry(_Entry): diff --git a/flowfile_core/flowfile_core/project/models.py b/flowfile_core/flowfile_core/project/models.py index ad0dfdba8..197fe06df 100644 --- a/flowfile_core/flowfile_core/project/models.py +++ b/flowfile_core/flowfile_core/project/models.py @@ -44,6 +44,7 @@ class SetupResult: imported_schedules: int = 0 placeholder_secrets: list[str] = field(default_factory=list) prune_errors: list[str] = field(default_factory=list) + warnings: list[str] = field(default_factory=list) recovery_sha: str | None = None def to_dict(self) -> dict: @@ -55,5 +56,6 @@ def to_dict(self) -> dict: }, "placeholder_secrets": sorted(set(self.placeholder_secrets)), "prune_errors": list(self.prune_errors), + "warnings": list(self.warnings), "recovery_sha": self.recovery_sha, } diff --git a/flowfile_core/flowfile_core/project/projection.py b/flowfile_core/flowfile_core/project/projection.py index ab613d646..bfedcb27e 100644 --- a/flowfile_core/flowfile_core/project/projection.py +++ b/flowfile_core/flowfile_core/project/projection.py @@ -63,7 +63,27 @@ "verify_ssl", ) # Non-secret database-connection fields that round-trip verbatim. -_DB_PLAIN_FIELDS = ("database_type", "host", "port", "database", "username", "ssl_enabled") +_DB_PLAIN_FIELDS = ( + "database_type", + "host", + "port", + "database", + "username", + "ssl_enabled", + "auth_method", + "oauth_client_id", + "oauth_authorize_endpoint", + "oauth_token_endpoint", + "oauth_redirect_uri", +) +# Secret-backed database-connection fields: (file field, model FK column), like _CLOUD_SECRETS. +# The OAuth refresh token is deliberately absent: it is minted interactively, expires, and +# must never reach the projection even as a placeholder — imports re-authenticate instead. +_DB_SECRETS = ( + ("private_key", "private_key_id"), + ("private_key_passphrase", "private_key_passphrase_id"), + ("oauth_client_secret", "oauth_client_secret_id"), +) def _secret_name(db: Session, secret_id: int | None) -> str | None: @@ -183,11 +203,17 @@ def remove_stale_flow_files(root: Path, flow_uuid: str, keep: Path) -> None: def _db_connection_dict(db: Session, conn: DatabaseConnection) -> dict: + from flowfile_core.flowfile.database_connection_manager.db_connections import parse_extra_params + secret_name = _secret_name(db, conn.password_id) d = {"kind": "database_connection", "connection_name": conn.connection_name} for f in _DB_PLAIN_FIELDS: d[f] = getattr(conn, f) + d["extra_params"] = parse_extra_params(conn.extra_params) d["password"] = make_placeholder(secret_name) if secret_name else None + for field, fk in _DB_SECRETS: + name = _secret_name(db, getattr(conn, fk)) + d[field] = make_placeholder(name) if name else None return d @@ -280,9 +306,15 @@ def regenerate_secret_manifest(db: Session, root: Path, owner_id: int) -> None: """secrets.yaml lists only standalone secrets; connection secrets are implied by the connection files (and recreated by their store functions on import).""" linked: set[int] = set() - for (sid,) in db.query(DatabaseConnection.password_id).filter(DatabaseConnection.user_id == owner_id): - if sid: - linked.add(sid) + db_secret_fk_columns = ( + DatabaseConnection.password_id, + DatabaseConnection.private_key_id, + DatabaseConnection.private_key_passphrase_id, + DatabaseConnection.oauth_client_secret_id, + DatabaseConnection.oauth_refresh_token_id, + ) + for row in db.query(*db_secret_fk_columns).filter(DatabaseConnection.user_id == owner_id): + linked.update(sid for sid in row if sid) for row in db.query(*[getattr(CloudStorageConnection, fk) for fk in _CLOUD_SECRET_FK_COLUMNS]).filter( CloudStorageConnection.user_id == owner_id ): diff --git a/flowfile_core/flowfile_core/routes/db_oauth.py b/flowfile_core/flowfile_core/routes/db_oauth.py new file mode 100644 index 000000000..c6df3f6e8 --- /dev/null +++ b/flowfile_core/flowfile_core/routes/db_oauth.py @@ -0,0 +1,198 @@ +"""FastAPI routes for the Snowflake SSO / OAuth sign-in flow on database connections. + +Clones the Google Analytics OAuth machinery: ``/oauth/start`` builds the +authorize URL with an HMAC-signed state (the callback is unauthenticated, so +the state JWT is what lets it trust the user id + connection name), and +``/oauth/callback`` exchanges the code for tokens and stores the refresh +token encrypted against the connection. The API never accepts or returns raw +token material. +""" + +from __future__ import annotations + +import html +import time +from urllib.parse import urlencode + +from fastapi import APIRouter, Depends, HTTPException, Query +from fastapi.responses import HTMLResponse +from jose import JWTError, jwt +from pydantic import BaseModel +from sqlalchemy.orm import Session + +from flowfile_core.auth.jwt import get_current_active_user, get_jwt_secret +from flowfile_core.configs import logger +from flowfile_core.configs.settings import ALGORITHM +from flowfile_core.database.connection import get_db +from flowfile_core.database.models import DatabaseConnection as DBConnectionModel +from flowfile_core.database.models import Secret +from flowfile_core.flowfile.database_connection_manager.db_connections import get_database_connection +from flowfile_core.flowfile.database_connection_manager.db_oauth import resolve_oauth_endpoints, store_refresh_token +from flowfile_core.secret_manager.secret_manager import decrypt_secret +from shared.snowflake_oauth import SnowflakeOAuthError, exchange_authorization_code + +router = APIRouter() + +_STATE_TTL_SECONDS = 600 +_STATE_TYPE = "db_oauth_state" +_DEFAULT_REDIRECT_URI = "http://localhost:63578/db_connection_lib/oauth/callback" +# Snowflake's built-in OAuth server wants its proprietary scope; external IdPs +# use the standard OpenID offline-access scope for a refresh token. +_SNOWFLAKE_SCOPE = "refresh_token" +_EXTERNAL_IDP_SCOPE = "offline_access" + + +class OAuthStartResponse(BaseModel): + auth_url: str + + +def _sign_oauth_state(*, user_id: int, connection_name: str) -> str: + payload = { + "type": _STATE_TYPE, + "user_id": user_id, + "connection_name": connection_name, + "exp": int(time.time()) + _STATE_TTL_SECONDS, + } + return jwt.encode(payload, get_jwt_secret(), algorithm=ALGORITHM) + + +def _verify_oauth_state(state: str) -> dict: + try: + payload = jwt.decode(state, get_jwt_secret(), algorithms=[ALGORITHM]) + except JWTError as e: + raise HTTPException(400, f"Invalid OAuth state: {e}") from e + if payload.get("type") != _STATE_TYPE: + raise HTTPException(400, "OAuth state has wrong type") + return payload + + +def _redirect_uri(db_connection: DBConnectionModel) -> str: + return db_connection.oauth_redirect_uri or _DEFAULT_REDIRECT_URI + + +def _client_config(db: Session, db_connection: DBConnectionModel) -> tuple[str, str]: + """The connection's OAuth client id + decrypted client secret, or 422.""" + if not db_connection.oauth_client_id or db_connection.oauth_client_secret_id is None: + raise HTTPException( + 422, + f"Connection '{db_connection.connection_name}' has no OAuth client configured. " + "Set the client id and client secret on the connection first.", + ) + secret = db.query(Secret).filter(Secret.id == db_connection.oauth_client_secret_id).first() + if secret is None: + raise HTTPException(422, f"Connection '{db_connection.connection_name}' OAuth client secret is missing.") + return db_connection.oauth_client_id, decrypt_secret(secret.encrypted_value).get_secret_value() + + +def _get_oauth_connection(db: Session, connection_name: str, user_id: int) -> DBConnectionModel: + db_connection = get_database_connection(db, connection_name, user_id) + if db_connection is None: + raise HTTPException(404, "Database connection not found") + if db_connection.auth_method != "oauth": + raise HTTPException(422, f"Connection '{connection_name}' does not use OAuth authentication.") + return db_connection + + +def _callback_html(status: str, message: str) -> str: + safe_status = html.escape(status) + safe_message = html.escape(message) + return f""" +{safe_message}
+You can close this window.
+ +""" + + +@router.get("/db_connection_lib/oauth/start", response_model=OAuthStartResponse, tags=["db_connections"]) +def oauth_start( + connection_name: str = Query(..., min_length=1), + current_user=Depends(get_current_active_user), + db: Session = Depends(get_db), +) -> OAuthStartResponse: + """Return the IdP authorize URL to open in a popup for a stored OAuth connection.""" + db_connection = _get_oauth_connection(db, connection_name, current_user.id) + client_id, _ = _client_config(db, db_connection) + try: + endpoints = resolve_oauth_endpoints(db_connection) + except ValueError as e: + raise HTTPException(422, str(e)) from e + # Explicit endpoint overrides signal an external IdP (Okta / Entra / Ping). + scope = _EXTERNAL_IDP_SCOPE if db_connection.oauth_token_endpoint else _SNOWFLAKE_SCOPE + state = _sign_oauth_state(user_id=current_user.id, connection_name=connection_name) + params = { + "response_type": "code", + "client_id": client_id, + "redirect_uri": _redirect_uri(db_connection), + "scope": scope, + "state": state, + } + return OAuthStartResponse(auth_url=f"{endpoints.authorize_endpoint}?{urlencode(params)}") + + +@router.get("/db_connection_lib/oauth/callback", tags=["db_connections"]) +def oauth_callback( + code: str | None = Query(None), + state: str | None = Query(None), + error: str | None = Query(None), + error_description: str | None = Query(None), + db: Session = Depends(get_db), +) -> HTMLResponse: + """The IdP redirects here with ``?code=...&state=...``; exchanges the code and + stores the refresh token encrypted under the connection owner's key.""" + if error: + logger.info("DB OAuth callback error: %s (%s)", error, error_description) + detail = f"{error}: {error_description}" if error_description else error + return HTMLResponse(_callback_html("error", f"The identity provider returned an error: {detail}")) + if not code or not state: + return HTMLResponse(_callback_html("error", "Missing code or state parameter")) + try: + state_payload = _verify_oauth_state(state) + except HTTPException as e: + return HTMLResponse(_callback_html("error", e.detail), status_code=400) + + connection_name = state_payload["connection_name"] + user_id = int(state_payload["user_id"]) + try: + db_connection = _get_oauth_connection(db, connection_name, user_id) + client_id, client_secret = _client_config(db, db_connection) + endpoints = resolve_oauth_endpoints(db_connection) + except (HTTPException, ValueError) as e: + detail = e.detail if isinstance(e, HTTPException) else str(e) + return HTMLResponse(_callback_html("error", str(detail)), status_code=400) + + try: + token_response = exchange_authorization_code( + endpoints.token_endpoint, client_id, client_secret, code, _redirect_uri(db_connection) + ) + except SnowflakeOAuthError as e: + logger.error("DB OAuth token exchange failed for %s: %s", connection_name, e) + return HTMLResponse(_callback_html("error", f"Token exchange failed: {e}"), status_code=400) + + if not token_response.refresh_token: + return HTMLResponse( + _callback_html( + "error", + "The identity provider did not return a refresh token. For Snowflake OAuth the " + "security integration must allow the 'refresh_token' scope; for an external IdP " + "request offline access.", + ), + status_code=400, + ) + + store_refresh_token(db, db_connection, token_response.refresh_token) + return HTMLResponse(_callback_html("ok", f"Connection '{connection_name}' is signed in.")) diff --git a/flowfile_core/flowfile_core/routes/routes.py b/flowfile_core/flowfile_core/routes/routes.py index 1f2ee5315..85e5c354c 100644 --- a/flowfile_core/flowfile_core/routes/routes.py +++ b/flowfile_core/flowfile_core/routes/routes.py @@ -80,9 +80,11 @@ delete_database_connection, get_all_database_connections_interface, get_database_connection, + parse_extra_params, store_database_connection, update_database_connection, ) +from flowfile_core.flowfile.database_connection_manager.db_oauth import ReconnectRequiredError from flowfile_core.flowfile.extensions import get_instant_func_results from flowfile_core.flowfile.flow_data_engine.column_stats import ColumnStatsUnavailable from flowfile_core.flowfile.flow_data_engine.flow_data_engine import FlowDataEngine @@ -759,8 +761,9 @@ def create_db_connection( _require_known_database_type(input_connection.database_type) try: store_database_connection(db, input_connection, current_user.id) - except ValueError: - raise HTTPException(422, "Connection name already exists") from None + except ValueError as e: + # Duplicate name or invalid credential shape (e.g. key_pair without a key). + raise HTTPException(422, str(e)) from None except Exception as e: logger.error(e) raise HTTPException(422, str(e)) from e @@ -786,16 +789,36 @@ def update_db_connection( changed = changed_target_fields( db_connection, input_connection, ("host", "port", "database", "database_type", "ssl_enabled") ) + # extra_params is a target too (e.g. the snowflake account): compare the row's + # JSON against the incoming dict, normalized, so unchanged params don't trip the guard. + if (parse_extra_params(db_connection.extra_params) or {}) != (input_connection.extra_params or {}): + changed.append("extra_params") + # auth_method needs normalization: a stored NULL and an incoming "password" are the same. + if (db_connection.auth_method or "password") != (input_connection.auth_method or "password"): + changed.append("auth_method") + # OAuth endpoints/client are targets too: repointing the token endpoint would + # harvest the refresh token on the next resolve. (The update path additionally + # drops the stored refresh token whenever these change, forcing a re-sign-in.) + for oauth_field in ("oauth_client_id", "oauth_authorize_endpoint", "oauth_token_endpoint"): + if (getattr(db_connection, oauth_field) or None) != (getattr(input_connection, oauth_field) or None): + changed.append(oauth_field) require_credentials_on_target_change( changed, - has_new_credentials=bool(input_connection.password.get_secret_value()), - has_bundled_secrets=db_connection.password_id is not None, + has_new_credentials=bool(input_connection.password.get_secret_value()) + or bool(input_connection.private_key and input_connection.private_key.get_secret_value()) + or bool(input_connection.oauth_client_secret and input_connection.oauth_client_secret.get_secret_value()), + has_bundled_secrets=db_connection.password_id is not None + or db_connection.private_key_id is not None + or db_connection.oauth_refresh_token_id is not None, ) try: # Owner's user_id keeps a rotated password encrypted under the OWNER's key. update_database_connection(db, input_connection, db_connection.user_id) - except ValueError: - raise HTTPException(404, "Database connection not found") from None + except ValueError as e: + if "not found" in str(e): + raise HTTPException(404, "Database connection not found") from None + # Invalid credential shape (e.g. switching to key_pair without a key). + raise HTTPException(422, str(e)) from None except Exception as e: logger.error(e) raise HTTPException(422, str(e)) from e @@ -2433,6 +2456,9 @@ async def validate_db_settings( sql_source = create_sql_source_from_db_settings(database_settings, user_id=current_user.id) sql_source.validate() return {"message": "Query settings are valid"} + except ReconnectRequiredError: + # Typed: main.py's handler maps it to 422 with error_code=RECONNECT_REQUIRED. + raise except Exception as e: raise HTTPException(status_code=422, detail=str(e)) from e @@ -2444,6 +2470,8 @@ async def get_db_schemas( """Returns available schema names for the given database connection.""" try: return list_db_schemas(database_settings, user_id=current_user.id) + except ReconnectRequiredError: + raise except Exception as e: raise HTTPException(status_code=422, detail=str(e)) from e @@ -2460,6 +2488,8 @@ async def get_db_tables( """ try: return list_db_tables(database_settings, user_id=current_user.id) + except ReconnectRequiredError: + raise except Exception as e: raise HTTPException(status_code=422, detail=str(e)) from e diff --git a/flowfile_core/flowfile_core/schemas/input_schema.py b/flowfile_core/flowfile_core/schemas/input_schema.py index 43e13cb88..be360f397 100644 --- a/flowfile_core/flowfile_core/schemas/input_schema.py +++ b/flowfile_core/flowfile_core/schemas/input_schema.py @@ -917,6 +917,28 @@ def get_default_description(self) -> str: return f"{name} ({rf.file_type})" +def _validate_extra_params(v: dict[str, str] | None) -> dict[str, str] | None: + """Normalize empty to None and reject keys that could override auth/target settings.""" + if not v: + return None + from shared.db_dialects import is_blocked_extra_param + + blocked = sorted(key for key in v if is_blocked_extra_param(key)) + if blocked: + raise ValueError(f"extra_params may not override connection auth or target settings: {', '.join(blocked)}") + return v + + +def _validate_auth_method(auth_method: str | None, database_type: str) -> None: + """Reject an auth method the connection's dialect does not support.""" + if auth_method and auth_method != "password": + from shared.db_dialects import get_dialect_or_generic + + dialect = get_dialect_or_generic(database_type) + if auth_method not in dialect.auth_methods: + raise ValueError(f"auth_method '{auth_method}' is not supported by database type '{database_type}'") + + class DatabaseConnection(BaseModel): """Defines the connection parameters for a database.""" @@ -927,6 +949,10 @@ class DatabaseConnection(BaseModel): port: int | None = None database: str | None = None url: str | None = None + extra_params: dict[str, str] | None = None + auth_method: str | None = None + private_key_ref: SecretRef | None = None + private_key_passphrase_ref: SecretRef | None = None @field_validator("database_type") @classmethod @@ -938,16 +964,44 @@ def known_database_type(cls, v: str) -> str: raise ValueError(f"Unsupported database type '{v}'. Supported types: {', '.join(KNOWN_DIALECT_NAMES)}") return low - @field_validator("password_ref", mode="before") + @field_validator("password_ref", "private_key_ref", "private_key_passphrase_ref", "auth_method", mode="before") @classmethod def empty_string_to_none(cls, v): if v == "": return None return v + @field_validator("extra_params") + @classmethod + def guard_extra_params(cls, v): + return _validate_extra_params(v) + + @model_validator(mode="after") + def check_auth_method(self): + _validate_auth_method(self.auth_method, self.database_type) + # getattr: the wire subclass (ExtDatabaseConnection) carries the key ciphertext + # in a `private_key` field this model does not have. + if self.auth_method == "key_pair" and not self.private_key_ref and not getattr(self, "private_key", None): + raise ValueError("key_pair authentication requires a private key") + # OAuth tokens are minted interactively against a stored connection; inline node + # settings cannot carry them. The wire subclass passes with its token ciphertext. + if self.auth_method == "oauth" and not getattr(self, "oauth_token", None): + raise ValueError( + "OAuth authentication requires a stored connection: save the connection, sign in, " + "and use connection_mode='reference'" + ) + return self + class FullDatabaseConnection(BaseModel): - """A complete database connection model including the secret password.""" + """A complete database connection model including the secret password. + + For key-pair auth, ``private_key`` holds the PEM text (and + ``private_key_passphrase`` its optional passphrase). Both mirror + ``password``'s update semantics: empty means "keep the existing secret". + The key is not required at the model level for that reason — creation-time + enforcement lives in ``store_database_connection``. + """ connection_name: str database_type: str = "postgresql" @@ -958,6 +1012,18 @@ class FullDatabaseConnection(BaseModel): database: str | None = None ssl_enabled: bool | None = False url: str | None = None + extra_params: dict[str, str] | None = None + auth_method: str | None = None + private_key: SecretStr | None = None + private_key_passphrase: SecretStr | None = None + oauth_client_id: str | None = None + oauth_client_secret: SecretStr | None = None + oauth_authorize_endpoint: str | None = None + oauth_token_endpoint: str | None = None + oauth_redirect_uri: str | None = None + # Read-side only: the refresh-token ciphertext when the connection is signed in. + # Ignored on create/update — the token is minted by the interactive OAuth flow. + oauth_refresh_token: SecretStr | None = None @field_validator("database_type") @classmethod @@ -965,9 +1031,43 @@ def normalize_database_type(cls, v: str) -> str: # lowercase only, no vocabulary check: legacy stored types (e.g. redshift) must keep loading return v.lower() + @field_validator("auth_method", mode="before") + @classmethod + def empty_auth_method_to_none(cls, v): + if v == "": + return None + return v + + @field_validator("oauth_client_id", "oauth_authorize_endpoint", "oauth_token_endpoint", "oauth_redirect_uri") + @classmethod + def empty_oauth_field_to_none(cls, v): + if isinstance(v, str): + v = v.strip() + return v or None + + @field_validator("private_key", "private_key_passphrase", "oauth_client_secret", mode="before") + @classmethod + def empty_key_string_to_none(cls, v): + # A raw "" (JSON clients sending the empty form field) means "no key material"; + # an explicit SecretStr("") (the project importer's placeholder refill rows) + # deliberately passes through and creates a linked empty secret. + if v == "": + return None + return v + + @field_validator("extra_params") + @classmethod + def guard_extra_params(cls, v): + return _validate_extra_params(v) + + @model_validator(mode="after") + def check_auth_method(self): + _validate_auth_method(self.auth_method, self.database_type) + return self + class FullDatabaseConnectionInterface(BaseModel): - """A database connection model intended for UI display, omitting the password.""" + """A database connection model intended for UI display, omitting the password and key material.""" connection_name: str database_type: str = "postgresql" @@ -977,6 +1077,13 @@ class FullDatabaseConnectionInterface(BaseModel): database: str | None = None ssl_enabled: bool | None = False url: str | None = None + extra_params: dict[str, str] | None = None + auth_method: str | None = None + oauth_client_id: str | None = None + oauth_authorize_endpoint: str | None = None + oauth_token_endpoint: str | None = None + oauth_redirect_uri: str | None = None + oauth_connected: bool = False id: int | None = None access: AccessInfo | None = None diff --git a/flowfile_core/tests/docs_examples/test_docs_examples.py b/flowfile_core/tests/docs_examples/test_docs_examples.py index 19b6db0b0..ca5f4c539 100644 --- a/flowfile_core/tests/docs_examples/test_docs_examples.py +++ b/flowfile_core/tests/docs_examples/test_docs_examples.py @@ -83,10 +83,18 @@ def _mssql_available() -> bool: return mssql_fixtures.can_connect_to_db() +def _snowflake_available() -> bool: + import os + + required = ("ACCOUNT", "USER", "PASSWORD", "WAREHOUSE") + return all(os.environ.get(f"FLOWFILE_TEST_SNOWFLAKE_{name}") for name in required) + + INTEGRATION_GATES = { "database_read": _postgres_available, "database_read_duckdb": lambda: True, # in-process, no backing service "database_read_mssql": _mssql_available, + "database_read_snowflake": _snowflake_available, # live account only; skipped in CI "database_transform_write": _postgres_available, "cloud_storage_s3": _minio_available, "kafka_read": _kafka_available, diff --git a/flowfile_core/tests/flowfile/external_sources/test_dialect_vocabulary.py b/flowfile_core/tests/flowfile/external_sources/test_dialect_vocabulary.py index 94aee9025..87e326a9c 100644 --- a/flowfile_core/tests/flowfile/external_sources/test_dialect_vocabulary.py +++ b/flowfile_core/tests/flowfile/external_sources/test_dialect_vocabulary.py @@ -28,6 +28,15 @@ def test_database_connection_accepts_mixed_case_and_normalizes_it(): assert DatabaseConnection(database_type="SQLITE").database_type == "sqlite" +def test_database_connection_rejects_blocked_extra_params(): + with pytest.raises(ValidationError, match="extra_params may not override"): + DatabaseConnection(database_type="snowflake", extra_params={"account": "a", "password": "x"}) + + +def test_database_connection_normalizes_empty_extra_params_to_none(): + assert DatabaseConnection(database_type="snowflake", extra_params={}).extra_params is None + + class TestSqlSourceDialectInference: """SqlSource resolves its dialect from the explicit database_type, else the URI scheme.""" @@ -51,6 +60,13 @@ def test_mssql_scheme_resolves_to_top_based_limits(self): assert source.dialect.name == "mssql" assert source.get_sample_query() == "SELECT TOP 1 * FROM t" + def test_snowflake_scheme_resolves_to_native_dialect(self): + from flowfile_core.flowfile.sources.external_sources.sql_source.sql_source import SqlSource + + source = SqlSource(connection_string="snowflake://u:p@acct/d?warehouse=WH", table_name="t") + assert source.dialect.name == "snowflake" + assert source.get_sample_query() == "SELECT * FROM t LIMIT 1" + def test_explicit_database_type_wins_over_scheme(self): from flowfile_core.flowfile.sources.external_sources.sql_source.sql_source import SqlSource diff --git a/flowfile_core/tests/flowfile/external_sources/test_snowflake_oauth.py b/flowfile_core/tests/flowfile/external_sources/test_snowflake_oauth.py new file mode 100644 index 000000000..7ae79b027 --- /dev/null +++ b/flowfile_core/tests/flowfile/external_sources/test_snowflake_oauth.py @@ -0,0 +1,641 @@ +"""Snowflake SSO / OAuth tests: connection CRUD, refresh-token custody, core-side +token minting (with rotation persistence), the reconnect-required surface, the +signed-state OAuth routes, and the projection's re-auth marking. + +The token-endpoint HTTP behavior itself is covered in +shared/tests/test_snowflake_oauth.py against a mock server; here the shared +helpers are monkeypatched so these tests exercise the core plumbing only. The +live leg is gated on FLOWFILE_TEST_SNOWFLAKE_OAUTH_* (a real security +integration) — fakesnow is irrelevant to auth. +""" + +import os + +import pytest +from fastapi.testclient import TestClient +from pydantic import SecretStr, ValidationError + +from flowfile_core import main +from flowfile_core.database.connection import get_db_context +from flowfile_core.database.models import Secret +from flowfile_core.flowfile.database_connection_manager import db_oauth +from flowfile_core.flowfile.database_connection_manager.db_connections import ( + delete_database_connection, + get_all_database_connections_interface, + get_database_connection, + get_database_connection_schema, + store_database_connection, + update_database_connection, +) +from flowfile_core.flowfile.database_connection_manager.db_oauth import ( + ReconnectRequiredError, + resolve_oauth_access_token, + resolve_oauth_endpoints, + store_refresh_token, +) +from flowfile_core.schemas.input_schema import DatabaseConnection, FullDatabaseConnection +from flowfile_core.secret_manager.secret_manager import decrypt_secret +from shared.snowflake_oauth import SnowflakeOAuthError, TokenResponse + +USER_ID = 1 +EXTRA_PARAMS = {"account": "myorg-myaccount", "warehouse": "COMPUTE_WH"} + + +def _oauth_connection( + name: str, + client_id: str | None = "client-abc", + client_secret: str | None = "sekrit", + token_endpoint: str | None = None, + authorize_endpoint: str | None = None, +) -> FullDatabaseConnection: + return FullDatabaseConnection( + connection_name=name, + database_type="snowflake", + username="user", + password=SecretStr(""), + database="ANALYTICS", + extra_params=EXTRA_PARAMS, + auth_method="oauth", + oauth_client_id=client_id, + oauth_client_secret=SecretStr(client_secret) if client_secret is not None else None, + oauth_authorize_endpoint=authorize_endpoint, + oauth_token_endpoint=token_endpoint, + ) + + +def _cleanup(name: str) -> None: + with get_db_context() as db: + if get_database_connection(db, name, USER_ID) is not None: + delete_database_connection(db, name, USER_ID) + + +def _store(connection: FullDatabaseConnection) -> None: + with get_db_context() as db: + store_database_connection(db, connection, USER_ID) + + +def _sign_in(name: str, refresh_token: str = "rt-initial") -> None: + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + store_refresh_token(db, row, refresh_token) + + +class TestOauthCrud: + def test_store_and_reload_round_trips_client_config(self): + name = "snowflake_oauth_store" + _cleanup(name) + try: + _store(_oauth_connection(name)) + with get_db_context() as db: + reloaded = get_database_connection_schema(db, name, USER_ID) + assert reloaded.auth_method == "oauth" + assert reloaded.oauth_client_id == "client-abc" + secret_ciphertext = reloaded.oauth_client_secret.get_secret_value() + assert secret_ciphertext.startswith("$ffsec$"), "schema must return ciphertext, never the secret" + assert decrypt_secret(secret_ciphertext).get_secret_value() == "sekrit" + assert reloaded.oauth_refresh_token is None, "no sign-in yet" + finally: + _cleanup(name) + + def test_store_requires_client_config(self): + _cleanup("snowflake_oauth_missing") + with get_db_context() as db: + with pytest.raises(ValueError, match="client id and client secret"): + store_database_connection(db, _oauth_connection("snowflake_oauth_missing", client_secret=None), USER_ID) + + def test_interface_reports_connected_state_without_secrets(self): + name = "snowflake_oauth_iface" + _cleanup(name) + try: + _store(_oauth_connection(name)) + with get_db_context() as db: + iface = next( + i for i in get_all_database_connections_interface(db, USER_ID) if i.connection_name == name + ) + assert iface.auth_method == "oauth" + assert iface.oauth_client_id == "client-abc" + assert iface.oauth_connected is False + assert not hasattr(iface, "oauth_client_secret") + assert not hasattr(iface, "oauth_refresh_token") + + _sign_in(name) + with get_db_context() as db: + iface = next( + i for i in get_all_database_connections_interface(db, USER_ID) if i.connection_name == name + ) + assert iface.oauth_connected is True + finally: + _cleanup(name) + + def test_update_rotates_client_secret_and_empty_keeps_existing(self): + name = "snowflake_oauth_rotate" + _cleanup(name) + try: + _store(_oauth_connection(name)) + with get_db_context() as db: + update_database_connection(db, _oauth_connection(name, client_secret=""), USER_ID) + with get_db_context() as db: + kept = get_database_connection_schema(db, name, USER_ID).oauth_client_secret.get_secret_value() + assert decrypt_secret(kept).get_secret_value() == "sekrit" + + with get_db_context() as db: + update_database_connection(db, _oauth_connection(name, client_secret="rotated"), USER_ID) + with get_db_context() as db: + rotated = get_database_connection_schema(db, name, USER_ID).oauth_client_secret.get_secret_value() + assert decrypt_secret(rotated).get_secret_value() == "rotated" + finally: + _cleanup(name) + + def test_client_change_drops_the_refresh_token(self): + name = "snowflake_oauth_repoint" + _cleanup(name) + try: + _store(_oauth_connection(name)) + _sign_in(name) + with get_db_context() as db: + token_secret_id = get_database_connection(db, name, USER_ID).oauth_refresh_token_id + assert token_secret_id is not None + + with get_db_context() as db: + update_database_connection(db, _oauth_connection(name, client_id="other-client"), USER_ID) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + assert row.oauth_refresh_token_id is None, "a changed client must force re-authentication" + assert db.query(Secret).filter(Secret.id == token_secret_id).count() == 0 + finally: + _cleanup(name) + + def test_unchanged_client_keeps_the_refresh_token(self): + name = "snowflake_oauth_keep" + _cleanup(name) + try: + _store(_oauth_connection(name)) + _sign_in(name) + with get_db_context() as db: + update_database_connection(db, _oauth_connection(name, client_secret=""), USER_ID) + with get_db_context() as db: + assert get_database_connection(db, name, USER_ID).oauth_refresh_token_id is not None + finally: + _cleanup(name) + + def test_switching_away_from_oauth_deletes_oauth_secrets(self): + name = "snowflake_oauth_switch" + _cleanup(name) + try: + _store(_oauth_connection(name)) + _sign_in(name) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + oauth_secret_ids = [row.oauth_client_secret_id, row.oauth_refresh_token_id] + assert all(secret_id is not None for secret_id in oauth_secret_ids) + + password_connection = FullDatabaseConnection( + connection_name=name, + database_type="snowflake", + username="user", + password=SecretStr("new-pass"), + database="ANALYTICS", + extra_params=EXTRA_PARAMS, + ) + with get_db_context() as db: + update_database_connection(db, password_connection, USER_ID) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + assert row.oauth_client_id is None + assert row.oauth_client_secret_id is None + assert row.oauth_refresh_token_id is None + assert db.query(Secret).filter(Secret.id.in_(oauth_secret_ids)).count() == 0 + finally: + _cleanup(name) + + def test_delete_removes_oauth_secrets(self): + name = "snowflake_oauth_delete" + _cleanup(name) + _store(_oauth_connection(name)) + _sign_in(name) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + secret_ids = [row.password_id, row.oauth_client_secret_id, row.oauth_refresh_token_id] + assert all(secret_id is not None for secret_id in secret_ids) + with get_db_context() as db: + delete_database_connection(db, name, USER_ID) + with get_db_context() as db: + assert db.query(Secret).filter(Secret.id.in_(secret_ids)).count() == 0 + + +class TestEndpointResolution: + def test_endpoints_derive_from_account(self): + name = "snowflake_oauth_derive" + _cleanup(name) + try: + _store(_oauth_connection(name)) + with get_db_context() as db: + endpoints = resolve_oauth_endpoints(get_database_connection(db, name, USER_ID)) + assert endpoints.authorize_endpoint == "https://myorg-myaccount.snowflakecomputing.com/oauth/authorize" + assert endpoints.token_endpoint == "https://myorg-myaccount.snowflakecomputing.com/oauth/token-request" + finally: + _cleanup(name) + + def test_explicit_endpoints_win(self): + name = "snowflake_oauth_external" + _cleanup(name) + try: + _store( + _oauth_connection( + name, + authorize_endpoint="https://idp.example.com/authorize", + token_endpoint="https://idp.example.com/token", + ) + ) + with get_db_context() as db: + endpoints = resolve_oauth_endpoints(get_database_connection(db, name, USER_ID)) + assert endpoints.authorize_endpoint == "https://idp.example.com/authorize" + assert endpoints.token_endpoint == "https://idp.example.com/token" + finally: + _cleanup(name) + + +class TestAccessTokenResolution: + def test_resolve_mints_encrypted_access_token(self, monkeypatch): + name = "snowflake_oauth_resolve" + _cleanup(name) + try: + _store(_oauth_connection(name)) + _sign_in(name, "rt-stored") + calls = {} + + def fake_refresh(token_endpoint, client_id, client_secret, refresh_token): + calls.update( + endpoint=token_endpoint, + client_id=client_id, + client_secret=client_secret, + refresh_token=refresh_token, + ) + return TokenResponse(access_token="at-fresh", expires_in=600, refresh_token=None) + + monkeypatch.setattr(db_oauth, "refresh_access_token", fake_refresh) + ciphertext = resolve_oauth_access_token(name, USER_ID) + assert ciphertext.startswith("$ffsec$") + assert decrypt_secret(ciphertext).get_secret_value() == "at-fresh" + assert calls["endpoint"] == "https://myorg-myaccount.snowflakecomputing.com/oauth/token-request" + assert calls["client_id"] == "client-abc" + assert calls["client_secret"] == "sekrit" + assert calls["refresh_token"] == "rt-stored" + finally: + _cleanup(name) + + def test_rotated_refresh_token_is_persisted(self, monkeypatch): + name = "snowflake_oauth_rotation" + _cleanup(name) + try: + _store(_oauth_connection(name)) + _sign_in(name, "rt-old") + monkeypatch.setattr( + db_oauth, + "refresh_access_token", + lambda *a, **k: TokenResponse(access_token="at", expires_in=600, refresh_token="rt-rotated"), + ) + resolve_oauth_access_token(name, USER_ID) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + stored = db.query(Secret).filter(Secret.id == row.oauth_refresh_token_id).first() + assert decrypt_secret(stored.encrypted_value).get_secret_value() == "rt-rotated" + finally: + _cleanup(name) + + def test_never_signed_in_raises_reconnect_required(self): + name = "snowflake_oauth_nosignin" + _cleanup(name) + try: + _store(_oauth_connection(name)) + with pytest.raises(ReconnectRequiredError, match="Reconnect"): + resolve_oauth_access_token(name, USER_ID) + finally: + _cleanup(name) + + def test_invalid_grant_raises_reconnect_required(self, monkeypatch): + name = "snowflake_oauth_expired" + _cleanup(name) + try: + _store(_oauth_connection(name)) + _sign_in(name) + + def fake_refresh(*args, **kwargs): + raise SnowflakeOAuthError("expired", error="invalid_grant", status_code=400) + + monkeypatch.setattr(db_oauth, "refresh_access_token", fake_refresh) + with pytest.raises(ReconnectRequiredError, match="sign-in has expired"): + resolve_oauth_access_token(name, USER_ID) + finally: + _cleanup(name) + + def test_transient_idp_error_is_not_reconnect(self, monkeypatch): + name = "snowflake_oauth_transient" + _cleanup(name) + try: + _store(_oauth_connection(name)) + _sign_in(name) + + def fake_refresh(*args, **kwargs): + raise SnowflakeOAuthError("upstream down", status_code=503) + + monkeypatch.setattr(db_oauth, "refresh_access_token", fake_refresh) + with pytest.raises(SnowflakeOAuthError): + resolve_oauth_access_token(name, USER_ID) + finally: + _cleanup(name) + + +class TestModelValidation: + def test_inline_oauth_without_token_is_rejected(self): + with pytest.raises(ValidationError, match="stored connection"): + DatabaseConnection(database_type="snowflake", username="u", auth_method="oauth") + + def test_oauth_rejected_for_non_supporting_dialect(self): + with pytest.raises(ValidationError, match="not supported"): + FullDatabaseConnection( + connection_name="x", + database_type="postgresql", + username="u", + password=SecretStr("p"), + auth_method="oauth", + ) + + def test_wire_model_with_token_ciphertext_passes(self): + from flowfile_core.flowfile.sources.external_sources.sql_source.models import ExtDatabaseConnection + + connection = ExtDatabaseConnection( + database_type="snowflake", + username="u", + auth_method="oauth", + extra_params=EXTRA_PARAMS, + oauth_token="$ffsec$1$1$abc", + ) + assert connection.oauth_token == "$ffsec$1$1$abc" + + +def _get_test_client() -> TestClient: + with TestClient(main.app) as auth_client: + token = auth_client.post("/auth/token").json()["access_token"] + client = TestClient(main.app) + client.headers = {"Authorization": f"Bearer {token}"} + return client + + +class TestOauthRoutes: + def test_state_round_trip_and_type_check(self): + from fastapi import HTTPException + + from flowfile_core.routes import db_oauth as routes + + state = routes._sign_oauth_state(user_id=7, connection_name="conn-a") + payload = routes._verify_oauth_state(state) + assert payload["user_id"] == 7 + assert payload["connection_name"] == "conn-a" + with pytest.raises(HTTPException): + routes._verify_oauth_state(state + "tampered") + + def test_start_builds_authorize_url(self): + name = "snowflake_oauth_start" + _cleanup(name) + try: + _store(_oauth_connection(name)) + client = _get_test_client() + response = client.get("/db_connection_lib/oauth/start", params={"connection_name": name}) + assert response.status_code == 200, response.text + auth_url = response.json()["auth_url"] + assert auth_url.startswith("https://myorg-myaccount.snowflakecomputing.com/oauth/authorize?") + assert "client_id=client-abc" in auth_url + assert "scope=refresh_token" in auth_url, "Snowflake OAuth uses its proprietary refresh scope" + assert "state=" in auth_url + finally: + _cleanup(name) + + def test_start_uses_offline_access_for_external_idp(self): + name = "snowflake_oauth_start_ext" + _cleanup(name) + try: + _store( + _oauth_connection( + name, + authorize_endpoint="https://idp.example.com/authorize", + token_endpoint="https://idp.example.com/token", + ) + ) + client = _get_test_client() + response = client.get("/db_connection_lib/oauth/start", params={"connection_name": name}) + assert response.status_code == 200, response.text + auth_url = response.json()["auth_url"] + assert auth_url.startswith("https://idp.example.com/authorize?") + assert "scope=offline_access" in auth_url + finally: + _cleanup(name) + + def test_start_rejects_non_oauth_connection(self): + name = "snowflake_oauth_start_pw" + _cleanup(name) + try: + _store( + FullDatabaseConnection( + connection_name=name, + database_type="snowflake", + username="u", + password=SecretStr("p"), + extra_params=EXTRA_PARAMS, + ) + ) + client = _get_test_client() + response = client.get("/db_connection_lib/oauth/start", params={"connection_name": name}) + assert response.status_code == 422 + finally: + _cleanup(name) + + def test_callback_stores_refresh_token(self, monkeypatch): + from flowfile_core.routes import db_oauth as routes + + name = "snowflake_oauth_callback" + _cleanup(name) + try: + _store(_oauth_connection(name)) + calls = {} + + def fake_exchange(token_endpoint, client_id, client_secret, code, redirect_uri): + calls.update(code=code, redirect_uri=redirect_uri) + return TokenResponse(access_token="at", expires_in=600, refresh_token="rt-from-code") + + monkeypatch.setattr(routes, "exchange_authorization_code", fake_exchange) + state = routes._sign_oauth_state(user_id=USER_ID, connection_name=name) + client = TestClient(main.app) + response = client.get( + "/db_connection_lib/oauth/callback", params={"code": "the-code", "state": state} + ) + assert response.status_code == 200, response.text + assert "signed in" in response.text + assert calls["code"] == "the-code" + assert calls["redirect_uri"] == "http://localhost:63578/db_connection_lib/oauth/callback" + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + stored = db.query(Secret).filter(Secret.id == row.oauth_refresh_token_id).first() + assert decrypt_secret(stored.encrypted_value).get_secret_value() == "rt-from-code" + finally: + _cleanup(name) + + def test_callback_rejects_invalid_state(self): + client = TestClient(main.app) + response = client.get("/db_connection_lib/oauth/callback", params={"code": "c", "state": "garbage"}) + assert response.status_code == 400 + assert "Invalid OAuth state" in response.text + + def test_callback_without_refresh_token_errors(self, monkeypatch): + from flowfile_core.routes import db_oauth as routes + + name = "snowflake_oauth_no_rt" + _cleanup(name) + try: + _store(_oauth_connection(name)) + monkeypatch.setattr( + routes, + "exchange_authorization_code", + lambda *a, **k: TokenResponse(access_token="at", expires_in=600, refresh_token=None), + ) + state = routes._sign_oauth_state(user_id=USER_ID, connection_name=name) + client = TestClient(main.app) + response = client.get("/db_connection_lib/oauth/callback", params={"code": "c", "state": state}) + assert response.status_code == 400 + assert "did not return a refresh token" in response.text + with get_db_context() as db: + assert get_database_connection(db, name, USER_ID).oauth_refresh_token_id is None + finally: + _cleanup(name) + + def test_reconnect_required_surfaces_as_422_error_code(self, monkeypatch): + name = "snowflake_oauth_422" + _cleanup(name) + try: + _store(_oauth_connection(name)) + client = _get_test_client() + # Never signed in -> resolving credentials for a browse call must 422. + response = client.post( + "/db_schemas", + json={ + "connection_mode": "reference", + "database_connection_name": name, + "query_mode": "table", + }, + ) + if response.status_code == 404: + pytest.skip("no /db_schemas route in this build") + assert response.status_code == 422, response.text + detail = response.json()["detail"] + assert detail["error_code"] == "RECONNECT_REQUIRED" + assert "Reconnect" in detail["message"] + finally: + _cleanup(name) + + +class TestOauthRepointGuard: + @staticmethod + def _guard(changed, has_new_credentials, has_bundled_secrets=True): + from fastapi import HTTPException + + from flowfile_core.routes._connection_sharing import require_credentials_on_target_change + + try: + require_credentials_on_target_change( + changed, has_new_credentials=has_new_credentials, has_bundled_secrets=has_bundled_secrets + ) + except HTTPException: + return False + return True + + def test_endpoint_change_without_credentials_blocked(self): + assert self._guard(["oauth_token_endpoint"], has_new_credentials=False) is False + + def test_endpoint_change_with_new_client_secret_allowed(self): + assert self._guard(["oauth_token_endpoint"], has_new_credentials=True) is True + + +class TestProjectionMarksReauth: + def test_import_of_oauth_connection_warns_and_skips_token(self): + from flowfile_core.project.importer import _import_db_connection + from flowfile_core.project.models import SetupResult + + name = "snowflake_oauth_projected" + _cleanup(name) + try: + result = SetupResult() + data = { + "kind": "database_connection", + "connection_name": name, + "database_type": "snowflake", + "username": "user", + "auth_method": "oauth", + "extra_params": EXTRA_PARAMS, + "password": f"${{secret:{name}}}", + "oauth_client_id": "client-abc", + "oauth_client_secret": f"${{secret:{name}_oauth_client_secret}}", + } + imported = _import_db_connection(data, USER_ID, {}, result) + assert imported == name + assert any("re-authentication" in w for w in result.warnings) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + assert row.auth_method == "oauth" + assert row.oauth_client_id == "client-abc" + assert row.oauth_refresh_token_id is None, "token material must never come from a projection" + finally: + _cleanup(name) + + def test_projection_never_writes_refresh_token(self): + from flowfile_core.project.projection import _db_connection_dict + + name = "snowflake_oauth_project_out" + _cleanup(name) + try: + _store(_oauth_connection(name)) + _sign_in(name, "rt-secret-material") + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + projected = _db_connection_dict(db, row) + flat = str(projected) + assert "rt-secret-material" not in flat + assert "refresh_token" not in flat + assert projected["oauth_client_id"] == "client-abc" + assert projected["oauth_client_secret"] == f"${{secret:{name}_oauth_client_secret}}" + finally: + _cleanup(name) + + +_LIVE_OAUTH_ENV = ("ACCOUNT", "USER", "DATABASE", "WAREHOUSE", "CLIENT_ID", "CLIENT_SECRET", "REFRESH_TOKEN") + +snowflake_oauth_available = pytest.mark.skipif( + not all(os.environ.get(f"FLOWFILE_TEST_SNOWFLAKE_OAUTH_{name}") for name in _LIVE_OAUTH_ENV), + reason="FLOWFILE_TEST_SNOWFLAKE_OAUTH_* credentials are not configured", +) + + +@snowflake_oauth_available +class TestSnowflakeOauthLive: + def test_refresh_and_read_round_trip(self): + """Real security integration: refresh the token, connect, run SELECT 1.""" + from shared.db_dialects import get_dialect + from shared.snowflake_oauth import derive_snowflake_endpoints, refresh_access_token + + env = {name: os.environ[f"FLOWFILE_TEST_SNOWFLAKE_OAUTH_{name}"] for name in _LIVE_OAUTH_ENV} + _, token_endpoint = derive_snowflake_endpoints(env["ACCOUNT"]) + token = refresh_access_token( + token_endpoint, env["CLIENT_ID"], env["CLIENT_SECRET"], env["REFRESH_TOKEN"] + ) + dialect = get_dialect("snowflake") + uri = dialect.build_uri( + username=env["USER"], + database=env["DATABASE"], + account=env["ACCOUNT"], + warehouse=env["WAREHOUSE"], + auth_method="oauth", + oauth_token=token.access_token, + ) + import logging + + df = dialect.read("SELECT 1 AS one", uri, logging.getLogger(__name__)) + assert df["one"].to_list() == [1] diff --git a/flowfile_core/tests/flowfile/external_sources/test_snowflake_source.py b/flowfile_core/tests/flowfile/external_sources/test_snowflake_source.py new file mode 100644 index 000000000..40140cd42 --- /dev/null +++ b/flowfile_core/tests/flowfile/external_sources/test_snowflake_source.py @@ -0,0 +1,518 @@ +"""Snowflake connection-model tests: extra_params CRUD round-trip, the repoint +guard's normalized comparison, and live reads/writes against a real account. + +URI shapes, the type map, and the fakesnow behavioral legs live in +shared/tests/db_dialects/test_snowflake_dialect.py. The live classes here are +gated on FLOWFILE_TEST_SNOWFLAKE_* credentials (account/user/password/database/ +warehouse) per the roadmap's testing decision, so CI skips them. +""" + +import logging +import os +import uuid + +import polars as pl +import pytest +from pydantic import SecretStr, ValidationError + +from flowfile_core.database.connection import get_db_context +from flowfile_core.flowfile.database_connection_manager.db_connections import ( + delete_database_connection, + get_all_database_connections_interface, + get_database_connection, + get_database_connection_schema, + parse_extra_params, + store_database_connection, + update_database_connection, +) +from flowfile_core.routes._connection_sharing import require_credentials_on_target_change +from flowfile_core.schemas.input_schema import ( + DatabaseConnection, + FullDatabaseConnection, + FullDatabaseConnectionInterface, +) +from flowfile_core.secret_manager.secret_manager import decrypt_secret +from shared.db_dialects import get_dialect + +logger = logging.getLogger(__name__) + +_LIVE_ENV = ("ACCOUNT", "USER", "PASSWORD", "DATABASE", "WAREHOUSE") + + +def _live_credentials_configured() -> bool: + return all(os.environ.get(f"FLOWFILE_TEST_SNOWFLAKE_{name}") for name in _LIVE_ENV) + + +snowflake_available = pytest.mark.skipif( + not _live_credentials_configured(), + reason="FLOWFILE_TEST_SNOWFLAKE_* credentials are not configured", +) + +USER_ID = 1 +EXTRA_PARAMS = {"account": "myorg-myaccount", "warehouse": "COMPUTE_WH", "role": "ANALYST"} + + +def _snowflake_connection(name: str, extra_params: dict[str, str] | None = EXTRA_PARAMS) -> FullDatabaseConnection: + return FullDatabaseConnection( + connection_name=name, + database_type="snowflake", + username="user", + password=SecretStr("pass"), + database="ANALYTICS", + extra_params=extra_params, + ) + + +def _cleanup(name: str) -> None: + with get_db_context() as db: + if get_database_connection(db, name, USER_ID) is not None: + delete_database_connection(db, name, USER_ID) + + +class TestExtraParamsCrud: + def test_store_and_reload_round_trips_extra_params(self): + name = "snowflake_crud_store" + _cleanup(name) + try: + with get_db_context() as db: + store_database_connection(db, _snowflake_connection(name), USER_ID) + with get_db_context() as db: + reloaded = get_database_connection_schema(db, name, USER_ID) + assert reloaded is not None + assert reloaded.extra_params == EXTRA_PARAMS + finally: + _cleanup(name) + + def test_update_persists_changed_extra_params(self): + name = "snowflake_crud_update" + _cleanup(name) + try: + with get_db_context() as db: + store_database_connection(db, _snowflake_connection(name), USER_ID) + updated = _snowflake_connection(name, {"account": "other-account"}) + with get_db_context() as db: + update_database_connection(db, updated, USER_ID) + with get_db_context() as db: + reloaded = get_database_connection_schema(db, name, USER_ID) + assert reloaded is not None and reloaded.extra_params == {"account": "other-account"} + finally: + _cleanup(name) + + def test_blocked_extra_params_rejected_at_model_boundary(self): + with pytest.raises(ValidationError, match="extra_params may not override"): + _snowflake_connection("snowflake_evil", {"account": "a", "private_key_file": "/tmp/k.p8"}) + + def test_parse_extra_params_normalizes(self): + assert parse_extra_params(None) is None + assert parse_extra_params("") is None + assert parse_extra_params("not json") is None + assert parse_extra_params("{}") is None + assert parse_extra_params('{"account": "a"}') == {"account": "a"} + + +class TestRepointGuardComparison: + """The PUT route treats a changed extra_params as a target change; the comparison + is normalized (row JSON vs incoming dict) so an unchanged dict never trips it.""" + + @staticmethod + def _changed(row_json: str | None, incoming: dict[str, str] | None) -> bool: + return (parse_extra_params(row_json) or {}) != (incoming or {}) + + def test_unchanged_extra_params_do_not_trip_the_guard(self): + assert self._changed('{"account": "a", "warehouse": "w"}', {"warehouse": "w", "account": "a"}) is False + assert self._changed(None, None) is False + assert self._changed(None, {}) is False + + def test_changed_account_trips_the_guard(self): + assert self._changed('{"account": "a"}', {"account": "attacker"}) is True + assert self._changed(None, {"account": "a"}) is True + + def test_guard_requires_credentials_on_change(self): + from fastapi import HTTPException + + with pytest.raises(HTTPException) as exc: + require_credentials_on_target_change( + ["extra_params"], has_new_credentials=False, has_bundled_secrets=True + ) + assert exc.value.status_code == 422 + require_credentials_on_target_change(["extra_params"], has_new_credentials=True, has_bundled_secrets=True) + require_credentials_on_target_change([], has_new_credentials=False, has_bundled_secrets=True) + + +def _make_pem() -> str: + """A fresh throwaway RSA key as unencrypted PKCS#8 PEM text.""" + from cryptography.hazmat.primitives import serialization + from cryptography.hazmat.primitives.asymmetric import rsa + + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + return key.private_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PrivateFormat.PKCS8, + encryption_algorithm=serialization.NoEncryption(), + ).decode("ascii") + + +def _key_pair_connection( + name: str, private_key: str | None, passphrase: str | None = None +) -> FullDatabaseConnection: + return FullDatabaseConnection( + connection_name=name, + database_type="snowflake", + username="svc_user", + password=SecretStr(""), + database="ANALYTICS", + extra_params={"account": "myorg-myaccount"}, + auth_method="key_pair", + private_key=SecretStr(private_key) if private_key is not None else None, + private_key_passphrase=SecretStr(passphrase) if passphrase is not None else None, + ) + + +class TestKeyPairCrud: + def test_store_and_reload_round_trips_ciphertexts(self): + name = "snowflake_kp_store" + pem = _make_pem() + _cleanup(name) + try: + with get_db_context() as db: + store_database_connection(db, _key_pair_connection(name, pem, "pass-phrase"), USER_ID) + with get_db_context() as db: + reloaded = get_database_connection_schema(db, name, USER_ID) + assert reloaded is not None + assert reloaded.auth_method == "key_pair" + key_ciphertext = reloaded.private_key.get_secret_value() + assert key_ciphertext.startswith("$ffsec$"), "schema must return ciphertext, never the PEM" + assert pem not in key_ciphertext + assert decrypt_secret(key_ciphertext).get_secret_value() == pem + passphrase_ciphertext = reloaded.private_key_passphrase.get_secret_value() + assert passphrase_ciphertext.startswith("$ffsec$") + assert decrypt_secret(passphrase_ciphertext).get_secret_value() == "pass-phrase" + finally: + _cleanup(name) + + def test_create_requires_private_key(self): + _cleanup("snowflake_kp_missing") + with get_db_context() as db: + with pytest.raises(ValueError, match="requires a private key"): + store_database_connection(db, _key_pair_connection("snowflake_kp_missing", None), USER_ID) + + def test_update_rotates_key_and_empty_keeps_existing(self): + name = "snowflake_kp_rotate" + pem = _make_pem() + _cleanup(name) + try: + with get_db_context() as db: + store_database_connection(db, _key_pair_connection(name, pem), USER_ID) + + with get_db_context() as db: + update_database_connection(db, _key_pair_connection(name, None), USER_ID) + with get_db_context() as db: + kept = get_database_connection_schema(db, name, USER_ID).private_key.get_secret_value() + assert decrypt_secret(kept).get_secret_value() == pem, "empty key on update must keep the existing secret" + + new_pem = _make_pem() + with get_db_context() as db: + update_database_connection(db, _key_pair_connection(name, new_pem), USER_ID) + with get_db_context() as db: + rotated = get_database_connection_schema(db, name, USER_ID).private_key.get_secret_value() + assert decrypt_secret(rotated).get_secret_value() == new_pem + finally: + _cleanup(name) + + def test_delete_removes_all_connection_secrets(self): + from flowfile_core.database.models import Secret + + name = "snowflake_kp_delete" + _cleanup(name) + with get_db_context() as db: + store_database_connection(db, _key_pair_connection(name, _make_pem(), "pp"), USER_ID) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + secret_ids = [row.password_id, row.private_key_id, row.private_key_passphrase_id] + assert all(secret_id is not None for secret_id in secret_ids) + with get_db_context() as db: + delete_database_connection(db, name, USER_ID) + with get_db_context() as db: + assert db.query(Secret).filter(Secret.id.in_(secret_ids)).count() == 0 + + def test_switching_to_password_detaches_and_deletes_key_secrets(self): + from flowfile_core.database.models import Secret + + name = "snowflake_kp_flip" + _cleanup(name) + try: + with get_db_context() as db: + store_database_connection(db, _key_pair_connection(name, _make_pem(), "pp"), USER_ID) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + key_secret_ids = [row.private_key_id, row.private_key_passphrase_id] + assert all(secret_id is not None for secret_id in key_secret_ids) + + flipped = FullDatabaseConnection( + connection_name=name, + database_type="snowflake", + username="svc_user", + password=SecretStr("new-pass"), + database="ANALYTICS", + extra_params={"account": "myorg-myaccount"}, + auth_method="password", + ) + with get_db_context() as db: + update_database_connection(db, flipped, USER_ID) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + assert row.private_key_id is None, "a rotated-away key must not linger on the row" + assert row.private_key_passphrase_id is None + assert db.query(Secret).filter(Secret.id.in_(key_secret_ids)).count() == 0 + reloaded = get_database_connection_schema(db, name, USER_ID) + assert reloaded.private_key is None, "resolvers must no longer see any key material" + finally: + _cleanup(name) + + def test_switching_to_password_drops_a_stray_incoming_key(self): + """auth_method="password" with a non-empty private_key in the same payload (e.g. a + client that didn't clear the hidden field): the stray key must be ignored AND the + existing key secrets deleted — build_uri infers key-pair from key presence, so a + persisted stray key would silently out-vote the password.""" + from flowfile_core.database.models import Secret + + name = "snowflake_kp_stray" + _cleanup(name) + try: + with get_db_context() as db: + store_database_connection(db, _key_pair_connection(name, _make_pem(), "pp"), USER_ID) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + old_key_ids = [row.private_key_id, row.private_key_passphrase_id] + + flipped = FullDatabaseConnection( + connection_name=name, + database_type="snowflake", + username="svc_user", + password=SecretStr("new-pass"), + database="ANALYTICS", + extra_params={"account": "myorg-myaccount"}, + auth_method="password", + private_key=SecretStr(_make_pem()), + private_key_passphrase=SecretStr("stray-pass"), + ) + with get_db_context() as db: + update_database_connection(db, flipped, USER_ID) + with get_db_context() as db: + row = get_database_connection(db, name, USER_ID) + assert row.private_key_id is None + assert row.private_key_passphrase_id is None + assert db.query(Secret).filter(Secret.id.in_(old_key_ids)).count() == 0 + reloaded = get_database_connection_schema(db, name, USER_ID) + assert reloaded.private_key is None, "a stray incoming key must never be persisted" + assert reloaded.private_key_passphrase is None + finally: + _cleanup(name) + + def test_update_to_key_pair_without_key_is_rejected(self): + name = "snowflake_kp_bad_flip" + _cleanup(name) + try: + with get_db_context() as db: + store_database_connection(db, _snowflake_connection(name), USER_ID) + with get_db_context() as db: + with pytest.raises(ValueError, match="requires a private key"): + update_database_connection(db, _key_pair_connection(name, None), USER_ID) + finally: + _cleanup(name) + + def test_interface_reports_auth_method_without_key_material(self): + name = "snowflake_kp_interface" + pem = _make_pem() + _cleanup(name) + try: + with get_db_context() as db: + store_database_connection(db, _key_pair_connection(name, pem), USER_ID) + with get_db_context() as db: + interfaces = get_all_database_connections_interface(db, USER_ID) + entry = next(i for i in interfaces if i.connection_name == name) + assert entry.auth_method == "key_pair" + assert pem not in str(entry.model_dump()) + finally: + _cleanup(name) + + +class TestKeyPairModelValidation: + def test_key_pair_rejected_for_non_supporting_dialect(self): + with pytest.raises(ValidationError, match="not supported by database type"): + FullDatabaseConnection( + connection_name="pg_kp", + database_type="postgresql", + username="u", + password=SecretStr(""), + auth_method="key_pair", + private_key=SecretStr("-----BEGIN PRIVATE KEY-----"), + ) + + def test_inline_key_pair_requires_private_key_ref(self): + with pytest.raises(ValidationError, match="requires a private key"): + DatabaseConnection(database_type="snowflake", username="u", auth_method="key_pair") + conn = DatabaseConnection( + database_type="snowflake", username="u", auth_method="key_pair", private_key_ref="kp_secret" + ) + assert conn.private_key_ref == "kp_secret" + + def test_raw_empty_key_strings_normalize_to_none(self): + # JSON clients send "" for the blank form fields; that must not create key secrets. + conn = FullDatabaseConnection( + connection_name="pw_conn", + database_type="postgresql", + username="u", + password=SecretStr("pw"), + private_key="", + private_key_passphrase="", + ) + assert conn.private_key is None + assert conn.private_key_passphrase is None + # The importer's explicit SecretStr("") placeholder rows pass through untouched. + refill = FullDatabaseConnection( + connection_name="kp_refill", + database_type="snowflake", + username="u", + password=SecretStr(""), + extra_params={"account": "acct"}, + auth_method="key_pair", + private_key=SecretStr(""), + ) + assert refill.private_key is not None + + def test_inline_empty_strings_normalize_to_none(self): + conn = DatabaseConnection( + database_type="snowflake", username="u", auth_method="", private_key_ref="", private_key_passphrase_ref="" + ) + assert conn.auth_method is None + assert conn.private_key_ref is None + assert conn.private_key_passphrase_ref is None + + def test_interface_model_never_exposes_key_material(self): + assert "private_key" not in FullDatabaseConnectionInterface.model_fields + assert "private_key_passphrase" not in FullDatabaseConnectionInterface.model_fields + assert "auth_method" in FullDatabaseConnectionInterface.model_fields + + +class TestAuthMethodRepointGuard: + """The PUT route appends auth_method via a normalized comparison (stored NULL == "password").""" + + @staticmethod + def _changed(row_value: str | None, incoming: str | None) -> bool: + return (row_value or "password") != (incoming or "password") + + def test_null_and_password_are_equivalent(self): + assert self._changed(None, "password") is False + assert self._changed("password", None) is False + assert self._changed(None, None) is False + assert self._changed("key_pair", "key_pair") is False + + def test_auth_method_flip_trips_the_guard(self): + from fastapi import HTTPException + + assert self._changed(None, "key_pair") is True + assert self._changed("key_pair", "password") is True + with pytest.raises(HTTPException) as exc: + require_credentials_on_target_change(["auth_method"], has_new_credentials=False, has_bundled_secrets=True) + assert exc.value.status_code == 422 + require_credentials_on_target_change(["auth_method"], has_new_credentials=True, has_bundled_secrets=True) + + +def _live_uri() -> str: + dialect = get_dialect("snowflake") + extra = { + "account": os.environ["FLOWFILE_TEST_SNOWFLAKE_ACCOUNT"], + "warehouse": os.environ["FLOWFILE_TEST_SNOWFLAKE_WAREHOUSE"], + } + if os.environ.get("FLOWFILE_TEST_SNOWFLAKE_ROLE"): + extra["role"] = os.environ["FLOWFILE_TEST_SNOWFLAKE_ROLE"] + if os.environ.get("FLOWFILE_TEST_SNOWFLAKE_SCHEMA"): + extra["schema"] = os.environ["FLOWFILE_TEST_SNOWFLAKE_SCHEMA"] + return dialect.build_uri( + username=os.environ["FLOWFILE_TEST_SNOWFLAKE_USER"], + password=os.environ["FLOWFILE_TEST_SNOWFLAKE_PASSWORD"], + database=os.environ["FLOWFILE_TEST_SNOWFLAKE_DATABASE"], + **extra, + ) + + +@snowflake_available +class TestSnowflakeLive: + def test_write_read_roundtrip_and_fast_schema_parity(self): + dialect = get_dialect("snowflake") + uri = _live_uri() + table = f"ff_test_{uuid.uuid4().hex[:12]}" + df = pl.DataFrame({"id": [1, 2, 3], "score": [0.5, 1.5, 2.5], "label": ["a", "b", "c"]}) + try: + dialect.write(df, uri=uri, table_name=table, if_exists="replace") + dialect.write(df, uri=uri, table_name=table, if_exists="append") + with pytest.raises(ValueError, match="already exists"): + dialect.write(df, uri=uri, table_name=table, if_exists="fail") + dialect.write(df, uri=uri, table_name=table, if_exists="replace") + + result = dialect.read(f"SELECT * FROM {table}", uri, logger) + assert result.height == df.height + assert result.columns == df.columns + + predicted = dialect.query_schema(uri, f"SELECT * FROM {table}") + assert predicted is not None and dict(predicted) == dict(result.schema) + predicted_table = dialect.table_schema(uri, table, None) + assert predicted_table is not None and dict(predicted_table) == dict(result.schema) + finally: + dialect._execute_rows(uri, f"DROP TABLE IF EXISTS {table}") + + def test_browse_lists_schemas_and_tables(self): + dialect = get_dialect("snowflake") + uri = _live_uri() + schemas = dialect.list_schemas(uri) + assert schemas, "expected at least one schema" + + def test_sql_source_schema_prediction(self): + from flowfile_core.flowfile.sources.external_sources.sql_source.sql_source import SqlSource + + source = SqlSource(connection_string=_live_uri(), query="SELECT 1 AS one", database_type="snowflake") + columns = source.get_schema() + assert [c.name for c in columns] == ["ONE"] + + +_KEY_PAIR_LIVE_ENV = ("ACCOUNT", "USER", "DATABASE", "WAREHOUSE") + + +def _live_key_pair_configured() -> bool: + return bool(os.environ.get("FLOWFILE_TEST_SNOWFLAKE_PRIVATE_KEY_PATH")) and all( + os.environ.get(f"FLOWFILE_TEST_SNOWFLAKE_{name}") for name in _KEY_PAIR_LIVE_ENV + ) + + +snowflake_key_pair_available = pytest.mark.skipif( + not _live_key_pair_configured(), + reason="FLOWFILE_TEST_SNOWFLAKE_PRIVATE_KEY_PATH and account credentials are not configured", +) + + +@snowflake_key_pair_available +class TestSnowflakeKeyPairLive: + def test_key_pair_read_round_trip(self): + dialect = get_dialect("snowflake") + with open(os.environ["FLOWFILE_TEST_SNOWFLAKE_PRIVATE_KEY_PATH"], encoding="utf-8") as f: + pem = f.read() + extra = { + "account": os.environ["FLOWFILE_TEST_SNOWFLAKE_ACCOUNT"], + "warehouse": os.environ["FLOWFILE_TEST_SNOWFLAKE_WAREHOUSE"], + } + if os.environ.get("FLOWFILE_TEST_SNOWFLAKE_ROLE"): + extra["role"] = os.environ["FLOWFILE_TEST_SNOWFLAKE_ROLE"] + if os.environ.get("FLOWFILE_TEST_SNOWFLAKE_SCHEMA"): + extra["schema"] = os.environ["FLOWFILE_TEST_SNOWFLAKE_SCHEMA"] + uri = dialect.build_uri( + username=os.environ["FLOWFILE_TEST_SNOWFLAKE_USER"], + database=os.environ["FLOWFILE_TEST_SNOWFLAKE_DATABASE"], + auth_method="key_pair", + private_key=pem, + private_key_passphrase=os.environ.get("FLOWFILE_TEST_SNOWFLAKE_PRIVATE_KEY_PASSPHRASE"), + **extra, + ) + result = dialect.read("SELECT 1 AS one", uri, logger) + assert result.height == 1 + assert result.columns == ["ONE"] diff --git a/flowfile_core/tests/project/test_roundtrip.py b/flowfile_core/tests/project/test_roundtrip.py index 16cb818b7..b64c4d891 100644 --- a/flowfile_core/tests/project/test_roundtrip.py +++ b/flowfile_core/tests/project/test_roundtrip.py @@ -1268,3 +1268,122 @@ def test_flow_writer_projects_portable_namespace(tmp_path): _delete_flow(flow_uuid) _cleanup_custom_namespaces(["WrCat"]) project_sync.close_project(OWNER) + + +def test_database_connection_extra_params_round_trip(tmp_path, monkeypatch): + """A snowflake connection's extra_params (account/warehouse) must survive + project → import → project, not silently drop out of the git projection.""" + project_sync.close_project(OWNER) + conn = "proj_db_snowflake" + extra_params = {"account": "myorg-myaccount", "warehouse": "COMPUTE_WH"} + with get_db_context() as db: + if get_database_connection(db, conn, OWNER) is None: + store_database_connection( + db, + input_schema.FullDatabaseConnection( + connection_name=conn, + database_type="snowflake", + username="etl_reader", + password="s3cr3t", + database="ANALYTICS", + extra_params=extra_params, + ), + OWNER, + ) + root = tmp_path / "project" + try: + project_sync.init_project(str(root), "Snowflake RT", OWNER) + conn_text = (root / "connections" / "database" / f"{conn}.yaml").read_text(encoding="utf-8") + assert "myorg-myaccount" in conn_text + assert "s3cr3t" not in conn_text + + project_sync.close_project(OWNER) + with get_db_context() as db: + delete_database_connection(db, conn, OWNER) + + monkeypatch.setenv("FLOWFILE_SECRET_PROJ_DB_SNOWFLAKE", "s3cr3t") + from flowfile_core.project.importer import import_project + + import_project(root, OWNER) + from flowfile_core.flowfile.database_connection_manager.db_connections import ( + get_database_connection_schema, + ) + + with get_db_context() as db: + restored = get_database_connection_schema(db, conn, OWNER) + assert restored is not None + assert restored.extra_params == extra_params + finally: + _cleanup([conn], []) + + +def test_database_connection_key_pair_round_trip(tmp_path, monkeypatch): + """A key-pair snowflake connection projects auth_method verbatim and the key/passphrase + as ${secret:...} placeholders (never PEM text or ciphertext), the secret manifest keeps + treating the linked key secrets as connection-implied, and import restores a working + connection from FLOWFILE_SECRET_* env vars.""" + import yaml + + project_sync.close_project(OWNER) + conn = "proj_db_snowflake_kp" + pem = "-----BEGIN PRIVATE KEY-----\nkey-material-for-round-trip\n-----END PRIVATE KEY-----\n" + with get_db_context() as db: + if get_database_connection(db, conn, OWNER) is None: + store_database_connection( + db, + input_schema.FullDatabaseConnection( + connection_name=conn, + database_type="snowflake", + username="svc_reader", + password="", + database="ANALYTICS", + extra_params={"account": "myorg-myaccount"}, + auth_method="key_pair", + private_key=pem, + private_key_passphrase="kp-pass", + ), + OWNER, + ) + root = tmp_path / "project" + try: + project_sync.init_project(str(root), "Snowflake KP RT", OWNER) + conn_text = (root / "connections" / "database" / f"{conn}.yaml").read_text(encoding="utf-8") + assert "auth_method: key_pair" in conn_text + assert f"${{secret:{conn}_private_key}}" in conn_text + assert f"${{secret:{conn}_private_key_passphrase}}" in conn_text + assert "BEGIN PRIVATE KEY" not in conn_text + assert "kp-pass" not in conn_text + assert "$ffsec$" not in conn_text + + manifest_data = yaml.safe_load((root / "secrets.yaml").read_text(encoding="utf-8")) + listed = set(manifest_data.get("required_secrets") or []) + assert f"{conn}_private_key" not in listed, "linked key secrets are connection-implied" + assert f"{conn}_private_key_passphrase" not in listed + + project_sync.close_project(OWNER) + with get_db_context() as db: + delete_database_connection(db, conn, OWNER) + + monkeypatch.setenv("FLOWFILE_SECRET_PROJ_DB_SNOWFLAKE_KP", "") + monkeypatch.setenv("FLOWFILE_SECRET_PROJ_DB_SNOWFLAKE_KP_PRIVATE_KEY", pem) + monkeypatch.setenv("FLOWFILE_SECRET_PROJ_DB_SNOWFLAKE_KP_PRIVATE_KEY_PASSPHRASE", "kp-pass") + from flowfile_core.project.importer import import_project + + import_project(root, OWNER) + from flowfile_core.flowfile.database_connection_manager.db_connections import ( + get_database_connection_schema, + ) + from flowfile_core.secret_manager.secret_manager import decrypt_secret + + with get_db_context() as db: + restored = get_database_connection_schema(db, conn, OWNER) + assert restored is not None + assert restored.auth_method == "key_pair" + assert restored.private_key is not None + assert decrypt_secret(restored.private_key.get_secret_value()).get_secret_value() == pem + assert restored.private_key_passphrase is not None + assert ( + decrypt_secret(restored.private_key_passphrase.get_secret_value()).get_secret_value() == "kp-pass" + ) + finally: + _cleanup([conn], []) diff --git a/flowfile_core/tests/test_db_dialect_endpoints.py b/flowfile_core/tests/test_db_dialect_endpoints.py index 8165d894d..5dc346f3f 100644 --- a/flowfile_core/tests/test_db_dialect_endpoints.py +++ b/flowfile_core/tests/test_db_dialect_endpoints.py @@ -43,6 +43,7 @@ def test_get_db_dialects(): assert by_name["sqlite"]["default_port"] is None assert by_name["postgresql"]["default_port"] == 5432 assert by_name["postgresql"]["supports_ssl"] is True + assert by_name["postgresql"]["auth_methods"] == ["password"] assert all(d["available"] is True for d in dialects) @@ -212,6 +213,145 @@ def test_mssql_in_dialect_catalog(): assert mssql_entry["available"] is True +def test_snowflake_in_dialect_catalog(): + response = client.get("/db_dialects") + assert response.status_code == 200, response.text + entry = next(d for d in response.json() if d["name"] == "snowflake") + assert entry["display_name"] == "Snowflake" + assert entry["file_based"] is False + assert entry["default_port"] == 443 + assert entry["available"] is True + assert [f["name"] for f in entry["extra_fields"]] == ["account", "warehouse", "role"] + assert entry["extra_fields"][0]["required"] is True + assert entry["hidden_fields"] == ["host", "port", "ssl"] + assert entry["auth_methods"] == ["password", "key_pair", "oauth"] + + +def test_create_snowflake_connection_with_extra_params(): + _cleanup_connection("snowflake_conn") + payload = { + "connection_name": "snowflake_conn", + "database_type": "snowflake", + "username": "user", + "password": "pass", + "database": "ANALYTICS", + "extra_params": {"account": "myorg-myaccount", "warehouse": "COMPUTE_WH", "role": "ANALYST"}, + } + response = client.post("/db_connection_lib", json=payload) + assert response.status_code == 200, response.text + listed = client.get("/db_connection_lib").json() + entry = next(c for c in listed if c["connection_name"] == "snowflake_conn") + assert entry["extra_params"] == payload["extra_params"] + _cleanup_connection("snowflake_conn") + + +def test_create_connection_rejects_blocked_extra_params(): + payload = { + "connection_name": "snowflake_evil", + "database_type": "snowflake", + "username": "user", + "password": "pass", + "extra_params": {"account": "acct", "authenticator": "externalbrowser"}, + } + response = client.post("/db_connection_lib", json=payload) + assert response.status_code == 422, response.text + assert "authenticator" in response.text + _cleanup_connection("snowflake_evil") + + +def test_create_connection_rejects_auth_method_extra_param(): + payload = { + "connection_name": "snowflake_evil_auth", + "database_type": "snowflake", + "username": "user", + "password": "pass", + "extra_params": {"account": "acct", "auth_method": "key_pair"}, + } + response = client.post("/db_connection_lib", json=payload) + assert response.status_code == 422, response.text + assert "auth_method" in response.text + _cleanup_connection("snowflake_evil_auth") + + +_TEST_PEM = ( + "-----BEGIN PRIVATE KEY-----\nMIIB-not-a-real-key-but-round-trips-fine\n-----END PRIVATE KEY-----\n" +) + + +def test_snowflake_key_pair_connection_http_round_trip(): + _cleanup_connection("snowflake_kp_http") + payload = { + "connection_name": "snowflake_kp_http", + "database_type": "snowflake", + "username": "svc_user", + "password": "", + "database": "ANALYTICS", + "extra_params": {"account": "myorg-myaccount"}, + "auth_method": "key_pair", + "private_key": _TEST_PEM, + } + response = client.post("/db_connection_lib", json=payload) + assert response.status_code == 200, response.text + try: + listed = client.get("/db_connection_lib").json() + entry = next(c for c in listed if c["connection_name"] == "snowflake_kp_http") + assert entry["auth_method"] == "key_pair" + assert "private_key" not in entry, "the interface payload must never carry key material" + assert _TEST_PEM not in str(listed) + + # The OWNER may flip the auth method freely — the anti-repoint guard applies to + # manage-grantees only (helper-level coverage in test_snowflake_source.py). + rotate = {**payload, "auth_method": "password", "private_key": "", "password": "new-pass"} + response = client.put("/db_connection_lib", json=rotate) + assert response.status_code == 200, response.text + listed = client.get("/db_connection_lib").json() + entry = next(c for c in listed if c["connection_name"] == "snowflake_kp_http") + assert entry["auth_method"] == "password" + + # The flip must fully shed the key material — a stale key must never keep authenticating. + from flowfile_core.flowfile.database_connection_manager.db_connections import ( + get_database_connection_schema, + ) + + with get_db_context() as db: + restored = get_database_connection_schema(db, "snowflake_kp_http", 1) + assert restored.private_key is None + assert restored.private_key_passphrase is None + finally: + _cleanup_connection("snowflake_kp_http") + + +def test_create_key_pair_connection_requires_private_key(): + _cleanup_connection("snowflake_kp_incomplete") + payload = { + "connection_name": "snowflake_kp_incomplete", + "database_type": "snowflake", + "username": "svc_user", + "password": "", + "extra_params": {"account": "acct"}, + "auth_method": "key_pair", + } + response = client.post("/db_connection_lib", json=payload) + assert response.status_code == 422, response.text + assert "private key" in response.text + _cleanup_connection("snowflake_kp_incomplete") + + +def test_create_key_pair_connection_rejected_for_password_only_dialect(): + payload = { + "connection_name": "pg_kp_http", + "database_type": "postgresql", + "username": "u", + "password": "", + "host": "h", + "auth_method": "key_pair", + "private_key": _TEST_PEM, + } + response = client.post("/db_connection_lib", json=payload) + assert response.status_code == 422, response.text + _cleanup_connection("pg_kp_http") + + def test_create_duckdb_connection_without_credentials(): _cleanup_connection("duckdb_conn") payload = { diff --git a/flowfile_core/tests/test_migration.py b/flowfile_core/tests/test_migration.py index 76e487b25..82db3f5f3 100644 --- a/flowfile_core/tests/test_migration.py +++ b/flowfile_core/tests/test_migration.py @@ -913,3 +913,211 @@ def test_upgrade_is_guarded_when_the_column_already_exists(self, tmp_path, monke _run_migration(db_path, monkeypatch) assert "scd2_config" in self._columns(db_path, "catalog_tables") + + +# Migration 031: database_connections key-pair auth columns + + +class TestKeyPairColumnsMigration: + _NEW_COLUMNS = ("auth_method", "private_key_id", "private_key_passphrase_id") + + @staticmethod + def _columns(db_path: Path, table: str) -> set[str]: + engine = create_engine(f"sqlite:///{db_path}") + names = {c["name"] for c in inspect(engine).get_columns(table)} + engine.dispose() + return names + + def test_fresh_install_has_the_columns(self, tmp_path, monkeypatch): + db_path = tmp_path / "catalog.db" + _run_migration(db_path, monkeypatch) + columns = self._columns(db_path, "database_connections") + assert set(self._NEW_COLUMNS) <= columns + + def test_upgrade_from_030_adds_the_columns(self, tmp_path, monkeypatch): + from alembic import command + + from flowfile_core.database.migration import _get_alembic_config + + db_path = tmp_path / "catalog.db" + monkeypatch.setenv("FLOWFILE_DB_PATH", str(db_path)) + command.upgrade(_get_alembic_config(), "030") + assert not set(self._NEW_COLUMNS) & self._columns(db_path, "database_connections") + + _run_migration(db_path, monkeypatch) + assert set(self._NEW_COLUMNS) <= self._columns(db_path, "database_connections") + + def test_existing_rows_read_back_null(self, tmp_path, monkeypatch): + """Pre-031 connections become password connections (NULL auth_method), never half-migrated.""" + from alembic import command + + from flowfile_core.database.migration import _get_alembic_config + + db_path = tmp_path / "catalog.db" + monkeypatch.setenv("FLOWFILE_DB_PATH", str(db_path)) + command.upgrade(_get_alembic_config(), "030") + + engine = create_engine(f"sqlite:///{db_path}") + with engine.connect() as conn: + conn.execute( + text( + "INSERT INTO database_connections (connection_name, database_type, username, user_id) " + "VALUES ('legacy_conn', 'postgresql', 'u', 1)" + ) + ) + conn.commit() + engine.dispose() + + _run_migration(db_path, monkeypatch) + + engine = create_engine(f"sqlite:///{db_path}") + with engine.connect() as conn: + row = conn.execute( + text( + "SELECT auth_method, private_key_id, private_key_passphrase_id " + "FROM database_connections WHERE connection_name = 'legacy_conn'" + ) + ).one() + engine.dispose() + assert tuple(row) == (None, None, None) + + def test_downgrade_then_upgrade_round_trips(self, tmp_path, monkeypatch): + from alembic import command + + from flowfile_core.database.migration import _get_alembic_config + + db_path = tmp_path / "catalog.db" + _run_migration(db_path, monkeypatch) + cfg = _get_alembic_config() + + command.downgrade(cfg, "030") + assert not set(self._NEW_COLUMNS) & self._columns(db_path, "database_connections") + + command.upgrade(cfg, "031") + assert set(self._NEW_COLUMNS) <= self._columns(db_path, "database_connections") + + def test_upgrade_is_guarded_when_a_column_already_exists(self, tmp_path, monkeypatch): + """Dev DBs that added a column out of band must not fail the startup upgrade.""" + from alembic import command + + from flowfile_core.database.migration import _get_alembic_config + + db_path = tmp_path / "catalog.db" + monkeypatch.setenv("FLOWFILE_DB_PATH", str(db_path)) + command.upgrade(_get_alembic_config(), "030") + + engine = create_engine(f"sqlite:///{db_path}") + with engine.connect() as conn: + conn.execute(text("ALTER TABLE database_connections ADD COLUMN auth_method VARCHAR")) + conn.commit() + engine.dispose() + + _run_migration(db_path, monkeypatch) + assert set(self._NEW_COLUMNS) <= self._columns(db_path, "database_connections") + + +# Migration 032: database_connections OAuth (SSO) columns + + +class TestOAuthColumnsMigration: + _NEW_COLUMNS = ( + "oauth_client_id", + "oauth_authorize_endpoint", + "oauth_token_endpoint", + "oauth_redirect_uri", + "oauth_client_secret_id", + "oauth_refresh_token_id", + ) + + @staticmethod + def _columns(db_path: Path, table: str) -> set[str]: + engine = create_engine(f"sqlite:///{db_path}") + names = {c["name"] for c in inspect(engine).get_columns(table)} + engine.dispose() + return names + + def test_fresh_install_has_the_columns(self, tmp_path, monkeypatch): + db_path = tmp_path / "catalog.db" + _run_migration(db_path, monkeypatch) + assert set(self._NEW_COLUMNS) <= self._columns(db_path, "database_connections") + + def test_upgrade_from_031_adds_the_columns(self, tmp_path, monkeypatch): + from alembic import command + + from flowfile_core.database.migration import _get_alembic_config + + db_path = tmp_path / "catalog.db" + monkeypatch.setenv("FLOWFILE_DB_PATH", str(db_path)) + command.upgrade(_get_alembic_config(), "031") + assert not set(self._NEW_COLUMNS) & self._columns(db_path, "database_connections") + + _run_migration(db_path, monkeypatch) + assert set(self._NEW_COLUMNS) <= self._columns(db_path, "database_connections") + + def test_existing_rows_read_back_null(self, tmp_path, monkeypatch): + """Pre-032 connections carry no OAuth config; never half-migrated.""" + from alembic import command + + from flowfile_core.database.migration import _get_alembic_config + + db_path = tmp_path / "catalog.db" + monkeypatch.setenv("FLOWFILE_DB_PATH", str(db_path)) + command.upgrade(_get_alembic_config(), "031") + + engine = create_engine(f"sqlite:///{db_path}") + with engine.connect() as conn: + conn.execute( + text( + "INSERT INTO database_connections (connection_name, database_type, username, user_id) " + "VALUES ('legacy_conn_oauth', 'postgresql', 'u', 1)" + ) + ) + conn.commit() + engine.dispose() + + _run_migration(db_path, monkeypatch) + + engine = create_engine(f"sqlite:///{db_path}") + with engine.connect() as conn: + row = conn.execute( + text( + "SELECT oauth_client_id, oauth_token_endpoint, oauth_client_secret_id, oauth_refresh_token_id " + "FROM database_connections WHERE connection_name = 'legacy_conn_oauth'" + ) + ).one() + engine.dispose() + assert tuple(row) == (None, None, None, None) + + def test_downgrade_then_upgrade_round_trips(self, tmp_path, monkeypatch): + from alembic import command + + from flowfile_core.database.migration import _get_alembic_config + + db_path = tmp_path / "catalog.db" + _run_migration(db_path, monkeypatch) + cfg = _get_alembic_config() + + command.downgrade(cfg, "031") + assert not set(self._NEW_COLUMNS) & self._columns(db_path, "database_connections") + + command.upgrade(cfg, "032") + assert set(self._NEW_COLUMNS) <= self._columns(db_path, "database_connections") + + def test_upgrade_is_guarded_when_a_column_already_exists(self, tmp_path, monkeypatch): + """Dev DBs that added a column out of band must not fail the startup upgrade.""" + from alembic import command + + from flowfile_core.database.migration import _get_alembic_config + + db_path = tmp_path / "catalog.db" + monkeypatch.setenv("FLOWFILE_DB_PATH", str(db_path)) + command.upgrade(_get_alembic_config(), "031") + + engine = create_engine(f"sqlite:///{db_path}") + with engine.connect() as conn: + conn.execute(text("ALTER TABLE database_connections ADD COLUMN oauth_client_id VARCHAR")) + conn.commit() + engine.dispose() + + _run_migration(db_path, monkeypatch) + assert set(self._NEW_COLUMNS) <= self._columns(db_path, "database_connections") diff --git a/flowfile_frame/flowfile_frame/database/connection_manager.py b/flowfile_frame/flowfile_frame/database/connection_manager.py index d33bd544e..8eda19eac 100644 --- a/flowfile_frame/flowfile_frame/database/connection_manager.py +++ b/flowfile_frame/flowfile_frame/database/connection_manager.py @@ -40,13 +40,17 @@ def create_database_connection( password: str | SecretStr | None = None, ssl_enabled: bool = False, url: str | None = None, + extra_params: dict[str, str] | None = None, + auth_method: str | None = None, + private_key: str | SecretStr | None = None, + private_key_passphrase: str | SecretStr | None = None, ) -> FullDatabaseConnection: """Create and store a new database connection. Args: connection_name: Unique name for this connection. database_type: Type of database (one of shared.db_dialects.KNOWN_DIALECT_NAMES, - e.g. postgresql, mysql, sqlite, duckdb, mssql). + e.g. postgresql, mysql, sqlite, duckdb, mssql, snowflake). host: Database server hostname. port: Database server port. database: Database name. @@ -54,13 +58,22 @@ def create_database_connection( password: Database password (not needed for file-based types like sqlite/duckdb). ssl_enabled: Whether to use SSL for the connection. url: Full database URL (overrides other connection parameters). + extra_params: Dialect-specific connection parameters, e.g. for Snowflake + ``{"account": "myorg-myaccount", "warehouse": "COMPUTE_WH", "role": "ANALYST"}``. + Keys that could override credentials (user, password, host, ...) are rejected. + auth_method: Authentication method; must be supported by the dialect + (``"password"`` everywhere, ``"key_pair"`` for Snowflake JWT auth). + private_key: Private key PEM *text* (never a path) for key-pair auth, e.g. + ``open("rsa_key.p8").read()``. Stored as an encrypted secret. + private_key_passphrase: Optional passphrase when the PEM is encrypted. Returns: FullDatabaseConnection: The created connection object. Raises: - ValueError: If a connection with this name already exists, or the - database_type is not a supported dialect. + ValueError: If a connection with this name already exists, the + database_type is not a supported dialect, or key-pair auth is + requested without a private key. """ if database_type.lower() not in KNOWN_DIALECT_NAMES: raise ValueError( @@ -70,11 +83,17 @@ def create_database_connection( if isinstance(password, str): password = SecretStr(password) + if isinstance(private_key, str): + private_key = SecretStr(private_key) + if isinstance(private_key_passphrase, str): + private_key_passphrase = SecretStr(private_key_passphrase) if get_dialect_or_generic(database_type).file_based: # No credentials for file-based databases; the stored model requires strings username = username or "" password = password if password is not None else SecretStr("") + if auth_method == "key_pair" and password is None: + password = SecretStr("") connection = FullDatabaseConnection( connection_name=connection_name, @@ -86,6 +105,10 @@ def create_database_connection( password=password, ssl_enabled=ssl_enabled, url=url, + extra_params=extra_params, + auth_method=auth_method, + private_key=private_key, + private_key_passphrase=private_key_passphrase, ) with get_db_context() as db: @@ -105,13 +128,17 @@ def create_database_connection_if_not_exists( password: str | SecretStr | None = None, ssl_enabled: bool = False, url: str | None = None, + extra_params: dict[str, str] | None = None, + auth_method: str | None = None, + private_key: str | SecretStr | None = None, + private_key_passphrase: str | SecretStr | None = None, ) -> FullDatabaseConnection: """Create a database connection if it doesn't already exist. Args: connection_name: Unique name for this connection. database_type: Type of database (one of shared.db_dialects.KNOWN_DIALECT_NAMES, - e.g. postgresql, mysql, sqlite, duckdb, mssql). + e.g. postgresql, mysql, sqlite, duckdb, mssql, snowflake). host: Database server hostname. port: Database server port. database: Database name. @@ -119,6 +146,10 @@ def create_database_connection_if_not_exists( password: Database password (not needed for file-based types like sqlite/duckdb). ssl_enabled: Whether to use SSL for the connection. url: Full database URL (overrides other connection parameters). + extra_params: Dialect-specific connection parameters (see create_database_connection). + auth_method: Authentication method (see create_database_connection). + private_key: Private key PEM text for key-pair auth (see create_database_connection). + private_key_passphrase: Optional passphrase when the PEM is encrypted. Returns: FullDatabaseConnection: The existing or newly created connection. @@ -139,6 +170,10 @@ def create_database_connection_if_not_exists( password=password, ssl_enabled=ssl_enabled, url=url, + extra_params=extra_params, + auth_method=auth_method, + private_key=private_key, + private_key_passphrase=private_key_passphrase, ) @@ -163,6 +198,7 @@ def get_all_available_database_connections() -> list[FullDatabaseConnectionInter List of database connection interfaces (without passwords). """ from flowfile_core.database.models import DatabaseConnection as DBConnectionModel + from flowfile_core.flowfile.database_connection_manager.db_connections import parse_extra_params user_id = get_current_user_id() with get_db_context() as db: @@ -177,6 +213,8 @@ def get_all_available_database_connections() -> list[FullDatabaseConnectionInter port=conn.port, database=conn.database, ssl_enabled=conn.ssl_enabled, + extra_params=parse_extra_params(conn.extra_params), + auth_method=conn.auth_method, ) for conn in connections ] @@ -197,11 +235,15 @@ def del_database_connection(connection_name: str) -> bool: with get_db_context() as db: connection = get_database_connection(db, connection_name, user_id) if connection: - # Delete the associated password secret - if connection.password_id: - secret = db.query(Secret).filter(Secret.id == connection.password_id).first() - if secret: - db.delete(secret) + # Delete every associated secret (password + key-pair material) + all_secret_ids = ( + connection.password_id, + connection.private_key_id, + connection.private_key_passphrase_id, + ) + secret_ids = [secret_id for secret_id in all_secret_ids if secret_id is not None] + if secret_ids: + db.query(Secret).filter(Secret.id.in_(secret_ids)).delete(synchronize_session=False) db.delete(connection) db.commit() diff --git a/flowfile_frame/flowfile_frame/database/connection_manager.pyi b/flowfile_frame/flowfile_frame/database/connection_manager.pyi index cd58bfa5d..42bd8897a 100644 --- a/flowfile_frame/flowfile_frame/database/connection_manager.pyi +++ b/flowfile_frame/flowfile_frame/database/connection_manager.pyi @@ -6,8 +6,8 @@ from pydantic import SecretStr from flowfile_core.schemas.input_schema import FullDatabaseConnection, FullDatabaseConnectionInterface def get_current_user_id() -> int: ... -def create_database_connection(connection_name: str, *, database_type: str='postgresql', host: str | None=None, port: int | None=None, database: str | None=None, username: str | None=None, password: str | SecretStr | None=None, ssl_enabled: bool=False, url: str | None=None) -> FullDatabaseConnection: ... -def create_database_connection_if_not_exists(connection_name: str, *, database_type: str='postgresql', host: str | None=None, port: int | None=None, database: str | None=None, username: str | None=None, password: str | SecretStr | None=None, ssl_enabled: bool=False, url: str | None=None) -> FullDatabaseConnection: ... +def create_database_connection(connection_name: str, *, database_type: str='postgresql', host: str | None=None, port: int | None=None, database: str | None=None, username: str | None=None, password: str | SecretStr | None=None, ssl_enabled: bool=False, url: str | None=None, extra_params: dict[str, str] | None=None, auth_method: str | None=None, private_key: str | SecretStr | None=None, private_key_passphrase: str | SecretStr | None=None) -> FullDatabaseConnection: ... +def create_database_connection_if_not_exists(connection_name: str, *, database_type: str='postgresql', host: str | None=None, port: int | None=None, database: str | None=None, username: str | None=None, password: str | SecretStr | None=None, ssl_enabled: bool=False, url: str | None=None, extra_params: dict[str, str] | None=None, auth_method: str | None=None, private_key: str | SecretStr | None=None, private_key_passphrase: str | SecretStr | None=None) -> FullDatabaseConnection: ... def get_database_connection_by_name(connection_name: str) -> FullDatabaseConnection | None: ... def get_all_available_database_connections() -> list[FullDatabaseConnectionInterface]: ... def del_database_connection(connection_name: str) -> bool: ... diff --git a/flowfile_frontend/src/renderer/app/api/dbDialects.test.ts b/flowfile_frontend/src/renderer/app/api/dbDialects.test.ts index 6acef794b..fc2f445b1 100644 --- a/flowfile_frontend/src/renderer/app/api/dbDialects.test.ts +++ b/flowfile_frontend/src/renderer/app/api/dbDialects.test.ts @@ -80,6 +80,31 @@ describe("getDbDialects", () => { "sqlite", "duckdb", "mssql", + "snowflake", ]); }); + + it("carries the snowflake form metadata in the fallback entry", async () => { + const { FALLBACK_DIALECTS } = await loadModule(); + const snowflake = FALLBACK_DIALECTS.find((dialect) => dialect.name === "snowflake"); + expect(snowflake?.extra_fields?.map((field) => field.name)).toEqual([ + "account", + "warehouse", + "role", + ]); + expect(snowflake?.extra_fields?.[0].required).toBe(true); + expect(snowflake?.hidden_fields).toEqual(["host", "port", "ssl"]); + }); + + it("carries auth methods in every fallback entry, key_pair only on snowflake", async () => { + const { FALLBACK_DIALECTS } = await loadModule(); + for (const dialect of FALLBACK_DIALECTS) { + expect(dialect.auth_methods, dialect.name).toContain("password"); + if (dialect.name === "snowflake") { + expect(dialect.auth_methods).toEqual(["password", "key_pair", "oauth"]); + } else { + expect(dialect.auth_methods).toEqual(["password"]); + } + } + }); }); diff --git a/flowfile_frontend/src/renderer/app/api/dbDialects.ts b/flowfile_frontend/src/renderer/app/api/dbDialects.ts index 19604da7f..6acbe4d67 100644 --- a/flowfile_frontend/src/renderer/app/api/dbDialects.ts +++ b/flowfile_frontend/src/renderer/app/api/dbDialects.ts @@ -1,5 +1,11 @@ import axios from "axios"; +export interface DbDialectFieldInfo { + name: string; + label: string; + required: boolean; +} + export interface DbDialectInfo { name: string; display_name: string; @@ -7,6 +13,12 @@ export interface DbDialectInfo { default_port: number | null; supports_ssl: boolean; available: boolean; + // Dialect-specific connection fields (stored in extra_params) and standard + // form fields the dialect hides; absent on older cores, so keep optional. + extra_fields?: DbDialectFieldInfo[]; + hidden_fields?: string[]; + // Supported authentication methods (e.g. ["password", "key_pair"]); absent on older cores. + auth_methods?: string[]; } // Rendered when the catalog request fails (offline, older core). Must mirror the @@ -19,6 +31,9 @@ export const FALLBACK_DIALECTS: DbDialectInfo[] = [ default_port: 5432, supports_ssl: true, available: true, + extra_fields: [], + hidden_fields: [], + auth_methods: ["password"], }, { name: "mysql", @@ -27,6 +42,9 @@ export const FALLBACK_DIALECTS: DbDialectInfo[] = [ default_port: 3306, supports_ssl: false, available: true, + extra_fields: [], + hidden_fields: [], + auth_methods: ["password"], }, { name: "sqlite", @@ -35,6 +53,9 @@ export const FALLBACK_DIALECTS: DbDialectInfo[] = [ default_port: null, supports_ssl: false, available: true, + extra_fields: [], + hidden_fields: [], + auth_methods: ["password"], }, { name: "duckdb", @@ -43,6 +64,9 @@ export const FALLBACK_DIALECTS: DbDialectInfo[] = [ default_port: null, supports_ssl: false, available: true, + extra_fields: [], + hidden_fields: [], + auth_methods: ["password"], }, { name: "mssql", @@ -51,6 +75,24 @@ export const FALLBACK_DIALECTS: DbDialectInfo[] = [ default_port: 1433, supports_ssl: false, available: true, + extra_fields: [], + hidden_fields: [], + auth_methods: ["password"], + }, + { + name: "snowflake", + display_name: "Snowflake", + file_based: false, + default_port: 443, + supports_ssl: false, + available: true, + extra_fields: [ + { name: "account", label: "Account", required: true }, + { name: "warehouse", label: "Warehouse", required: false }, + { name: "role", label: "Role", required: false }, + ], + hidden_fields: ["host", "port", "ssl"], + auth_methods: ["password", "key_pair", "oauth"], }, ]; diff --git a/flowfile_frontend/src/renderer/app/components/nodes/node-types/elements/databaseReader/DatabaseConnectionSettings.vue b/flowfile_frontend/src/renderer/app/components/nodes/node-types/elements/databaseReader/DatabaseConnectionSettings.vue index d9c771274..a45efb978 100644 --- a/flowfile_frontend/src/renderer/app/components/nodes/node-types/elements/databaseReader/DatabaseConnectionSettings.vue +++ b/flowfile_frontend/src/renderer/app/components/nodes/node-types/elements/databaseReader/DatabaseConnectionSettings.vue @@ -15,9 +15,7 @@ id="database-type" :value="modelValue.database_type" class="form-control" - @change=" - (e: Event) => updateField('database_type', (e.target as HTMLSelectElement).value) - " + @change="(e: Event) => updateDatabaseType((e.target as HTMLSelectElement).value)" >