Skip to content

feat(sfdc_formula_view): add query_engine inference and use run_query for non-Redshift adapters#136

Open
fivetran-jacklowery wants to merge 1 commit into
mainfrom
feature/query-engine-inference-catalog-linked-snowflake
Open

feat(sfdc_formula_view): add query_engine inference and use run_query for non-Redshift adapters#136
fivetran-jacklowery wants to merge 1 commit into
mainfrom
feature/query-engine-inference-catalog-linked-snowflake

Conversation

@fivetran-jacklowery

Copy link
Copy Markdown

Summary

This PR updates sfdc_formula_view to select the generated formula model by query engine and avoid a relation-existence check that can fail for Snowflake catalog-linked databases.

The Salesforce connector produces multiple rows in fivetran_formula_model for the same Salesforce object — one per query engine. The previous macro filtered only by object and used dbt_utils.get_column_values, whose load_relation check can return None for catalog-linked (Iceberg/Managed Data Lake) sources even when direct SQL against the source succeeds.

Changes:

  • Adds an optional query_engine argument to sfdc_formula_view
  • Adds a salesforce_formula_query_engine project var as a fallback
  • Infers query engine from target.type if neither is provided (snowflakeSnowflake, bigqueryBigQuery, databricks/sparkDatabricks, redshiftRedshift, else Generic)
  • Replaces dbt_utils.get_column_values on the non-Redshift path with a direct run_query call that filters by both object and query_engine
  • Preserves all existing Redshift model_large / SUPER behavior unchanged
  • Preserves the materialization argument; documents that catalog-linked Snowflake databases require materialization='table' because view creation is unsupported

Validation

Smoke-tested against a Snowflake catalog-linked database backed by Fivetran Managed Data Lake using fivetran/dbt_snowflake_iceberg_writer:

dbt run --select smoke_salesforce_formula_account smoke_salesforce_formula_user_role
# 2 pass, 0 fail

Both models materialized as BASE TABLE relations against lowery_linked_writer.dbt_core_dev.

Test plan

  • Confirm dbt parse passes with updated macro
  • Confirm dbt compile --select smoke_salesforce_formula_account smoke_salesforce_formula_user_role passes
  • Confirm dbt run --select smoke_salesforce_formula_account smoke_salesforce_formula_user_role passes with materialization='table' on Snowflake catalog-linked database
  • Confirm existing Redshift integration tests still pass
  • Add/update integration tests: multiple query_engine rows for same object selects adapter-specific row; explicit query_engine override works; default adapter inference works

🤖 Generated with Claude Code

… for non-Redshift adapters

Adds an optional `query_engine` argument (with `salesforce_formula_query_engine` var
fallback and adapter-based inference) so the macro correctly filters
`fivetran_formula_model` by both object and query engine.

Replaces `dbt_utils.get_column_values` on the non-Redshift path with a direct
`run_query` call, avoiding `load_relation` existence checks that silently return
None for Snowflake catalog-linked (Iceberg) sources even when direct SQL succeeds.
Redshift model_large / SUPER behavior is preserved unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant