blastradius

Column-level lineage that does not stop at the edge of the warehouse. Ask what breaks if you change a column, and get back the downstream models and the application code that reads them.

The problem

Someone opens a pull request that drops a column. It is unused, as far as anyone can tell. dbt agrees: nothing downstream references it. The tests pass, it merges, and on Monday the field team's quota report is empty and the nightly push into the CRM has been writing nulls for two days.

The column was not unused. It was read by a pandas script, a tablet app endpoint and an export job — none of which are dbt models, so none of which dbt can see.

This is the ordinary shape of a data incident, and the tools do not cover it. SQL parsers like dbt, sqlglot and SQLFlow see inside the warehouse and stop at its boundary. Runtime capture tools like OpenLineage see whatever passed through an instrumented orchestrator, which DataHub's own documentation is candid about: logic in ad-hoc scripts that never runs through the orchestrator stays invisible. And a runtime tool answers in the past tense anyway, while the question — what will break if I do this — is asked before the change ships.

So the code most likely to break is the code least likely to be covered.

The approach

Parse both sides and join them into one graph.

SQL gives the lineage inside the warehouse: DDL for the schema, model SQL for column-to-column derivation, with SELECT * resolved against the schema and CTE chains walked rather than collapsed. Python gives the other half: string SQL, df["col"] subscripts, df.col attribute access, and column names in groupby, agg and sort_values. Once both live in the same graph, "what breaks" is a reachability query.

Nothing is imported and no query is executed. Analysis is entirely static, so the tool runs on a repository the moment it is cloned, with no warehouse credentials and no orchestrator. See ADR 1.

What it looks like

$ blastradius impact fixtures/pharma raw.territories.target_quota

raw.territories.target_quota: 6 downstream dependencies

warehouse columns derived from it
    analytics.dim_prescriber.target_quota
    analytics.mart_territory_performance.attainment_pct
    analytics.mart_territory_performance.target_quota

application code that reads it
    app/api/endpoints.py
        attainment_pct:5   attainment_pct
    app/etl/export_to_crm.py
        target_quota:29    subscript 'target_quota'
    app/reporting/quota_report.py
        attainment_pct:16  subscript 'attainment_pct'
        target_quota:15    subscript 'target_quota'

might also be affected, column list not statically resolvable
    app/etl/export_to_crm.py       (select_star)
    app/reporting/quota_report.py  (select_star)

The top block is what dbt already tells you. The bottom two are the reason this exists.

Accuracy

Measured against 30 hand-labelled column references in the fixture project. The labels were written by reading the source, not by running the tool.

references	precision	recall	tp / fp / fn
resolvable statically	100%	100%	27 / 0 / 0
not statically resolvable	0%	0%	0 / 0 / 3
all	100%	90%	27 / 0 / 3

The split is the honest part. Three references are labelled unrecoverable by static analysis: one column name arrives as a function argument, two are dict keys iterated at runtime. Reporting a single blended 90% would hide which half the failures are in.

Zero false positives matters more than the recall number. A missing reference shows up as an unresolved warning the user can see. A fabricated one is invisible, and it makes a blast radius look complete when it is not — so the resolver declines to attribute a column whenever two joined relations both define it, rather than picking one. test_detector_produces_no_false_positives guards this.

It works on real projects

The fixture is hand-written flat SQL. Real dbt models are Jinja over CTE chains, and CI clones dbt-labs/jaffle-shop-classic on every push to check that analytics.stg_payments.amount still reaches analytics.customers.customer_lifetime_value through five intermediate nodes.

That check exists because the first version scored a green test suite, a clean type check, and zero columns on all five of those models. Details in ADR 2.

Running it

Needs uv. Nothing else — no database, no API key.

make install

blastradius scan     fixtures/pharma
blastradius impact   fixtures/pharma raw.territories.target_quota
blastradius impact   fixtures/pharma raw.prescriptions.quantity --json
blastradius evaluate fixtures/pharma

On your own project, point it at a directory containing warehouse/ (DDL plus a models/ subdirectory) and app/.

Docker, against a mounted repository:

docker build -t blastradius .
docker run --rm -v "$PWD:/work" blastradius impact /work raw.orders.status

The data

fixtures/pharma is a synthetic pharmaceutical commercial-analytics warehouse: four raw tables, four models, and three application modules that read the marts the way real code does — a SQL constant, an implicitly concatenated string, an f-string with an interpolated ORDER BY, pandas subscripts and attribute access, and a loop over a dict of column names.

ground_truth.yaml labels all 30 references by hand with a difficulty tag, plus the mart columns nothing reads, so false positives are measurable and not just absent from the report.

Tests

make check   # ruff, mypy, pytest

42 tests, 95% line coverage. Most of them check that the tool stays quiet when it should rather than that it finds things, because over-reporting is the failure mode that actually hurts here.

The ones that matter most:

• test_pandas_methods_are_not_mistaken_for_columns — df.groupby and df.new_starts are identical syntax; only one is a column.
• test_ambiguous_unqualified_column_is_not_guessed — when two joined tables both define id, no edge is emitted.
• test_star_expansion_ignores_tables_inside_ctes — the bug that gave a real model 39 columns instead of 7.
• test_models_resolve_regardless_of_filename_order — customers sorts before the models it depends on.

Known limitations

• Python only. A warehouse read from Scala, R, a Looker view or a notebook is invisible.
• Column names built at runtime cannot be recovered. They are reported as unresolved rather than guessed, but a user who ignores those warnings gets an understated radius.
• SELECT * in application code marks the consumer as depending on the whole relation. That is correct and it is coarse.
• dbt Jinja is substituted with a regex, not compiled. ref, source and config are handled; a macro that generates SQL structure is flagged and left as a placeholder. dbt compile first if you need exactness.
• One SQL dialect at a time, defaulting to Postgres. Warehouse-specific syntax will need the dialect set.
• No incremental mode. Everything is reparsed on each run, which is fine at fixture and jaffle-shop scale and untested on a thousand-model project.

Insights and learning

I picked this because the tooling gap is unusually clean. Column-level lineage is a solved problem inside the warehouse and an unsolved one a single step outside it, and the step outside is where the incidents happen. Every tool I looked at either parses SQL and stops at the schema boundary, or watches an orchestrator and therefore only sees the jobs somebody already thought worth orchestrating. The pandas script that a analyst wrote and scheduled with cron is in neither category, and it is exactly the thing that breaks.

The design decision I care about is refusing to guess. It would have been easy to attribute an unqualified column to the first joined table that defines it, and the recall number would have gone up. It would also have made the tool actively dangerous, because the entire value proposition is a person reading the output and concluding it is safe to ship. A missing reference is visible — it shows up as an unresolved query the user can investigate. An invented one is invisible and it tells them the coast is clear. So the resolver stays silent on ambiguity and the test suite has more assertions about what the tool must not report than about what it must find.

The part I did not expect was how badly my own fixture misled me. I wrote the warehouse fixture and the parser, the tests passed, mypy was clean, coverage was 75%, and the accuracy score against my own hand-labelled ground truth was a perfect 100% on everything statically resolvable. Then I cloned jaffle-shop-classic, which is the most standard dbt project in existence, and got zero columns out of all five models. Four separate bugs: Jinja was not SQL, the models were CTE chains rather than flat SELECTs, filename order had nothing to do with dependency order, and my star expansion was walking into CTE bodies and inflating one model from 7 columns to 39.

None of those were subtle, and none of them could have been caught by the fixture, because I wrote the fixture with the same assumptions I wrote the parser with. A test suite authored alongside the code tests the code against its author's mental model, and mine was wrong in four places at once. That is now a CI job that clones somebody else's repository on every push, which is the only part of the test suite that can tell me something I did not already believe.

If I took this further, the next thing is not more patterns. It is constant propagation inside a module, which would recover two of the three references I currently miss — both are dict keys defined a few lines above the loop that uses them, and a human reading the file resolves them instantly. After that, other languages, since Python-only covers a fraction of what actually reads a warehouse, and a --fail-on-impact mode so this can sit in CI and block the pull request that drops the column rather than explaining the incident afterwards.

License

MIT