Add pipelines preview-lineage command - #2
Open
tanishgupta-db wants to merge 4 commits into
Open
Conversation
tanishgupta-db
force-pushed
the
pipelines_lineage_command
branch
from
July 9, 2026 21:37
3cb0023 to
fd896c9
Compare
tanishgupta-db
force-pushed
the
pipelines_datasets_command
branch
from
July 9, 2026 22:50
25834f6 to
a2e43e6
Compare
tanishgupta-db
force-pushed
the
pipelines_lineage_command
branch
from
July 9, 2026 22:52
fd896c9 to
ed56284
Compare
tanishgupta-db
force-pushed
the
pipelines_datasets_command
branch
from
July 9, 2026 23:04
a2e43e6 to
7dcddc1
Compare
tanishgupta-db
force-pushed
the
pipelines_lineage_command
branch
from
July 9, 2026 23:12
ed56284 to
59e23b7
Compare
tanishgupta-db
force-pushed
the
pipelines_datasets_command
branch
from
July 10, 2026 20:52
7dcddc1 to
eee1577
Compare
tanishgupta-db
force-pushed
the
pipelines_lineage_command
branch
2 times, most recently
from
July 10, 2026 23:59
41c244b to
53e7de0
Compare
tanishgupta-db
force-pushed
the
pipelines_datasets_command
branch
from
July 11, 2026 00:00
d1d163a to
57f5a7c
Compare
tanishgupta-db
force-pushed
the
pipelines_lineage_command
branch
from
July 11, 2026 00:00
53e7de0 to
097998c
Compare
tanishgupta-db
force-pushed
the
pipelines_datasets_command
branch
from
July 21, 2026 22:58
57f5a7c to
e3460b2
Compare
tanishgupta-db
force-pushed
the
pipelines_lineage_command
branch
from
July 21, 2026 22:58
097998c to
3d33d8a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds a hidden
databricks pipelines preview-lineagecommand that shows a table's upstream and downstream datasets within a pipeline. It reuses the same dry-run resolution and graph-fetching aspreview-datasets, then walks the graph's flows to compute lineage for the requested table. Supports the same--force-dry-run/--no-dry-run/--timeoutflags and text/JSON output.Why
Lineage answers "what feeds this table, and what does it feed?" from the same dry-run graph that powers
preview-datasets, so it shares that machinery rather than duplicating it.Key decisions:
unresolved_refscountFuture work (not in this PR): figure out how to surface external sources; table name duplication across nodes --> can it happen and how to handle.
Tests
Unit tests for the lineage computation (upstream/downstream resolution, multi-input flows, sink handling, unresolved-ref counting deduped across directions) and rendering (text + JSON, with and without unresolved refs); a full-flow test over the fake pipeline API covering the sink-downstream path; and acceptance tests running the real CLI against the fake on both deployment engines.