fix: use full table schema when producing the prune map - #380
Open
gruuya wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Instead of the partition schema, which contains only a subset of columns for partitioned tables in general, or no columns at all for unpartitioned tables, compute the prune map with the full table arrow schema instead.
Note that this didn't impact correctness, since row filtering was still applied correctly and moreover the files were pruned dynamically after inspecting their stats during execution. it's just that unpartitioned tables got zero plan-time file pruning and partitioned tables got none on non-partition columns.
Instead, now we just skip processing those files at plan time altogether, which can be especially useful for tables with many files (previously we payed a footer fetch during execution for each one just to discard it based on the stats).