Fix filtering on parquet file containing a struct column#11665
Fix filtering on parquet file containing a struct column#11665
Conversation
Unit Test ResultsSee test report for an extended history of previous test failures. This is useful for diagnosing flaky tests. 15 files ± 0 15 suites ±0 4h 28m 29s ⏱️ - 1m 25s Results for commit f09cc81. ± Comparison against base commit 1e6ed84. |
|
Thanks! I did not know that we aren't supporting filtering on struct columns, do you know if we pay attention not to push filters into read_parquet if they are applied to a struct column? |
Yeah, I honestly hadn't thought much about filtering on a struct column until today :/ |
Dask does not support filtering on a PyArrow struct column, but it should support filtering on other columns when a struct column is present. This PR fixes a bug in the way we align field names with field indices when struct columns are present.
pre-commit run --all-files