Use polars.scan_parquet

[mrocklin]

cc @ritchie46

[mrocklin]

This is still not great. Performance is somewhat slow and I'm getting errors I don't yet understand on query 2:

  File "/Users/mrocklin/workspace/benchmarks/tests/tpch/conftest.py", line 349, in _run
  File "/Users/mrocklin/workspace/benchmarks/tests/tpch/test_polars.py", line 152, in _
  File "/opt/coiled/env/lib/python3.11/site-packages/polars/utils/deprecation.py", line 96, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/coiled/env/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 1788, in collect
    return wrap_df(ldf.collect())
                   ^^^^^^^^^^^^^

PanicException('not yet supported')

  File "/Users/mrocklin/workspace/benchmarks/tests/tpch/conftest.py", line 349, in _run
  File "/Users/mrocklin/workspace/benchmarks/tests/tpch/test_polars.py", line 115, in _
  File "/opt/coiled/env/lib/python3.11/site-packages/polars/utils/deprecation.py", line 96, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/coiled/env/lib/python3.11/site-packages/polars/lazyframe/frame.py", line 1788, in collect
    return wrap_df(ldf.collect())
                   ^^^^^^^^^^^^^

ShapeError('unable to append to a dataframe of width 4 with a dataframe of width 9')

@ritchie46 I'm going to add you to this Coiled team in case you want to look at these exceptions in context (and look at performance metrics). No obligation to use it though if you don't want to.

[mrocklin]

@ritchie46 invite sent.

Cluster link: https://cloud.coiled.io/clusters/299556/information?account=dask-benchmarks&computation=