feat: Add spark-compat mode to integrate datafusion-spark features au…

[mattcuento]

…tomatically

Which issue does this PR close?

Closes #1397.

Rationale for this change

Exposing datafusion-spark functions as a 'bundled' feature with Ballista. This simplifies augmenting Ballista with spark features for quicker and more simple experimentation/adoption.

Documentation has been added in the user guide to describe how to enable these functions in builds.

What changes are included in this PR?

• Introduce spark-compat feature to register all spark scalar/agg/window/table functions by default
• Created helper datafusion_x_functions to register default (and spark) functions if applicable in the SessionState for scheduler/client usage
• Extended most usages of session state builder to register functions using the helper
• Register spark functions if applicable in the executor via BallistaFunctionRegistry::default()

I've rendered the docs locally to ensure all looks well.

Are there any user-facing changes?

• New feature flag spark-compat to automatically register Spark-compatible scalar, aggregate, and window functions from datafusion-spark

[mattcuento]

I'm not certain that we have a well-defined place to document compile-time features, just docs for runtime configurations. I'd like to follow this up with a markdown file to describe our features

[milenkovicm]

I'm not certain that we have a well-defined place to document compile-time features, just docs for runtime configurations. I'd like to follow this up with a markdown file to describe our features

maybe section in the readme file?

[mattcuento]

I'm not certain that we have a well-defined place to document compile-time features, just docs for runtime configurations. I'd like to follow this up with a markdown file to describe our features

maybe section in the readme file?

Threw up a separate PR: https://github.com/apache/datafusion-ballista/pull/1418/changes

[milenkovicm]

thanks @mattcuento, great addition

i'm not sure if we have to add table functions into registry. also if we there are table functions to be supported, we may need to create encoder for them. so I'd suggest to ignore table functions for now and take them as a follow up if you agree

[andygrove]

I'm not certain that we have a well-defined place to document compile-time features, just docs for runtime configurations. I'd like to follow this up with a markdown file to describe our features

maybe section in the readme file?

Adding Spark compatibility is a pretty major feature(!), so I think there should be a documentation update as part of this PR. I would be careful to explain it as "using Spark-compatible expressions" where available, rather than full compatibility.

[mattcuento]

@milenkovicm

also if we there are table functions to be supported, we may need to create encoder for them. so I'd suggest to ignore table functions for now and take them as a follow up if you agree

Yup agreed, happy to take the easy wins for scalar/window/aggregate for now and can see what we can do with table functions as a follow up!

@andygrove

Adding Spark compatibility is a pretty major feature(!), so I think there should be a documentation update as part of this PR. I would be careful to explain it as "using Spark-compatible expressions" where available, rather than full compatibility.

Good point, I can write something up as a part of this PR 👍

[milenkovicm]

sorry for delay @mattcuento I'm behind reviews (and everything else :) ) had a quick look and it looks ok with me. Will try to do proper review tomorrow or over the weekend.

maybe @andygrove has some more ideas what should be addressed

[mattcuento]

No worries! Sounds good to me. If there's any PRs I can help with reviewing as a new contributor to help alleviate a bit, let me know!

[milenkovicm]

thanks @mattcuento
just one comment to address, otherwise its a good in my opinion

[milenkovicm]

we need to add

[[example]]
name = "remote-spark-functions"
required-features = ["spark-compat"]

at the bottom of the file, so it triggers so it triggers build with spark-compact

please add same for substrait example and remove it from default

[mattcuento]

ah good call, added for both, did substrait in a separate commit. Ran both locally. Thanks!

[milenkovicm]

Thanks @mattcuento
If we want to improve this further we can open new PRs