[SPARK-49828][SQL] Make Column(expression) usable outside of the org.apache.spark package

[peter-toth]

What changes were proposed in this pull request?

This makes the intended Column(expression) conversion usable from outside the org.apache.spark package:

1. object Column (sql/api) becomes public. Every member of it is already private[spark] or narrower, so no new member becomes visible -- the object simply becomes nameable, which is all the extension in point 2 needs. The two apply overloads that relied on the object's own visibility are now marked private[spark] explicitly.
2. A named @DeveloperApi ClassicConversions.column(e: Expression): Column is added, for callers who would rather not rely on an implicit.
3. A new ExpressionToColumnSuite in package test.org.apache.spark.sql, i.e. outside org.apache.spark, pins both entry points and the round trip.

Why are the changes needed?

ClassicConversions.ColumnConstructorExt is already a public @DeveloperApi:

implicit class ColumnConstructorExt(val c: Column.type) {
  def apply(e: Expression): Column = ExpressionUtils.column(e)
}

It is meant to let extension developers write Column(expr), but it cannot be used outside org.apache.spark, because object Column is itself private[spark]. So Column(expr) does not even compile there:

package com.example.ext
// error: not found: value Column
val c = Column(Literal(1))

ExpressionUtils, ExpressionColumnNode and ColumnNode are package private as well, and SPARK-49022 removed the public new Column(expr: Expression) constructor, so there is currently no public Expression -> Column path at all. The reverse direction is public and works fine (ColumnConversions.expression(col), col.expr), which makes the gap asymmetric. Today the only workaround is to put the helper in a package under org.apache.spark, which is what mllib's ml/stat/Summarizer.scala does.

This also makes an existing review comment on #48306 true: @hvanhovell suggested "You could call org.apache.spark.sql.classic.ClassicConversions.column directly if you want to avoid implicits" -- that method did not exist until now.

Prior art: #48306 by @holdenk took the broader approach of exposing the ColumnNode AST types (ExpressionColumnNode, ColumnNodeToExpressionConverter) and moving columnNodeSupport.scala out of sql.internal. It was approved by @hvanhovell, then the approval was dismissed over the package choice, and the stale bot closed it. Most of that diff is now obsolete, since the file has meanwhile moved to org.apache.spark.sql.classic. This PR intentionally stays narrower and leaves the AST types internal, so Spark keeps freedom over their shape.

Three points reviewers may want to weigh in on:

• The named factory is deliberately on object ClassicConversions rather than on the ClassicConversions trait. On the trait it shadows functions.column(colName: String) for anyone who mixes the trait in, which breaks existing call sites -- 6 of them in sql/core's own tests, via QueryTest.testImplicits. The trade-off is that a cross-version shim implementing the trait does not inherit it.
• import ClassicConversions._ combined with import functions.column is now ambiguous. It is opt-in and a compile error rather than anything silent, but happy to rename (columnOf, toColumn) if preferred; column was chosen to mirror ColumnConversions.expression.
• object Column has no public members, so it shows up as an empty entry in the generated API docs. I left it without an @since tag, since the object itself has existed since 4.0, just not publicly.

Does this PR introduce any user-facing change?

Yes, it adds public API. No existing behavior changes.

• object Column becomes public (no new members).
• ClassicConversions.column(e: Expression): Column is new, tagged @since 4.4.0.

Extension developers outside org.apache.spark can now write either:

import org.apache.spark.sql.classic.ClassicConversions._
val c = Column(myExpression)
// or, without the implicit:
val c = ClassicConversions.column(myExpression)

How was this patch tested?

New sql/core/src/test/scala/test/org/apache/spark/sql/ExpressionToColumnSuite.scala. Its package is outside org.apache.spark, so compiling it is as much of the test as running it -- that is the piece #48306 lacked, which is how the visibility gap went unnoticed in the first place.

• The suite fails to compile on unmodified master with not found: value Column at both Column(...) sites, which is exactly the error a user hits.
• 3 tests pass with the change: both entry points, plus use in an actual query.
• ColumnExpressionSuite, JavaColumnExpressionSuite, DataFrameSuite, DatasetSuite, JavaDatasetSuite and the full sql-api test suite pass (~900 tests).
• MiMa is clean against spark-sql_2.13:4.0.0 and spark-sql-api_2.13:4.0.0, and compile is clean across all modules, including mllib, which consumes the Column(node) overload.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

[uros-b]

LGTM

[dongjoon-hyun]

If we change the visibility, we need annotations like we did for class Column.

spark/sql/api/src/main/scala/org/apache/spark/sql/Column.scala

Lines 139 to 140 in 496d20f

	@Stable
	class Column(val node: ColumnNode) extends Logging with TableValuedFunctionArgument {

In this case, which one do you propose in this PR? Stable or DeveloperApi, @peter-toth ?

[peter-toth]

Good catch, thanks. I went with @DeveloperApi (plus @since 4.4.0) in f4b2d23.

Reasoning: everything that makes Column(expression) work is already @DeveloperApi — ClassicConversions and ColumnConstructorExt — so the path is only as stable as those, and @Stable on the companion would over-promise. The object also has no public members; it is public purely so the type can be named as a value.

The counter-argument, in case you'd rather have @Stable: @DeveloperApi does let us make the object private again in a minor release, which would re-break the use case this PR fixes. Happy to switch if you prefer that guarantee.

[dongjoon-hyun]

+1

[dongjoon-hyun]

This new documentation might be misleading to the Spark Connect users because org.apache.spark.sql.classic.* is a classic-specific API.

When a Spark Connect user traverses API doc, he/she reaches this object Column doc and will see this. Can we revise this doc into more nuetural way?

[peter-toth]

Agreed, that was a mistake on my side — Column.scala is in sql/api, which Connect shares, so pointing at a classic-only API from there is wrong. Dropped the ClassicConversions.ColumnConstructorExt reference in f4b2d23; the doc now just says an implementation can add the factory via an extension method.

[dongjoon-hyun]

+1 for the direction. LGTM (with two minor comments).