[SPARK-57521][ML][CONNECT] Exclude parent from Model.estimatedSize to fix overcounting in ML cache

[mkincaid]

What changes were proposed in this pull request?

This patch unsets parent before calling the SizeEstimator.

Why are the changes needed?

Currently SizeEstimator includes the size of the SparkSession because it traverses the parent object which (in the case of many estimators that use DataFrame operations when fitting, like StringIndexer) eventually refers to the session. The session is there anyway and its size isn't attributable to fitting this specific model (and this results in double-counting when more models are fit), so it shouldn't be included in the size estimate.

The impact of the bug is largest when the SparkSession is large. For example, in Databricks, my testing shows that a 300-800M SparkSession is typical. In some configurations, like Databricks serverless, the size limit for a single model object might be 256M, so this bug causes such models to fail to train regardless of the state of the cache otherwise.

The Jira ticket includes a simple script that reproduces the condition locally, though the session is much smaller in that case (maybe 300k).

Does this PR introduce any user-facing change?

Yes, a favorable one, in that the model cache would fill less quickly (and the reported sizes of cached models would be smaller, if they are among the affected models).

How was this patch tested?

A test is added: training a StringIndexer should estimate at no larger than 50k, in the trivial test case with 3 strings. This test fails before the patch and passes after it. Another similar test is provided for MinMaxScaler. A ModelSuite is added to hold these since the bug is at the Model level, not that of individual models (so the StringIndexer and MinMaxScaler suites aren't really the right place for these tests, although they are examples).

Was this patch authored or co-authored using generative AI tooling?

Yes, the bug was discovered and initial patch/tests were created by pair programming with Claude. I wrote the bug/docs myself and validated the approach and final patch.

Generated-by: Claude Opus 4.6

[mkincaid]

Fixed actions configuration on my fork. Closing and reopening to trigger the checks to rerun