Recommend the RAPIDS cache serializer for cached SQL plans

[parthosa]

Summary

This PR adds a shared AutoTuner heuristic that recommends spark.sql.cache.serializer=com.nvidia.spark.ParquetCachedBatchSerializer when an application uses Spark SQL caching. It collects cache evidence from every observed SQL plan version, including AQE updates, and from TableCacheQueryStage RDD scopes when Spark omits cache operators from SQL plan events.

Approach

• Track normalized node names from every SQL plan version in SQLPlanModelManager, including versions replaced by later AQE updates.
• Recognize InMemoryRelation, InMemoryTableScan, and TableCacheQueryStage as alternate signals for Spark SQL caching.
• Inspect structured stage RDD scopes as a fallback for event logs whose SQL plan events omit the cache operators.
• Expose the shared evidence to both Qualification and Profiling AutoTuner providers.
• Store the recommended serializer and Spark default in the existing tuning YAML files.

Key design decisions

• The stage fallback matches only known SQL cache scope names; unrelated persisted RDDs do not trigger the recommendation.
• Missing values and Spark's default serializer are replaced with the configured RAPIDS serializer.
• An already-correct value is unchanged. Any other custom serializer is preserved with an advisory instead of being overridden.
• Cache evidence remains available after SQL cleanup so completed or replaced plan versions cannot erase the signal.

Reviewer attention areas

• SQLPlanModelManager.collectObservedRDDScopes filters structured RDD scope names against the same cache-node set used by AutoTuner.
• AppBase.getOrCreateStage records stage-scope evidence for both Qualification and Profiling processing paths.
• recommendCacheSerializer respects excluded properties and distinguishes Spark's default serializer from a user-provided custom value.

Files changed

Area	What changed
bootstrap/tuningConfigs.yaml and tuningTable.yaml	Added the configurable RAPIDS serializer value, Spark default, comments, and bootstrap eligibility.
SQLPlanModelManager.scala and AppBase.scala	Added cache evidence collection across SQL plan versions and stage RDD scopes.
Profiling and Qualification providers	Exposed the shared observed-node signal to AutoTuner.
AutoTuner.scala	Added the cache serializer recommendation and custom-value advisory logic.
AutoTuner, provider, and plan-model suites	Added behavior, AQE, stage-scope, configuration, and provider-parity coverage.

What is NOT affected

• Applications without recognized SQL cache evidence receive no new recommendation.
• The change does not enable spark.rapids.sql.exec.InMemoryTableScanExec or alter plugin fallback behavior.
• Unsupported cached data types can continue to fall back to CPU processing.

Testing

• AppSummaryInfoProviderSuite: 5 tests passed.
• QualificationAutoTunerSuite: 42 tests passed.
• ProfilingAutoTunerSuiteV2: 43 tests passed.
• Scalastyle passed with no errors or warnings.
• mvn package -DskipTests completed successfully for the Scala 2.12 JAR.
• CLI validation confirmed the recommendation in one Qualification run and two Profiling runs where cache evidence was retained in stage RDD scopes.

Known limitations / follow-up

• The recommendation establishes GPU cache compatibility but does not estimate whether a workload will benefit materially from caching.
• Spark 3.5.0 and 3.5.1 retain the plugin's existing AQE fallback behavior for GPU cache scans; this PR does not change that execution rule.