Skip to content

[SPARK-56462][SQL]Fix UPDATE/INSERT schema evolution when source col contains dot#55329

Open
jiwen624 wants to merge 1 commit intoapache:masterfrom
jiwen624:fix-merge-into-schema-evolution-dot-column
Open

[SPARK-56462][SQL]Fix UPDATE/INSERT schema evolution when source col contains dot#55329
jiwen624 wants to merge 1 commit intoapache:masterfrom
jiwen624:fix-merge-into-schema-evolution-dot-column

Conversation

@jiwen624
Copy link
Copy Markdown
Contributor

@jiwen624 jiwen624 commented Apr 13, 2026

What changes were proposed in this pull request?

In Analyzer.scala, the UPDATE * and INSERT * schema-evolution paths inside ResolveReferences built assignment keys for new source columns using UnresolvedAttribute(sourceAttr.name). That constructor splits the name on dots, so a column named `job.title` produced nameParts = Seq("job", "title") instead of Seq("job.title").

The fix replaces both call sites with UnresolvedAttribute.quoted(sourceAttr.name), which treats the whole name as a single quoted identifier.

Why are the changes needed?

With a dot in the source column name, isSameColumnAssignment compared a two-part key against a single-part value and returned false, blocking schema evolution. Analysis then failed with UNRESOLVED_COLUMN.
image

Does this PR introduce any user-facing change?

Yes. MERGE INTO … WHEN MATCHED UPDATE * WHEN NOT MATCHED INSERT * with schema evolution now correctly handles source columns whose names contain a dot.

How was this patch tested?

Added a testEvolution case in MergeIntoSchemaEvolutionBasicTests covering both schema-evolution-enabled (dot-named column is added and populated) and schema-evolution-disabled (extra column is silently ignored).

Was this patch authored or co-authored using generative AI tooling?

Yes.

@jiwen624 jiwen624 marked this pull request as ready for review April 14, 2026 05:27
@jiwen624
Copy link
Copy Markdown
Contributor Author

@dongjoon-hyun @cloud-fan @HyukjinKwon could you let me know your thoughts on this fix? Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant