Skip to content

Commit

Permalink
[Spark] Handle NullType in normalizeColumnNames()
Browse files Browse the repository at this point in the history
The sanity check in normalizeColumnNamesInDataType() introduced by that change is a bit too restrictive, and fails to handle NullType correctly.

Closes #2634

GitOrigin-RevId: faaf3d981c57ef3ceb4081e0bc94d457359fc9d8
  • Loading branch information
ala authored and vkorukanti committed Feb 21, 2024
1 parent 360e066 commit 0ee57b7
Showing 1 changed file with 4 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,10 @@ def normalizeColumnNamesInDataType(
keyType = normalizedKeyType,
valueType = normalizedValueType
)
case (_: NullType, _) =>
// When schema evolution adds a new column during MERGE, it can be represented with
// a NullType in the schema of the data written by the MERGE.
sourceDataType
case _ =>
if (Utils.isTesting) {
assert(sourceDataType == tableDataType,
Expand Down

0 comments on commit 0ee57b7

Please sign in to comment.