[SPARK-52582][SQL] Improve the memory usage of XML parser

[xiaonanyang-db]

What changes were proposed in this pull request?

Today, the XML parser is not memory efficient. It loads each XML record into memory first before parsing, which causes an OOM if the input XML record is large. This PR improves the parser to parse an XML record token by token to avoid copying the entire XML record into memory ahead of time.

The optimization is governed by an SQL conf and disabled by default because it comes with two consequences:

1. XSD validation is not supported in the optimized parser. The current XSD validation works only on a full XML record string, which violates the optimization here.
2. Behavior change of corrupted record handling. Currently, good records in an XML file can be parsed correctly even if there is a corrupted XML record in between, but they won't be parsed anymore in the optimized parser. For example, given the following XML file:

<ROWS>
  <ROW><b>1</b></ROW>
  <ROW><b></ROW>
  <ROW><b>2</b></ROW>
</ROWS>

In the current parser, both the 1st and 3rd records will be parsed correctly, and the second one will be moved to the corrupted data column. In the new parser, only the 1st record will be parsed and the whole document will be moved to the corrupted data column as the second record.

Why are the changes needed?

Solve the OOM issue in XML ingestion.

Does this PR introduce any user-facing change?

No. The new behavior is disabled by default for now.

How was this patch tested?

New UTs.

Was this patch authored or co-authored using generative AI tooling?

No.

[LuciferYang]

Will the optimized XML parser have any negative impacts? Why is it necessary to add a switch instead of directly replacing the old implementation with the optimized XML parser?

[xiaonanyang-db]

Yes, there are two implications for the new optimized parser, please check the PR description.