[WIP][GLUTEN-10215][VL] Delta 4 native write support for Spark 4.0

[malinjawi]

What changes are proposed in this pull request?

This PR adds native Delta Lake write support for Spark 4.0 + Delta 4.0 in the Velox backend.

Main changes:

• Add Delta 4 write offload integration in Delta write operators and command offload path.
• Wire Delta-specific parquet file format and optimistic transaction write path for native execution.
• Add native write statistics tracking plumbing used by Delta write.
• Add Delta optimized writer offload integration for the Delta 4 path.

Scope note:

• This PR is focused on native write support only.
• Delta 4 test-suite expansion is split into separate PR(s) for reviewability.

Part of #10215.

How was this patch tested?

• Compile/build validation with Spark 4.0 + Delta 4 profile:

./build/mvn -pl backends-velox -am -Pbackends-velox,delta,spark-4.0,scala-2.13,java-17 -DskipTests -Dcheckstyle.skip=true -Dspotless.skip=true -Dscalastyle.skip=true compile

[zhztheplayer]

@malinjawi looks good.

I think we'd first open and merge a PR which adds tests, then switch to this one?

[zhztheplayer]

A out-of-topic question: Do you think we can possibly support e.g., src-spark4 / src-delta4 folder-activation in Gluten's Maven settings? So we might not have to port these code again for Spark 4.1 if lucky.

Something like:

backends-velox
 |- src-spark3
 |- src-spark34
 |- src-spark35
 |- src-spark4
 |- src-spark40
 |- src-spark41
 |- src-delta
 |- src-delta4
 |- src-delta40
 |- src-delta41
 ...

Current code is a bit inflexible, it was introduced in #9996.

[zhztheplayer]

A component file is needed to activate the component: https://github.com/apache/gluten/blob/main/backends-velox/src-delta33/main/resources/META-INF/gluten-components/org.apache.gluten.component.VeloxDelta33WriteComponent

[malinjawi]

@malinjawi looks good.

I think we'd first open and merge a PR which adds tests, then switch to this one?

Hey @zhztheplayer! Agreed I had added the UT for this path in separate branches.

I am raising the base delta 4.0 UT here: #11938

Also planned to expand testing as needed in separate patches.