[VL][Delta] Add native deletion vector scan foundation

[malinjawi]

Issue: #11901

What changes are proposed in this pull request?

This PR adds the native Delta deletion vector scan foundation for the Velox backend in Gluten.

The architecture is based of the design of Delta Lake's: Deletion Vectors High Level Design.

The scope is intentionally limited to the read path. It introduces the JVM and native plumbing required to read Delta tables with deletion vectors natively, while keeping DML and DV write-path work out of scope for a follow-up PR.

The main changes are:

• Add JVM-side Delta scan preparation and metadata normalization for DV-aware scans.
• Add native Delta scan/read components under cpp/velox/compute/delta, including:
  • Delta connector and datasource plumbing
  • Delta split representation
  • deletion vector reader
  • split reader integration
  • UUID / Base85 utilities for Delta DV handling
• Add explicit Roaring dependency wiring and a local RoaringBitmapArray helper needed by the native DV read path and tests.
• Wire the Delta read path into the Velox backend and scan execution flow.
• Add focused native tests for:
  • Delta connector
  • deletion vector reader
  • split handling
  • UUID utilities

This PR does not include:

• row-index finder integration
• DELETE / UPDATE / MERGE DV write-path changes
• compaction / utility command support

How was this patch tested?

• Added native tests under cpp/velox/compute/delta/tests:
  • DeltaConnectorTest
  • DeltaDeletionVectorReaderTest
  • DeltaSplitTest
  • DeltaUuidUtilsTest
• Locally validated that the new Delta read sources are integrated into the native velox target and compile through the new Delta translation units.
• Locally validated explicit Roaring discovery and removed unresolved CRoaring symbols after wiring the dependency.

Was this patch authored or co-authored using generative AI tooling?

Co-authored: IBM Bob

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[zhztheplayer]

@malinjawi Thanks.

A broad design question: Can you check if it's worth it to maintain native deletion vector reader vs. passing serialized deletion vector from Java to C++? In terms of the overall reading performance.

[github-actions]

Run Gluten Clickhouse CI on x86

[malinjawi]

@malinjawi Thanks.

A broad design question: Can you check if it's worth it to maintain native deletion vector reader vs. passing serialized deletion vector from Java to C++? In terms of the overall reading performance.

Thanks for the design query @zhztheplayer.

I checked this with 2 benchmark layers:

1. a focused DV-path microbenchmark
2. a full end-to-end Spark benchmark on a Delta table with deletion vectors

I compared native path Vs. JVM-materialized serialized DV handoff path, where I setup as follows:

• native side loads/materializes the deletion vector and applies it in the read path

• JVM materializes the deletion vector first using Delta’s JVM path, serializes it as a portable Roaring payload, then passes it to native through Gluten’s existing split metadata path to decode/deserialize it and apply it in the read path

Here what I found:

Microbenchmark:

• When full cost is included, the JVM-materialized handoff path performs essentially the same as the native path.
• Across the inline and stored DV cases I tested, the observed difference stayed within normal benchmark noise, approximately -1% to +3% relative to the native path.
• The only clearly faster result came from an optimistic consumer-only setup where native receives serialized DV bytes at zero cost, which does not reflect the real end-to-end design.

End-to-end Spark benchmark:

• 1M rows, 125k deleted rows in the DV

  • native median: 627.81 ms
  • JVM handoff median: 596.75 ms
  • JVM handoff was approximately 4.9% faster in this run

• 4M rows, 500k deleted rows in the DV

  • native median: 1556.15 ms
  • JVM handoff median: 1541.18 ms
  • JVM handoff was approximately 1.0% faster in this run

From my observation, the smaller run shows a modest advantage for the JVM-materialized handoff path, but the larger run narrows that gap to about 1%. Taken together, these results do not show a strong end-to-end performance advantage for replacing the native DV reader with a JVM-side serialized-DV handoff design. Therefore, the two approaches are effectively performance-neutral for this test, so JVM handoff path is not clearly better then native in these testing.

Based on that, I think keeping the native DV reader is still the right foundation for this PR.

I think we can introduce caching / avoiding repeated DV loads, rather than shifting the DV load boundary from native to JVM. Although what we could also checkout is a zero-copy JNI byte-transfer design.

What do you think @zhztheplayer ?

cc: @zhouyuan

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[github-actions]

Run Gluten Clickhouse CI on x86

[zhztheplayer]

@malinjawi many thanks for taking time for testing.

Therefore, the two approaches are effectively performance-neutral for this test, so JVM handoff path is not clearly better then native in these testing.

If the performance is at the same level, should we just pass DVs from Java to C++? So we can save some energy from maintaining the C++ DV reader by ourselves. Or do you have other reasons to add the C++ DV reader?

[malinjawi]

malinjawi many thanks for taking time for testing.

Therefore, the two approaches are effectively performance-neutral for this test, so JVM handoff path is not clearly better then native in these testing.

If the performance is at the same level, should we just pass DVs from Java to C++? So we can save some energy from maintaining the C++ DV reader by ourselves. Or do you have other reasons to add the C++ DV reader?

Thanks @zhztheplayer .

I looked at this more closely and tried to separate the architectural reasoning from the benchmark question.

Based on the results so far, I do not think we have a strong performance argument for keeping the C++ DV reader by itself as it stands now the observed advantage is small and actually shrank as total query work grew.

The more important point is the breakdown inside that DV slice:

• inline cases were almost entirely dominated by native applyDeletionFilter(...)
• for moderate stored DVs, native read/load was usually only about 5% to 8% of native current total
• only in denser stored cases did native load rise to about 17% to 26%
• even in those denser cases, zero-copy handoff still stayed close to parity with native current

So from an Amdahl's law perspective, even if we made the native reader perfect, the upside is usually limited because the hotter path is still native apply/filter, not materialization/loading.

So I think my takeaways are:

1. Right now we dont have a strong performance argument for keeping the C++ DV reader by itself.
2. I also dont think the data supports the current metadata/string-style handoff path, because that path gets clearly more expensive as payload size grows.
3. The dominant cost is still native DV consume/apply/filter in the scan path.

I think this also helps separate the native path into more specific responsibilities:

• descriptor / protocol handling
• materialization / loading
• native consume / deserialization
• scan-time apply / filter
• caching / reuse

Based on the current measurements, the hottest and most valuable native parts are:

• native consume / deserialization
• scan-time apply / filter
• scan integration

The native reader/materialization side is easier to question, because it does not currently show a strong speed win.

So my current view is:

-The lowest maintenance design would be to let JVM / Delta utilities own DV materialization/loading and keep native focused on consume/apply/filter

-The most self-contained approach would be to let native scan as we can build on top of it as shown in Velox POC and Gluten POC

The main architectural reasons I still see for keeping the native DV reader are:

• it is the cleanest base for future native caching
• it gives native full control over DV file I/O, prefetch, and loading policy for future work
• it aligns better with the broader native Velox DV direction from previous efforts made
• it keeps the scan path more self-contained and less dependent on JVM materialization at execution time
• it is potentially more reusable for broader Velox-native DV patches beyond the Gluten

But this is not currently evident by this PR but rather future architectural not currently demonstrated benchmark wins.

So, I think we could go either way with this implementation and I think the design choice between native reader and JVM should now be made mostly on maintenance / integration / future tradeoffs. What do you think would be the best approach all things considered @zhouyuan @zhztheplayer ?

[zhztheplayer]

@malinjawi thanks for your write-up. I understand the current status better now.

Overall, I thought a native DV IO layer might possibly become beneficial in the future, when we meet more DV-heavy workloads than the time being. So we don't have to stick on one single approach against the other. I would be inclined to start from a simpler, easier-to-maintain approach, i.e., passing DVs from Java to C++, and preserve the possibility of switching to native DV layer in our design and roadmap. Just my 2 cents, insights are welcome here.