Iceberg scan fallback - Spark 4, Iceberg 1.10, Gluten 1.6.0

[1b5d]

Hi, I'm trying to read an iceberg table using Gluten (velox backend) and it's falling back, even though the table is not partitioned, and there are no deletions:

26/04/01 19:31:39 WARN GlutenFallbackReporter: Validation failed for plan: BatchScan spark-test.b7c164ad-9483-45ba-9235-8f60e6d94a19.EKKO_2y, due to:
Unsupported scan IcebergScan(table=spark-test.b7c164ad-9483-45ba-9235-8f60e6d94a19.EKKO_2y, branch=null, type=struct<1: MANDT: optional string, 2: EBELN: optional string, 3: BUKRS: optional string, 4: BSTYP: optional string, 5: BSART: optional string, 6: LOEKZ: optional string, 7: STATU: optional string, 8: AEDAT: optional date, 9: ERNAM: optional string, 10: LIFNR: optional string, 11: ZTERM: optional string, 12: ZBD1T: optional double, 13: ZBD2T: optional double, 14: ZBD3T: optional double, 15: ZBD1P: optional double, 16: ZBD2P: optional double, 17: EKORG: optional string, 18: EKGRP: optional string, 19: WAERS: optional string, 20: WKURS: optional double, 21: BEDAT: optional date, 22: KDATB: optional date, 23: KDATE: optional date, 24: BWBDT: optional date, 25: ANGDT: optional date, 26: BNDDT: optional date, 27: GWLDT: optional date, 28: IHRAN: optional date, 29: IHREZ: optional string, 30: VERKF: optional string, 31: LIFRE: optional string, 32: UNSEZ: optional string, 33: STCEG: optional string, 34: RELEASE_DATE: optional date, 35: ZZPOSOURCE: optional string, 36: ZZPOCLOSE: optional string, 37: ZZSHIPPOINT: optional string, 38: EQ_EINDT: optional date, 39: _CHANGE_DATE: optional timestamptz, 40: _rescued_data: optional string>, filters=[], runtimeFilters=], caseSensitive=false)

I'm confused because I'm running Spark 4 with the latest version of gluten-velox-bundle (jar downloaded from here https://dlcdn.apache.org/gluten/1.6.0/ from what I understand that is already built with the iceberg profile, please correct me if I'm wrong)

Can someone help me please figure out why I'm not able to enable the native scan?

[zhouyuan]

@1b5d
In 1.6.0 release spark4.0 does not have iceberg support included, mostly due to not fully tested. Please have a try with below nightly 1.7 package:
https://nightlies.apache.org/gluten/nightly-release-jdk17-enhanced/

[1b5d]

Thank you, the nightly package worked, but I guess we have to build our own release artifact if we want to use 1.6.0? I can only find snapshot versions

[1b5d]

@zhouyuan a different quick question, do we know if credential vending is supported for iceberg tables?

[zhouyuan]

@1b5d Hi, yeah you may need to build the Gluten package and enable iceberg module. Note that it's not fully tested on Spark-4.0 in the community as most users are still on Spark3.x
Here's the reference script for packaging:
https://github.com/apache/gluten/blob/main/dev/release/build-release.sh

The credential vending is not support now afaik, the native iceberg in Velox are now mostly focusing on I/O(read/write) - it should be the next stage goal as well as V3 support