GH-3070: Add Variant logical type annotation to parquet-java

[aihuaxu]

Rationale for this change

This is to add Variant logical type in parquet-java to be used by dependent projects.

What changes are included in this PR?

Variant logical type is added to LogicalTypeAnnotation.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes. Variant logical type is available.

Closes #3070

[aihuaxu]

@Fokko, @wgtmac Can you help check if this is right implementation? Thanks.

[wgtmac]

Do we need to support writing and reading variant data?

[aihuaxu]

The Variant reading and writing are getting implemented in Iceberg and/or the engines themselves. I think later we can think of pulling the implementation to Parquet if needed.

[emkornfield]

The Variant reading and writing are getting implemented in Iceberg and/or the engines themselves. I think later we can think of pulling the implementation to Parquet if needed.

I think this is problematic if the spec lives in parquet and doesn't have a complete implementation per previously agreed upon guidelines for new parquet features. This probably warrants a discussion on the mailing list. CC @julienledem @rdblue @RussellSpitzer

[Fokko]

@aihuaxu I agree with @emkornfield that the iceberg-java implementation should be able to read and write the variant type.

It would also be great to drop some example parquet files in https://github.com/apache/parquet-testing, this will also help the adoption of other implementations, see apache/parquet-format#456 (comment)

[wgtmac]

Usually we need two reference implementations for spec changes like this. I'm not sure if there is any chance to have another implementation ready in a timely manner. IMO, at least parquet-java should support basic roundtrip read and write.

[aihuaxu]

I see. Per guideline, we need to have the implementation in parquet-java and then another one. Do we usually include the implementation with this annotation change or should be separate?

Completeness: The goal of this phase is to ensure the feature is viable, there is no ambiguity in its specification by demonstrating compatibility between implementations. Once a change has lazy consensus, two implementations of the feature demonstrating interopability must also be provided. One implementation MUST be parquet-java. It is preferred that the second implementation be parquet-cpp or parquet-rs,

[wgtmac]

I think it should be in one change. The parquet-format cannot be released without concrete PoC implementation in parquet-java. Without that release, separate changes may break CI and thus cannot be merged.

[aihuaxu]

@wgtmac With https://github.com/apache/parquet-java/pull/3117/files implementing encoding/decoding, should we consider merging this separately?

[wgtmac]

I think at least it needs the conversion from/to thrift definition of the variant type. So we need to wait for the release of parquet-format 2.11.0.

[rdblue]

Can you move this to a separate PR?

[rdblue]

The other cases create new instances of the logical type, but I agree with using the LogicalTypes constant. This just doesn't need to be a method call. It would also be nice to follow up with a PR that converts all of these to the correct constant rather than creating new instances.

[aihuaxu]

Will do.