PARQUET-2417: Add support for geometry logical type

[zhangfengcdt]

This PR is to provide a POC to support the proposed changes to the parquet-format to add geometry type to parquet.

Here is the proposal: apache/parquet-format#240

Jira

• My PR addresses the following Parquet Jira issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR"
  • https://issues.apache.org/jira/browse/PARQUET-2471

Tests

• My PR adds the following unit tests: TestGeometryTypeRoundTrip

Commits

• My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain Javadoc that explain what it does

[zhangfengcdt]

CC: @jiayuasu @Kontinuation @wgtmac

[wgtmac]

Thanks for the update! I have left some comments. I think we are reaching the finish line!

[wgtmac]

Thanks for adding these tests!

I think we are missing tests in following cases:

• verify geometry type metadata is well preserved.
• verify all kinds of geometry stats are preserved, including bbox, covering and geometry types.
• verify geo stats in the column index have been generated.

I can do these later.

[Kontinuation]

I am depending on this PR to build geo support for iceberg. I got lots of test failures when building this branch locally:

java.lang.NullPointerException
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:965)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.buildColumnChunkMetaData(ParquetMetadataConverter.java:1750)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:1848)
	at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:1728)
	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:629)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:934)
	at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:925)
	at org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:698)

NPE is thrown when reading parquet files without geo columns. Can we apply the following patch to resolve this problem?

diff --git a/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java b/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
index 3efc9345..22e51783 100644
--- a/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
+++ b/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java
@@ -961,6 +961,9 @@ public class ParquetMetadataConverter {
 
   static org.apache.parquet.column.statistics.geometry.GeospatialStatistics fromParquetStatistics(
       GeospatialStatistics formatGeomStats, PrimitiveType type) {
+    if (formatGeomStats == null) {
+      return null;
+    }
     org.apache.parquet.column.statistics.geometry.BoundingBox bbox = null;
     if (formatGeomStats.isSetBbox()) {
       BoundingBox formatBbox = formatGeomStats.getBbox();

[wgtmac]

Thanks for the update! Please see my inline comments.

[Kontinuation]

Should we always normalize the x coordinate and handle wrap-around? The geometry could be in a projected CRS such as EPSG:3857, normalizing longitude or wrap-around should not be applied to such geometries.

The parquet implementation may need to be aware of the crs, or have some other options to turn on/off this longitude normalization and wrap-around behavior. I found that the C++ implementation does not support wrap-around, which may be a better default for geometry types. CC @jiayuasu

[wgtmac]

cc @paleolimbot

[jiayuasu]

In that case, how will the system handle geometries that cross the antimeridian? Should they be flagged as unsupported? It seems reasonable to continue performing wraparound for antimeridian-crossing geometries, rather than simply rejecting them.

[jiayuasu]

@paleolimbot I thought the C++ implementation should also support wraparound?