Support Rest Catalog Metrics Endpoint

[sungwy]

Feature Request / Improvement

Similar to Java, support the publication of ScanReport and CommitReport to the Rest Catalog:

Metrics Endpoint Spec

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'

[DerGut]

I'd like to pick this back up because we started a discussion about in apache/iceberg-rust#1466 too.

On the draft implementation, @sdd raised a good point that we now have other, often more idiomatic interfaces available apache/iceberg-rust#1496 (comment). In Rust for example, we've decided on using the facade metrics which users can back by any exporter they like, offering simple integrations with existing observability systems. In Python, opentelemetry offers similar functionality.

Using existing telemetry APIs, reporting code could look much simpler and backing integrations will be easier (no custom code needed).

Metric Names

Emitting metrics straight from the library will mean we also need to standardize on metric names or implementations could diverge, defeating the idea of a unified way of monitoring Iceberg clients.

I would like to propose a naming system similar to @sdd's PoC comprised of

iceberg.<operation>.<resource>.<count-type>

for example iceberg.scan.data_files.scanned, iceberg.scan.delete_manifests.skipped or iceberg.commit.delete_files.added. Existing metrics can be taken from ScanMetricsResult.java and CommitMetricsResult.java.

Catalog Spec

The Metrics Reporting API is part of the catalog spec which suggests that we should consider implementing it anyway. If we can prove with an experiment that (for example) an opentelemetry exporter can consume a spec-compliant reporter interface, we should be good. If we can't, we need to take this into consideration.
With the spec's API, multiple metrics are bundled together into a single report. This doesn't seem natural for other metrics APIs and could become an implementation burden.

---

I can help with an implementation, but I would first like to extend the discussion about following the Java implementation vs. using more idiomatic approaches. Note, I've also created apache/iceberg-go#485 in an attempt to get different implementations move into a similar direction.

[kevinjqliu]

Thanks for bringing this up! I think we should have a wider discussion to standardize the approach between the different language implementations (and perhaps also align the java implementation)

Let's bring this to the devlist! I'm happy to help kick the conversation off. I see there are several separate discussions already :P

cc @DerGut @sdd @zeroshade

[DerGut]

Agreed, makes sense to consolidate this somewhere!

Thanks for the offer Kevin! Are there any formal considerations to start a discussion on the dev list, or is a mail to dev@iceberg.apache.org containing an overview of the discussion (similar to the three implementation-specific issues) all it needs?

[DerGut]

Update: sent out this doc. Looking forward to the discussion 🎉