Add Iceberg support as an alternative storage backend

[thevijayshekhawat]

Summary

Currently the shadow proxy writes query logs as newline-delimited JSON to GCS in a Hive-partitioned layout. Add support for writing logs directly as Iceberg tables, enabling queryability from StarRocks and other Iceberg-compatible engines without needing BigQuery external tables.

Motivation

• Iceberg tables can be queried directly from StarRocks, eliminating the need for a separate BigQuery external table setup
• Better schema evolution support as the log format changes over time
• Compaction and snapshot management give more control over storage lifecycle

Considerations

• The proxy writes continuously in near-real-time, so frequent Iceberg metadata commits could create contention
• A batched write approach (e.g. buffering and committing every few minutes) would keep metadata overhead manageable while preserving near-real-time queryability
• Should be additive — GCS/S3 JSONL remains the default backend