Wishing list: Pulsar SQL should support readCompact on topics

[Hugues514]

So far, when a Pulsar topic is compacted, the Pulsar SQL query will read all the records, not considering the compacted option

It could be interesting to be able to query such a topic using a readCompact option so that the reader will skip all the 'compacted records'.

[jiazhai]

We need a special handle in PulsarSplitManager.getSplitsForTopic to split based also on compactionHorizon(contained in CompactedTopicImpl); so that for data before compactionHorizon, we could read from compacted topic, while data after compactionHorizon, read from non-compacted topic.

[jiazhai]

more info:
https://github.com/apache/pulsar/wiki/PIP-19:-Pulsar-SQL
https://github.com/apache/pulsar/wiki/PIP-14:-Topic-compaction

[pointearth]

Can you guys open this ticket and finish it or move it to a suitable appointment? appreciate!
Add readCompact option #7461

[longtengz]

Does this also imply pulsar sql cannot process empty payload messages for the time being? Because I'm experiencing #13127

[tisonkun]

Moved to general discussions since there's a wishing list item and no one works on it now.