You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm building a TableProvider that represents a custom external data source that stores data in partitions in a consistent hash ring. The partitions are trivial to discern from the filters provided.
I'm trying to implement filter pushdown and I can't seem to find good "best practices" guidance on how much work should be done in the logical versus physical stages.
Should I determine which shards I need to scan in my logical plan and pass it into the new physical plan I create in the constructor? Or should the logical plan just hand off the raw filters and leave it to the physical plan to sort out?
Should I represent each shard scan as a separate Exec node in the physical plan? For example, a logical plan could determine it needs to talk to shards 1, 3, and 5 and therefore create 3 Exec nodes - for each individual scan from a shard. Or should I treat my sharding as a black box, and just put all this logic into the single Exec node to be determined at runtime?
In general, how much "pre-work" should I do in the logical plan? As much as possible? As little as possible?
Any links to readings or presentations on this subject would be appreciated.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
I'm building a TableProvider that represents a custom external data source that stores data in partitions in a consistent hash ring. The partitions are trivial to discern from the filters provided.
I'm trying to implement filter pushdown and I can't seem to find good "best practices" guidance on how much work should be done in the logical versus physical stages.
Any links to readings or presentations on this subject would be appreciated.
Thanks in advance!
Beta Was this translation helpful? Give feedback.
All reactions