Closed
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Follow-on issue to address feedback in #3009
In particular:
- Run some performance test and see if it would be better to use builders in the aggregate state rather than arrays. Should we just use
ScalarValue::List
instead? - Should we have a configurable limit on the number of values that can be collected to prevent OOM?
- Can we leverage the
concat
andtake
kernels? - Should we add an
average
kernel to arrow-rs? - Can we reduce memory overhead with some form of compression (such as run-length encoding or maintaining a map of unique values with counts)
Describe the solution you'd like
See above
Describe alternatives you've considered
None
Additional context
None