Open
Description
Is your feature request related to a problem? Please describe.
Currently, all batching is hard coded to a maximum of 1024
. In certain circumstances, it might be beneficial to specify smaller batch sizes to improve concurrency.
Describe the solution you'd like
Add a cli arg, --batch-size
which controls the max batch size.