Configure batch size on KafkaConnect BigQuery sink connector

[icyBlue27]

Using the KafkaConnect BigQuery sink connector, from time to time, we observe the following error since the streaming API has a limit on the size of the batch it can write to BigQuery com.google.cloud.bigquery.BigQueryException: Request size is too big: 12705398 limitation: 12582912. Would it be possible to have the batch size configurable on the sink connector to stay under certain limits?

Kafka offers different parameters to configure on the producer at the moment (https://kafka.apache.org/documentation/#producerconfigs).

[b-goyal]

@icyBlue27 , could you try setting consumer.override.max.poll.records to a lower number ? Default value for this config is 500.

[icyBlue27]

thanks @b-goyal . If I understand correctly max.poll.records controls the number of records which are pulled. This is not exactly what we would like to have, which is a limit on the size in bytes of the batch. True that less records would probably be smaller in size, but this is not an optimal solution.