Avro Input Format decoding fails on negative array block sizes

[genzgd]

Describe what's wrong

Arrays encoded in binary Avro format start with a "block size" Avro long that indicates the number of elements to be decoded in the following block data. Recent versions of the Avro specification allow for a "negative" block size, as described here:

Arrays are encoded as a series of blocks. Each block consists of a long count value, followed by that many array items. A block with count zero indicates the end of the array. Each item is encoded per the array's item schema.

If a block's count is negative, its absolute value is used, and the count is followed immediately by a long block size indicating the number of bytes in the block.

ClickHouse throws an error when reading data in Avro format if it is encoded with a negative block size, for example: avro::Exception: Cannot have negative length: -932348185.

Does it reproduce on the most recent release?

Yes

How to reproduce

The attached strarray.avro file contains 10 records with the following schema:

{
  "doc": "Testing for complex types",
  "fields": [
    {
      "name": "str_array",
      "type": {
        "items": "string",
        "type": "array"
      }
    }
  ],
  "name": "complex_types",
  "namespace": "com.clickhouse.clickpipes",
  "type": "record"
}

Reading this file in clickhouse or clickhouse-local results in the following error:

SELECT *
FROM file('strarrays.avro')

Query id: 9e4c9e5a-0144-47ab-84e7-4fedeefc1e0a

↓ Progress: 0.00 rows, 0.00 B (0.00 rows/s., 0.00 B/s.)  0%
Elapsed: 0.317 sec. 

Received exception from server (version 24.1.1):
Code: 1001. DB::Exception: Received from localhost:9007. DB::Exception: avro::Exception: Cannot have negative length: -81. (STD_EXCEPTION)

No stack trace is recorded in the clickhouse logs:

2024.02.27 01:00:32.745596 [ 225479 ] {9e4c9e5a-0144-47ab-84e7-4fedeefc1e0a} <Debug> executeQuery: (from [::1]:59422) select * from file('strarrays.avro'); (stage: Complete)
2024.02.27 01:00:33.304303 [ 225479 ] {9e4c9e5a-0144-47ab-84e7-4fedeefc1e0a} <Error> executeQuery: std::exception. Code: 1001, type: avro::Exception, e.what() = Cannot have negative length: -81 (version 24.1.1.1455 (official build)) (from [::1]:59422) (in query: select * from file('strarrays.avro');), Stack trace (when copying this message, always include the lines below):

<Empty trace>

If the "negative block size" feature is turned off, ClickHouse works correctly. The attached strarrays_ok.avro file has a similar avro payload with the same schema, but does not use negative block sizes.

Other Thoughts
This is presumably a result of the contrib avro library being out of date with the current spec. Note that most encoders don't use this feature, but it is becoming more common in current Avro libraries.

strarrays.avro.zip
strarrays_ok.avro.zip

[genzgd]

The issue looks to be only in the second block or later blocks (which is why arrays have to be very large before the error is observed). In the following code, arrayStart() correctly handles the negative block size by calling doDecodeItemCount(), but arrayNext() inexplicably doesn't. So this should be a one line fix?

size_t BinaryDecoder::arrayStart()
{
    return doDecodeItemCount();
}

size_t BinaryDecoder::doDecodeItemCount()
{
    int64_t result = doDecodeLong();
    if (result < 0) {
        doDecodeLong();
        return static_cast<size_t>(-result);
    }
    return static_cast<size_t>(result);
}

size_t BinaryDecoder::arrayNext()
{
    return static_cast<size_t>(doDecodeLong());
}

[genzgd]

This code also looks broken in the main apache project?