Switch OTLP endpoints to ingest v2

[rdettai]

Description

Closes #5209

How was this PR tested?

Integration tests

[github-actions]

On SSD:

Average search latency is 0.99x that of the reference (lower is better).
Ref run id: 2963, ref commit: 82c475b
Link

On GCS:

Average search latency is 0.939x that of the reference (lower is better).
Ref run id: 2964, ref commit: 82c475b
Link

[rdettai]

Integration tests for OTLP are hanging on shutdown

[INFO  quickwit_ingest::ingest_v2::ingester] decommissioning ingester

Likely related to #5068

[rdettai]

After some further analysis, it seems that the ingester decomissioning freeze happens when we stop the cluster right after the first ingestions. Likely race condition:

• indexer has no shard, no ingest v2 indexing pipeline is scheduled
• indexer receives events, creates a shard
• the control plane is shut down before rebuilding the indexing plan, so the pipeline is never scheduled for the shard
• the indexer hangs on shutdown because the shard is never indexed

[rdettai]

Adding a wait after the ingest to give time to the control plane to schedule an indexing pipeline works but:

• on some runs, the indexing pipeline becomes irresponsive (no extra logs)
• on runs where the indexing goes through, the decommissioning of the indexer still fails. It looks like it tries to index a new empty batch right in the middle of the teardown, and that gets stuck forever because the control plane and metastore are not available anymore:

[INFO  quickwit-indexing] index-doc-batches; index_id=otel-logs-v0_7 source_id=_ingest-source pipeline_uid=01J47CHTZPH4SX2J4D6HSVY5RH workbench_id=01J47CJDJEAQBP1NHP0GJTGDMP

This can be reproduced with:

cargo test --package quickwit-integration-tests --lib --all-features --tests::index_tests::test_ingest_logs_with_otlp_grpc_api --exact --show-output

Note: in the test_ingest_v2_happy_path test if we don't delete the index before shutting down the cluster we end up with the same error (trying to index an empty batch late in the tear down process)

[rdettai]

Note: the issue mentioned above

on some runs, the indexing pipeline becomes irresponsive (no extra logs)

was not addressed. I didn't manage to consistently reproduce or get more logs.

[fmassot]

See comments.

[fmassot]

idea: we could have a add_json_doc function on the DocBatchV2Builder.

[rdettai]

I copied over the JsonDocBatchBuilder logic from V1. To use DocBatchV2Builder directly, we would either:

1. need to first serialize the struct then add thoses bytes to the BytesMut, but that means an extra copy
2. also use a Writer<BytesMut> in DocBatchV2Builder, but I think it is not 0 cost, otherwise it would have been done that way in V1 (going through the writer involves some extra if statements)

[rdettai]

Following the other review comments, I simplified the JsonDocBatchV2Builder as much as possible by avoiding the unnecessary forth and back with DocBatchV2Builder. I would prefer not to unify the two structures without ensuring that there isn't a performance impact as this is on the critical write path. But measuring this could be a bit complicated. What do you think?

[fmassot]

fine for me!