Error (code: 288, message: Double-distributed IN/JOIN subqueries is denied (distributed_product_mode = 'deny')

[mk-raven]

Clickhouse version: 24.8.13.16
Use cluster mode with DDL.
Qryn image: ghcr.io/metrico/gigapipe:latest
Logs in qryn:

Error (code: 288, message: Double-distributed IN/JOIN subqueries is denied (distributed_product_mode = 'deny'). You may rewrite query to use local tables in subqueries, or use GLOBAL keyword, or set distributed_product_mode to suitable value.). Please check the server logs for more details.

Logs of example failed query in clickhouse:

1. │ 2025-03-18 12:01:55 │ qryn_test │ WITH trace_ids as ( SELECT DISTINCT trace_id as trace_id FROM qryn_test.tempo_traces_dist as traces WHERE (((timestamp_ns) >= (1742299015000000000)) and ((timestamp_ns) <= (1742299315000000000))) ORDER BY timestamp_ns desc LIMIT 20),trace_and_span_ids as ( SELECT trace_id as trace_id, groupArray(100)(span_id) as span_id FROM qryn_test.tempo_traces_dist as traces WHERE (((timestamp_ns) >= (1742299015000000000)) and ((timestamp_ns) < (1742299315000000000)) and (trace_id IN (trace_ids))) GROUP BY trace_id),trace_and_span_ids_unnested as ( SELECT trace_id as trace_id, _span_id as span_id FROM trace_and_span_ids array JOIN trace_and_span_ids.span_id as _span_id ),index_search as ( SELECT trace_id as trace_id, span_id as span_id, duration_ns as duration, timestamp_ns as timestamp_ns FROM qryn_test.tempo_traces_dist as traces WHERE (((timestamp_ns) >= (1742299015000000000)) and ((timestamp_ns) < (1742299315000000000)) and ((traces.trace_id, traces.span_id) IN (trace_and_span_ids_unnested))) ORDER BY timestamp_ns desc),index_grouped as ( SELECT trace_id as trace_id, groupArray(100)(span_id) as span_id FROM index_search GROUP BY trace_id ORDER BY max(index_search.timestamp_ns) desc LIMIT 20),trace_span_ids as ( SELECT trace_id, span_id FROM index_grouped array JOIN span_id ),traces_info as ( SELECT traces.trace_id as trace_id, min(traces.timestamp_ns) as _start_time_unix_nano, toFloat64(max(traces.timestamp_ns + traces.duration_ns) - min(traces.timestamp_ns)) / 1000000 as _duration_ms, argMin(traces.service_name, traces.timestamp_ns) as _root_service_name, argMin(traces.name, traces.timestamp_ns) as _root_trace_name FROM qryn_test.tempo_traces as traces WHERE (traces.trace_id IN (trace_ids)) GROUP BY traces.trace_id) SELECT lower(hex(traces.trace_id)) as trace_id, arrayMap(x -> lower(hex(x)), groupArray(traces.span_id)) as span_id, groupArray(traces.duration_ns) as duration, groupArray(traces.timestamp_ns) as timestamp_ns, min(_start_time_unix_nano) as start_time_unix_nano, min(_duration_ms) as duration_ms, min(_root_service_name) as root_service_name, min(_root_trace_name) as root_trace_name FROM qryn_test.tempo_traces_dist as traces any left JOIN traces_info ON (traces.trace_id) == (traces_info.trace_id) WHERE (traces.trace_id IN (trace_ids)) and ((traces.trace_id, traces.span_id) IN (trace_span_ids)) GROUP BY traces.trace_id ORDER BY start_time_unix_nano desc LIMIT 20 │ qryn_test │

[akvlad]

@mk-raven the issue should be fixed in the gigapipe:v4.0.1-beta version. Please retry.