Invalid request sent from clickhouse-cpp #426

IyeOnline · 2025-05-23T13:24:12Z

I am sending data to a local clickhouse instance using clickhouse-cpp. The
program successfully send some data, but at some point fails. The point where it fails seems to be random.

My local client then throws a std::system_error:

fail to send 8192 bytes of data: Connection reset by peer

which originates from socket.cpp:462

My clickhouse sever is a local instance simply started via ./clickhouse server.

Looking into the server's console output/log I can see

server log "Unrecognized token: Syntax error"

2025.05.23 14:28:23.925116 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Debug> executeQuery: (from 127.0.0.1:50400) INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES (stage: Complete)
2025.05.23 14:28:23.931055 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Debug> TCPHandler: Processed in 0.006340198 sec.
2025.05.23 14:28:23.931178 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Error> executeQuery: Code: 62. DB::Exception: Unrecognized token: Syntax error (data type): failed at position 1 (): JJJJJJJJJ. . (SYNTAX_ERROR) (version 25.6.1.781 (official build)) (from 127.0.0.1:50400) (in query: INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

2025.05.23 14:28:23.931449 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Error> TCPHandler: Can't send logs or exception to client. Close connection.: Code: 101. DB::Exception: Unexpected packet Hello received from client. (UNEXPECTED_PACKET_FROM_CLIENT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009e8968c
2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009e95a0b
3. DB::TCPHandler::processUnexpectedHello() @ 0x00000000153decbf
4. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e5bd0
5. DB::TCPHandler::skipData(DB::QueryState&) @ 0x00000000153dc670
6. DB::TCPHandler::runImpl() @ 0x00000000153cc7fe
7. DB::TCPHandler::run() @ 0x00000000153ea719
8. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
9. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
10. Poco::PooledThread::run() @ 0x0000000018aca19b
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
12. start_thread @ 0x000000000009caa4
13. __clone3 @ 0x0000000000129c3c
 (version 25.6.1.781 (official build))
2025.05.23 14:28:23.931557 [ 46881 ] {} <Error> TCPHandler: Code: 62. DB::Exception: Unrecognized token: Syntax error (data type): failed at position 1 (): JJJJJJJJJ. . (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

I have also seen a variation of this where the error was Empty Query:, but i got the same failure in the client:

alternative server log "Empty query. (SYNTAX_ERROR)"

2025.05.23 14:55:32.203191 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Debug> executeQuery: (from 127.0.0.1:46478) INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES (stage: Complete)
2025.05.23 14:55:32.211155 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Debug> TCPHandler: Processed in 0.008380116 sec.
2025.05.23 14:55:32.211270 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Error> executeQuery: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR) (version 25.6.1.781 (official build)) (from 127.0.0.1:46478) (in query: INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

2025.05.23 14:55:32.211535 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Error> TCPHandler: Can't send logs or exception to client. Close connection.: Code: 101. DB::Exception: Unexpected packet Hello received from client. (UNEXPECTED_PACKET_FROM_CLIENT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009e8968c
2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009e95a0b
3. DB::TCPHandler::processUnexpectedHello() @ 0x00000000153decbf
4. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e5bd0
5. DB::TCPHandler::skipData(DB::QueryState&) @ 0x00000000153dc670
6. DB::TCPHandler::runImpl() @ 0x00000000153cc7fe
7. DB::TCPHandler::run() @ 0x00000000153ea719
8. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
9. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
10. Poco::PooledThread::run() @ 0x0000000018aca19b
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
12. start_thread @ 0x000000000009caa4
13. __clone3 @ 0x0000000000129c3c
 (version 25.6.1.781 (official build))
2025.05.23 14:55:32.211631 [ 46881 ] {} <Error> TCPHandler: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

I also got back the exception from the server a few times:

server log where I get back the exception in my client

2025.05.27 14:56:24.999297 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Debug> executeQuery: (from 127.0.0.1:53512) INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata`,`answers`,`query`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint`,`dst_endpoint`,`connection_info`,`status`,`status_id`,`unmapped` ) VALUES (stage: Complete)
2025.05.27 14:56:25.006626 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Debug> TCPHandler: Processed in 0.007714557 sec.
2025.05.27 14:56:25.006968 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Error> executeQuery: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR) (version 25.6.1.781 (official build)) (from 127.0.0.1:53512) (in query: INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata`,`answers`,`query`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint`,`dst_endpoint`,`connection_info`,`status`,`status_id`,`unmapped` ) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

2025.05.27 14:56:25.015245 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Error> TCPHandler: Can't send logs or exception to client. Close connection.: Code: 101. DB::Exception: Unexpected packet Hello received from client. (UNEXPECTED_PACKET_FROM_CLIENT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009e8968c
2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009e95a0b
3. DB::TCPHandler::processUnexpectedHello() @ 0x00000000153decbf
4. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e5bd0
5. DB::TCPHandler::skipData(DB::QueryState&) @ 0x00000000153dc670
6. DB::TCPHandler::runImpl() @ 0x00000000153cc7fe
7. DB::TCPHandler::run() @ 0x00000000153ea719
8. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
9. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
10. Poco::PooledThread::run() @ 0x0000000018aca19b
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
12. start_thread @ 0x000000000009caa4
13. __clone3 @ 0x0000000000129c3c
 (version 25.6.1.781 (official build))
2025.05.27 14:56:25.015351 [ 27578 ] {} <Error> TCPHandler: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

As far as I can tell, the Block I am sending is well-formed.

I can provide a full end-to-end setup/demo if you are interested. Our source
code is open-source here: https://github.com/tenzir/tenzir/tree/main/plugins/clickhouse
and I can also give you the pipeline & data used for testing. Unfortunately I
was not yet able to craft a smaller scale test case. For smaller handcrafted
tests, I do not have any issues.

Let me know if I can assist in tracking this down in any way, I'd be happy to
help!

The text was updated successfully, but these errors were encountered:

mshustov · 2025-05-27T06:56:35Z

I can provide a full end-to-end setup/demo if you are interested. Our source

yes, a minimal repro scenario would be helpful.

IyeOnline · 2025-05-28T16:05:24Z

I have mostly figured it out now.

It is caused by trying to send invalid Arrays with missmatching offsets + data arrays. While the invalid arrays are clearly created by my code, there seem to be some basic sanity checks for this missing in clickhouse-cpp .

More importantly, the Clickhouse sever itself also does not properly guard against this/validate the request it gets, causing it to essentially "go out of bounds", leading to almost random errors.

Once I have my own thing fixed, I will create a simple repro and file proper issues for both of these points.

IyeOnline · 2025-06-03T10:48:25Z

I opened up a PR that fixes my issue and added a reproduction to that:

#428

IyeOnline linked a pull request Jun 3, 2025 that will close this issue

Introduce validation to the ColumnArray constructor #428

Open

IyeOnline mentioned this issue Jun 3, 2025

Missing validation for data send by an external client ClickHouse/ClickHouse#81199

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Invalid request sent from clickhouse-cpp #426

Invalid request sent from clickhouse-cpp #426

IyeOnline commented May 23, 2025 •

edited

Loading

mshustov commented May 27, 2025

Uh oh!

IyeOnline commented May 28, 2025

Uh oh!

IyeOnline commented Jun 3, 2025

Uh oh!

Invalid request sent from clickhouse-cpp #426

Invalid request sent from clickhouse-cpp #426

Comments

IyeOnline commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mshustov commented May 27, 2025

Uh oh!

IyeOnline commented May 28, 2025

Uh oh!

IyeOnline commented Jun 3, 2025

Uh oh!

IyeOnline commented May 23, 2025 •

edited

Loading