Skip to content

Invalid request sent from clickhouse-cpp #426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
IyeOnline opened this issue May 23, 2025 · 3 comments · May be fixed by #428
Open

Invalid request sent from clickhouse-cpp #426

IyeOnline opened this issue May 23, 2025 · 3 comments · May be fixed by #428

Comments

@IyeOnline
Copy link

IyeOnline commented May 23, 2025

I am sending data to a local clickhouse instance using clickhouse-cpp. The
program successfully send some data, but at some point fails. The point where it fails seems to be random.

My local client then throws a std::system_error:

fail to send 8192 bytes of data: Connection reset by peer

which originates from socket.cpp:462

My clickhouse sever is a local instance simply started via ./clickhouse server.

Looking into the server's console output/log I can see

server log "Unrecognized token: Syntax error"
2025.05.23 14:28:23.925116 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Debug> executeQuery: (from 127.0.0.1:50400) INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES (stage: Complete)
2025.05.23 14:28:23.931055 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Debug> TCPHandler: Processed in 0.006340198 sec.
2025.05.23 14:28:23.931178 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Error> executeQuery: Code: 62. DB::Exception: Unrecognized token: Syntax error (data type): failed at position 1 (): JJJJJJJJJ. . (SYNTAX_ERROR) (version 25.6.1.781 (official build)) (from 127.0.0.1:50400) (in query: INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

2025.05.23 14:28:23.931449 [ 46881 ] {40ff087d-cda1-4b93-9c57-e5fbfabfb96a} <Error> TCPHandler: Can't send logs or exception to client. Close connection.: Code: 101. DB::Exception: Unexpected packet Hello received from client. (UNEXPECTED_PACKET_FROM_CLIENT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009e8968c
2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009e95a0b
3. DB::TCPHandler::processUnexpectedHello() @ 0x00000000153decbf
4. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e5bd0
5. DB::TCPHandler::skipData(DB::QueryState&) @ 0x00000000153dc670
6. DB::TCPHandler::runImpl() @ 0x00000000153cc7fe
7. DB::TCPHandler::run() @ 0x00000000153ea719
8. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
9. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
10. Poco::PooledThread::run() @ 0x0000000018aca19b
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
12. start_thread @ 0x000000000009caa4
13. __clone3 @ 0x0000000000129c3c
 (version 25.6.1.781 (official build))
2025.05.23 14:28:23.931557 [ 46881 ] {} <Error> TCPHandler: Code: 62. DB::Exception: Unrecognized token: Syntax error (data type): failed at position 1 (): JJJJJJJJJ. . (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

I have also seen a variation of this where the error was Empty Query:, but i got the same failure in the client:

alternative server log "Empty query. (SYNTAX_ERROR)"
2025.05.23 14:55:32.203191 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Debug> executeQuery: (from 127.0.0.1:46478) INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES (stage: Complete)
2025.05.23 14:55:32.211155 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Debug> TCPHandler: Processed in 0.008380116 sec.
2025.05.23 14:55:32.211270 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Error> executeQuery: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR) (version 25.6.1.781 (official build)) (from 127.0.0.1:46478) (in query: INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata_log_name`,`metadata_logged_time`,`metadata_product_name`,`metadata_product_vendor_name`,`metadata_product_cpe_name`,`metadata_uid`,`metadata_version`,`answers_rdata`,`answers_ttl`,`query_class`,`query_hostname`,`query_type`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint_ip`,`src_endpoint_port`,`dst_endpoint_ip`,`dst_endpoint_port`,`connection_info_direction`,`connection_info_direction_id`,`connection_info_protocol_name`,`connection_info_protocol_num`,`connection_info_protocol_ver_id`,`status`,`status_id`,`unmapped` ) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

2025.05.23 14:55:32.211535 [ 46881 ] {18a4db8a-80fb-4234-bff7-e4f5fa4526d9} <Error> TCPHandler: Can't send logs or exception to client. Close connection.: Code: 101. DB::Exception: Unexpected packet Hello received from client. (UNEXPECTED_PACKET_FROM_CLIENT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009e8968c
2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009e95a0b
3. DB::TCPHandler::processUnexpectedHello() @ 0x00000000153decbf
4. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e5bd0
5. DB::TCPHandler::skipData(DB::QueryState&) @ 0x00000000153dc670
6. DB::TCPHandler::runImpl() @ 0x00000000153cc7fe
7. DB::TCPHandler::run() @ 0x00000000153ea719
8. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
9. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
10. Poco::PooledThread::run() @ 0x0000000018aca19b
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
12. start_thread @ 0x000000000009caa4
13. __clone3 @ 0x0000000000129c3c
 (version 25.6.1.781 (official build))
2025.05.23 14:55:32.211631 [ 46881 ] {} <Error> TCPHandler: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

I also got back the exception from the server a few times:

server log where I get back the exception in my client
2025.05.27 14:56:24.999297 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Debug> executeQuery: (from 127.0.0.1:53512) INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata`,`answers`,`query`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint`,`dst_endpoint`,`connection_info`,`status`,`status_id`,`unmapped` ) VALUES (stage: Complete)
2025.05.27 14:56:25.006626 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Debug> TCPHandler: Processed in 0.007714557 sec.
2025.05.27 14:56:25.006968 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Error> executeQuery: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR) (version 25.6.1.781 (official build)) (from 127.0.0.1:53512) (in query: INSERT INTO ocsf_dns ( `activity_id`,`activity_name`,`category_uid`,`category_name`,`class_uid`,`class_name`,`severity_id`,`severity`,`type_uid`,`time`,`start_time`,`metadata`,`answers`,`query`,`query_time`,`response_time`,`rcode`,`rcode_id`,`src_endpoint`,`dst_endpoint`,`connection_info`,`status`,`status_id`,`unmapped` ) VALUES), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

2025.05.27 14:56:25.015245 [ 27578 ] {3fefab3e-45b5-4be3-a9a6-dbba9ea53f02} <Error> TCPHandler: Can't send logs or exception to client. Close connection.: Code: 101. DB::Exception: Unexpected packet Hello received from client. (UNEXPECTED_PACKET_FROM_CLIENT), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009e8968c
2. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000009e95a0b
3. DB::TCPHandler::processUnexpectedHello() @ 0x00000000153decbf
4. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e5bd0
5. DB::TCPHandler::skipData(DB::QueryState&) @ 0x00000000153dc670
6. DB::TCPHandler::runImpl() @ 0x00000000153cc7fe
7. DB::TCPHandler::run() @ 0x00000000153ea719
8. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
9. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
10. Poco::PooledThread::run() @ 0x0000000018aca19b
11. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
12. start_thread @ 0x000000000009caa4
13. __clone3 @ 0x0000000000129c3c
 (version 25.6.1.781 (official build))
2025.05.27 14:56:25.015351 [ 27578 ] {} <Error> TCPHandler: Code: 62. DB::Exception: Empty query. (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fc7139b
1. DB::Exception::createDeprecated(String const&, int, bool) @ 0x000000000fd3a94d
2. DB::parseQueryAndMovePosition(DB::IParser&, char const*&, char const*, String const&, bool, unsigned long, unsigned long, unsigned long) @ 0x0000000015ee7b52
3. std::shared_ptr<DB::IDataType const> DB::DataTypeFactory::getImpl<false>(String const&) const @ 0x0000000012e4edf6
4. DB::NativeReader::read() @ 0x0000000015433b67
5. DB::TCPHandler::receivePacketsExpectData(DB::QueryState&) @ 0x00000000153e4fe0
6. DB::TCPHandler::processInsertQuery(DB::QueryState&) @ 0x00000000153daea3
7. DB::TCPHandler::runImpl() @ 0x00000000153cb748
8. DB::TCPHandler::run() @ 0x00000000153ea719
9. Poco::Net::TCPServerConnection::start() @ 0x0000000018afea67
10. Poco::Net::TCPServerDispatcher::run() @ 0x0000000018afeeb9
11. Poco::PooledThread::run() @ 0x0000000018aca19b
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000018ac867d
13. start_thread @ 0x000000000009caa4
14. __clone3 @ 0x0000000000129c3c

As far as I can tell, the Block I am sending is well-formed.

I can provide a full end-to-end setup/demo if you are interested. Our source
code is open-source here: https://github.com/tenzir/tenzir/tree/main/plugins/clickhouse
and I can also give you the pipeline & data used for testing. Unfortunately I
was not yet able to craft a smaller scale test case. For smaller handcrafted
tests, I do not have any issues.

Let me know if I can assist in tracking this down in any way, I'd be happy to
help!

@mshustov
Copy link
Member

I can provide a full end-to-end setup/demo if you are interested. Our source

yes, a minimal repro scenario would be helpful.

@IyeOnline
Copy link
Author

I have mostly figured it out now.

It is caused by trying to send invalid Arrays with missmatching offsets + data arrays. While the invalid arrays are clearly created by my code, there seem to be some basic sanity checks for this missing in clickhouse-cpp .

More importantly, the Clickhouse sever itself also does not properly guard against this/validate the request it gets, causing it to essentially "go out of bounds", leading to almost random errors.

Once I have my own thing fixed, I will create a simple repro and file proper issues for both of these points.

@IyeOnline IyeOnline linked a pull request Jun 3, 2025 that will close this issue
@IyeOnline
Copy link
Author

I opened up a PR that fixes my issue and added a reproduction to that:

#428

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants