Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore(deps): Update ClickHouse SQL driver to v2 #16423

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

chore(deps): Update ClickHouse SQL driver to v2 #16423

wants to merge 1 commit into from
+167 −39

Conversation

AndreKR
Copy link

@AndreKR AndreKR commented Jan 22, 2025

Summary

This updates the clickhouse SQL driver (and its dependencies) to v2.

This is mainly because the v1 driver doesn't support the HTTP protocol, but it also helps with some issues regarding DSN parsing that v1 suffers from.

This includes another commit that changes the default TableTemplate so that it works with ClickHouse. Previously everyone who wanted to use ClickHouse had to override TableTemplate.

Checklist

  • No AI generated code was used in this PR

Related issues

resolves #16422

@AndreKR AndreKR force-pushed the clickhouse-upgrade branch 2 times, most recently from f1c28d0 to f797456 Compare January 22, 2025 11:47
@AndreKR AndreKR marked this pull request as draft January 22, 2025 11:51
@AndreKR AndreKR changed the title Update ClickHouse driver chore(deps): Update ClickHouse SQL driver to v2 Jan 22, 2025
@telegraf-tiger telegraf-tiger bot added the chore label Jan 22, 2025
@AndreKR AndreKR force-pushed the clickhouse-upgrade branch 6 times, most recently from 80069ac to d6a09dd Compare January 26, 2025 21:21
@AndreKR AndreKR marked this pull request as ready for review January 26, 2025 21:41
@AndreKR AndreKR force-pushed the clickhouse-upgrade branch from d6a09dd to 3c523dd Compare January 26, 2025 22:11
@AndreKR AndreKR marked this pull request as draft January 26, 2025 22:39
@AndreKR AndreKR force-pushed the clickhouse-upgrade branch 2 times, most recently from 5cd941f to d2682fd Compare January 28, 2025 01:24
@AndreKR AndreKR marked this pull request as ready for review January 28, 2025 11:59
@AndreKR AndreKR force-pushed the clickhouse-upgrade branch from a5cd7e3 to d2682fd Compare January 28, 2025 12:30
@srebhan srebhan self-assigned this Jan 28, 2025
@srebhan srebhan added the dependencies Pull requests that update a dependency file label Jan 28, 2025
@srebhan
Copy link
Member

srebhan commented Jan 28, 2025

@AndreKR do current v1 DSNs work with the new driver?

@AndreKR
Copy link
Author

AndreKR commented Jan 28, 2025

No, I don't think so. The main difference is that in v1 database was a parameter (tcp://.../?database=<database>) while in v2 it's the path (tcp://.../<database>?).

@srebhan
Copy link
Member

srebhan commented Jan 29, 2025

Do you see some way of converting old DSNs? Maybe DSNs can be converted by parsing them with the v1 parser and (if successful) then create a v2 DSN from it? I'm asking because Telegraf cannot do these kind of breaking changes, therefore we need to find a way to keep the old configs (and thus DSNs) working...

@AndreKR
Copy link
Author

AndreKR commented Jan 29, 2025

Yes, I think that would be possible. I'll prepare something.

@AndreKR AndreKR marked this pull request as draft January 29, 2025 12:05
@AndreKR AndreKR force-pushed the clickhouse-upgrade branch 4 times, most recently from 58bd017 to 88fce4f Compare January 31, 2025 03:09
@AndreKR
Copy link
Author

AndreKR commented Jan 31, 2025

@srebhan I have added code that converts parameters in the DSN that have been renamed or have a different format to their v2 equivalents. A warning is logged for each legacy parameter. A number of parameters (tls_config, no_delay, write_timeout, block_size, check_connection_liveness) have no equivalent in v2. These are simply removed from the DSN and a warning is logged as well.

@AndreKR AndreKR marked this pull request as ready for review January 31, 2025 03:42
srebhan
srebhan reviewed Jan 31, 2025
View reviewed changes
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @AndreKR for the amazing work! I have some comments in the code...

Additionally, did you test that the conversion function leaves newer v2 DSNs unchanged? Would be good to get some testing there...

Last but not least, please also add a note in the Important Changes section of the Changelog about the change, something like

- PR [#16423](https://github.com/influxdata/telegraf/pull/16423) converts the clickhouse drivers to the v2 version.
  This new version also requires a [new format for the DSN](url). The plugin tries
  its best to convert the old DSN to the new format but might not be able to do so.
  Please check for warnings in your log file and convert to the new format as soon
  as possible

plugins/outputs/sql/sample.conf Outdated Show resolved Hide resolved
plugins/outputs/sql/sql.go Outdated Show resolved Hide resolved
plugins/outputs/sql/sql.go Outdated Show resolved Hide resolved
plugins/outputs/sql/sql.go Outdated Show resolved Hide resolved
plugins/outputs/sql/sql.go Outdated Show resolved Hide resolved
plugins/outputs/sql/sql.go Outdated Show resolved Hide resolved
plugins/outputs/sql/README.md Show resolved Hide resolved
plugins/outputs/sql/sql_test.go Outdated Show resolved Hide resolved
@AndreKR AndreKR marked this pull request as draft February 1, 2025 22:41
@AndreKR AndreKR force-pushed the clickhouse-upgrade branch 3 times, most recently from 6aa4837 to d73a52e Compare February 2, 2025 00:08
@AndreKR
Copy link
Author

AndreKR commented Feb 2, 2025

Additionally, did you test that the conversion function leaves newer v2 DSNs unchanged? Would be good to get some testing there...

I had a test about leaving a v2 URL untouched, I have added another one for a v2 URL containing parameters that would be changed if they had incompatible values but they have already compatible values.

Last but not least, please also add a note in the Important Changes section of the Changelog about the change

Done.

@AndreKR AndreKR marked this pull request as ready for review February 2, 2025 00:19
@AndreKR
chore(deps): Update clickhouse driver to v2
b787dec 
This updates the clickhouse SQL driver for the SQL output plugin to v2.

This keeps some compatibility with v1 DSNs by converting parameters from
the v1 to the v2 format while logging a warning. Some parameters are no
longer supported, those are removed while logging a warning as well.
@AndreKR AndreKR force-pushed the clickhouse-upgrade branch from d73a52e to b787dec Compare February 2, 2025 20:24
@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented Feb 2, 2025

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 1.20 % for linux amd64 (new size: 287.4 MB, nightly size 284.0 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB RPM TAR GZ ZIP
amd64.deb aarch64.rpm darwin_amd64.tar.gz windows_amd64.zip
arm64.deb armel.rpm darwin_arm64.tar.gz windows_arm64.zip
armel.deb armv6hl.rpm freebsd_amd64.tar.gz windows_i386.zip
armhf.deb i386.rpm freebsd_armv7.tar.gz
i386.deb ppc64le.rpm freebsd_i386.tar.gz
mips.deb riscv64.rpm linux_amd64.tar.gz
mipsel.deb s390x.rpm linux_arm64.tar.gz
ppc64el.deb x86_64.rpm linux_armel.tar.gz
riscv64.deb linux_armhf.tar.gz
s390x.deb linux_i386.tar.gz
linux_mips.tar.gz
linux_mipsel.tar.gz
linux_ppc64le.tar.gz
linux_riscv64.tar.gz
linux_s390x.tar.gz

srebhan
srebhan approved these changes Feb 4, 2025
View reviewed changes
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @AndreKR for the update and for splitting out the other topics into separate PRs! It's a joy to work with you...

@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Feb 4, 2025
@srebhan srebhan assigned DStrand1 and unassigned srebhan Feb 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srebhan srebhan srebhan approved these changes

Assignees

@DStrand1 DStrand1

Labels
chore dependencies Pull requests that update a dependency file ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update ClickHouse driver to support HTTP
3 participants
@AndreKR @srebhan @DStrand1