Support sending custom tags when using pollDBStats

[KaibutsuX]

What does this PR do?

Motivation

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.
• For internal contributors, a matching PR should be created to the v2-dev branch and reviewed by @DataDog/apm-go.

Unsure? Have a question? Request a review!

[darccio]

@mtoffl01 Was there any reason behind not allowing tags from the beginning?

Checking the code, I see that for the purpose of this PR, it should be enough to set DD_TAGS or OTEL_PROPAGATORS (if using OTel) to set these global tags.

WDYT @KaibutsuX?

[KaibutsuX]

I don't have enough context or documentation to know what DD_TAGS is or how I would set it as an end-user.

[darccio]

DD_TAGS is properly explained in our Go tracer public documentation.

We understand this should be enough to achieve what you are trying with this PR.

[KaibutsuX]

That links says: "A list of default tags to be added to every span and profile." From my tracing through the code, I could not see that this same list of user-supplied tags was included in these metrics used in pollDBStats

[darccio]

Yes, we understand it may not that be clear from the code. Sorry about that. It inherits the global tags from the tracer config, set by globalconfig.SetStatsTags, which is initialized by newConfig, and retrieved when setting up the DB statsd client.

We'll update the public documentation to make sure it reflects this detail.

[mtoffl01]

Hi @KaibutsuX ,

As @darccio pointed out, tags set in DD_TAGS are applied to DBStats metrics. We are updating the Go tracer public documentation to include "metrics" in the list of data to which tags are applied to, so this should be clear for future readers.

Are you trying to pass your global tags down to DBStats metrics, or are you looking to apply a unique set of tags specifically to those metrics? If the former, DD_TAGS will do the trick. If the latter, this feature is not currently supported, but we'd be happy to learn more about your use case so we can consider implementing it.

Thank you!

[KaibutsuX]

@mtoffl01 I will see if using DD_TAGS works as expected.

You may want to also update the documentation for DD_TAGS as it is marked as an array, but in go code it accepts a map[string]interface{} which implicitly only respects the entries that can be asserted as strings. Further, it's unclear that the labels will be formatted as labelKey:labelValue.(string) especially since the documentation explicitly says that DD_TAGS supports multiple values as csv using commas or spaces.

[mtoffl01]

Hi @KaibutsuX ,

Thanks for the tip! We will change the Default value in the docs and update the description to specify that keys and values should be string-convertible types. In the meantime, keep us posted on your success with DD_TAGS.

[darccio]

@KaibutsuX Is everything solved with the tips we shared? Do you want to still push this into the codebase? Thanks!

[KaibutsuX]

Using the alternative suggestion does not allow me to set the metrics namespace. Instead all tags are sent to the default namespace, which in my use-case makes the tagging unhelpful and results in just more noise. For the time being, I'm still using a local version of this PR to support my own pollDBStats in which I send my tags AND set my own custom namespace.

[mtoffl01]

@KaibutsuX ,
What do you mean by "metric namespace" and "all tags are sent to the default namespace?" Indeed the expectation with DD_TAGS is that all configured key-values will be applied to all traces, metrics and logs submitted by this tracer instance.

If you would like to configure tags to apply only to your DB Stats metrics, please tell us some more about how you use these tags. If I've not understood the "namespace" term, please expand.

Thank you!

[KaibutsuX]

dd-trace-go/contrib/database/sql/option.go

Line 43 in 137578f

sc, err := internal.NewStatsdClient(globalconfig.DogstatsdAddr(), statsTags(c))

Since absolutely everything is being sent to the same namespace, if I'm only interested in 30 / 1000 tags is makes it hard to filter and use.

NewStatsdClient accepts options, one of which would support custom namespaces would be statsd.WithNamespace("my-custom-metrics")

[rarguelloF]

Suggested change

	func WithDBStatsTags(tags []string) Option {
	// WithDBStatsTags enables DBStats metrics collection and sets the given tags for those metrics.
	func WithDBStatsTags(tags map[string]string) Option {

Could we make this a map[string]string? Since this is how we model tags generally everywhere else.
We can keep cfg.statsTags as a []string and do the transformation here.

[KaibutsuX]

dd-trace-go/contrib/database/sql/metrics.go

Line 60 in 137578f

func statsTags(c *config) []string {

dd-trace-go/internal/statsd.go

Line 16 in c345adb

type StatsdClient interface {

That does make more sense, but since your public interfaces accept []string with an only an expected convention of using : as the delimiter between keys/values, I think it's probably better to address those public places first and/or update documentation.

I'm closing this as I no longer have the time to spend on it.

[mtoffl01]

@KaibutsuX , Upon revisiting this request I noticed that we already support tag customization at the integration level (that is, not global) via the WithCustomTag option: https://pkg.go.dev/gopkg.in/DataDog/dd-trace-go.v1/contrib/database/sql#WithCustomTag

WithCustomTag sets the provided key-value pair on the config's "tags" map, which is then used to populate the integration's statsd client here via the statsTags helper function.

Key differences between this option and the WithDBStatsTags option that you proposed:

1. The existing option accepts a single key and value at a time, while your proposed option accepts a list of key-vals
2. The existing option applies the provided tags to spans as well as metrics, while your proposed option applies the provided tags only to metrics.

Are these differences significant enough to you to warrant a new API? Please let us know and we will consider this further.

By the way, the documentation for the existing WithCustomTag option predates the DBStats feature, so it misleadingly implies that the tags provided will apply only to spans, not metrics. To address this, I've opened this PR to clarify the documentation: #3246