feat(contrib/sarama): option to add custom consumer/producer span tags

[fabiomargarido]

What does this PR do?

Add WithConsumerCustomTag and WithProducerCustomTag options to enable adding custom tags to spans generated by wrapped consumers and producers.

Motivation

Including extra information (e.g. Kafka message keys) as tags on spans can greatly enhance observability, enabling users to filter, group, and analyze traces based on specific message properties. This improves debugging capabilities and provides a clearer understanding of message flow within Kafka-based systems.

Reviewer's Checklist

• Changed code has unit tests for its functionality at or near 100% coverage.
• System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
• There is a benchmark for any new code, or changes to existing code.
• If this interacts with the agent in a new way, a system test has been added.
• New code is free of linting errors. You can check this by running ./scripts/lint.sh locally.
• Add an appropriate team label so this PR gets put in the right place for the release notes.
• Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

[fabiomargarido]

Discussing this with colleagues we also came up with the idea of changing this to a more generic WithCustomTags which takes a func that returns a slice of tags to be added to the span. I can change it to that if you think it makes more sense.

[kakkoyun]

Discussing this with colleagues we also came up with the idea of changing this to a more generic WithCustomTags which takes a func that returns a slice of tags to be added to the span. I can change it to that if you think it makes more sense.

This sounds more future-proof TBH.

[fabiomargarido]

Cool, thanks for the feedback @kakkoyun. I'm on vacation for the next few weeks so not sure when I'll be able to update this, but I'll try to make some time to do so.

[gubtos]

@kakkoyun Could you please let me know if there are any additional changes needed, or if the implemented code looks good to you?

[fabiomargarido]

@rarguelloF I have pushed the requested changes, can you please check it again? Thanks