Add support for MSK with IAM authentication

[madebydna]

This adds support for using MSK IAM authentication with the rdkafka2 output type. Authentication and authorization with an MSK cluster are facilitated through a base64-encoded signed URL, which is generated by the aws-msk-iam-sasl-signer-ruby library.

[madebydna]

Reopening it based on conversation here.

[madebydna]

I fixed the dependencies that require Ruby > 3.0, so that the CI tests using Ruby 2.7 now also pass. I also added rdkafka to the gemspec file instead of the Gemfile, but leaving the USE_RDKAFKA conditional intact. Was there a particular reason that dependency was only in the Gemfile?

If there is interest at all in merging it in, I'd be happy to continue making improvements to the code with some input. Please let me know if this feature is not something that is considered useful and I can close the PR.

In terms of possible improvements, I was wondering if it would be better to have a config param such as use_aws_msk_iam_auth and not require the two rdkafka_options of security.protocol and sasl.mechanisms to be set explicitly. So instead of

<match *>
    @type rdkafka2
    # other options
    
    # AWS Region (required)
    aws_msk_region us-east-1
    
    # MSK IAM authentication settings (required)
    rdkafka_options {
      "security.protocol": "sasl_ssl",
      "sasl.mechanisms": "OAUTHBEARER"
    }
</match>

the config could look like this:

<match *>
    @type rdkafka2
    # ... other options ....

    use_aws_msk_iam_auth true
    aws_msk_region us-east-1
</match>

Also, I was wondering if there is a way to not use the @shared_producer but use the per-thread producers instead, since that is the default. Currently, there is a deadlock when calling get_producer from the oauthbearer_token_refresh_callback and the the share_producer setting is not set to true.

[github-actions]

This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days

[yogrr]

Any updates?

[madebydna]

I'd be happy to continue to work on it to get this merged. Is the DCO the reason this PR has not received any attention, @daipom?

[daipom]

Sorry I haven’t been able to make time for this PR.

I'd be happy to continue to work on it to get this merged. Is the DCO the reason this PR has not received any attention, @daipom?

Thanks!!
Yes! We need to add DCO.
In addition, we need to fix the conflict.

[madebydna]

@daipom I have fixed the gemspec conflicts and signed off my commits. However, there is still a DCO issue that I don't know how to fix:

Commit sha: [632faa2]..., Author: Kentaro Hayashi, Committer: Andrea Singh; 
Expected "Kentaro Hayashi [[email protected]](mailto:[email protected])", 
but got "Kentaro Hayashi [[email protected]](mailto:[email protected])".

[madebydna]

Closing this PR in favor of this one with a cleaner git history.

[daipom]

Sorry for my late response.
Thanks!
It looks like the issue was that the Git history had become complicated.
So, it looks better to either rebase and force push or recreating the PR.
#537 looks good!