Run soak tests from EC2

[anchitj]

What

Adds OTEL config which would be used by OTEL agent to export the metrics and logs of Soak tests to Datadog.

Instructions to install OTEL agent on EC2:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_RESOURCE_ATTRIBUTES=service.name=njc-py-soak-tests-normal-2-2-1,service.version=v2.2.1-RC1-dev_soaktest_k8s_on_2.2.1
export OTEL_TRACES_EXPORTER=none

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.117.0/otelcol-contrib_0.117.0_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.117.0_linux_amd64.deb
otelcol-contrib --config otel_config.yaml

This would start OTEL agent and tests would start exporting metrics and logs to it.

Other changes to run with EC2

export HOSTNAME=$(hostname)
PYTHONPATH=/home/ubuntu/confluent-kafka-python/tests opentelemetry-instrument confluent-kafka-python/tests/soak/soakclient.py -t $topic -r 80 -f confluent-kafka-python/ccloud.config -i t1
pip install psutil

Checklist

• Contains customer facing changes? Including API/behavior changes
• Did you add sufficient unit test and/or integration test coverage for this PR?
  • If not, please explain why it is not required

References

JIRA:

Test & Review

Open questions / Follow-ups

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ anchitj
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[Copilot]

PR Overview

This PR adds a configuration file for the OTEL agent to export metrics and logs from soak tests to Datadog.

• Adds a new YAML file for OTEL configuration.
• Configures receivers, processors, exporters, and service pipelines for both metrics and logs.

Reviewed Changes

File	Description
tests/soak/otel-config.yaml	Added OTEL configuration with receivers, processors, exporters, and service pipelines for Datadog integration

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[sonarqube-confluent]

Analysis Details

5 Issues

• 0 Bugs
• 0 Vulnerabilities
• 5 Code Smells

Coverage and Duplications

• No coverage information (66.10% Estimated after merge)
• No duplication information (5.40% Estimated after merge)

Project ID: confluent-kafka-python

View in SonarQube

[k-raina]

Thanks for PR! Left few comments.

[k-raina]

Can we move out versions in script to common config file?

[k-raina]

Maybe change it to FILL_IN_REGION_HERE?

[k-raina]

Why have we commented out 2.10.x version?

[k-raina]

Should this be updated to master?

[k-raina]

Maybe add validation for otel config eg.

validate_config() {
    local errors=0
    
    if grep -q "FILL_IN_ROLE_ARN_HERE" otel-config.yaml; then
        echo "ERROR: AWS Role ARN not configured in otel-config.yaml"
        echo "Please set AWS_ROLE_ARN environment variable or edit the file"
        errors=$((errors + 1))
    fi
    
    if grep -q "FILL_IN_REMOTE_WRITE_ENDPOINT_HERE" otel-config.yaml; then
        echo "ERROR: Prometheus endpoint not configured"
        echo "Please set PROMETHEUS_ENDPOINT environment variable"
        errors=$((errors + 1))
    fi
    
    if [[ $errors -gt 0 ]]; then
        echo "Configuration validation failed. Please fix the above errors."
        exit 1
    fi
    
    echo "Configuration validation passed."
}

[k-raina]

Suggested change

	sudo make uninstall
	make uninstall

[k-raina]

Maybe below can be followup tasks?

• Add log rotation.
• Add Log cleanup weekly or monthly.

[k-raina]

What's your opinion on having master branch also? This will help catch issues before release?