-
Notifications
You must be signed in to change notification settings - Fork 3.3k
feat(events): improve events kafka pool & client retry #15429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
25e55e6 to
3de852e
Compare
3de852e to
f778943
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
f778943 to
ed3dd9a
Compare
ed3dd9a to
c325865
Compare
|
✅ Meticulous spotted 0 visual differences across 998 screens tested: view results. Meticulous evaluated ~8 hours of user flows against your PR. Expected differences? Click here. Last updated for commit c325865. This comment will update as new commits are pushed. |
0483129 to
72d0d62
Compare
72d0d62 to
2da26e2
Compare
2da26e2 to
44f8fcf
Compare
* Check consumers on access from kafka consumer pool * Allow infinite retries from python actions consumer * Locking fixes, race condition fixes with consumer pool
44f8fcf to
39044eb
Compare
Improve Kafka Consumer Pool and Python Client Retry Logic
Summary
This PR improves the Kafka consumer pool's efficiency and thread-safety, and enhances the Python client's retry capabilities for handling transient connection failures.
Changes
Kafka Consumer Pool (
KafkaConsumerPool.java)Fixes:
totalConsumersCreateddecrement withpoolManagementLockto prevent exceedingmaxPoolSizeunder concurrent accessshutdownPool()to avoid re-acquiring locks unnecessarilyactiveConsumers.clear()callOptimizations:
remainingTimecalculation to compute once per loop iterationPython Client Retry Logic (
datahub_cloud_events_consumer.py)Enhancements:
infinite_retryoption to allow indefinite retries on connection failuresConfiguration:
infinite_retry: false(default): Retries up to 15 times with exponential backoffinfinite_retry: true: Retries indefinitely with exponential backoff until connection restoredDocumentation
datahub-cloud-event-source.mdwithinfinite_retryconfiguration detailsImpact
Includes: #15417