Skip to content

Conversation

@jordotech
Copy link

Add persistent task-to-queue mapping cache to maintain queue associations across different event types. Queue information is not always available in all Celery events, leading to inaccurate or missing queue labels in metrics.

Changes:

  • Add task_queue_map dictionary to cache task UUID to queue mappings
  • Implement multi-source queue name detection (event fields, task attributes, and cached mappings)
  • Add null check for tasks to handle unknown task events gracefully
  • Update queue_cache when queue names are identified
  • Clean up task_queue_map when tasks reach ready states to prevent memory leaks

This ensures consistent and accurate queue_name labels in Prometheus metrics even when queue information is missing from certain event types.

Add persistent task-to-queue mapping cache to maintain queue
associations across different event types. Queue information is
not always available in all Celery events, leading to inaccurate
or missing queue labels in metrics.

Changes:
- Add task_queue_map dictionary to cache task UUID to queue mappings
- Implement multi-source queue name detection (event fields, task
  attributes, and cached mappings)
- Add null check for tasks to handle unknown task events gracefully
- Update queue_cache when queue names are identified
- Clean up task_queue_map when tasks reach ready states to prevent
  memory leaks

This ensures consistent and accurate queue_name labels in Prometheus
metrics even when queue information is missing from certain event types.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant