Skip to content

feat: add duplicate filter on log services (30s timeout) #1789

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

wdconinc
Copy link
Contributor

@wdconinc wdconinc commented Apr 12, 2025

Briefly, what does this PR introduce?

This PR adds a duplicate message filter on the logger output. If subsequent identical messages are emitted to the same logger instance within a timeout window (30 seconds by default, settable) they are eaten and replaced with a message Skipped %u duplicate messages.. (at info level, regrettably not possible at the level of the original message with the spdlog classes though we could extend the dup_filter to achieve that I guess).

Example:

13:20:41.158  [warn] Status: 82 events processed at 2.0 Hz (0.8 Hz avg)
13:20:41.658  [warn] Status: 82 events processed at 0.0 Hz (0.8 Hz avg)
13:20:42.158  [warn] Status: 82 events processed at 0.0 Hz (0.8 Hz avg)
[LOWQ2:TaggerTrackerClustering] [info] Skipped 55 duplicate messages..
[LOWQ2:TaggerTrackerClustering] [error] Could not find JFactoryT<edm4eic::TrackerHit> with tag=TaggerTrackerM1L0RawHits
13:20:42.658  [warn] Status: 83 events processed at 2.0 Hz (0.8 Hz avg)
13:20:43.158  [warn] Status: 83 events processed at 0.0 Hz (0.8 Hz avg)
[LOWQ2:TaggerTrackerClustering] [info] Skipped 55 duplicate messages..
[LOWQ2:TaggerTrackerClustering] [error] Could not find JFactoryT<edm4eic::TrackerHit> with tag=TaggerTrackerM1L0RawHits
13:20:43.659  [warn] Status: 84 events processed at 2.0 Hz (0.8 Hz avg)

What kind of change does this PR introduce?

  • Bug fix (issue #__)
  • New feature (issue: logger duplicate filtering)
  • Documentation update
  • Other: __

Please check if this PR fulfills the following:

  • Tests for the changes have been added
  • Documentation has been added / updated
  • Changes have been communicated to collaborators

Does this PR introduce breaking changes? What changes might users need to make to their code?

No.

Does this PR change default behavior?

Yes, less logger noise.

@veprbl
Copy link
Member

veprbl commented Apr 12, 2025

I don't like the idea of irreproducible log outputs. We should fix the info messages regardless. cc @simonge.

@wdconinc wdconinc force-pushed the log-dup-filter branch 2 times, most recently from c00d6c8 to cabfa6b Compare April 14, 2025 18:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants