fix(cortex): flip observability age check from oldest to newest write - #1920
Open
mcardinali-data wants to merge 1 commit into
Open
fix(cortex): flip observability age check from oldest to newest write#1920mcardinali-data wants to merge 1 commit into
mcardinali-data wants to merge 1 commit into
Conversation
…danielmiessler#1915) CortexHealth warned when the OLDEST observability log exceeded 30 days, but nothing in the tree prunes or rotates these logs, so the warning is permanently on for any install older than a month and cannot be cleared. It also measures the wrong property: an old, legitimately stable file is not a health defect. Flip the check's polarity to measure staleness of the NEWEST write: if nothing has written to the observability surface within the threshold, the telemetry pipeline is likely dead - which is actionable and self-clearing on healthy installs. Unbounded growth remains covered by the existing 256 MiB bytes check. The CORTEX_OBSERVABILITY_MAX_AGE_MS env knob keeps its name and shape. Finding id observability-age-exceeded becomes observability-stale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1915.
Problem
CortexHealth.ts(new in v7.40.4) warns when the oldest file underMEMORY/OBSERVABILITY/exceeds 30 days. Nothing in the tree prunes or rotates these logs, so on any install older than a month the warning is permanently on and cannot be cleared — a health signal with no remedy, which trains operators to stop reading the health line (same class as #1910).It also measures the wrong property. Observability holds append-only telemetry plus files that are legitimately stable; an old file is not a defect. The two real failure modes are:
Fix
Flip the polarity: track the newest mtime instead of the oldest, and warn as
observability-stalewhen nothing has written within the threshold. That warning is actionable (check hook wiring) and self-clearing on every healthy install.observabilityEvidence():oldestMs/Math.min→newestMs/Math.maxobservability-age-exceeded→observability-stale, message names the likely causeCORTEX_OBSERVABILITY_MAX_AGE_MSenv knob keeps its name and shapeThis is remedy option 3 from the issue (scope the measurement to what active logging proves), implemented as the minimal 6-line diff rather than shipping a rotation mechanism and a retention policy every install would have to adopt.
Verification
Synthetic fixtures against
assessCortexEvidence:observability-staleobservability-bytes-exceededstill firesLive install (logs written minutes ago, some files months old): overall
ok, zero observability findings.