Skip to content

fix(cortex): flip observability age check from oldest to newest write - #1920

Open
mcardinali-data wants to merge 1 commit into
danielmiessler:mainfrom
mcardinali-data:fix/observability-stale-check
Open

fix(cortex): flip observability age check from oldest to newest write#1920
mcardinali-data wants to merge 1 commit into
danielmiessler:mainfrom
mcardinali-data:fix/observability-stale-check

Conversation

@mcardinali-data

Copy link
Copy Markdown

Fixes #1915.

Problem

CortexHealth.ts (new in v7.40.4) warns when the oldest file under MEMORY/OBSERVABILITY/ exceeds 30 days. Nothing in the tree prunes or rotates these logs, so on any install older than a month the warning is permanently on and cannot be cleared — a health signal with no remedy, which trains operators to stop reading the health line (same class as #1910).

It also measures the wrong property. Observability holds append-only telemetry plus files that are legitimately stable; an old file is not a defect. The two real failure modes are:

  • unbounded growth — already covered directly by the existing 256 MiB bytes check, untouched here
  • a dead telemetry pipeline — nothing writes anymore; this is what an age signal can actually detect

Fix

Flip the polarity: track the newest mtime instead of the oldest, and warn as observability-stale when nothing has written within the threshold. That warning is actionable (check hook wiring) and self-clearing on every healthy install.

  • observabilityEvidence(): oldestMs/Math.minnewestMs/Math.max
  • finding id observability-age-exceededobservability-stale, message names the likely cause
  • CORTEX_OBSERVABILITY_MAX_AGE_MS env knob keeps its name and shape
  • bytes check unchanged

This is remedy option 3 from the issue (scope the measurement to what active logging proves), implemented as the minimal 6-line diff rather than shipping a rotation mechanism and a retention policy every install would have to adopt.

Verification

Synthetic fixtures against assessCortexEvidence:

  • dir with a 80-day-old file plus a fresh file → no observability finding (previously: permanent warn)
  • dir where the newest file is >30 days old → observability-stale
  • 300 MiB evidence → observability-bytes-exceeded still fires

Live install (logs written minutes ago, some files months old): overall ok, zero observability findings.

…danielmiessler#1915)

CortexHealth warned when the OLDEST observability log exceeded 30 days,
but nothing in the tree prunes or rotates these logs, so the warning is
permanently on for any install older than a month and cannot be cleared.
It also measures the wrong property: an old, legitimately stable file is
not a health defect.

Flip the check's polarity to measure staleness of the NEWEST write:
if nothing has written to the observability surface within the threshold,
the telemetry pipeline is likely dead - which is actionable and
self-clearing on healthy installs. Unbounded growth remains covered by
the existing 256 MiB bytes check. The CORTEX_OBSERVABILITY_MAX_AGE_MS
env knob keeps its name and shape.

Finding id observability-age-exceeded becomes observability-stale.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant