[$30 BOUNTY] Fix HighMemoryUsage alert ratio expression + add validation

[lee-muriithi-kingori]

Description

Fixed the HighMemoryUsage Prometheus alert that was using a self-dividing expression (process_resident_memory_bytes / process_resident_memory_bytes), which always evaluates to 1 for any non-zero value.

Changes

1. Fixed the alert expression:

   • Before: process_resident_memory_bytes / process_resident_memory_bytes > 0.9 (always true)
   • After: process_resident_memory_bytes / machine_memory_bytes > 0.9 (meaningful ratio)

2. Added alert expression validation:

   • validate_alert_expression() - catches self-dividing patterns using regex
   • validate_all_alert_rules() - bulk validation of all rules
   • --validate-alerts CLI flag for standalone validation
   • Validation runs before any rule upload, blocking on errors

3. Validation catches:

   • Self-dividing expressions: metric / metric
   • Empty expressions
   • Expressions without valid PromQL identifiers

Validation

$ python3 tools/monitoring_setup.py --validate-alerts
Validating alert rules...
  [OK] HighErrorRate
  [OK] HighLatency
  [OK] ServiceDown
  [OK] HighCPUUsage
  [OK] HighMemoryUsage
  [OK] LowDiskSpace
  [OK] CertificateExpiring
  [OK] HighDBConnections
  [OK] QueueBacklog
  [OK] GoroutineLeak
  [OK] GCPauseTime
  [OK] RateLimitExceeded

Bounty Claim

/claim #1

[SysCallVirtuoso]

"Really great work on implementing the validate_alert_expression() logic here. Catching the self-dividing patterns (metric / metric) via regex is a much more robust approach than simply manual review.

Have you considered also expanding the validation to catch cases where the denominator might be a constant that evaluates to zero? I’ve been playing with similar telemetry mitigation logic in my own projects, and adding a check for const == 0 edge cases could really harden the whole suite against false alerts.
Looking forward to seeing how this scales in the main branch!"