Skip to content

Fix HighMemoryUsage alert ratio validation#23

Open
shihuajin111 wants to merge 1 commit into
weilixiong:mainfrom
shihuajin111:fix/high-memory-alert-ratio
Open

Fix HighMemoryUsage alert ratio validation#23
shihuajin111 wants to merge 1 commit into
weilixiong:mainfrom
shihuajin111:fix/high-memory-alert-ratio

Conversation

@shihuajin111

Copy link
Copy Markdown

Summary

Fixes the recommended HighMemoryUsage Prometheus alert so it no longer divides process_resident_memory_bytes by itself. The alert now compares resident process memory to machine_memory_bytes, matching the existing memory usage recording rule in this monitoring setup.

Fixes #1.

Changes

  • Replaced the self-dividing HighMemoryUsage expression with process_resident_memory_bytes / machine_memory_bytes > 0.9.
  • Added alert rule validation that catches missing expressions and exact self-dividing PromQL series expressions.
  • Runs validation before alert rule upload, including dry-run mode.
  • Wires validation failures through CLI exit codes by returning upload failures from --alerts / --init and calling sys.exit(main()).

Testing

  • python3 -m py_compile tools/monitoring_setup.py - passed.
  • python3 tools/monitoring_setup.py --alerts --dry-run - passed; dry-run output now includes expr: process_resident_memory_bytes / machine_memory_bytes > 0.9.
  • File-level validation check confirmed validate_alert_rules(RECOMMENDED_ALERT_RULES) returns [].
  • File-level validation check confirmed validate_alert_rules([{"name": "BadMemory", "expr": "process_resident_memory_bytes / process_resident_memory_bytes > 0.9"}]) reports the self-dividing expression.
  • python3 tools/monitoring_setup.py --validate --prometheus-url http://127.0.0.1:1 --alertmanager-url http://127.0.0.1:1 prints recommended alert rules: OK and exits 1 only because no local Prometheus/Alertmanager is listening on that test port.
  • python3 build.py was attempted on Windows. It did not reach the diagnostic finalization step because build.py launches frontend npm install as subprocess.run(["npm", ...]); on this Windows environment Node provides npm.cmd, which shutil.which("npm") finds but subprocess.run(["npm", ...]) cannot execute directly.
  • python3 build.py -m backend was run to generate diagnostic artifacts. It exits 1 because Rust/Cargo is not installed in this environment, and it generated:
    • diagnostic/build-2b54872c.logd
    • diagnostic/build-2b54872c.json

Checklist

  • Relevant modules affected by these changes build locally
  • Tests pass locally
  • Diagnostic build log is committed in this PR
  • Documentation has been updated, if applicable
  • Configuration or schema changes are documented, if applicable
  • No generated build artifacts are committed, except the required diagnostic build log
  • Changes are scoped to the PR purpose and avoid unrelated cleanup
  • Security, privacy, and error-handling implications have been considered

  • I would like to request that my diagnostic build log is removed before merging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[$30 BOUNTY] [Python] Fix HighMemoryUsage alert ratio expression

1 participant