Phase 3: Health Probe Enhancement - Complete

What Was Done

Phase 3 enhanced the health probe logic for the diagnostic-mcp HTTP server, making it production-ready with Kubernetes-style startup, liveness, and readiness probes.

Key Deliverables

1. Enhanced HealthMonitor Class ✅

File: http_server.py (lines 78-317)

Features:

Startup probe (DOWN during initialization, UP after startup duration)
Liveness probe (DOWN after 10 consecutive failures)
Readiness probe (DOWN if too many rejections or still starting)
Degraded state detection (error rate threshold)
Configurable thresholds (rejections, sampling interval, recovery time)
Comprehensive metadata (uptime, timestamps, metrics)

Bug Fix:

Fixed overall status priority (now: critical > starting > unready > degraded > healthy)

2. MCP Tools Integration ✅

File: server.py (lines 1000-1028, 1770-1975)

Tools:

check_readiness_probe - Query readiness status
check_liveness_probe - Query liveness status
get_probe_status - Get comprehensive status (all probes)

Implementation:

HTTP-based querying (localhost:5555)
Structured responses
Error handling for unavailable server

3. CLI Integration ✅

File: cli.py (lines 90-100, 416-483)

Commands:

python cli.py --check readiness      # Readiness probe only
python cli.py --check liveness       # Liveness probe only
python cli.py --check probes         # All probes
python cli.py --check probes --format json    # JSON output
python cli.py --check probes --format summary # Summary only

Features:

Text, JSON, and summary output formats
Detailed probe status display
Error rate and metrics visualization

4. HTTP Endpoints ✅

File: http_server.py (lines 359-412)

Endpoints:

GET /health - Basic health (always 200)
GET /health?live - Liveness probe (200/503)
GET /health?ready - Readiness probe (200/503)
GET /health?startup - Startup probe (200/503)
GET /health?status - Comprehensive status (200/503)
GET /health/startup - Direct startup route
GET /health/status - Direct status route

Status Codes:

200 OK - Probe is UP
503 Service Unavailable - Probe is DOWN

5. Configuration Support ✅

File: http_server.py (lines 674-729)

Options:

python http_server.py \
  --port 5555 \
  --host 0.0.0.0 \
  --startup-duration 30 \
  --allowed-rejections 100 \
  --sampling-interval 10 \
  --recovery-interval 20 \
  --degraded-threshold 0.25

Environment Variables:

MCP_HTTP_PORT - Server port (default: 5555)
MCP_HTTP_HOST - Server host (default: 0.0.0.0)

6. Comprehensive Testing ✅

File: tests/test_health_monitor.py

Coverage:

16 tests, all passing
Unit tests for HealthMonitor
Integration tests for probe transitions
Edge case testing (recovery, degradation)
Metadata validation

Test Results:

16 passed in 6.76s

7. Documentation ✅

Files Created:

HEALTH_PROBES.md (8.5 KB)
- Complete probe documentation
- Configuration guide
- Integration examples (Kubernetes, NGINX, HAProxy, Prometheus)
- Troubleshooting guide
INSTALL_HTTP_SERVER.md (6.2 KB)
- Installation instructions
- Configuration examples
- Monitoring setup
- Security considerations
PHASE3_COMPLETION_SUMMARY.md (11.3 KB)
- Detailed implementation summary
- Test verification
- Next steps
scripts/test_probes.sh (executable)
- Automated probe testing script
- Colorized output
- Comprehensive status checks
diagnostic-mcp-http.service
- Systemd service file
- Production-ready configuration

8. Systemd Service ✅

File: diagnostic-mcp-http.service

Installation:

sudo cp diagnostic-mcp-http.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable diagnostic-mcp-http
sudo systemctl start diagnostic-mcp-http

Quick Start

1. Run Tests

cd /srv/latvian_mcp/servers/diagnostic-mcp
python -m pytest tests/test_health_monitor.py -v

2. Start HTTP Server

python http_server.py --port 5555

3. Test Probes

# Automated test script
./scripts/test_probes.sh

# Or manual tests
curl http://localhost:5555/health?live
curl http://localhost:5555/health?ready
curl http://localhost:5555/health?startup
curl http://localhost:5555/health?status | jq

4. Use CLI

python cli.py --check probes
python cli.py --check readiness --format json
python cli.py --check liveness --format summary

5. Install Systemd Service

sudo cp diagnostic-mcp-http.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable diagnostic-mcp-http
sudo systemctl start diagnostic-mcp-http
sudo systemctl status diagnostic-mcp-http

File Summary

File	Purpose	Lines	Status
`http_server.py`	HealthMonitor class + HTTP server	774	✅ Enhanced
`server.py`	MCP tools integration	1991	✅ Complete
`cli.py`	CLI probe checks	710	✅ Complete
`tests/test_health_monitor.py`	Comprehensive tests	335	✅ New
`HEALTH_PROBES.md`	Probe documentation	8.5 KB	✅ New
`INSTALL_HTTP_SERVER.md`	Installation guide	6.2 KB	✅ New
`PHASE3_COMPLETION_SUMMARY.md`	Implementation summary	11.3 KB	✅ New
`scripts/test_probes.sh`	Automated test script	267 lines	✅ New
`diagnostic-mcp-http.service`	Systemd service	41 lines	✅ New

Probe Logic Summary

Startup Probe

Initial: DOWN (during startup duration)
After startup: UP
Use: Delay traffic until initialized
Config: --startup-duration (default: 30s)

Liveness Probe

Default: UP (alive)
Trigger: 10 consecutive failures
Result: DOWN (critical - needs restart)
Use: Detect deadlock/critical issues

Readiness Probe

Initial: DOWN (during startup)
Ready: UP (after startup, low error rate)
Degraded: UP but degraded: true (high error rate)
Unready: DOWN (too many rejections)
Recovery: UP after recovery interval
Use: Load balancer health checks

Overall Status

Priority: critical > starting > unready > degraded > healthy
States:
- healthy - All probes UP, no issues
- starting - Startup probe DOWN
- unready - Readiness probe DOWN
- degraded - Ready but high error rate
- critical - Liveness probe DOWN

Configuration Examples

Development (Permissive)

python http_server.py \
  --startup-duration 10 \
  --allowed-rejections 200 \
  --sampling-interval 30 \
  --degraded-threshold 0.50

Production (Strict)

python http_server.py \
  --startup-duration 60 \
  --allowed-rejections 50 \
  --sampling-interval 5 \
  --degraded-threshold 0.10

High-Traffic (Tolerant)

python http_server.py \
  --startup-duration 30 \
  --allowed-rejections 500 \
  --sampling-interval 60 \
  --degraded-threshold 0.30

Integration Examples

Kubernetes

startupProbe:
  httpGet:
    path: /health/startup
    port: 5555
  failureThreshold: 30
  periodSeconds: 1

livenessProbe:
  httpGet:
    path: /health?live
    port: 5555
  failureThreshold: 3
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health?ready
    port: 5555
  failureThreshold: 3
  periodSeconds: 5

NGINX

upstream diagnostic_mcp {
    server localhost:5555 max_fails=3 fail_timeout=30s;
    check interval=5000 rise=2 fall=3 timeout=1000 type=http;
    check_http_send "GET /health?ready HTTP/1.0\r\n\r\n";
    check_http_expect_alive http_2xx;
}

HAProxy

backend diagnostic_mcp
    option httpchk GET /health?ready
    http-check expect status 200
    server server1 localhost:5555 check inter 5s fall 3 rise 2

Testing Verification

Unit Tests

cd /srv/latvian_mcp/servers/diagnostic-mcp
python -m pytest tests/test_health_monitor.py -v

# Result: 16 passed in 6.76s

Integration Tests

# Start server
python http_server.py &

# Run probe tests
./scripts/test_probes.sh

# Check status
python cli.py --check probes

MCP Tool Tests

# Via MCP protocol
await session.call_tool("check_readiness_probe", {})
await session.call_tool("check_liveness_probe", {})
await session.call_tool("get_probe_status", {})

Troubleshooting

Startup Probe Stuck on DOWN

Check uptime: curl http://localhost:5555/health?startup | jq '.uptime_seconds'
Wait for startup duration to elapse
Or restart with shorter duration: --startup-duration 10

Readiness Probe Flapping

Check metrics: curl http://localhost:5555/health?ready | jq '.metrics'
Increase allowed rejections: --allowed-rejections 200
Increase sampling interval: --sampling-interval 30

Degraded State Persistent

Check error rate: curl http://localhost:5555/health?ready | jq '.metrics.error_rate'
Investigate why requests are failing
Adjust threshold if acceptable: --degraded-threshold 0.40

Liveness Probe DOWN

Critical issue - 10+ consecutive failures
Check logs: sudo journalctl -u diagnostic-mcp-http -n 100
Restart server: sudo systemctl restart diagnostic-mcp-http
Investigate root cause

Next Steps (Future)

Potential enhancements for future phases:

Metrics Export
- Prometheus exporter
- Grafana dashboards
- Time-series tracking
Advanced Health Checks
- Dependency health (database, MCP servers)
- Custom health check plugins
- Circuit breaker integration
Auto-Recovery
- Automatic service restart
- Self-healing mechanisms
- Alert notifications
Historical Tracking
- Probe status history in Supabase
- Trend analysis
- SLA compliance reporting

References

Full Documentation: HEALTH_PROBES.md
Installation Guide: INSTALL_HTTP_SERVER.md
Implementation Summary: PHASE3_COMPLETION_SUMMARY.md
Test Suite: tests/test_health_monitor.py
Test Script: scripts/test_probes.sh
Main README: README.md

Conclusion

Phase 3 is 100% complete. The diagnostic-mcp HTTP server now has production-ready health probes with:

✅ 3 probe types (startup, liveness, readiness) ✅ 7 HTTP endpoints with proper status codes ✅ 3 MCP tools for programmatic access ✅ CLI integration with multiple output formats ✅ Configurable thresholds for all probe types ✅ 16 comprehensive tests (all passing) ✅ 4 documentation files (26 KB total) ✅ Systemd service for production deployment ✅ Automated test script for quick verification

The implementation is robust, well-tested, and ready for integration with Kubernetes, load balancers, and monitoring systems.

Version: 2.1.0 Date: 2025-12-21 Phase: 3 of 3 (Complete) Test Coverage: 16/16 tests passing Documentation: Complete Production Ready: ✅ Yes

FilesExpand file tree

PHASE3_README.md

Latest commit

History