Add observability

[sjmiller609]

Tracing, metrics, configurable logging, local observability stack

[mesa-dot-dev]

Mesa Description

TL;DR

Integrated OpenTelemetry for comprehensive tracing and metrics across the application, and implemented a new structured, configurable logging system.

Why we made these changes

To significantly enhance the observability of the Hypeman system, enabling better monitoring, debugging, and performance analysis through detailed traces, metrics, and flexible logging.

What changed?

OpenTelemetry Integration

• Core API (cmd/api):
  • Added OpenTelemetry configuration fields to Config struct.
  • Initialized OTel providers for traces and metrics in main.go.
  • Integrated otelchi.Middleware for HTTP request tracing and a custom HTTP metrics middleware.
• Exec Operations (lib/exec):
  • Enhanced ExecIntoInstance to record session duration, exit code, and byte transfer metrics.
  • Introduced lib/exec/metrics.go for exec specific OTel metrics (sessions, bytes, duration).
• Image Management (lib/images):
  • Integrated OTel metrics into manager.go to track build durations, pull counts, build queue length, and cached image status.
  • Added QueueLength() to BuildQueue for metric reporting.
• Instance Management (lib/instances):
  • Integrated OTel tracing and metrics for instance creation, restoration, and standby operations.
  • Introduced a Metrics struct in manager.go to hold OTel instruments for instance-related metrics (durations, state transitions, total instances by state).
• Network Management (lib/network):
  • Integrated OTel metrics to track TAP device lifecycle operations and total active network allocations.
• VMM Operations (lib/vmm):
  • Introduced lib/vmm/metrics.go for OTel metrics to monitor Cloud Hypervisor API call durations and errors.
• Volume Management (lib/volumes):
  • Integrated OTel metrics into manager.go to track volume creation duration, total volume count, allocated, and used bytes.
• Shared OTel Utilities (lib/otel):
  • Added lib/otel/otel.go for comprehensive OTel initialization, configuration, and runtime/application metrics registration.
  • Updated lib/otel/metrics.go for OTel metrics definitions.
  • Added lib/otel/README.md documenting OTel integration and configuration.
• Dependencies:
  • Updated go.mod and go.sum to include new OpenTelemetry related packages.

Configurable Structured Logging

• Logger Package (lib/logger):
  • Refactored lib/logger/logger.go to use slog for structured logging with subsystem-specific log levels.
  • Implemented traceContextHandler to automatically inject OpenTelemetry trace and span IDs into log records.
  • Added lib/logger/README.md documenting the new logging system.
• Configuration (cmd/api/config):
  • Extended Config struct with a general LogLevel field.
• Provider Refactoring (lib/providers):
  • Updated lib/providers/providers.go to utilize the new structured logger package.

Test Adaptations

• Updated various test files (cmd/api/api/api_test.go, lib/images/manager_test.go, lib/instances/exec_test.go, lib/instances/manager_test.go, lib/instances/resource_limits_test.go, lib/instances/volumes_test.go, lib/volumes/manager_test.go) to pass nil for new optional context or dependency parameters in manager constructors, reflecting updated function signatures.

Validation

{List of tests or validations performed}

^{Description generated by Mesa. Update settings}

[mesa-dot-dev]

Performed full review of 2e42f22...8658eca

Analysis

1. Global State Anti-Pattern: The exec package uses a global ExecMetrics variable with a SetMetrics() function, creating race conditions, testing difficulties, and inconsistency with the dependency injection pattern used throughout the rest of the codebase.

2. Dead Metrics Implementation: Despite adding metrics support to all managers, the Wire dependency injection system passes nil to all manager constructors except exec, rendering most of the added metrics infrastructure non-functional.

3. Silent Error Suppression: Multiple observable gauge callbacks silently return nil when encountering errors, masking underlying issues with metric collection and making debugging difficult.

4. OTel Shutdown Timing Issues: The OpenTelemetry shutdown uses a 5-second timeout that isn't coordinated with server shutdown, risking metric loss if the server takes longer to shut down gracefully.

5. Incomplete Context Propagation: While metric recording uses request contexts for trace propagation, there's no verification that contexts are properly propagated through all asynchronous operations.

Tip

Help

Slash Commands:

• /review - Request a full code review
• /review latest - Review only changes since the last review
• /describe - Generate PR description. This will update the PR body or issue comment depending on your configuration
• /help - Get help with Mesa commands and configuration options

^{30 files reviewed | 0 comments | Edit Agent Settings • Read Docs}

[sjmiller609]

/review

[mesa-dot-dev]

Performed full review of 2e42f22...a029711

Analysis

1. Dependency Management Issues: Global metric singletons in exec/metrics.go and vmm/metrics.go break dependency injection patterns, creating hidden dependencies that complicate testing. The providers.go file directly uses global OTel state instead of the passed provider instance.

2. Error Handling Gaps: Metric collection errors are silently ignored in several components, and there's a lack of dedicated error counters for critical failure paths like instance creation or volume allocation failures.

3. Performance and Scaling Concerns: Several metrics use high-cardinality labels that could cause memory issues in the metrics backend. Volume metrics perform syscalls for every volume on each collection cycle, potentially adding significant overhead on systems with many volumes.

4. Security Configuration: OTEL_INSECURE defaults to true, disabling TLS which creates a security vulnerability in production environments.

5. Architectural Inconsistencies: The HTTP metrics middleware uses complex inline functions for endpoint exclusion that won't scale well, and tracing is improperly coupled to metrics initialization rather than being independently configurable.

Tip

Help

Slash Commands:

• /review - Request a full code review
• /review latest - Review only changes since the last review
• /describe - Generate PR description. This will update the PR body or issue comment depending on your configuration
• /help - Get help with Mesa commands and configuration options

^{39 files reviewed | 0 comments | Edit Agent Settings • Read Docs}

[rgarcia]

noice