Fix: Refactor Metric Collection and Reporting

[laplaque]

Description

This pull request resolves several issues related to metric collection, reporting, and formatting. It also introduces new, more detailed process metrics for enhanced monitoring. Additionally, this PR includes significant improvements to the database schema creation and migration logic.

Metric Collection and Reporting Fixes

• Cleaner Service Names: Service names in Instana no longer have repetitive prefixes (e.g., m8mulprc/m8mulprc is now just m8mulprc)
• Accurate CPU Usage: The cpu_usage metric now correctly shows the average CPU usage across all monitored processes, not the sum
• Better Metric Names: Metric names with parameters are displayed correctly
• Correct Metric Types: Metrics appear with their proper types (e.g., Gauge, Counter) as defined in configuration
• Proper Percentage Formatting: Percentage-based metrics are sent correctly for proper Instana display

New Features

• Enhanced Process Metrics: Added new metrics for in-depth analysis:
  • Total CPU user time
  • Total CPU system time
  • Total RSS memory
  • Total virtual memory

Database Schema Management Improvements

• Simplified Schema Creation Logic: Replaced complex migration paths with a straightforward 5-step process that handles all database states consistently
• Better Code Organization: Functions clustered into logical sections (Database Initialization, Schema Version Management, Database Connection Management, Core CRUD Operations)
• Enhanced Reliability: Clear handling of edge cases including legacy databases without version tables and new database creation
• Improved Performance: New databases created directly with latest schema version (2.0) without unnecessary intermediate steps
• Reduced Complexity: Eliminated redundant code paths and duplicate method definitions

Technical Changes

• Process Monitor Refactoring: Reorganized code for better modularity, readability, and maintainability
• Database Schema Refactoring: Implemented simplified 5-step initialization logic with automatic legacy database handling
• Migration Strategy: Single migration path (1.0 → 2.0) with automatic handling of legacy databases
• Code Organization: Functions grouped into logical sections with clear separation of concerns

Documentation

• ADR-002: Documents metric collection and reporting refactoring decisions
• ADR-003: Documents simplified database schema creation approach
• Release Notes: Updated with comprehensive details for v0.1.03
• TAG File: Enhanced with both metric fixes and schema improvements

Testing

• All existing tests continue to pass (6/6 for metadata_store, 8/8 for schema_migration)
• Comprehensive test coverage validates new database creation, migration scenarios, and legacy database handling

Checklist before requesting a review

• I have performed a self-review of my code
• If this is a merge to master, I will create a version tag (v*..)
• The version tag will be higher than the previous version

Tag Information

Planned tag: v0.1.03

[Copilot]

Pull Request Overview

This PR refactors the metric collection and reporting functionality while also simplifying database schema management and migration logic. Key changes include:

• Refactored sensor tests to align version retrieval with manifest.toml changes.
• Updated schema migration tests and adjusted expected metric formatting for service names.
• Major code reorganization in common/process_monitor.py and common/metadata_store.py to support enhanced process metrics and a simplified 5-step schema creation strategy.

Reviewed Changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_sensor.py	Updated version retrieval and constant verification to align with manifest.toml
tests/test_schema_migration.py	Updated expected schema version and service formatting changes
tests/test_process_monitor.py	Removed references to obsolete per-core CPU testing and updated process metric tests
tests/test_otel_connector.py	Added tests for percentage value handling and adjusted observable metric registration
tests/test_metadata_store.py	Updated CPU core metric formatting and metric name consistency changes
docs/releases/* and docs/adr/*	Updated release notes and ADRs to document refactor and schema creation improvements
common/process_monitor.py	Reorganized process metric collection and removed outdated per-core CPU usage calculations
common/otel_connector.py	Adjusted metric formatting and error logging for unsupported otel types
common/metadata_store.py	Simplified schema creation logic and improved database migration to version 2.0