Confidence indicators

docs/usage.md

@@ -285,6 +285,147 @@ Today is 27 Aug:
 
 If a changepoint is detected in the first 5 data points, Orion expands the lookback window, re-runs the analysis, and reports based on that expanded result.
 
+## Confidence Indicators
+
+Every detected changepoint is automatically annotated with a statistical confidence indicator that combines two measures:
+
+- **Cohen's d effect size** (primary) — magnitude of the difference relative to data variability. This drives the label classification.
+- **Welch's t-test p-value** (descriptive) — probability of observing a difference this extreme if there were no real before/after change. Reported alongside the effect size but does not gate the label.
+
+Cohen's d is used as the primary indicator because the changepoint was already detected by the analysis algorithm — re-testing the same data with a t-test would inflate significance (post-selection bias). The effect size describes *how large* the shift is, independent of the detection method.
+
+These produce a human-readable label shown in the "Affected Metrics" summary table and embedded in JSON output.
+
+### Label Format
+
+Labels are driven by Cohen's d effect size thresholds (Cohen 1988: 0.2 small, 0.5 medium, 0.8 large). The p-value is included as descriptive context only — it is **not** calibrated for post-selection inference (the changepoint was already selected from this same data):
+
+| Label | Meaning |
+|-------|---------|
+| `Large shift (d=1.20, p=0.001)` | d >= 0.8 — strong evidence of a meaningful metric shift |
+| `Moderate shift (d=0.60, p=0.03)` | 0.5 <= d < 0.8 — moderate-magnitude shift |
+| `Small shift (d=0.30, p=0.02)` | 0.2 <= d < 0.5 — small but detectable effect |
+| `Negligible shift (d=0.10, p=0.01)` | d < 0.2 — negligible practical impact |
+| `Degenerate variance — shift detected but effect size undefined` | Both segments have zero variance with different means; Cohen's d is undefined |
+| `Insufficient data` | Fewer than 2 data points on either side of the changepoint |
+| `Anomaly detection — shift confidence not applicable` | IsolationForest detects outliers, not sustained shifts |
+
+The p-value is formatted with `:.2g` to preserve precision (e.g., `5e-10` instead of `0.00`).
+
+### Effect Size (Cohen's d)
+
+Cohen's d is computed using a pooled standard deviation, which assumes the before and after segments have roughly similar variance. This is appropriate for CI performance data where the variance structure tends to be stable across a shift. An alternative (Glass's delta, using only the before-segment standard deviation) could be considered if shifts are expected to also change variance.
+
+When both segments have zero variance but different means, Cohen's d is mathematically undefined. In this case, `cohens_d` is reported as `null` with a "Degenerate variance" label.
+
+### How Segments Are Split
+
+The before/after data segments depend on the algorithm:
+
+- **Hunter (E-Divisive)**: segments are bounded by neighboring changepoints. For a changepoint at index *i* with previous changepoint at *p* and next changepoint at *n*, the before-segment is `data[p:i]` and the after-segment is `data[i:n]`. If there is no previous changepoint, `p` defaults to 0 (start of data). If there is no next changepoint, `n` defaults to the end of data. This prevents a later recovery or reversal from masking an earlier genuine shift.
+- **CMR**: all previous runs vs. the most recent run (CMR typically produces "Insufficient data" since the after segment has only one point)
+- **Anomaly Detection (IsolationForest)**: confidence is not computed. IsolationForest identifies unusual individual observations, not sustained before/after shifts. A separate "Anomaly detection" label is assigned.
+
+### Text Output
+
+The "Affected Metrics" summary table includes a Confidence column:
+
+```text
+Affected Metrics
++---------+-------+----------+------------------------------------------+--------+
+| Metric  | Value | % Change | Confidence                               | Labels |
++---------+-------+----------+------------------------------------------+--------+
+| ovnCPU  | 2.43  | 64.14%   | Large shift (d=1.20, p=0.001)            | [infra] |
+| etcdCPU | 3.50  | 1.18%    | Negligible shift (d=0.15, p=0.42)        | [etcd] |
++---------+-------+----------+------------------------------------------+--------+
+```
+
+### JSON Output
+
+Each changepoint entry in JSON output includes a `confidence` object (illustrative values):
+
+```json
+{
+  "is_changepoint": true,
+  "metrics": {
+    "ovnCPU_avg": {
+      "value": 2.43,
+      "percentage_change": 64.14,
+      "confidence": {
+        "p_value": 0.001,
+        "cohens_d": 1.2,
+        "label": "Large shift (d=1.20, p=0.001)",
+        "sufficient_data": true,
+        "sample_size_before": 15,
+        "sample_size_after": 5,
+        "mean_before": 1.48,
+        "mean_after": 2.43,
+        "std_before": 0.12,
+        "std_after": 0.18,
+        "ci_95": [0.82, 1.08]
+      }
+    }
+  }
+}
+```
+
+The `ci_95` field is always present. When confidence cannot be computed, it is `null`:
+
+```json
+{
+  "confidence": {
+    "p_value": null,
+    "cohens_d": null,
+    "label": "Insufficient data",
+    "sufficient_data": false,
+    "sample_size_before": 3,
+    "sample_size_after": 1,
+    "mean_before": 10.5,
+    "mean_after": 20.0,
+    "std_before": 0.5,
+    "std_after": null,
+    "ci_95": null
+  }
+}
+```
+
+### Confidence Fields Reference
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `p_value` | float\|null | Welch's t-test p-value (descriptive only — not calibrated for post-selection inference). Null when insufficient data or degenerate variance. |
+| `cohens_d` | float\|null | Cohen's d effect size (pooled std). Measures the magnitude of the shift relative to data variability. Null when insufficient data, degenerate variance, or IsolationForest. |
+| `label` | string | Human-readable confidence label (see Label Format above). |
+| `sufficient_data` | bool | Whether both segments had at least 2 data points for statistical computation. |
+| `sample_size_before` | int | Number of data points before the changepoint (after NaN removal). |
+| `sample_size_after` | int | Number of data points from the changepoint onward (after NaN removal). |
+| `mean_before` | float\|null | Mean of the before-segment values. |
+| `mean_after` | float\|null | Mean of the after-segment values. |
+| `std_before` | float\|null | Sample standard deviation of the before-segment (ddof=1). Null if fewer than 2 points. |
+| `std_after` | float\|null | Sample standard deviation of the after-segment (ddof=1). Null if fewer than 2 points. |
+| `ci_95` | [float, float]\|null | 95% confidence interval for the mean difference (mean_after − mean_before), computed using the Welch-Satterthwaite degrees of freedom. Descriptive only — not calibrated for post-selection inference. Null when insufficient data or when standard error is zero. Always present in the output (never omitted). |
+
+The `mean_before`, `mean_after`, `std_before`, and `std_after` fields are the exact values used to compute both `p_value` and `cohens_d`. Combined with the sample sizes, any consumer can independently reproduce the pooled standard deviation, t-statistic, and confidence interval.
+
+The `ci_95` and `p_value` fields are provided as descriptive context. Because the changepoint was selected from the same data, these values are subject to post-selection bias and should not be interpreted as calibrated hypothesis tests. Use Cohen's d (effect size) as the primary indicator of shift magnitude.
+
+### Standalone Reports
+
+When generating reports from JSON files with `--report`, confidence data is propagated from the JSON into the summary tables automatically — no additional flags needed.
+
+### Interpreting Results
+
+Use confidence indicators to triage changepoints for evidence of meaningful metric shifts:
+
+1. **Large shift** — investigate immediately; strong evidence of a meaningful metric shift
+2. **Moderate shift** — investigate; meaningful shift that warrants attention
+3. **Small shift** — detectable but small; may be acceptable depending on the metric's sensitivity
+4. **Negligible shift** — the detected change is too small to matter in practice
+5. **Insufficient data** — not enough data points to compute statistics (common with CMR or very recent runs)
+6. **Anomaly detection** — IsolationForest results; the algorithm detects outliers, not sustained shifts
+
+A confidence label indicates evidence of a metric shift, not necessarily that product code caused a regression. Environment changes, workload variations, or measurement differences could also explain the shift — always investigate the cause.
+
 ## Node Count Filtering
 
 ### Relaxed Matching

orion/algorithms/algorithm.py

@@ -22,6 +22,15 @@ def __init__(
         self.regression_flag = False
         self._acked_logged = False
         self._cached_analysis = None
+        self.raw_change_points_by_metric = None
+        self._original_dataframe = None
+
+    @property
+    def original_dataframe(self):
+        """Pre-analysis dataframe. CMR sets _original_dataframe before collapsing."""
+        if self._original_dataframe is not None:
+            return self._original_dataframe
+        return self.dataframe
 
     def get_analysis_results(self):
         """Return (series, change_points_by_metric) from _analyze(),

orion/algorithms/cmr/cmr.py

@@ -2,7 +2,6 @@
 
 # pylint: disable = line-too-long
 import pandas as pd
-import numpy
 
 from otava.analysis import TTestStats
 from otava.series import  ChangePoint
@@ -37,6 +36,7 @@ def _analyze(self):
             series.data = self.dataframe
             return series, {}
         # if larger than 2 rows, need to get the mean of 0 through -2
+        self._original_dataframe = self.dataframe.copy()
         self.dataframe = self.combine_and_average_runs(self.dataframe)
 
         series= self.setup_series()
@@ -76,14 +76,19 @@ def run_cmr(self, dataframe_list: pd.DataFrame):
         change_points_by_metric={ k:[] for k in metric_columns }
 
         for column in metric_columns:
+            try:
+                m1 = float(dataframe_list[column][0])
+                m2 = float(dataframe_list[column][1])
+            except (ValueError, TypeError):
+                continue
 
             change_point = ChangePoint(metric=column,
                                 index=1,
                                 qhat=0.0,
                                 time=0,
                                 stats=TTestStats(
-                                        mean_1=dataframe_list[column][0],
-                                        mean_2=dataframe_list[column][1],
+                                        mean_1=m1,
+                                        mean_2=m2,
                                         std_1=0.0,
                                         std_2=0.0,
                                         pvalue=1.0
@@ -112,15 +117,17 @@ def combine_and_average_runs(self, dataFrame: pd.DataFrame):
 
         metric_columns = list(dataFrame.columns)
         for column in metric_columns:
-
-            if isinstance(dF.loc[0, column], (numpy.float64, numpy.int64)):
-                mean = dF[column].mean()
-                data2[column] = [mean]
-            else:
-                column_list = dF[column].tolist()
-                # Convert each item to string to handle lists, UUIDs, and other non-string types
-                non_numeric_joined_list = ','.join(str(item) for item in column_list)
-                data2[column] = [non_numeric_joined_list]
+            try:
+                numeric_col = pd.to_numeric(dF[column])
+                data2[column] = [numeric_col.mean()]
+            except (ValueError, TypeError):
+                numeric_col = pd.to_numeric(dF[column], errors='coerce')
+                if numeric_col.notna().any():
+                    data2[column] = [numeric_col.mean()]
+                else:
+                    column_list = dF[column].tolist()
+                    non_numeric_joined_list = ','.join(str(item) for item in column_list)
+                    data2[column] = [non_numeric_joined_list]
             i += 1
         df2 = pd.DataFrame(data2)
 

orion/algorithms/edivisive/edivisive.py

@@ -38,6 +38,10 @@ def _analyze(self):
             logger.info("ACKed UUIDs: %s", ", ".join(acked_uuids))
             self._acked_logged = True
 
+        self.raw_change_points_by_metric = {
+            metric: list(cps) for metric, cps in change_points_by_metric.items()
+        }
+
         # filter by direction and ack'ed issues
         for metric, changepoint_list in change_points_by_metric.items():
             for i in range(len(changepoint_list)-1, -1, -1):