# [FEATURE] Add TrustReport comparison for model drift & reliability deltas

[Khanz9664]

Overview

TrustLens answers one question well:

"Can I trust this model?"

But in real-world ML, that question isn't asked once — it's asked every time a model changes.

Teams retrain on new data, ship updated checkpoints, run A/B experiments, and gate releases through CI/CD. A model's accuracy may stay flat while its calibration silently degrades, its fairness worsens, or its failure rate quietly climbs.

Today, TrustLens has no first-class way to surface that. Users must manually diff JSON outputs across runs and draw their own conclusions.

This issue proposes a native report comparison API — so TrustLens can answer the question that actually matters in production:

"Did this model become more or less trustworthy?"

---

Problem

Given two reports:

baseline  = analyze(model=v1_model, X=X_test, y_true=y_test, y_prob=y_prob_v1)
candidate = analyze(model=v2_model, X=X_test, y_true=y_test, y_prob=y_prob_v2)

Users currently cannot easily answer:

Question	Today
Did calibration improve or regress?	Manual JSON diff
Which diagnostic dimension shifted most?	Manual inspection
Did fairness get worse?	Manual inspection
Did deployment readiness change?	Manual inspection
What specifically drove the Trust Score delta?	Guesswork

This becomes a real pain point in:

• Retraining pipelines — did the new checkpoint actually improve?
• CI/CD model validation — did this PR regress reliability?
• A/B experimentation — which variant is more trustworthy?
• Production monitoring — is the live model drifting?
• Release gating — is the candidate safe to promote?

---

Proposed Solution: TrustComparison

Introduce a lightweight, standalone comparison object that takes two TrustReport instances and computes structured reliability deltas.

Preferred API (Option A — method chaining)

comparison = baseline.compare(candidate)

Alternative API (Option B — standalone function)

from trustlens import compare_reports

comparison = compare_reports(baseline, candidate)

Both should return a TrustComparison object.

---

TrustComparison Object

class TrustComparison:
    def __init__(self, baseline: TrustReport, candidate: TrustReport): ...

    def summary(self) -> str:       # human-readable narrative
    def to_dict(self) -> dict:      # structured serialization
    def save(self, path: str):      # write to JSON
    def plot(self):                 # optional visualization

Keeping comparison logic in its own class avoids polluting TrustReport and makes the design easy to extend.

---

Output Specification

Top-Level Score Delta

{
  "trust_score_delta": +8,
  "grade_change": "C → B",
  "deployment_status_change": "Blocked → Caution"
}

Per-Dimension Breakdown

{
  "calibration": { "before": 71.2, "after": 84.5, "delta": +13.3 },
  "failure":     { "before": 42.1, "after": 39.5, "delta":  -2.6 },
  "fairness":    { "before": 88.0, "after": 82.9, "delta":  -5.1 },
  "separation":  { "before": 65.0, "after": 67.4, "delta":  +2.4 }
}

Human-Readable Summary (comparison.summary())

Model reliability improved overall (+8 Trust Score)  C → B

Biggest improvement:
  ✓ Calibration    +13.3  (71.2 → 84.5)

Biggest regression:
  ✗ Fairness        -5.1  (88.0 → 82.9)

Deployment verdict:
  Blocked → Caution

Note: Fairness regression may require review before promotion.

---

Architecture Notes

• Decouple from TrustReport — TrustComparison is a separate class; TrustReport.compare() is a thin convenience wrapper
• No hardcoded dimension lists — iterate over available modules dynamically to handle missing/optional diagnostics gracefully
• Policy-aware (if Policy Profiles lands first) — comparisons should note if reports were generated under different policies and warn accordingly
• Serialization parity — to_dict() output schema should mirror TrustReport.to_dict() conventions

---

Requirements

• TrustReport.compare(candidate) method
• Standalone compare_reports(a, b) function
• TrustComparison class with .summary(), .to_dict(), .save()
• Trust Score delta + grade change computation
• Per-diagnostic dimension delta breakdown
• Deployment verdict before/after diff
• Graceful handling of missing/mismatched diagnostic modules
• Warning when comparing reports generated under different policies
• Unit tests covering improvement, regression, and mixed-change scenarios
• Documentation with end-to-end example
• 100% backward compatible — no changes to existing analyze() or TrustReport surface area

---

Nice-to-Have (Out of Scope)

Support comparing persisted reports from disk:

comparison = compare_json_reports(
    "reports/v1_report.json",
    "reports/v2_report.json"
)

Optional visualization:

comparison.plot()  # delta bar chart, before/after dashboard

Neither is required for this issue.

---

Non-Goals

This issue is scoped to report-to-report comparison only. It does not introduce:

• Real-time or streaming drift detection
• Feature-level or dataset-level drift analysis
• Statistical process control or alerting
• Online monitoring pipelines

---

Why This Matters

A model can improve in accuracy while becoming measurably less trustworthy.

TrustLens already makes trust visible. This feature makes trust trackable — turning one-shot evaluations into a longitudinal reliability signal that fits naturally into how real ML teams actually ship models.