Problem Statement
As TrustLens scales in enterprise and production ML environments, teams are running model evaluations in automated CI/CD pipelines and tracking experiments using MLOps platforms like Weights & Biases (W&B) and MLflow. Currently, the TrustReport object generates rich visual and tabular data, but users have to write cumbersome custom parsers to log these metrics, calibration charts, and trust scores into their tracking platforms. This creates unnecessary friction and prevents TrustLens from seamlessly dropping into modern ML workflows.
Proposed Solution
We should introduce native, plug-and-play integrations for W&B and MLflow directly into the TrustReport API.
The proposed solution would add an export_to_wandb() and export_to_mlflow() method (or a unified log_to(backend="wandb") interface) that automatically:
- Logs the aggregate Trust Score and sub-component metrics (calibration, fairness, robustness).
- Uploads the generated HTML/JSON artifact directly to the experiment run.
- Automatically pushes our native Plotly visualisations (like the reliability diagram and class distribution) as interactive charts to the W&B/MLflow UI.
Alternatives Considered
- Keeping it manual: Users continue using
report.to_dict() and looping through the dictionary to log metrics. This is highly repetitive and leads to poor UX.
- Third-party community plugins: Relying on separate repos (e.g.,
trustlens-wandb) instead of first-party support. Given the ubiquity of these tools, native support makes TrustLens far more attractive to data science teams.
Additional Context
This feature would likely require abstracting a BaseLogger in trustlens/backends/ to make it extensible for future integrations (like Neptune or CometML). We can mark wandb and mlflow as optional dependencies in pyproject.toml (e.g., pip install trustlens[mlops]) to avoid bloating the core package. We'd love a contributor to spearhead this architecture!
Problem Statement
As TrustLens scales in enterprise and production ML environments, teams are running model evaluations in automated CI/CD pipelines and tracking experiments using MLOps platforms like Weights & Biases (W&B) and MLflow. Currently, the
TrustReportobject generates rich visual and tabular data, but users have to write cumbersome custom parsers to log these metrics, calibration charts, and trust scores into their tracking platforms. This creates unnecessary friction and prevents TrustLens from seamlessly dropping into modern ML workflows.Proposed Solution
We should introduce native, plug-and-play integrations for W&B and MLflow directly into the
TrustReportAPI.The proposed solution would add an
export_to_wandb()andexport_to_mlflow()method (or a unifiedlog_to(backend="wandb")interface) that automatically:Alternatives Considered
report.to_dict()and looping through the dictionary to log metrics. This is highly repetitive and leads to poor UX.trustlens-wandb) instead of first-party support. Given the ubiquity of these tools, native support makes TrustLens far more attractive to data science teams.Additional Context
This feature would likely require abstracting a
BaseLoggerintrustlens/backends/to make it extensible for future integrations (like Neptune or CometML). We can markwandbandmlflowas optional dependencies inpyproject.toml(e.g.,pip install trustlens[mlops]) to avoid bloating the core package. We'd love a contributor to spearhead this architecture!