From 794a07d00c1b6ca0ddfce37abd41e00ce9153051 Mon Sep 17 00:00:00 2001 From: Miro Dudik Date: Fri, 28 Feb 2020 09:47:04 -0500 Subject: [PATCH 1/8] add metrics API proposal Signed-off-by: Miro Dudik --- api/METRICS.md | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 api/METRICS.md diff --git a/api/METRICS.md b/api/METRICS.md new file mode 100644 index 0000000..f5c77b0 --- /dev/null +++ b/api/METRICS.md @@ -0,0 +1,91 @@ +# API proposal for metrics + +## Example + +```python +# For most sklearn metrics, we would have their group version that returns a Bunch with fields +# * overall: overall metric value +# * by_group: a dictionary that maps sensitive feature values to metric values + +summary = accuracy_score_by_group(y_true, y_pred, sensitive_features=sf, **other_kwargs) + +# Exporting into pd.Series or pd.DataFrame in not too complicated + +series = pd.Series({**summary.by_group, 'overall': summary.overall}) +df = pd.DataFrame({"model accuracy": {**summary.by_group, 'overall': summary.overall}}) + +# Several types of scalar metrics for group fairness can be obtained from `summary` via transformation functions + +acc_difference = difference_from_summary(summary) +acc_ratio = ratio_from_summary(summary) +acc_group_min = group_min_from_summary(summary) + +# Most common disparity metrics should be predefined + +demo_parity_difference = demographic_parity_difference(y_true, y_pred, sensitive_features=sf, **other_kwargs) +demo_parity_ratio = demographic_parity_ratio(y_true, y_pred, sensitive_features=sf, **other_kwargs) +eq_odds_difference = equalized_odds_difference(y_true, y_pred, sensitive_features=sf, **other_kwargs) + +# For predefined disparities based on sklearn metrics, we adopt a consistent naming conventions + +acc_difference = accuracy_score_difference(y_true, y_pred, sensitive_features=sf, **other_kwargs) +acc_ratio = accuracy_score_ratio(y_true, y_pred, sensitive_features=sf, **other_kwargs) +acc_group_min = accuracy_score_group_min(y_true, y_pred, sensitive_features=sf, **other_kwargs) +``` + +## Functions + +*Function signatures* + +```python +metric_by_group(metric, y_true, y_pred, *, sensitive_features, **other_kwargs) +# return the summary for the provided metrics + +make_metric_by_group(metric) +# return a callable object _by_group: +# _by_group(...) = metric_by_group(, ...) + +# Transformation functions returning scalars +difference_from_summary(summary) +ratio_from_summary(summary) +group_min_from_summary(summary) +group_max_from_summary(summary) + +# Metric-specific functions returing summary and scalars +_by_group(y_true, y_pred, *, sensitive_features, **other_kwargs) +_difference(y_true, y_pred, *, sensitive_features, **other_kwargs) +_ratio(y_true, y_pred, *, sensitive_features, **other_kwargs) +_group_min(y_true, y_pred, *, sensitive_features, **other_kwargs) +_group_max(y_true, y_pred, *, sensitive_features, **other_kwargs) +``` + +*Summary of transformations* + +|transformation function|output|metric-specific function|code|aif360| +|-----------------------|------|------------------------|----|------| +|`difference_from_summary`|max - min|`_difference`|D|unprivileged - privileged| +|`ratio_from_summary`|min / max|`_ratio`|R| unprivileged / privileged| +|`group_min_from_summary`|min|`_group_min`|Min| N/A | +|`group_max_from_summary`|max|`_group_max`|Max| N/A | + +*Supported metric-specific functions* + +|metric|variants|task|notes|aif360| +|------|--------|-----|----|------| +|`selection_rate`| G,D,R,Min | class | | ✓ | +|`demographic_parity`| D,R | class | `selection_rate_difference`, `selection_rate_ratio` | `statistical_parity_difference`, `disparate_impact`| +|`accuracy_score`| G,D,R,Min | class | sklearn | `accuracy` | +|`balanced_accuracy_score` | G | class | sklearn | - | +|`mean_absolute_error` | G,D,R,Max | class,reg | sklearn | class only: `error_rate` +|`false_positive_rate` | G,D,R | class | | ✓ | +|`false_negative_rate` | G | class | | ✓ | +|`true_positive_rate` | G,D,R | class | | ✓ | +|`true_negative_rate` | G | class | | ✓ | +|`equalized_odds` | D,R | class | max of difference or ratio under `true_positive_rate`, `false_positive_rate` | - | +|`precision_score`| G | class | sklearn | ✓ | +|`recall_score`| G | class | sklearn | ✓ | +|`f1_score`| G | class | sklearn | - | +|`roc_auc_score`| G | prob | sklearn | - | +|`log_loss`| G | prob | sklearn | - | +|`mean_squared_error`| G | prob,reg | sklearn | - | +|`r2_score`| G | reg | sklearn | - | From 3b93629f7088fe0f2f7be254411a908a1c8b72ad Mon Sep 17 00:00:00 2001 From: Miro Dudik Date: Tue, 3 Mar 2020 18:14:58 -0500 Subject: [PATCH 2/8] add clarifications and confusion_matrix Signed-off-by: Miro Dudik --- api/METRICS.md | 35 ++++++++++++++++++++++++++++------- 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/api/METRICS.md b/api/METRICS.md index f5c77b0..391a660 100644 --- a/api/METRICS.md +++ b/api/METRICS.md @@ -33,13 +33,14 @@ acc_ratio = accuracy_score_ratio(y_true, y_pred, sensitive_features=sf, **other_ acc_group_min = accuracy_score_group_min(y_true, y_pred, sensitive_features=sf, **other_kwargs) ``` -## Functions +## Proposal *Function signatures* ```python metric_by_group(metric, y_true, y_pred, *, sensitive_features, **other_kwargs) -# return the summary for the provided metrics +# return the summary for the provided `metric`, where `metric` has the signature +# metric(y_true, y_pred, **other_kwargs) make_metric_by_group(metric) # return a callable object _by_group: @@ -51,7 +52,7 @@ ratio_from_summary(summary) group_min_from_summary(summary) group_max_from_summary(summary) -# Metric-specific functions returing summary and scalars +# Metric-specific functions returning summary and scalars _by_group(y_true, y_pred, *, sensitive_features, **other_kwargs) _difference(y_true, y_pred, *, sensitive_features, **other_kwargs) _ratio(y_true, y_pred, *, sensitive_features, **other_kwargs) @@ -59,7 +60,7 @@ group_max_from_summary(summary) _group_max(y_true, y_pred, *, sensitive_features, **other_kwargs) ``` -*Summary of transformations* +*Summary of transformations and transformation codes* |transformation function|output|metric-specific function|code|aif360| |-----------------------|------|------------------------|----|------| @@ -68,7 +69,18 @@ group_max_from_summary(summary) |`group_min_from_summary`|min|`_group_min`|Min| N/A | |`group_max_from_summary`|max|`_group_max`|Max| N/A | -*Supported metric-specific functions* +*Summary of tasks and task codes* + +|task|definition|code| +|----|----------|----| +|binary classification|labels and predictions are in {0,1}|class| +|probabilistic binary classification|labels are in {0,1}, predictions are in [0,1] and correspond to estimates of P(y\|x)|prob| +|randomized binary classification|labels are in {0,1}, predictions are in [0,1] and represent the probability of drawing y=1 in a randomized decision|class-rand| +|regression|labels and predictions are real-valued|reg| + +*Predefined metric-specific functions* + +* variants: D, R, Min, Max refer to the transformations from the table above; G refers to `_by_group`. |metric|variants|task|notes|aif360| |------|--------|-----|----|------| @@ -76,7 +88,8 @@ group_max_from_summary(summary) |`demographic_parity`| D,R | class | `selection_rate_difference`, `selection_rate_ratio` | `statistical_parity_difference`, `disparate_impact`| |`accuracy_score`| G,D,R,Min | class | sklearn | `accuracy` | |`balanced_accuracy_score` | G | class | sklearn | - | -|`mean_absolute_error` | G,D,R,Max | class,reg | sklearn | class only: `error_rate` +|`mean_absolute_error` | G,D,R,Max | class, reg | sklearn | class only: `error_rate` | +|`confusion_matrix` | G | class | sklearn | `binary_confusion_matrix` | |`false_positive_rate` | G,D,R | class | | ✓ | |`false_negative_rate` | G | class | | ✓ | |`true_positive_rate` | G,D,R | class | | ✓ | @@ -87,5 +100,13 @@ group_max_from_summary(summary) |`f1_score`| G | class | sklearn | - | |`roc_auc_score`| G | prob | sklearn | - | |`log_loss`| G | prob | sklearn | - | -|`mean_squared_error`| G | prob,reg | sklearn | - | +|`mean_squared_error`| G | prob, reg | sklearn | - | |`r2_score`| G | reg | sklearn | - | + +## Dashboard questions + +1. Should we enable regression metrics for probabilistic classification? + * `mean_absolute_error`, `mean_squared_error`, `mean_squared_error(...,squared=False)` +1. Should we introduce balanced error metrics for probabilistic classification? + * `balanced_mean_{squared,absolute}_error`, `balanced_log_loss` +1. Do we keep `mean_prediction` and `mean_{over,under}prediction`? From 9359f135a372c90d093f36bc8a7c76ca144ddfe8 Mon Sep 17 00:00:00 2001 From: Miro Dudik Date: Tue, 3 Mar 2020 18:21:43 -0500 Subject: [PATCH 3/8] fix list markdown Signed-off-by: Miro Dudik --- api/METRICS.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/api/METRICS.md b/api/METRICS.md index 391a660..ec11657 100644 --- a/api/METRICS.md +++ b/api/METRICS.md @@ -106,7 +106,7 @@ group_max_from_summary(summary) ## Dashboard questions 1. Should we enable regression metrics for probabilistic classification? - * `mean_absolute_error`, `mean_squared_error`, `mean_squared_error(...,squared=False)` + * `mean_absolute_error`, `mean_squared_error`, `mean_squared_error(...,squared=False)` 1. Should we introduce balanced error metrics for probabilistic classification? - * `balanced_mean_{squared,absolute}_error`, `balanced_log_loss` + * `balanced_mean_{squared,absolute}_error`, `balanced_log_loss` 1. Do we keep `mean_prediction` and `mean_{over,under}prediction`? From ddde2ff751d2a9aee17ba42d0090a9e4c182283e Mon Sep 17 00:00:00 2001 From: Miro Dudik Date: Thu, 12 Mar 2020 11:48:43 -0400 Subject: [PATCH 4/8] rename _by_group to _group_summary for consistency Signed-off-by: Miro Dudik --- api/METRICS.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/api/METRICS.md b/api/METRICS.md index ec11657..e497e1d 100644 --- a/api/METRICS.md +++ b/api/METRICS.md @@ -3,18 +3,20 @@ ## Example ```python -# For most sklearn metrics, we would have their group version that returns a Bunch with fields +# For most sklearn metrics, we will have their group version that returns +# the summary of its performance across groups as well as the overall +# performance, represented as a Bunch object with fields # * overall: overall metric value # * by_group: a dictionary that maps sensitive feature values to metric values -summary = accuracy_score_by_group(y_true, y_pred, sensitive_features=sf, **other_kwargs) +summary = accuracy_score_group_summary(y_true, y_pred, sensitive_features=sf, **other_kwargs) # Exporting into pd.Series or pd.DataFrame in not too complicated series = pd.Series({**summary.by_group, 'overall': summary.overall}) df = pd.DataFrame({"model accuracy": {**summary.by_group, 'overall': summary.overall}}) -# Several types of scalar metrics for group fairness can be obtained from `summary` via transformation functions +# Several types of scalar metrics for group fairness can be obtained from the group summary via transformation functions acc_difference = difference_from_summary(summary) acc_ratio = ratio_from_summary(summary) @@ -38,13 +40,13 @@ acc_group_min = accuracy_score_group_min(y_true, y_pred, sensitive_features=sf, *Function signatures* ```python -metric_by_group(metric, y_true, y_pred, *, sensitive_features, **other_kwargs) -# return the summary for the provided `metric`, where `metric` has the signature +group_summary(metric, y_true, y_pred, *, sensitive_features, **other_kwargs) +# return the group summary for the provided `metric`, where `metric` has the signature # metric(y_true, y_pred, **other_kwargs) -make_metric_by_group(metric) -# return a callable object _by_group: -# _by_group(...) = metric_by_group(, ...) +make_metric_group_summary(metric) +# return a callable object _group_summary: +# _group_summary(...) = group_summary(, ...) # Transformation functions returning scalars difference_from_summary(summary) @@ -52,15 +54,15 @@ ratio_from_summary(summary) group_min_from_summary(summary) group_max_from_summary(summary) -# Metric-specific functions returning summary and scalars -_by_group(y_true, y_pred, *, sensitive_features, **other_kwargs) +# Metric-specific functions returning group summary and scalars +_group_summary(y_true, y_pred, *, sensitive_features, **other_kwargs) _difference(y_true, y_pred, *, sensitive_features, **other_kwargs) _ratio(y_true, y_pred, *, sensitive_features, **other_kwargs) _group_min(y_true, y_pred, *, sensitive_features, **other_kwargs) _group_max(y_true, y_pred, *, sensitive_features, **other_kwargs) ``` -*Summary of transformations and transformation codes* +*Transformations and transformation codes* |transformation function|output|metric-specific function|code|aif360| |-----------------------|------|------------------------|----|------| @@ -69,7 +71,7 @@ group_max_from_summary(summary) |`group_min_from_summary`|min|`_group_min`|Min| N/A | |`group_max_from_summary`|max|`_group_max`|Max| N/A | -*Summary of tasks and task codes* +*Tasks and task codes* |task|definition|code| |----|----------|----| @@ -80,7 +82,7 @@ group_max_from_summary(summary) *Predefined metric-specific functions* -* variants: D, R, Min, Max refer to the transformations from the table above; G refers to `_by_group`. +* variants: D, R, Min, Max refer to the transformations from the table above; G refers to `_group_summary`. |metric|variants|task|notes|aif360| |------|--------|-----|----|------| From 0b86e6d333bcc76d04c6cdce51c3256684e47944 Mon Sep 17 00:00:00 2001 From: Miro Dudik Date: Mon, 16 Mar 2020 11:36:06 -0400 Subject: [PATCH 5/8] remove dashboard questions Signed-off-by: Miro Dudik --- api/METRICS.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/api/METRICS.md b/api/METRICS.md index e497e1d..98a2be2 100644 --- a/api/METRICS.md +++ b/api/METRICS.md @@ -104,11 +104,3 @@ group_max_from_summary(summary) |`log_loss`| G | prob | sklearn | - | |`mean_squared_error`| G | prob, reg | sklearn | - | |`r2_score`| G | reg | sklearn | - | - -## Dashboard questions - -1. Should we enable regression metrics for probabilistic classification? - * `mean_absolute_error`, `mean_squared_error`, `mean_squared_error(...,squared=False)` -1. Should we introduce balanced error metrics for probabilistic classification? - * `balanced_mean_{squared,absolute}_error`, `balanced_log_loss` -1. Do we keep `mean_prediction` and `mean_{over,under}prediction`? From ace56bafbfde61d82f947cf3ef9b6cd88c3d6a9b Mon Sep 17 00:00:00 2001 From: Miro Dudik Date: Tue, 24 Mar 2020 18:11:51 -0400 Subject: [PATCH 6/8] start proposal for metrics/dashboard api Signed-off-by: Miro Dudik --- api/METRICS-DASHBOARD.md | 73 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 api/METRICS-DASHBOARD.md diff --git a/api/METRICS-DASHBOARD.md b/api/METRICS-DASHBOARD.md new file mode 100644 index 0000000..6f6acbd --- /dev/null +++ b/api/METRICS-DASHBOARD.md @@ -0,0 +1,73 @@ +# Proposal for metrics/dashboard API + +Goals: +* Unified treatment of many different kinds of metrics in dashboard +* A clean separation between dashboard (for visualization) and different kinds of backends (for metric calculation) + +Status quo: +* Dashboard currently keeps track of the metric values in the cache dictionary `fairlearn.widget._fairlearn_widget.FairlearnWidget._response` +* The dictionary is updated in `fairlearn.widget._fairlearn_dashboard.FairlearnDashboard._on_request` +* There's a PR out to fill the whole data structure in `fairlearn.metrics.create_dashboard_dictionary` + +Issues with the status quo: +* Code duplication / redundancy / brittleness due to copy-paste errors +* Currently only one kind of a metric (`_summary`) is supported and hard-wired into the dashboard dictionary + +## Proposal + +### Part I: More general dashboard dictionary + +``` +{ + "prediction_type": "binary_classification" or "probabilistic_binary_classification" or "regression", + "y_true": float[], + "y_pred": float[][], + "sensitive_features": [ + { + "column_name": String, + "column_data": int[], + "value_names": string[] + }, + ... + ], + "metrics": [ + [ + { + "" : , + ... + }, + ... + ] + ] +} +``` + +Where `metrics[i][j][""]` should correspond to the result of the call +``` +(y_true, y_pred[i], sensitive_features[j]) +``` +If the result of the call is a dictionary, then we use that dictionary as ``. If the result is a scalar `x`, we could use the dictionary `{ "value": x }`. + +#### What's still missing here: + +The above is incomplete, because we need to enable caching results for the calls of the form +``` +(y_true, y_pred[i], sensitive_features[j], arg1=val_arg1, arg2=val_arg2, ...) +``` +One idea would be to use something like: +``` +{ + "": [ + { + "arguments": { "arg1": val_arg1, "arg2": val_arg2, ...}, + "result": + }, + ... + ] +} +``` + + +### Part II: How to remove duplication + +`` From 366c3e09d76548f7786e14e957431ef2c10ee68f Mon Sep 17 00:00:00 2001 From: Miro Dudik Date: Thu, 16 Apr 2020 10:11:06 -0400 Subject: [PATCH 7/8] flatten the dictionary Signed-off-by: Miro Dudik --- api/METRICS-DASHBOARD.md | 94 ++++++++++++++++++++++------------------ 1 file changed, 53 insertions(+), 41 deletions(-) diff --git a/api/METRICS-DASHBOARD.md b/api/METRICS-DASHBOARD.md index 6f6acbd..f1c79ef 100644 --- a/api/METRICS-DASHBOARD.md +++ b/api/METRICS-DASHBOARD.md @@ -17,55 +17,67 @@ Issues with the status quo: ### Part I: More general dashboard dictionary -``` -{ +```python +{ "prediction_type": "binary_classification" or "probabilistic_binary_classification" or "regression", - "y_true": float[], - "y_pred": float[][], - "sensitive_features": [ - { - "column_name": String, - "column_data": int[], - "value_names": string[] + "array_bindings": { # all 1D arrays, including features and predictior vectors, are here + "" : { # the keys can be arbitrary strings; not sure we need to force any convention, but see examples below + "name": string, # the name of a feature would be the feature name, of a prediction vector would be the model name + "values": number[], + "value_names": string[], # an optional field to encode categorical data + }, + "sensitive_feature gender" : { # an example feature + "name": "gender", + "values": [0, 1, 0, 0, 2], + "value_names": ["female", "male", "non-binary"], + }, + "y_pred model0" : { # an example prediction vector + "name": "model0", + "values": [0, 0, 1, 1, 0], + }, + "y_true": { + "name": "y_true", + "values": [0, 1, 1, 1, 0], }, + "sample_weight": { + "name": "sample_weight", + "values": [0.1, 0.3, 1, 0.9], + } ... - ], - "metrics": [ - [ - { - "" : , - ... - }, - ... - ] - ] -} -``` - -Where `metrics[i][j][""]` should correspond to the result of the call -``` -(y_true, y_pred[i], sensitive_features[j]) -``` -If the result of the call is a dictionary, then we use that dictionary as ``. If the result is a scalar `x`, we could use the dictionary `{ "value": x }`. - -#### What's still missing here: - -The above is incomplete, because we need to enable caching results for the calls of the form -``` -(y_true, y_pred[i], sensitive_features[j], arg1=val_arg1, arg2=val_arg2, ...) -``` -One idea would be to use something like: -``` -{ - "": [ + }, + "cache" : [ { - "arguments": { "arg1": val_arg1, "arg2": val_arg2, ...}, - "result": + "function": string, # python function name; we could either limit to fairlearn.metrics + # or use fully qualified names + "arguments": { + "": "" or null, # array-valued arguments are matched with array bindings + "": number or null, # we should also support numeric arguments, strings, booleans + "": string or null, # null corresponds to None + "": boolean or null, + }, + "return_value": number or string or boolean or null or dict, + # dict could be encoded as { "keys": any[], "values": any[] } + }, + { # an example + "function": "fbeta_score_group_summary", + "arguments": { + "y_true": "y_true", + "y_pred": "y_pred model0", + "sensitive_features": "sensitive_feature gender", + "sample_weight": "sample_weight", + "beta": 0.3, + }, + "return_value": { + "overall": 0.11, + "by_group": { + "keys": [0, 1, 2], + "values": [0.15, 0.04, 0.03], + } + }, }, ... ] } -``` ### Part II: How to remove duplication From c3172e218bc51eba3779387aa1e85a4965c33d95 Mon Sep 17 00:00:00 2001 From: Miro Dudik Date: Thu, 16 Apr 2020 10:13:33 -0400 Subject: [PATCH 8/8] close code block Signed-off-by: Miro Dudik --- api/METRICS-DASHBOARD.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api/METRICS-DASHBOARD.md b/api/METRICS-DASHBOARD.md index f1c79ef..3f05213 100644 --- a/api/METRICS-DASHBOARD.md +++ b/api/METRICS-DASHBOARD.md @@ -78,7 +78,7 @@ Issues with the status quo: ... ] } - +``` ### Part II: How to remove duplication