Add max false positive rate option to tf.keras.metrics.AUC

[9las]

System information.

TensorFlow version (you are using): 2.12.0
Are you willing to contribute it (Yes/No) : No

Describe the feature and the current behavior/state.

I would like an option for tf.keras.metrics.AUC to control the upper bound when integrating the ROC curve. The standardized partial AUC over the false positive rate range [0, upper bound] should be returned. The option should work like the max_fpr option for sklearn.metrics.roc_auc_score.

The option would be useful if you e.g. are only interested in high model performance for low false positive rates. Then you could calculate AUC over e.g. the range [0, 0.1].

Will this change the current api? How?
I do not know

Who will benefit from this feature?
Keras users

Contributing

• Do you want to contribute a PR? (yes/no): no
• If yes, please read this page for instructions
• Briefly describe your candidate solution(if contributing):

Edit
I originally wrote that the option should control the upper bound of the false positive rate. But I think it is better if the option control both the upper and lower bound so that the option is more general.

[tilakrayal]

@9las,
Could you please elaborate about your Feature. Also, please specify the Use Cases for this feature. Thank you!

[9las]

Describe the feature

Let us look at the following diagram:

Here the receiver operating characteristic (ROC) curve is shown in blue. The false positive rate (FPR), wich is equal to 1 - specificity is on the first axis. The true positive rate (TPR) which is equal to sensitivity is on the second axis. The area under the ROC curve (AUC) is then the entire area under the blue curve from a FPR of 0 to a FPR of 1 (or from 0 to 100% on the diagram).

The partial AUC (pAUC) is then obtained by constraining FPR to a given interval. On the diagram the interval is constrained to $[A = 0.2, B = 0.4]$ and the pAUC is the area ABCD. The pAUC corresponding to a random model is the area ABgh. The pAUC corresponding to a perfect model is the area ABef. The random model area ABgh is the minimum pAUC ($pAUC_{min}$), and the perfect model area ABef is the maximum pAUC ($pAUC_{max}$). The pAUC can be interpreted as the average sensitivity in the pre-specified specificity interval.

The pAUC can be standardized so that it has the same scale as the total area under the ROC curve. This can be achieved by the McClish correction. The standardized pAUC ($pAUC_s$) is given by:
$$pAUC_s = \frac{1}{2} \left(1 + \frac{pAUC - pAUC_{min}}{pAUC_{max}-pAUC_{min}}\right)$$

This transformation makes the pAUC in the FPR interval [0, 1] equal to the AUC, since in this case $pAUC_{max} = 1$ and $pAUC_{min} = \frac{1}{2}$:
$$pAUC_s = \frac{1}{2} \left(1 + \frac{pAUC - \frac{1}{2}}{1-\frac{1}{2}}\right) = \frac{1}{2} \left(1 + 2pAUC - 1\right) = pAUC = AUC $$

In addition it has a maximum value of 1 (when $pAUC = pAUC_{max}$) and a minimum value of $\frac{1}{2}$ (when $pAUC = pAUC_{min}$). Therefore it allows to view the partial area on the same scale as the total area under the ROC curve.

$pAUC_{max}$ is the area of a rectangle ABef with sides of length 1 and $B - A$ and thus is equal to $B - A$.

$pAUC_{min}$ is the area of the trapezoid ABgh and is found as follows:
$$pAUC_{min} = \frac{1}{2}B^2 - \frac{1}{2}A^2 = \frac{1}{2}\left(B^2-A^2\right)$$

So I would like the feature to be an option for tf.keras.metrics.AUC to be able to control the lower and upper bound of FPR. The pAUC should be returned. Maybe it could be an option to standardize the pAUC or not.

The pAUC is also described here and here.

Use Cases

I have different neural network architectures that work as binary classifiers. I can use AUC to compare the models and for early stopping. However, I am not interested in a FPR above 0.1 and thus i makes sense to optimise the model based on pAUC in the interval $[0, 0.1]$ instead.

When comparing two ROC curves, the curves may be different over some range of clinical interest, while one may be the same as the other over the remainder of the curve. This may not be reflected in a global test such as the entire area under the curve (AUC).

[qlzh727]

Thanks for filing this request. Just curious why the false positive rate option is for max value, rather than an range which has [min, max], eg the [A=0.2 and B = 0.4] from your diagram above.

[9las]

Thanks for filing this request. Just curious why the false positive rate option is for max value, rather than an range which has [min, max], eg the [A=0.2 and B = 0.4] from your diagram above.

I think it is better if the false positive range option is for the range, so that it is more general. The reason why I wrote in my first post that it should be for max value was because that was what they had done for sklearn.metrics.roc_auc_score.

[qlzh727]

I see. That will make more sense. So far I think this FR is nice to have, and we are little bit short of eng resource at the moment. Any contribution from community is welcome, otherwise we will do it when we have time.

Thanks.