Skip to content

[Bug]: F1AdaptiveThreshold incorrectly selects minimum value when no anomalous images are in validation set #2433

@tanemaki

Description

@tanemaki

Describe the bug

The F1AdaptiveThreshold is designed to select the optimal threshold for separating normal and anomalous classes based on the F1 score. Ideally, this optimal threshold should correspond to the "maximum" predicted value when all images are normal. However, the current implementation incorrectly uses the "minimum" predicted value instead. This error leads to an abnormally low threshold when the model is trained exclusively with normal images, resulting in an excessively high false alarm rate.

Dataset

N/A

Model

N/A

Steps to reproduce the behavior

labels = torch.tensor([0, 0, 0, 0])
preds = torch.tensor([1.0, 2.0, 3.0, 4.0])

adaptive_threshold = F1AdaptiveThreshold(default_value=0.5)
threshold = adaptive_threshold(preds, labels)

# now threshold is tensor(1.0) but it should be tensor(4.0) because all images are normal

OS information

OS information:

  • OS: macOS Sequoia 15.1.1
  • Python version: 3.10.14
  • Anomalib version: 2.0.0dev
  • PyTorch version: 2.5.1
  • CUDA/cuDNN version: N/A
  • GPU models and configuration: N/A
  • Any other relevant information: N/A

Expected behavior

When all images are normal, the optimized value of F1AdaptiveThreshold should be the "maximum" predicted value.

In other words, the F1AdaptiveThreshold should pass the following test case:

labels = torch.tensor([0, 0, 0, 0])
preds = torch.tensor([1.0, 2.0, 3.0, 4.0])

adaptive_threshold = F1AdaptiveThreshold(default_value=0.5)
adaptive_threshold.update(preds, labels)
threshold_value = adaptive_threshold.compute()

assert threshold_value == 4.0

Screenshots

No response

Pip/GitHub

GitHub

What version/branch did you use?

bcc0b43

Configuration YAML

N/A

Logs

N/A

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions