CoSy: Evaluating Textual Explanations of Neurons

PyTorch

This repository contains the code and experiments for the paper CoSy: Evaluating Textual Explanations of Neurons by Kopf et al., 2024.

Table of Contents

1. About
2. Repository overview
3. Installation
4. How to run

About

Evaluating Neuron Explanations In Explainable AI (XAI) there are methods which connect neurons in Deep Neural Networks (DNNs) to human-understandable textual descriptions. Evaluating the quality of these explanations is challenging due to the lack of a unified quantitative approach. We introduce CoSy (Concept Synthesis), a novel, architecture-agnostic framework for evaluating textual explanations of latent neurons. Given textual explanations, our proposed framework uses a generative model conditioned on textual input to create data points representing the explanations, comparing the neuron's response to these and control data points to estimate explanation quality. We validate our framework through meta-evaluation experiments and benchmark various concept-based textual explanation methods for Computer Vision tasks, revealing significant differences in quality.

Repository overview

The repository is organized for ease of use:

• The src/ folder contains all necessary functions.
• The notebooks/ folder includes notebooks for generating the plots in the paper.
• The assets/explanations/ folder includes pre-computed explanations for various explanation methods across

Installation

Install the necessary packages using the provided requirements.txt:

pip install -r requirements.txt

How to run

First, download your control dataset, e,g, ImageNet or Places365, and set the path to your dataset here.

0. Collect Explanations with Explanation Methods

If you wish to evaluate other explanations than the pre-computed ones in assets/explanations/, follow the instuctions of the project pages listed below to generate textual explanations of neurons:

• MILAN (Hernandez et al., 2022)
• FALCON (Kalibhat et al., 2023)
• CLIP-Dissect (Oikarinen et al., 2023)
• INVERT (Bykov et al., 2023)

Place the generated csv file with the textual explanations in the assets/explanations/{method_name}/ folder.

1. Collect Activations

Collect activations for your model specifying your control dataset. In the example below, we select resnet18 as target model, the avgpool layer for inspection, and the ImageNet validation dataset as the control dataset.

python src/activation_collector.py --target_model resnet18 --target_layer avgpool --dataset imagenet

2. Generate Explanation Images

In this step, generate the explanation images for your neuron explanations. Define METHOD and MODEL_NAME. For our example, set :

METHOD = "INVERT"
MODEL_NAME = "resnet18-avgpool"

Use the --nproc_per_node argument to specify the number of GPUs to use. For more information on distributed inference follow these instructions.

torchrun src/image_generator.py --nproc_per_node=3

3. Evaluate Explanations

In the last step, evaluate the neuron explanations with CoSy. For example, evaluate a set of neuron explanations of the avgpool layer from resnet18 generated by the INVERT explanation method.

python src/evaluation.py --target_model resnet18 --target_layer avgpool --method INVERT

Citation

If you find this work interesting or useful in your research, use the following Bibtex annotation to cite us:

@misc{kopf2024cosy,
      title={{CoSy: Evaluating Textual Explanations of Neurons}}, 
      author={Laura Kopf and Philine Lou Bommer and Anna Hedström and Sebastian Lapuschkin and Marina M. -C. Höhne and Kirill Bykov},
      year={2024},
      eprint={2405.20331},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2405.20331}
}

This work is in review.

Thank you

We hope our repository is beneficial to your work and research. If you have any feedback, questions, or ideas, please feel free to raise an issue in this repository. Alternatively, you can reach out to us directly via email for more in-depth discussions or suggestions.

📧 Contact us:

• Laura Kopf: kopf[at]tu-berlin.de

Thank you for your interest and support!