The Illusion of Competence: Evaluating the Effect of Explanations on Users’ Mental Models of Visual Question Answering Systems

This is the data and code for our paper "The Illusion of Competence: Evaluating the Effect of Explanations on Users’ Mental Models of Visual Question Answering Systems" (EMNLP 2024)

Files

• data/Data_selection.xlsx: This file contains the data used for both experiments, along with the encoding of the experimental conditions and the distribution across the experimental lists.
• data/Experiment_A_results.csv: This file contains the results from the human evaluation study of Experiment A.
• data/Experiment_X_results.csv: This file contains the results from the human evaluation study of Experiment X.
• scripts/evaluate_human_results.ipynb: This script can be used to evaluate the results from both experiments.

• Source code for NLX-GPT is available from GitHub at github.com/fawazsammani/nlxgpt
• Source code for Uni-NLX is available from GitHub at github.com/fawazsammani/uni-nlx
• Source code for PJ-X and VQA-X data is available from GitHub at github.com/Seth-Park/MultimodalExplanations
• COCO Images for VQA-X are available here: cocodataset.org
• CLEVR-X data is available from GitHub at github.com/ExplainableML/CLEVR-X
• CLEVR images for \mbox{CLEVR-X} are available here: cs.stanford.edu/people/jcjohns/clevr