Explainable AI (XAI) has the potential to enhance decision-making in human-AI collaborations, yet existing research indicates that explanations can also lead to undue reliance on AI recommendations, a dilemma often referred to as the ‘white box paradox.’ This paradox illustrates how persuasive explanations for incorrect advice might foster inappropriate trust in AI systems. Our study extends beyond the traditional scope of the white box paradox by proposing a framework for examining explanation inadequacy. We specifically investigate how accurate AI advice, when paired with misleading explanations, affects decision-making in logic puzzle tasks. Our findings introduce the concept of the ‘XAI halo effect,’ where participants were influenced by the misleading explanations to the extent that they did not verify the correctness of the advice, despite its accuracy. This effect reveals a nuanced challenge in XAI, where even correct advice can lead to misjudgment if the accompanying explanations are not coherent and contextually relevant. The study highlights the critical need for explanations to be both accurate and relevant, especially in contexts where decision accuracy is paramount. This calls into question the use of explanations in situations where their potential to mislead outweighs their transparency or educational value.

Cabitza, F., Fregosi, C., Campagner, A., Natali, C. (2024). Explanations Considered Harmful: The Impact of Misleading Explanations on Accuracy in Hybrid Human-AI Decision Making. In Explainable Artificial Intelligence Second World Conference, xAI 2024, Valletta, Malta, July 17–19, 2024, Proceedings, Part IV (pp.255-269). Springer Cham [10.1007/978-3-031-63803-9_14].

Explanations Considered Harmful: The Impact of Misleading Explanations on Accuracy in Hybrid Human-AI Decision Making

Cabitza, F
Primo
;
Fregosi, C
Secondo
;
Campagner, A
Penultimo
;
Natali, C
Ultimo
2024

Abstract

Explainable AI (XAI) has the potential to enhance decision-making in human-AI collaborations, yet existing research indicates that explanations can also lead to undue reliance on AI recommendations, a dilemma often referred to as the ‘white box paradox.’ This paradox illustrates how persuasive explanations for incorrect advice might foster inappropriate trust in AI systems. Our study extends beyond the traditional scope of the white box paradox by proposing a framework for examining explanation inadequacy. We specifically investigate how accurate AI advice, when paired with misleading explanations, affects decision-making in logic puzzle tasks. Our findings introduce the concept of the ‘XAI halo effect,’ where participants were influenced by the misleading explanations to the extent that they did not verify the correctness of the advice, despite its accuracy. This effect reveals a nuanced challenge in XAI, where even correct advice can lead to misjudgment if the accompanying explanations are not coherent and contextually relevant. The study highlights the critical need for explanations to be both accurate and relevant, especially in contexts where decision accuracy is paramount. This calls into question the use of explanations in situations where their potential to mislead outweighs their transparency or educational value.
paper
Explainability paradox; Explainable artificial Intelligence (XAI); Human-AI Interaction;
English
Second World Conference, xAI 2024 - July 17–19, 2024
2024
Longo, L.;Lapuschkin, S; Seifert, C
Explainable Artificial Intelligence Second World Conference, xAI 2024, Valletta, Malta, July 17–19, 2024, Proceedings, Part IV
9783031638022
2024
2156 CCIS
255
269
https://link.springer.com/content/pdf/10.1007/978-3-031-63803-9_14.pdf?pdf=inline link
partially_open
Cabitza, F., Fregosi, C., Campagner, A., Natali, C. (2024). Explanations Considered Harmful: The Impact of Misleading Explanations on Accuracy in Hybrid Human-AI Decision Making. In Explainable Artificial Intelligence Second World Conference, xAI 2024, Valletta, Malta, July 17–19, 2024, Proceedings, Part IV (pp.255-269). Springer Cham [10.1007/978-3-031-63803-9_14].
File in questo prodotto:
File Dimensione Formato  
Cabitza-2024-Second World Conference, xAI 2024-CCIS-preprint.pdf

accesso aperto

Tipologia di allegato: Submitted Version (Pre-print)
Licenza: Creative Commons
Dimensione 3.06 MB
Formato Adobe PDF
3.06 MB Adobe PDF Visualizza/Apri
Cabitza-2024-II World Conf xAI 2024-CCIS-VoR.pdf

Solo gestori archivio

Tipologia di allegato: Publisher’s Version (Version of Record, VoR)
Licenza: Tutti i diritti riservati
Dimensione 1.74 MB
Formato Adobe PDF
1.74 MB Adobe PDF   Visualizza/Apri   Richiedi una copia

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/10281/499419
Citazioni
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
Social impact