This typically takes the form of high-level discussions of how examples of generated explanations align with human intuition. 

This includes cases where the output of a single explainability approach is examined
in isolation, or when explanations are compared to those of other reference approaches.

Informal examination of explanations

This compares generated explanations to ground truth data in order to quantify the performance of explainability techniques.

Approaches employed to address this issue involve having multiple annotators and reporting inter-annotator agreement or mean human performance, as well as evaluating the explanations at different granularities to account for disagreements on the precise value of the ground truth.

Comparison to ground truth

A more direct way to assess the explanation quality is to ask humans to evaluate
the effectiveness of the generated explanations.

This has the advantage of avoiding the assumption that there is only one good explanation that could serve as ground truth, as well as sidestepping the need to measure similarity of explanations. It is important to have multiple annotators, report
inter-annotator agreement, and correctly deal with subjectivity and variance in the responses.

Human evaluation

The major categories of evaluation techniques include: informal examination of explanations, comparison to ground truth, human evaluation, etc.

Explanations are multi-faceted objects that can be evaluated on multiple aspects, such as fidelity, comprehensibility, and others. Therefore, understanding the target of the evaluation is important for interpreting the evaluation results.

University of Michigan - Ann Arbor

A model’s quality should be evaluated not only by its accuracy and performance, but also by how well it provides explanations for its predictions. The state of the field in terms of defining and measuring explanation quality includes evaluation and predictive process coverage.

XAI quality

Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020). *A survey of the state of explainable AI for natural language processing*. arXiv preprint arXiv:2010.00711.

A Survey of the State of Explainable AI for Natural Language Processing

Evaluation methods of XAI

An important and often overlooked aspect of explanation quality is the part of the prediction process covered by an explanation.

The level of coverage may be an artifact of explainability techniques used: provenance-based approaches tend to provide more coverage, while example-driven approaches, may provide little to no coverage.

Higher coverage, while in general a positive aspect, should always be considered in combination with the target use and audience of the produced explanations.

Learn Before

Related

Learn After