The major categories of evaluation techniques include: informal examination of explanations, comparison to ground truth, human evaluation, etc.

Explanations are multi-faceted objects that can be evaluated on multiple aspects, such as fidelity, comprehensibility, and others. Therefore, understanding the target of the evaluation is important for interpreting the evaluation results.

Evaluation methods of XAI

An important and often overlooked aspect of explanation quality is the part of the prediction process covered by an explanation.

The level of coverage may be an artifact of explainability techniques used: provenance-based approaches tend to provide more coverage, while example-driven approaches, may provide little to no coverage.

Higher coverage, while in general a positive aspect, should always be considered in combination with the target use and audience of the produced explanations.

Predictive Process Coverage

A model’s quality should be evaluated not only by its accuracy and performance, but also by how well it provides explanations for its predictions. The state of the field in terms of defining and measuring explanation quality includes evaluation and predictive process coverage.

University of Michigan - Ann Arbor

Black box techniques, such as deep learning models and the use of language embeddings as features can substantially advance model quality, while their obfuscation of the process by which a model arrives at its results can be problematic. Therefore, In the broader AI community, this growing understanding of the importance of explainability has created an emerging field called Explainable AI (XAI).

Explanations can be categorized along two main aspects:
- The first distinguishes whether the explanation is for an individual prediction (local) or the model’s prediction process as a whole (global).
- The second differentiates between the explanation emerging directly from the prediction process (selfexplaining) versus requiring post-processing (posthoc).

Explainable AI for Natural Language Processing

Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020). A survey of the state of explainable AI for natural language processing. arXiv preprint arXiv:2010.00711.

A Survey of the State of Explainable AI for Natural Language Processing

- Local and global: A local explanation provides information or justification for the model’s prediction on a specific input, while a global explanation provides similar justification by revealing how the model’s prediction process works, independently of any particular input.
- Self-Explaining and Post-Hoc: A self-explaining approach generates the explanation
at the same time as the prediction. In contrast, a post-hoc approach requires that an additional operation is performed after making predictions.

Categorization of explanations

There are five major explainability techniques that differ in the mechanisms they adopt to generate the raw mathematical justifications.

- Feature importance

- Surrogate model

- Example-driven

- Provenance-based

- Declarative induction

A complex NLP approach consisting of different components may employ more than one of these explainability techniques.

Explainability Techniques

The most common set of operations that are used to enable explainability:

- First-derivative saliency

- Layer-wise relevance propagation

- Input perturbations

- Attention

etc.

Operations to Enable Explainability

The major visualization techniques include:

- Saliency

- Raw declarative representations

- Natural language explanation

Learn Before

Related

Learn After