This has been primarily used to visualize the importance scores of different types of
elements in XAI learning systems, such as showing input-output word alignment, highlighting words in input text or displaying extracted relations. 

Saliency-based visualizations are popular because they present visually perceptive
explanations, and they can be easily understood by different types of end users.

Visualization technique: Saliency

This visualization technique directly presents the learned declarative representations,
such as logic rules, trees, and programs. Such techniques assume that end users can
understand specific representations, such as first order logic rules and reasoning trees, and therefore may implicitly target more advanced users.

Raw declarative representations

It is verbalized in human-comprehensible natural language. The natural language can be generated using sophisticated deep learning models, or by using simple template based approaches.

Natural language explanation

The major visualization techniques include:

- Saliency

- Raw declarative representations

- Natural language explanation

University of Michigan - Ann Arbor

Black box techniques, such as deep learning models and the use of language embeddings as features can substantially advance model quality, while their obfuscation of the process by which a model arrives at its results can be problematic. Therefore, In the broader AI community, this growing understanding of the importance of explainability has created an emerging field called Explainable AI (XAI).

Explanations can be categorized along two main aspects:
- The first distinguishes whether the explanation is for an individual prediction (local) or the model’s prediction process as a whole (global).
- The second differentiates between the explanation emerging directly from the prediction process (selfexplaining) versus requiring post-processing (posthoc).

Explainable AI for Natural Language Processing

Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020). *A survey of the state of explainable AI for natural language processing*. arXiv preprint arXiv:2010.00711.

A Survey of the State of Explainable AI for Natural Language Processing

- Local and global: A local explanation provides information or justification for the model’s prediction on a specific input, while a global explanation provides similar justification by revealing how the model’s prediction process works, independently of any particular input.
- Self-Explaining and Post-Hoc: A self-explaining approach generates the explanation
at the same time as the prediction. In contrast, a post-hoc approach requires that an additional operation is performed after making predictions.

Categorization of explanations

There are five major explainability techniques that differ in the mechanisms they adopt to generate the raw mathematical justifications.

- Feature importance

- Surrogate model

- Example-driven

- Provenance-based

- Declarative induction

A complex NLP approach consisting of different components may employ more than one of these explainability techniques.

Explainability Techniques

The most common set of operations that are used to enable explainability:

- First-derivative saliency

- Layer-wise relevance propagation

- Input perturbations

- Attention

etc.

Operations to Enable Explainability

XAI Visualization Techniques

A model’s quality should be evaluated not only by its accuracy and performance, but also by how well it provides explanations for its predictions. The state of the field in terms of defining and measuring explanation quality includes evaluation and predictive process coverage.

Learn Before

Related

Learn After