It's another way to attribute relevance to features computed in any intermediate layer of an NN. Layer-wise relevance propagation has been used to enable feature importance explainability and example-driven explainability.

Layer-wise relevance propagation

Gradient-based explanations estimate the contribution of input $i$ towards output $o$ by computing the partial derivative of $o$ with respect to $i$.

This operation is particularly convenient for NN-based models because these can be computed for any layer using a single call to auto-differentiation, which most deep learning engines provide out-of-the-box.

First-derivative saliency

Input perturbations can explain the output for input x by generating random perturbations of x and training an explainable model. They are mainly used to enable surrogate models.

Input perturbations

Attention layers can be added to most NN architectures and, because they appeal to human intuition, can help indicate where the NN model is “focusing”.

Attention Layers

It is common to mine the outputs of LSTM cells to explain outputs, but there may also be information present in the outputs of the gates produced within the cells. It's possible to utilize other operations to interpret gating signals to aid feature importance explainability.

LSTM gating signals

The most common set of operations that are used to enable explainability:

- First-derivative saliency

- Layer-wise relevance propagation

- Input perturbations

- Attention

etc.

University of Michigan - Ann Arbor

Black box techniques, such as deep learning models and the use of language embeddings as features can substantially advance model quality, while their obfuscation of the process by which a model arrives at its results can be problematic. Therefore, In the broader AI community, this growing understanding of the importance of explainability has created an emerging field called Explainable AI (XAI).

Explanations can be categorized along two main aspects:
- The first distinguishes whether the explanation is for an individual prediction (local) or the model’s prediction process as a whole (global).
- The second differentiates between the explanation emerging directly from the prediction process (selfexplaining) versus requiring post-processing (posthoc).

Explainable AI for Natural Language Processing

Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020). A survey of the state of explainable AI for natural language processing. arXiv preprint arXiv:2010.00711.

A Survey of the State of Explainable AI for Natural Language Processing

- Local and global: A local explanation provides information or justification for the model’s prediction on a specific input, while a global explanation provides similar justification by revealing how the model’s prediction process works, independently of any particular input.
- Self-Explaining and Post-Hoc: A self-explaining approach generates the explanation
at the same time as the prediction. In contrast, a post-hoc approach requires that an additional operation is performed after making predictions.

Categorization of explanations

There are five major explainability techniques that differ in the mechanisms they adopt to generate the raw mathematical justifications.

- Feature importance

- Surrogate model

- Example-driven

- Provenance-based

- Declarative induction

A complex NLP approach consisting of different components may employ more than one of these explainability techniques.

Explainability Techniques

Operations to Enable Explainability

The major visualization techniques include:

- Saliency

- Raw declarative representations

- Natural language explanation

XAI Visualization Techniques

A model’s quality should be evaluated not only by its accuracy and performance, but also by how well it provides explanations for its predictions. The state of the field in terms of defining and measuring explanation quality includes evaluation and predictive process coverage.

Learn Before

Related

Learn After