Attention layers can be added to most NN architectures and, because they appeal to human intuition, can help indicate where the NN model is “focusing”.

University of Michigan - Ann Arbor

The most common set of operations that are used to enable explainability:

- First-derivative saliency

- Layer-wise relevance propagation

- Input perturbations

- Attention

etc.

Operations to Enable Explainability

Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020). *A survey of the state of explainable AI for natural language processing*. arXiv preprint arXiv:2010.00711.

A Survey of the State of Explainable AI for Natural Language Processing

It's another way to attribute relevance to features computed in any intermediate layer of an NN. Layer-wise relevance propagation has been used to enable feature importance explainability and example-driven explainability.

Layer-wise relevance propagation

Gradient-based explanations estimate the contribution of input $i$ towards output $o$ by computing the partial derivative of $o$ with respect to $i$.

This operation is particularly convenient for NN-based models because these can be computed for any layer using a single call to auto-differentiation, which most deep learning engines provide out-of-the-box.

First-derivative saliency

Input perturbations can explain the output for input x by generating random perturbations of x and training an explainable model. They are mainly used to enable surrogate models.

Input perturbations

Attention Layers

It is common to mine the outputs of LSTM cells to explain outputs, but there may also be information present in the outputs of the gates produced within the cells. It's possible to utilize other operations to interpret gating signals to aid feature importance explainability.

Learn Before

Related