The main idea is to derive explanation by investigating the importance scores of different features used to output the final prediction. Such approaches can be built on features like manual features obtained from feature engineering, lexical features including word/tokens and n-gram, etc.

Feature importance

Model predictions are explained by learning a second, usually more explainable model, as a proxy. Surrogate model-based approaches are model-agnostic and can be used to achieve either local or global explanations. 

Surrogate model

It explains the prediction of an input instance by identifying and presenting other instances, usually from available labeled data, that are semantically similar to the input instance.

Example-driven technique

Explanations are provided by illustrating some or all of the prediction derivation process, which is an intuitive and effective explainability technique when the final prediction is the result of a series of reasoning steps.

Provenance-based

It involves human-readable representations, such as rules, trees, and programs to deliver explainability.

Declarative induction

There are five major explainability techniques that differ in the mechanisms they adopt to generate the raw mathematical justifications.

- Feature importance

- Surrogate model

- Example-driven

- Provenance-based

- Declarative induction

A complex NLP approach consisting of different components may employ more than one of these explainability techniques.

University of Michigan - Ann Arbor

Black box techniques, such as deep learning models and the use of language embeddings as features can substantially advance model quality, while their obfuscation of the process by which a model arrives at its results can be problematic. Therefore, In the broader AI community, this growing understanding of the importance of explainability has created an emerging field called Explainable AI (XAI).

Explanations can be categorized along two main aspects:
- The first distinguishes whether the explanation is for an individual prediction (local) or the model’s prediction process as a whole (global).
- The second differentiates between the explanation emerging directly from the prediction process (selfexplaining) versus requiring post-processing (posthoc).

Explainable AI for Natural Language Processing

Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020). A survey of the state of explainable AI for natural language processing. arXiv preprint arXiv:2010.00711.

A Survey of the State of Explainable AI for Natural Language Processing

- Local and global: A local explanation provides information or justification for the model’s prediction on a specific input, while a global explanation provides similar justification by revealing how the model’s prediction process works, independently of any particular input.
- Self-Explaining and Post-Hoc: A self-explaining approach generates the explanation
at the same time as the prediction. In contrast, a post-hoc approach requires that an additional operation is performed after making predictions.

Categorization of explanations

Explainability Techniques

The most common set of operations that are used to enable explainability:

- First-derivative saliency

- Layer-wise relevance propagation

- Input perturbations

- Attention

etc.

Operations to Enable Explainability

The major visualization techniques include:

- Saliency

- Raw declarative representations

- Natural language explanation

XAI Visualization Techniques

A model’s quality should be evaluated not only by its accuracy and performance, but also by how well it provides explanations for its predictions. The state of the field in terms of defining and measuring explanation quality includes evaluation and predictive process coverage.

Learn Before

Related

Learn After