A bottleneck to using MIXUP for NLP tasks was the requirement of continuous inputs.

University of Michigan - Ann Arbor

This class of techniques is also sometimes referred to as Mixed Sample Data Augmentation (MSDA). In the nlp area, related work often interpolates inner components, more general mixing schemes, and add adversaries.

Example Interpolation in NLP

https://arxiv-download.xixiaoyao.cn/pdf/2105.03075.pdf

A Survey of Data Augmentation Approaches for NLP

Bottleneck of Using Mixup on NLP

It mixes embeddings or higher hidden layers to tackle the continuous problem.

MixText

https://aclanthology.org/2020.acl-main.194/

MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

https://aclanthology.org/2020.acl-main.529/

AdvAug: Robust Adversarial Augmentation for Neural Machine Translation

It minimizes the vicinal risk over virtual sentences sampled from two vicinity distributions, in which the crucial one is a novel vicinity distribution for adversarial sentences that describes a smooth interpolated embedding space centered around observed training sentence pairs.

AdvAug

It generalizes MIXUP for sequence transduction tasks in two ways - the "hard" version samples a binary mask and picks from one of two sequences at each token position,
while the "soft" version softly interpolates between sequences based on a coefficient sampled from a Bernoulli prior.  The "soft" version is found to outperform the "hard" version and earlier interpolation-based techniques.

SEQ2MIXUP

https://aclanthology.org/2020.emnlp-main.447/

Learn Before

Related