A bottleneck to using MIXUP for NLP tasks was the requirement of continuous inputs.

Bottleneck of Using Mixup on NLP

It mixes embeddings or higher hidden layers to tackle the continuous problem.

MixText

https://aclanthology.org/2020.acl-main.194/

MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification

https://aclanthology.org/2020.acl-main.529/

AdvAug: Robust Adversarial Augmentation for Neural Machine Translation

It minimizes the vicinal risk over virtual sentences sampled from two vicinity distributions, in which the crucial one is a novel vicinity distribution for adversarial sentences that describes a smooth interpolated embedding space centered around observed training sentence pairs.

AdvAug

It generalizes MIXUP for sequence transduction tasks in two ways - the "hard" version samples a binary mask and picks from one of two sequences at each token position,
while the "soft" version softly interpolates between sequences based on a coefficient sampled from a Bernoulli prior.  The "soft" version is found to outperform the "hard" version and earlier interpolation-based techniques.

SEQ2MIXUP

https://aclanthology.org/2020.emnlp-main.447/

Sequence-Level Mixed Sample Data Augmentation

This class of techniques is also sometimes referred to as Mixed Sample Data Augmentation (MSDA). In the nlp area, related work often interpolates inner components, more general mixing schemes, and add adversaries.

University of Michigan - Ann Arbor

Data augmentation (DA) refers to strategies for increasing the diversity of training examples without explicitly collecting new data. Related methods are first explored in computer vision (CV), and DA’s adaptation for natural language processing (NLP) seems secondary and comparatively underexplored.

Data Augmentation in NLP

Interpolation is the process of estimating unknown values that fall between known values.

Interpolation

https://arxiv-download.xixiaoyao.cn/pdf/2105.03075.pdf

A Survey of Data Augmentation Approaches for NLP

The trade-off is often based on development cost and data variation.

Trade-off of DA Tricks in NLP

The typical explanation of DA is that it serves as regularization. Some researcher believes that it's insufficient. Existing work on this topic is mainly surface-level, and rarely investigates the theoretical underpinnings and principles.

Interpretation of DA in NLP

Rule-based data augmentation is to apply basic changes to the dataset which enlarges the divergence that the neural network sees.

Rule-based Data Augmentation

Example Interpolation in NLP

Seq2seq and language models have also been used for DA. Techniques include applying selective gates, generating different samples, and augmenting word representations with a context-sensitive attention-based mixture. 

Model-Based DA Techniques in NLP

It's widely used in Low-Resource NMT, Mitigating Bias, Fixing Class Imbalance, Few-shot Learning, and Adversarial Examples (AVEs).

Applications of DA in NLP

1. Summarization: Common Crawl, WikiRef, QMDSCNN, QMDSIR
2. QA: XLDA, QANet
3. Sequence Tagging Tasks: DAGA, dependency tree morphing, SEQMIX
4. Parsing Tasks: Data Recombination, GRAPPA, compositionality
5. Grammatical Error Correction: Use annotated data or confusion sets
6. Neural Machine Translation: BackTranslation, SwitchOut, Soft Contextual, Data Diversification
7. Data-to-Text NLG: perturbing language game scores or inputs, or adding noise.
8. Open-Ended & Conditional Generation: GEAUNG
9. Dialogue: LIGHTWEIGHT AUGMENTATION, diversity rank, MADA, spoken language understanding
10. Multimodal Tasks: Augment parallel data seperately and reinforce the alignment.

Downstream NLP Tasks applying DA Techniques 

1. Dissonance between empirical novelties and theoretical narrative
2. Minimal benefit for pretrained models on indomain data
3. Multimodal challenges
4. Span-based tasks
5. Working in specialized domains
6. Working with low-resource languages
7. More vision-inspired techniques
8. Self-supervised learning
9. Offline versus online data augmentation
10. Lack of unification
11. Good data augmentation practices

Challenges & Future Directions of DA in NLP

In machine learning, interpolation of training data occurs when an over-parameterized model perfectly fits the training dataset, achieving zero training error. Because large neural networks possess vastly more parameters than necessary to fit the data, they have the capacity to memorize every training example. This tendency to interpolate is a key mechanism that allows deep neural networks to behave more like nonparametric models.

Interpolation of Training Data

Bilinear interpolation is a commonly used upsampling technique in image processing to scale up an image. To calculate a pixel at coordinate $$(x, y)$$ in the upsampled output image, the technique maps $$(x, y)$$ to a corresponding real-valued coordinate $$(x', y')$$ on the original input image based on the ratio of the input size to the output size. It then identifies the four pixels closest to $$(x', y')$$ on the input image and calculates the final output pixel value based on these four pixels and their relative distances from $$(x', y')$$. In deep learning, this technique is frequently used to initialize transposed convolutional layers.

Learn Before

Related

Learn After