The popular BACKTRANSLATION method translates a sequence into another language and then back into the original language.

BackTranslation

Improving Neural Machine Translation Models with Monolingual Data

https://aclanthology.org/2020.findings-emnlp.90/

Generative Data Augmentation for Commonsense Reasoning

It generates synthetic examples using pretrained transformer language models and selects the most informative and diverse set for augmentation.

G-DAUG

Keep Calm and Switch On! Preserving Sentiment and Fluency in Semantic Text Exchange

It involves adjusting the overall semantics of a text to fit the context of a new word/phrase that is inserted called the replacement entity (RE).

SEMANTIC TEXT EXCHANGE (STE) 

It learns a label-conditioned generator by finetuning GPT-2 on the training data, using this to generate candidate examples per class. A classifier trained on the original training set is then used to select top k candidate examples that confidently belong to the respective class for augmentation.

LAMBADA

Not Enough Data? Deep Learning to the Rescue!

In natural language processing (NLP), model-based data augmentation techniques customized for downstream tasks can have strong positive effects on performance, but they are often difficult to develop and utilize.

Trade-off of Model-Based Data Augmentation Techniques in NLP

Seq2seq and language models have also been used for DA. Techniques include applying selective gates, generating different samples, and augmenting word representations with a context-sensitive attention-based mixture. 

University of Michigan - Ann Arbor

Data augmentation (DA) refers to strategies for increasing the diversity of training examples without explicitly collecting new data. Related methods are first explored in computer vision (CV), and DA’s adaptation for natural language processing (NLP) seems secondary and comparatively underexplored.

Data Augmentation in NLP

https://arxiv-download.xixiaoyao.cn/pdf/2105.03075.pdf

A Survey of Data Augmentation Approaches for NLP

The trade-off is often based on development cost and data variation.

Trade-off of DA Tricks in NLP

The typical explanation of DA is that it serves as regularization. Some researcher believes that it's insufficient. Existing work on this topic is mainly surface-level, and rarely investigates the theoretical underpinnings and principles.

Interpretation of DA in NLP

Rule-based data augmentation is to apply basic changes to the dataset which enlarges the divergence that the neural network sees.

Rule-based Data Augmentation

This class of techniques is also sometimes referred to as Mixed Sample Data Augmentation (MSDA). In the nlp area, related work often interpolates inner components, more general mixing schemes, and add adversaries.

Example Interpolation in NLP

Model-Based DA Techniques in NLP

It's widely used in Low-Resource NMT, Mitigating Bias, Fixing Class Imbalance, Few-shot Learning, and Adversarial Examples (AVEs).

Applications of DA in NLP

1. Summarization: Common Crawl, WikiRef, QMDSCNN, QMDSIR
2. QA: XLDA, QANet
3. Sequence Tagging Tasks: DAGA, dependency tree morphing, SEQMIX
4. Parsing Tasks: Data Recombination, GRAPPA, compositionality
5. Grammatical Error Correction: Use annotated data or confusion sets
6. Neural Machine Translation: BackTranslation, SwitchOut, Soft Contextual, Data Diversification
7. Data-to-Text NLG: perturbing language game scores or inputs, or adding noise.
8. Open-Ended & Conditional Generation: GEAUNG
9. Dialogue: LIGHTWEIGHT AUGMENTATION, diversity rank, MADA, spoken language understanding
10. Multimodal Tasks: Augment parallel data seperately and reinforce the alignment.

Downstream NLP Tasks applying DA Techniques 

1. Dissonance between empirical novelties and theoretical narrative
2. Minimal benefit for pretrained models on indomain data
3. Multimodal challenges
4. Span-based tasks
5. Working in specialized domains
6. Working with low-resource languages
7. More vision-inspired techniques
8. Self-supervised learning
9. Offline versus online data augmentation
10. Lack of unification
11. Good data augmentation practices

Learn Before

Related

Learn After