Large-scale research on pre-training in natural language processing began with the development of self-supervised language models, such as BERT and GPT. These models operate on the shared principle that general language understanding and generation can be achieved by training them to predict masked words within massive amounts of text. Despite the simplicity of this approach, the resulting models demonstrate a remarkable ability to model linguistic structure without explicit training for it. The generality of these self-supervised pre-training tasks leads to systems that exhibit strong performance across a wide variety of problems, often outperforming well-developed supervised systems.