The success of universal language models is largely driven by a specific architectural strategy in pre-training. This approach involves separating the common components shared across many neural network-based systems and training these shared structures on massive amounts of unlabeled data using self-supervision, rather than building separate full systems from scratch for every distinct task.