In a Many-to-Many translation setting, the amount of parallel data grows quadratically with the number of languages, so a standard-capacity neural network underfits rapidly. To supply enough capacity, model size is scaled far beyond typical bilingual systems: models over 50 times larger than current bilingual models are trained using model parallelism, building on scaling results from Kaplan et al. (2020) and Arora et al. (2018).