• WMT — The majority of language pairs from WMT go through English and the data is from the news domain. We consider data for 13 languages (Ondrej et al., 2017; Bojar et al., 2018; Barrault et al., 2019).

• WAT — The WAT competition covers Asian languages paired with English. We consider data for Burmese-English (Riza et al., 2016), which contains news articles.

WAT contains many other evaluation directions, but many of those are covered by WMT or in a specific domain, so we focus on Burmese-English for WAT only. • IWSLT — The IWSLT translation competition contains data from TED talks paired with English translations. We use data for 4 languages (Cettolo et al., 2017).

• FLORES — FLORES2(Guzmán et al., 2019) pairs two low resource languages, Sinhala and Nepali, with English in the Wikipedia domain.

• TED — The TED Talks dataset3(Ye et al., 2018) contains translations between more than 50 languages; most of the pairs do not include English. The evaluation data is n-way parallel and contains thousands of directions.

• Autshumato — Autshumato4is an 11-way parallel dataset comprising 10 African languages and English from the government domain. There is no standard valid/test split, so we use the first half of the dataset for valid and second half for test.

• Tatoeba — Tatoeba Challenge5covers 692 test pairs from mixed domains where sentences are contributed and translated by volunteers online. The evaluation pairs we use from Tatoeba cover 85 different languages