(1) a large corpus of text is preprocessed and divided into different languages,

 (2) candidate pairs of aligned sentences are embedded and stored in a index, 

(3) indexed sentences are compared to form potential pairs, (4) the resulting candidate pairs are filtered in post-processing

Steps of generic data mining pipeline

 This search requires a measure that captures the semantic similarity between sentences in different languages.

University of Michigan - Ann Arbor

Supervised translation systems rely on large quantities of parallel sentences

Hence, we use mining technique

Covering the Language Matrix

Mining parallel data with LASER

We apply a filtering step to remove sentences of greater than 50% punctuation. The data is then deduplicated, and we remove any sentence that appears in any validation or test dataset – even if it is associated with another language pair. 

Finally, we apply length and language-specific filtering

Learn Before

Related

Learn After