The data used to train the model are known as parallel texts, or bitexts.
bitexts = source + </s> + target
, where source is the text being translated, target is the translation output, and </s> is the end-of-sentence token.

University of Michigan - Ann Arbor

Instead of generating a sentence from scratch, the language model can complete a sequence given a specified prefix as following.

1. Pass a specified prefix to the language model using forward inference to produce a sequence of hidden states;
2. Apply autoregressive generation using the hidden state of the last word of the prefix as the starting point of generation;
3. The result of this process is a sequence of words that should be a reasonable completion given the prefix input.

A variation on autoregressive generation

Machine translation can use a scoring function for each possible translation and a heuristic search algorithm because the set of possible output sentences is too large. Optimization Verification can then diagnose whether an incorrect translation came from search failure or from the score computation.

Machine Translation

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Application of autoregressive generation given a prefix: Machine translation

The abstracted architecture of the twisted autoregressive generation is referred to as the encoder-decoder architecture.

It consists of three parts:
- encoder: accepts a sequence as its input and generates a corresponding sequence of contextualized representations (hidden states);
- context vector: is a function of the vector of contextualized representations generated from the encoder and conveys the essence of the input to the decoder;
- decoder: takes context vector as input and generates an arbitrary length sequence of hidden states, therefore obtains a corresponding sequence of output states.

The encoder and decoder networks are typically implemented with the same architecture, often using recurrent networks, but there are some other possibilities in each part.

Encoder-decoder networks

Backtranslation is a way of making use of monolingual corpora in the target language by creating synthetic bitexts. In backtranslation, we train an intermediate target-to-source MT system on the small bitext to translate the monolingual target data to the source language. 
Backtranslation has various parameters. One is how we generate the backtranslated data; we can run the decoder in greedy inference, or use beam search. Or we can do sampling, or Monte Carlo search. In Monte Carlo decoding, at each search timestep, instead of always generating the word with the highest softmax probability, we roll a weighted die, and use it to choose the next word according to its softmax probability.

Backtranslation

Translations are evaluated along two dimensions:
1. adequacy: how well the translation captures the exact meaning of the source sentence. 
 2. fluency: how fluent the translation is in the target language.

The most accurate evaluations use human raters. An alternative is to do ranking: give the raters a pair of candidate translations, and ask them which one they prefer.
While humans produce the best evaluations of machine translation output, running a
human evaluation can be time consuming and expensive. For this reason automatic
metrics are often used. 


 MT Evaluation

Machine translation models are trained on a parallel corpus, sometimes called a bitext, a text that appears in two (or more) languages. Some examples of parallel corpora are:
- The Europarl Corpus, extracted from the proceedings of the European Parliament
- The United Nations Parallel Corpus, extracted from official records and other parliamentary documents of the United Nations
- The OpenSubtitles Corpus, extracted from movie and TV subtitles
- The ParaCrawl Corpus, extracted from general web text

MT Corpora

A company is developing an automated translation feature for its real-time customer support chat. They are testing two prototype systems on a Spanish sentence from a customer: 'Mi pedido no ha llegado y estoy muy enojado.' (My order has not arrived and I am very angry.)

- **System A** translates this to English as: 'The delivery of my order is delayed and I am experiencing frustration.'
- **System B** translates this to English as: 'My order not arrived and I am very angry.'

System A produces a grammatically perfect and formal sentence, but it softens the customer's emotional tone. System B's output is grammatically flawed but directly conveys the customer's message and strong emotion. For the specific goal of a customer support agent needing to quickly understand the customer's problem and emotional state, which system's translation is more effective? Justify your reasoning by comparing the two outputs.

Assessing Translation Effectiveness for a Specific Use Case

A company is developing a translation service for legal documents, where preserving the precise meaning and complex sentence structure of the original text is the highest priority. The company has access to a massive parallel corpus of legal texts. Given these requirements, which approach would be more suitable and why?

An automated system is tasked with translating the English idiom 'The early bird gets the worm' into Spanish. It produces two options:

*   **Translation A:** 'El pájaro temprano consigue el gusano.' (A word-for-word translation.)
*   **Translation B:** 'A quien madruga, Dios le ayuda.' (An idiomatic equivalent, meaning 'God helps those who rise early.')

Evaluate these two translations. Which one is a better translation and why? In your answer, analyze the strengths and weaknesses of each option in terms of how well it captures the original meaning and how natural it sounds in the target language.

Evaluating Machine Translation Quality

In sequence-to-sequence learning tasks like machine translation, models often must handle unaligned data, where the input and output sequences do not share a strict one-to-one correspondence. Unaligned data introduces two primary challenges: the input and output sequences may have entirely different lengths, and the corresponding regions of meaning may appear in a different order, as seen when translating sentences between languages with different grammatical structures.

Unaligned Data in Sequence Learning

Statistical Machine Translation (SMT) is a machine translation paradigm where translations are generated based on statistical models whose parameters are estimated from the analysis of bilingual text corpora. SMT typically uses phrase-based models to translate input word or phrase units by finding the most statistically likely corresponding units in the target language.

Statistical Machine Translation (SMT)

Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) are two key paradigms in machine translation:

* **Statistical Machine Translation (SMT):** A phrase-based statistical approach that divides input sentences into word or phrase units and maps them to the most statistically similar translations stored in advance.
* **Neural Machine Translation (NMT):** An end-to-end deep learning approach that applies a single large artificial neural network to translate entire sentences. NMT typically consumes less memory, is faster during inference, and captures contextual information more effectively than the multi-stage pipeline of SMT.

Learn Before

Related