The data used to train the model are known as parallel texts, or bitexts.
bitexts = source + </s> + target
, where source is the text being translated, target is the translation output, and </s> is the end-of-sentence token.

University of Michigan - Ann Arbor

Instead of generating a sentence from scratch, the language model can complete a sequence given a specified prefix as following.

1. Pass a specified prefix to the language model using forward inference to produce a sequence of hidden states;
2. Apply autoregressive generation using the hidden state of the last word of the prefix as the starting point of generation;
3. The result of this process is a sequence of words that should be a reasonable completion given the prefix input.

A variation on autoregressive generation

Machine translation is a sequence-to-sequence learning problem that involves ingesting a sentence in a source language and predicting its corresponding translation in a target language. A major challenge in this task is unaligned data, because the input and output sequences can have varying lengths and the corresponding words or regions may appear in completely different orders.

Machine Translation

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Application of autoregressive generation given a prefix: Machine translation

The abstracted architecture of the twisted autoregressive generation is referred to as the encoder-decoder architecture.

It consists of three parts:
- encoder: accepts a sequence as its input and generates a corresponding sequence of contextualized representations (hidden states);
- context vector: is a function of the vector of contextualized representations generated from the encoder and conveys the essence of the input to the decoder;
- decoder: takes context vector as input and generates an arbitrary length sequence of hidden states, therefore obtains a corresponding sequence of output states.

The encoder and decoder networks are typically implemented with the same architecture, often using recurrent networks, but there are some other possibilities in each part.

Encoder-decoder networks

SMT
- Phrase-based statistical approach.
- Store several meanings of particular word in advance.
- When the user enters input value, it divides it into the word. or phrase unit.
- Find and translate the most statistically similar meaning.


NMT
- How human brain trains itself.
- Applies a large artificial neural network 
- Takes up less memory, less time
- NMT models fit a single model rather than a pipeline of fine-tuned models.
- Currently achieves comparably precise translation results.
- End-to-End model design

Statistical Machine Translation vs 
Neural Machine Translation

Backtranslation is a way of making use of monolingual corpora in the target language by creating synthetic bitexts. In backtranslation, we train an intermediate target-to-source MT system on the small bitext to translate the monolingual target data to the source language. 
Backtranslation has various parameters. One is how we generate the backtranslated data; we can run the decoder in greedy inference, or use beam search. Or we can do sampling, or Monte Carlo search. In Monte Carlo decoding, at each search timestep, instead of always generating the word with the highest softmax probability, we roll a weighted die, and use it to choose the next word according to its softmax probability.

Backtranslation

Translations are evaluated along two dimensions:
1. adequacy: how well the translation captures the exact meaning of the source sentence. 
 2. fluency: how fluent the translation is in the target language.

The most accurate evaluations use human raters. An alternative is to do ranking: give the raters a pair of candidate translations, and ask them which one they prefer.
While humans produce the best evaluations of machine translation output, running a
human evaluation can be time consuming and expensive. For this reason automatic
metrics are often used. 


 MT Evaluation

Machine translation models are trained on a parallel corpus, sometimes called a bitext, a text that appears in two (or more) languages. Some examples of parallel corpora are:
- The Europarl Corpus, extracted from the proceedings of the European Parliament
- The United Nations Parallel Corpus, extracted from official records and other parliamentary documents of the United Nations
- The OpenSubtitles Corpus, extracted from movie and TV subtitles
- The ParaCrawl Corpus, extracted from general web text

MT Corpora

A company is developing an automated translation feature for its real-time customer support chat. They are testing two prototype systems on a Spanish sentence from a customer: 'Mi pedido no ha llegado y estoy muy enojado.' (My order has not arrived and I am very angry.)

- **System A** translates this to English as: 'The delivery of my order is delayed and I am experiencing frustration.'
- **System B** translates this to English as: 'My order not arrived and I am very angry.'

System A produces a grammatically perfect and formal sentence, but it softens the customer's emotional tone. System B's output is grammatically flawed but directly conveys the customer's message and strong emotion. For the specific goal of a customer support agent needing to quickly understand the customer's problem and emotional state, which system's translation is more effective? Justify your reasoning by comparing the two outputs.

Assessing Translation Effectiveness for a Specific Use Case

A company is developing a translation service for legal documents, where preserving the precise meaning and complex sentence structure of the original text is the highest priority. The company has access to a massive parallel corpus of legal texts. Given these requirements, which approach would be more suitable and why?

An automated system is tasked with translating the English idiom 'The early bird gets the worm' into Spanish. It produces two options:

*   **Translation A:** 'El pájaro temprano consigue el gusano.' (A word-for-word translation.)
*   **Translation B:** 'A quien madruga, Dios le ayuda.' (An idiomatic equivalent, meaning 'God helps those who rise early.')

Evaluate these two translations. Which one is a better translation and why? In your answer, analyze the strengths and weaknesses of each option in terms of how well it captures the original meaning and how natural it sounds in the target language.

Evaluating Machine Translation Quality

In sequence-to-sequence learning tasks like machine translation, models often must handle unaligned data, where the input and output sequences do not share a strict one-to-one correspondence. Unaligned data introduces two primary challenges: the input and output sequences may have entirely different lengths, and the corresponding regions of meaning may appear in a different order, as seen when translating sentences between languages with different grammatical structures.

Learn Before

Related