The data used to train the model are known as parallel texts, or bitexts.
bitexts = source + </s> + target
, where source is the text being translated, target is the translation output, and </s> is the end-of-sentence token.

Application of autoregressive generation given a prefix: Machine translation

SMT
- Phrase-based statistical approach.
- Store several meanings of particular word in advance.
- When the user enters input value, it divides it into the word. or phrase unit.
- Find and translate the most statistically similar meaning.


NMT
- How human brain trains itself.
- Applies a large artificial neural network 
- Takes up less memory, less time
- NMT models fit a single model rather than a pipeline of fine-tuned models.
- Currently achieves comparably precise translation results.
- End-to-End model design

Statistical Machine Translation vs 
Neural Machine Translation

Backtranslation is a way of making use of monolingual corpora in the target language by creating synthetic bitexts. In backtranslation, we train an intermediate target-to-source MT system on the small bitext to translate the monolingual target data to the source language. 
Backtranslation has various parameters. One is how we generate the backtranslated data; we can run the decoder in greedy inference, or use beam search. Or we can do sampling, or Monte Carlo search. In Monte Carlo decoding, at each search timestep, instead of always generating the word with the highest softmax probability, we roll a weighted die, and use it to choose the next word according to its softmax probability.

Backtranslation

Translations are evaluated along two dimensions:
1. adequacy: how well the translation captures the exact meaning of the source sentence. 
 2. fluency: how fluent the translation is in the target language.

The most accurate evaluations use human raters. An alternative is to do ranking: give the raters a pair of candidate translations, and ask them which one they prefer.
While humans produce the best evaluations of machine translation output, running a
human evaluation can be time consuming and expensive. For this reason automatic
metrics are often used. 


 MT Evaluation

Machine translation models are trained on a parallel corpus, sometimes called a bitext, a text that appears in two (or more) languages. Some examples of parallel corpora are:
- The Europarl Corpus, extracted from the proceedings of the European Parliament
- The United Nations Parallel Corpus, extracted from official records and other parliamentary documents of the United Nations
- The OpenSubtitles Corpus, extracted from movie and TV subtitles
- The ParaCrawl Corpus, extracted from general web text

MT Corpora

A company is developing an automated translation feature for its real-time customer support chat. They are testing two prototype systems on a Spanish sentence from a customer: 'Mi pedido no ha llegado y estoy muy enojado.' (My order has not arrived and I am very angry.)

- **System A** translates this to English as: 'The delivery of my order is delayed and I am experiencing frustration.'
- **System B** translates this to English as: 'My order not arrived and I am very angry.'

System A produces a grammatically perfect and formal sentence, but it softens the customer's emotional tone. System B's output is grammatically flawed but directly conveys the customer's message and strong emotion. For the specific goal of a customer support agent needing to quickly understand the customer's problem and emotional state, which system's translation is more effective? Justify your reasoning by comparing the two outputs.

Assessing Translation Effectiveness for a Specific Use Case

A company is developing a translation service for legal documents, where preserving the precise meaning and complex sentence structure of the original text is the highest priority. The company has access to a massive parallel corpus of legal texts. Given these requirements, which approach would be more suitable and why?

An automated system is tasked with translating the English idiom 'The early bird gets the worm' into Spanish. It produces two options:

*   **Translation A:** 'El pájaro temprano consigue el gusano.' (A word-for-word translation.)
*   **Translation B:** 'A quien madruga, Dios le ayuda.' (An idiomatic equivalent, meaning 'God helps those who rise early.')

Evaluate these two translations. Which one is a better translation and why? In your answer, analyze the strengths and weaknesses of each option in terms of how well it captures the original meaning and how natural it sounds in the target language.

Evaluating Machine Translation Quality

In sequence-to-sequence learning tasks like machine translation, models often must handle unaligned data, where the input and output sequences do not share a strict one-to-one correspondence. Unaligned data introduces two primary challenges: the input and output sequences may have entirely different lengths, and the corresponding regions of meaning may appear in a different order, as seen when translating sentences between languages with different grammatical structures.

Unaligned Data in Sequence Learning

Machine translation can use a scoring function for each possible translation and a heuristic search algorithm because the set of possible output sentences is too large. Optimization Verification can then diagnose whether an incorrect translation came from search failure or from the score computation.

University of Michigan - Ann Arbor

Google

Claude

- Information extraction
- Machine translation
- Discourse, dialogue and pragmatics
- Natural language generation
- Speech recognition
- Lexical semantics
- Phonology / morphology
- Language resources

Natural language processing in ACM Computing Classification

Sequence-to-sequence learning is a specific subset of sequence learning problems where both the inputs and the outputs consist of variable-length sequences. In these tasks, the length of the input sequence does not necessarily match the length of the output sequence. Prominent applications include machine translation and speech-to-text transcription.

Sequence-to-Sequence Learning

An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition
https://www.cs.colorado.edu/~martin/slp2.html

Speech and Language Processing (2nd edition)

Reference of Foundations of Large Language Models Course

Dive into Deep Learning

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

NLG, a subfield of artificial intelligence (AI), is a software process that automatically transforms data into plain-English content.

Natural language generation

Speech recognition

Phonology / Phonetics

Knowledge of the meaningful components of words; more specifically, the way words break down into components that carry meanings like singular vs. plural, I'm, can't, etc.

Morphology

Lexical semantics

Knowledge about linguistic units larger than a single utterance

Discourse

Knowledge of the relationship of meaning to the goals and intentions of the speaker.

Pragmatics

Information extraction

Machine Translation

Automatic speech recognition is a sequence-to-sequence learning task where the input sequence is an audio recording of a speaker and the output is a text transcript of the spoken words. A significant challenge in this domain is that there is no one-to-one correspondence between audio frames and text, as thousands of audio samples may correspond to a single word, making the input sequence much longer than the output sequence.

Automatic Speech Recognition

Text to speech, or speech synthesis, is the inverse problem of automatic speech recognition. It is a sequence-to-sequence learning task where the input is a sequence of text and the output is a generated audio file, meaning the resulting output sequence is significantly longer than the input sequence.

Text to Speech

The `Seq2Seq` class is a concrete implementation that combines a separately defined RNN encoder and decoder into a single, trainable sequence-to-sequence model. It inherits from the `EncoderDecoder` base class and is initialized with four arguments: an `encoder` instance, a `decoder` instance, the index of the target-language padding token (`tgt_pad`), and the learning rate (`lr`). All of these constructor arguments are saved as class attributes via `save_hyperparameters()`. The class provides two key methods beyond the inherited forward pass. First, a `validation_step` method computes the model's predictions on a validation batch by calling the forward pass on all inputs except the labels, and then plots the resulting loss with `train=False` to track generalization performance. Second, a `configure_optimizers` method returns the Adam optimizer, parameterized with the specified learning rate, as the optimization algorithm for updating all learnable parameters during training.

Seq2Seq Model Implementation

In sequence generation tasks, the ultimate objective is typically to find the single most likely output sequence, which can differ from a sequence formed by simply selecting the most likely token at each individual step. If a sequence-to-sequence decoder accurately reflects the underlying generative process, the most likely translation is the complete sequence that maximizes the product of conditional probabilities over all time steps: $$\prod_{t'=1}^{T'} P(y_{t'} \mid y_1, \ldots, y_{t'-1}, \mathbf{c})$$ where $$\mathbf{c}$$ is the context vector. This expression represents the global optimal sequence based on the model's learned probability distribution.

Learn Before

Related

Learn After