The data used to train the model are known as parallel texts, or bitexts.
bitexts = source + </s> + target
, where source is the text being translated, target is the translation output, and </s> is the end-of-sentence token.

Application of autoregressive generation given a prefix: Machine translation

SMT
- Phrase-based statistical approach.
- Store several meanings of particular word in advance.
- When the user enters input value, it divides it into the word. or phrase unit.
- Find and translate the most statistically similar meaning.


NMT
- How human brain trains itself.
- Applies a large artificial neural network 
- Takes up less memory, less time
- NMT models fit a single model rather than a pipeline of fine-tuned models.
- Currently achieves comparably precise translation results.
- End-to-End model design

Statistical Machine Translation vs 
Neural Machine Translation

Backtranslation is a way of making use of monolingual corpora in the target language by creating synthetic bitexts. In backtranslation, we train an intermediate target-to-source MT system on the small bitext to translate the monolingual target data to the source language. 
Backtranslation has various parameters. One is how we generate the backtranslated data; we can run the decoder in greedy inference, or use beam search. Or we can do sampling, or Monte Carlo search. In Monte Carlo decoding, at each search timestep, instead of always generating the word with the highest softmax probability, we roll a weighted die, and use it to choose the next word according to its softmax probability.

Backtranslation

Translations are evaluated along two dimensions:
1. adequacy: how well the translation captures the exact meaning of the source sentence. 
 2. fluency: how fluent the translation is in the target language.

The most accurate evaluations use human raters. An alternative is to do ranking: give the raters a pair of candidate translations, and ask them which one they prefer.
While humans produce the best evaluations of machine translation output, running a
human evaluation can be time consuming and expensive. For this reason automatic
metrics are often used. 


 MT Evaluation

Machine translation models are trained on a parallel corpus, sometimes called a bitext, a text that appears in two (or more) languages. Some examples of parallel corpora are:
- The Europarl Corpus, extracted from the proceedings of the European Parliament
- The United Nations Parallel Corpus, extracted from official records and other parliamentary documents of the United Nations
- The OpenSubtitles Corpus, extracted from movie and TV subtitles
- The ParaCrawl Corpus, extracted from general web text

MT Corpora

A company is developing an automated translation feature for its real-time customer support chat. They are testing two prototype systems on a Spanish sentence from a customer: 'Mi pedido no ha llegado y estoy muy enojado.' (My order has not arrived and I am very angry.)

- **System A** translates this to English as: 'The delivery of my order is delayed and I am experiencing frustration.'
- **System B** translates this to English as: 'My order not arrived and I am very angry.'

System A produces a grammatically perfect and formal sentence, but it softens the customer's emotional tone. System B's output is grammatically flawed but directly conveys the customer's message and strong emotion. For the specific goal of a customer support agent needing to quickly understand the customer's problem and emotional state, which system's translation is more effective? Justify your reasoning by comparing the two outputs.

Assessing Translation Effectiveness for a Specific Use Case

A company is developing a translation service for legal documents, where preserving the precise meaning and complex sentence structure of the original text is the highest priority. The company has access to a massive parallel corpus of legal texts. Given these requirements, which approach would be more suitable and why?

An automated system is tasked with translating the English idiom 'The early bird gets the worm' into Spanish. It produces two options:

*   **Translation A:** 'El pájaro temprano consigue el gusano.' (A word-for-word translation.)
*   **Translation B:** 'A quien madruga, Dios le ayuda.' (An idiomatic equivalent, meaning 'God helps those who rise early.')

Evaluate these two translations. Which one is a better translation and why? In your answer, analyze the strengths and weaknesses of each option in terms of how well it captures the original meaning and how natural it sounds in the target language.

Evaluating Machine Translation Quality

In sequence-to-sequence learning tasks like machine translation, models often must handle unaligned data, where the input and output sequences do not share a strict one-to-one correspondence. Unaligned data introduces two primary challenges: the input and output sequences may have entirely different lengths, and the corresponding regions of meaning may appear in a different order, as seen when translating sentences between languages with different grammatical structures.

Unaligned Data in Sequence Learning

Machine translation is a sequence-to-sequence learning problem that involves ingesting a sentence in a source language and predicting its corresponding translation in a target language. A major challenge in this task is unaligned data, because the input and output sequences can have varying lengths and the corresponding words or regions may appear in completely different orders.

University of Michigan - Ann Arbor

Google

Claude

- Information extraction
- Machine translation
- Discourse, dialogue and pragmatics
- Natural language generation
- Speech recognition
- Lexical semantics
- Phonology / morphology
- Language resources

Natural language processing in ACM Computing Classification

Sequence-to-sequence learning is a specific subset of sequence learning problems where both the inputs and the outputs consist of variable-length sequences. In these tasks, the length of the input sequence does not necessarily match the length of the output sequence. Prominent applications include machine translation and speech-to-text transcription.

Sequence-to-Sequence Learning

An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition
https://www.cs.colorado.edu/~martin/slp2.html

Speech and Language Processing (2nd edition)

Reference of Foundations of Large Language Models Course

Dive into Deep Learning

NLG, a subfield of artificial intelligence (AI), is a software process that automatically transforms data into plain-English content.

Natural language generation

Speech recognition

Phonology / Phonetics

Knowledge of the meaningful components of words; more specifically, the way words break down into components that carry meanings like singular vs. plural, I'm, can't, etc.

Morphology

Lexical semantics

Knowledge about linguistic units larger than a single utterance

Discourse

Knowledge of the relationship of meaning to the goals and intentions of the speaker.

Pragmatics

Information extraction

Machine Translation

Automatic speech recognition is a sequence-to-sequence learning task where the input sequence is an audio recording of a speaker and the output is a text transcript of the spoken words. A significant challenge in this domain is that there is no one-to-one correspondence between audio frames and text, as thousands of audio samples may correspond to a single word, making the input sequence much longer than the output sequence.

Automatic Speech Recognition

Text to speech, or speech synthesis, is the inverse problem of automatic speech recognition. It is a sequence-to-sequence learning task where the input is a sequence of text and the output is a generated audio file, meaning the resulting output sequence is significantly longer than the input sequence.

Learn Before

Related

Learn After