Learn Before
Shannon's Foundational Work on Language Modeling
The concept of probabilistic language modeling dates back to early experiments conducted by Claude Shannon in 1951. In his work, a language model was designed to estimate the predictability of English by determining how well the next letter of a text can be predicted when the preceding letters are known. Although these experiments were preliminary, the fundamental goals and methods of language modeling have remained largely unchanged over the decades since then.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Types of Language Models
Evaluating language models
Shannon's Foundational Work on Language Modeling
Generalization of the Language Modeling Concept
Chain Rule for Sequence Probability
Deep Learning Approach to Language Modeling
Output Token Sequence in LLMs
Start of Sentence (SOS) Token
[CLS] Token as a Start Symbol
A system is designed to predict the probability of a sequence of words. For the sequence 'The dog ran', the system provides the following conditional probabilities:
- The probability of 'The' occurring at the start of a sequence is 0.2.
- The probability of 'dog' occurring after 'The' is 0.3.
- The probability of 'ran' occurring after 'The dog' is 0.7.
Based on the fundamental principle used by such systems to determine the likelihood of a full sequence, what is the overall probability of the sequence 'The dog ran'?
Analyzing Language Model Probability Assignments
A system's primary goal is to predict the probability of a sequence of tokens. To calculate the total probability for the sequence 'The quick brown fox', it breaks the problem down into a series of conditional probability calculations. Arrange the following calculations in the correct order that the system would use to find the total probability of the sequence.
Evaluating a Language Model's Probabilistic Output
Learn After
An early experiment in language modeling involved a human subject guessing the next letter in a sentence, given the preceding letters. The experimenter recorded how many guesses were needed for each letter. What fundamental principle about language was this experiment designed to investigate?
The Significance of Predictability in Language
An early experiment designed to estimate the predictability of a language involved a human subject guessing the next letter in a text, given the preceding letters. The number of guesses needed for each letter was recorded. What was the primary goal of measuring the number of guesses?
N-gram Language Modeling