Learn Before
[CLS] Token as a Start Symbol
The [CLS] token, a convention established by BERT models, can be used as the special start symbol for a sequence in language modeling. This serves as a concrete example of an initial token used to begin the probability calculation for a sequence.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
Types of Language Models
Evaluating language models
Shannon's Foundational Work on Language Modeling
Generalization of the Language Modeling Concept
Chain Rule for Sequence Probability
Deep Learning Approach to Language Modeling
Output Token Sequence in LLMs
Start of Sentence (SOS) Token
[CLS] Token as a Start Symbol
A system is designed to predict the probability of a sequence of words. For the sequence 'The dog ran', the system provides the following conditional probabilities:
- The probability of 'The' occurring at the start of a sequence is 0.2.
- The probability of 'dog' occurring after 'The' is 0.3.
- The probability of 'ran' occurring after 'The dog' is 0.7.
Based on the fundamental principle used by such systems to determine the likelihood of a full sequence, what is the overall probability of the sequence 'The dog ran'?
Analyzing Language Model Probability Assignments
A system's primary goal is to predict the probability of a sequence of tokens. To calculate the total probability for the sequence 'The quick brown fox', it breaks the problem down into a series of conditional probability calculations. Arrange the following calculations in the correct order that the system would use to find the total probability of the sequence.
Evaluating a Language Model's Probabilistic Output
Learn After
A researcher is preparing the sentence 'Language models are powerful.' for input into a specific type of model. This model architecture requires a special token, denoted as
[CLS], to be placed at the very beginning of every input sequence to mark its start. Which of the following options correctly formats the sentence for this model?The
[CLS]token, when used in a language model, is exclusively for aggregating sequence information for downstream classification tasks and does not serve as the initial token required to start the sequence's probability calculation.The Role of the Initial Token in Sequence Probability