Learn Before
Ranked Sequence Log-Probability Calculation
A language model is tasked with ranking a set of three candidate responses {R1, R2, R3} for a given prompt. The model has computed the following conditional log-probabilities for selecting the best response from a given set of available options:
-
log Pr(R1 | {R1, R2, R3}) = -0.6 -
log Pr(R2 | {R1, R2, R3}) = -1.1 -
log Pr(R3 | {R1, R2, R3}) = -1.5 -
log Pr(R2 | {R2, R3}) = -0.4 -
log Pr(R3 | {R2, R3}) = -1.3 -
log Pr(R1 | {R1, R3}) = -0.2 -
log Pr(R3 | {R1, R3}) = -1.8 -
log Pr(R1 | {R1, R2}) = -0.8 -
log Pr(R2 | {R1, R2}) = -0.9
Note that the log-probability of selecting the single remaining item from a set of one is always 0 (e.g., log Pr(R3 | {R3}) = 0).
Based on these values, calculate the total log-probability for the specific ranked sequence R1 > R3 > R2.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Set of Remaining Items in a Ranked Sequence
Plackett-Luce Loss Function
A model is designed to rank a set of three documents {Doc A, Doc B, Doc C} for a given user query. To calculate the log-probability of the specific ranked sequence 'Doc A > Doc B > Doc C', a developer proposes calculating the total log-probability as the sum of the log-probabilities of each document being chosen first from the full set of three documents. Why is this approach fundamentally flawed for modeling a sequential ranking process?
Ranked Sequence Log-Probability Calculation
Calculating Log-Probability for a Ranked List