Investigating a Production Scoring Bug: Softmax Normalization vs. Autoregressive Sequence Log-Probability
You are reviewing a production LLM feature that ranks two candidate continuations for the same user prompt x by computing a score for each full continuation y and choosing the higher-scoring one. The intended scoring rule is to choose (\hat{\mathbf{y}} = \arg\max_{\mathbf{y}} \log \Pr(\mathbf{y}\mid\mathbf{x})), using the autoregressive decomposition (\log \Pr(\mathbf{y}\mid\mathbf{x}) = \sum_{i=1}^{n} \log \Pr(y_i\mid\mathbf{x}, \mathbf{y}_{<i})). The model produces next-token logits at each step, which should be converted to probabilities with softmax over the full vocabulary.
A teammate implemented the scorer as follows: at each step i, they take the logit for the candidate’s next token (u_{y_i}) and subtract (\log\big(\sum_{t \in S_i} \exp(u_t)\big)), where (S_i) is a small, request-specific shortlist of 50 tokens (not the full vocabulary). They then sum these per-token values across the continuation to get the sequence score.
In an A/B test, the system starts preferring verbose, low-quality continuations that repeat common tokens, even when the model’s raw logits for the “good” continuation look higher at several positions.
As the on-call ML engineer, analyze whether the teammate’s scoring method is mathematically consistent with maximizing (\log \Pr(\mathbf{y}\mid\mathbf{x})) under an autoregressive language model. In your answer, explain (1) how softmax normalization affects the conditional next-token probability (\Pr(y_i\mid\mathbf{x},\mathbf{y}_{<i})), (2) why using a changing shortlist (S_i) can distort the summed log-likelihood comparison between two full sequences, and (3) what concrete change you would make to the scoring computation to correctly rank candidates by (\log \Pr(\mathbf{y}\mid\mathbf{x})).
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.1 Pre-training - Foundations of Large Language Models
Ch.5 Inference - Foundations of Large Language Models
Data Science
Related
Conditional Probability in Sequence-to-Sequence Generation
Next-Token Probability Calculation in Autoregressive Decoders
Example of Autoregressive Generation and Log-Probability Calculation
An auto-regressive language model is generating text following the input 'The cat sat on the'. The model's objective is to find the output sequence with the highest total log-probability. It is considering two possible two-word continuations:
Path A: 'warm mat'
- log Pr('warm' | 'The cat sat on the') = -0.9
- log Pr('mat' | 'The cat sat on the warm') = -1.5
Path B: 'plush rug'
- log Pr('plush' | 'The cat sat on the') = -1.2
- log Pr('rug' | 'The cat sat on the plush') = -1.1
Based on the provided conditional log-probabilities, which path will the model choose and why?
Debugging a Generation Model's Choice
Greedy Decoding vs. Optimal Sequence Probability
Reconciling Training Log-Likelihood with Inference-Time Sequence Selection
Diagnosing a “High-Confidence Wrong Token” Bug in Autoregressive Scoring
Explaining a Counterintuitive Decoding Outcome Using Softmax, Next-Token Conditionals, and Sequence Log-Probability
Auditing a Candidate Completion Using Softmax Next-Token Probabilities and Autoregressive Log-Probability
Investigating a Production Scoring Bug: Softmax Normalization vs. Autoregressive Sequence Log-Probability
Root-Cause Analysis: Why a “More Likely” Token-by-Token Completion Loses on Total Sequence Score
Design a Correct Sequence-Scoring Function for Autoregressive LLM Outputs
Your team is building an internal tool that ranks ...
You’re reviewing an internal evaluation script tha...
You’re reviewing an internal LLM evaluation pipeli...
Direct Computation of Output Sequence Log-Probability in LLMs
Mathematical Justification for Greedy Search
A language model needs to compute the total log-probability for generating the specific three-token sequence
y = (y_1, y_2, y_3)given an inputx. Based on the standard autoregressive formulation, which of the following expressions correctly represents this calculation?Calculating Sequence Log-Probability
Analysis of Text Generation Approaches
You’re reviewing an internal evaluation script tha...
Your team is building an internal tool that ranks ...
You’re reviewing an internal LLM evaluation pipeli...
Reconciling Training Log-Likelihood with Inference-Time Sequence Selection
Explaining a Counterintuitive Decoding Outcome Using Softmax, Next-Token Conditionals, and Sequence Log-Probability
Diagnosing a “High-Confidence Wrong Token” Bug in Autoregressive Scoring
Investigating a Production Scoring Bug: Softmax Normalization vs. Autoregressive Sequence Log-Probability
Design a Correct Sequence-Scoring Function for Autoregressive LLM Outputs
Root-Cause Analysis: Why a “More Likely” Token-by-Token Completion Loses on Total Sequence Score
Auditing a Candidate Completion Using Softmax Next-Token Probabilities and Autoregressive Log-Probability
Schematic of Probability Calculation in Causal Language Modeling
An autoregressive language model is given the sequence of tokens: 'The', 'cat', 'sat', 'on', 'the'. It is now tasked with predicting the very next token. Which of the following expressions correctly represents the primary calculation the model performs to determine the likelihood of the word 'mat' appearing next?
Contextual Influence on Token Probability
Analyzing Contextual Influence on Next-Token Probability
You’re reviewing an internal evaluation script tha...
Your team is building an internal tool that ranks ...
You’re reviewing an internal LLM evaluation pipeli...
Reconciling Training Log-Likelihood with Inference-Time Sequence Selection
Explaining a Counterintuitive Decoding Outcome Using Softmax, Next-Token Conditionals, and Sequence Log-Probability
Diagnosing a “High-Confidence Wrong Token” Bug in Autoregressive Scoring
Investigating a Production Scoring Bug: Softmax Normalization vs. Autoregressive Sequence Log-Probability
Design a Correct Sequence-Scoring Function for Autoregressive LLM Outputs
Root-Cause Analysis: Why a “More Likely” Token-by-Token Completion Loses on Total Sequence Score
Auditing a Candidate Completion Using Softmax Next-Token Probabilities and Autoregressive Log-Probability
Neural Network-Based Next-Token Probability Distribution
Initial Token Probability Assumption
Calculating Sequence Log-Likelihood
A language model is being trained on the sentence '
The cat sat'. The model calculates the following conditional log-probabilities at each step, where ' ' is a fixed start-of-sequence token: log P('The' | '<BOS>') = -1.5log P('cat' | '<BOS>', 'The') = -0.9log P('sat' | '<BOS>', 'The', 'cat') = -1.2
Based on the standard training objective for this single sequence, what is the total log-likelihood value that the model aims to maximize?
Model Output Evaluation
You’re reviewing an internal evaluation script tha...
Your team is building an internal tool that ranks ...
You’re reviewing an internal LLM evaluation pipeli...
Reconciling Training Log-Likelihood with Inference-Time Sequence Selection
Explaining a Counterintuitive Decoding Outcome Using Softmax, Next-Token Conditionals, and Sequence Log-Probability
Diagnosing a “High-Confidence Wrong Token” Bug in Autoregressive Scoring
Investigating a Production Scoring Bug: Softmax Normalization vs. Autoregressive Sequence Log-Probability
Design a Correct Sequence-Scoring Function for Autoregressive LLM Outputs
Root-Cause Analysis: Why a “More Likely” Token-by-Token Completion Loses on Total Sequence Score
Auditing a Candidate Completion Using Softmax Next-Token Probabilities and Autoregressive Log-Probability
Pros and Cons of Softmax Function
Softmax Regression (Activation)
Parameterized Softmax Layer
Plackett-Luce Selection Probability Formula
Conditional Probability Formula for Autoregressive Models using Softmax
A neural network's final layer produces the raw output scores (logits)
[2.0, 1.0, 0.1]for three possible classes. To convert these scores into class probabilities, a function is applied that first exponentiates each score and then normalizes these new values by dividing each by their sum. What is the resulting probability distribution? (Values are rounded to three decimal places).A function is used to convert a vector of raw, unnormalized scores
z = [z_1, z_2, ..., z_K]into a probability distribution. This function operates by first applying the standard exponential function to each score and then normalizing these new values by dividing each by their sum. If a constant valueCis added to every score in the input vectorz, resulting in a new vectorz' = [z_1+C, z_2+C, ..., z_K+C], how will the resulting output probability distribution be affected?Consider two input vectors of raw scores (logits) for a 3-class classification problem: Vector A =
[1, 2, 3]and Vector B =[1, 5, 10]. Both vectors are passed through a function that exponentiates each score and then normalizes the results by dividing by their sum. How will the resulting probability distribution for Vector B compare to the one for Vector A?You’re reviewing an internal evaluation script tha...
Your team is building an internal tool that ranks ...
You’re reviewing an internal LLM evaluation pipeli...
Reconciling Training Log-Likelihood with Inference-Time Sequence Selection
Explaining a Counterintuitive Decoding Outcome Using Softmax, Next-Token Conditionals, and Sequence Log-Probability
Diagnosing a “High-Confidence Wrong Token” Bug in Autoregressive Scoring
Investigating a Production Scoring Bug: Softmax Normalization vs. Autoregressive Sequence Log-Probability
Design a Correct Sequence-Scoring Function for Autoregressive LLM Outputs
Root-Cause Analysis: Why a “More Likely” Token-by-Token Completion Loses on Total Sequence Score
Auditing a Candidate Completion Using Softmax Next-Token Probabilities and Autoregressive Log-Probability
Derivative of Softmax Cross-Entropy Loss with Respect to Logits
Numerical Overflow in Softmax Function