Representation-based Repetition Penalty
A representation-based repetition penalty is a technique to discourage repetitive text by operating on the model's internal representations. This method calculates a penalty based on the maximum distance between the representation of the token being predicted and the representations of tokens that have already been generated. By penalizing tokens whose representations are too close to previous ones, the search objective is guided away from producing degenerate, repetitive outputs.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
Representation-based Repetition Penalty
A language model is tasked with writing a short story and produces the following output: 'The knight rode his horse through the dark forest. The forest was very dark. The knight was a brave knight, and the dark forest did not scare him.' Which of the following adjustments to the generation process would be most effective at discouraging this kind of repetitive phrasing?
Tuning Text Generation for Poetry
Consequences of Over-Tuning a Repetition Penalty
Representation-based Repetition Penalty
A developer wants to ensure a language model generates multi-paragraph text that maintains a consistent theme, penalizing outputs that start on one topic and then drift into an unrelated one. Why is a penalty function that assesses the model's internal hidden states generally more effective for this specific task than a function that only evaluates the final, complete text?
Designing a Penalty Function for Safe AI
A researcher aims to guide a language model to generate text with a consistently positive sentiment, penalizing it the moment its internal thought process begins to drift towards negativity, even before negative words are explicitly written. Which approach to designing a penalty function is most suitable for this real-time, internal-state intervention?
Learn After
Diagnosing Semantic Repetition
Diagnosing Semantic Repetition
A language model is configured with a penalty mechanism to reduce repetitive output. During generation, it produces the phrase '...the large canine...' and is now considering the word 'dog' for a subsequent part of the sentence. Which of the following statements best analyzes how a representation-based repetition penalty would function in this specific situation, as compared to a penalty based solely on token frequency?