Learn Before
Critique of a Masking Implementation
A developer implements a text corruption process for a language model. Instead of selecting positions randomly from the input sequence, their algorithm always replaces the final token of every sequence with a special [MASK] symbol. Based on the formal definition of the masking process, identify the key component that is incorrectly implemented and explain why this deviation is problematic for the model's learning objective.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Example of Masked Language Modeling with Single and Multiple Masks
In a masked modeling approach, an input sequence
xis transformed into a modified sequencex̄by replacing tokens at a randomly selected set of positionsA(x)with a special[MASK]symbol. Given an input sequencex = (T1, T2, T3, T4, T5, T6)and a set of selected positionsA(x) = {2, 5}(using 0-based indexing), what is the resulting modified sequencex̄?Critique of a Masking Implementation
In the formal definition of the masking process used in language models, several components are used to describe the transformation of an input sequence. Match each symbolic component with its correct description.