Learn Before
Analyzing a Model's Command Interpretation Failure
A language model is trained to convert simple commands into action sequences. During its training, it correctly processes commands such as 'jump', 'run', 'jump twice', and 'run thrice'. When tested on the new command 'run twice', a command it has never seen before, it produces the incorrect sequence: 'RUN, JUMP, JUMP'. Based on this information, analyze the most probable cause of this specific failure.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
SCAN Tasks for Evaluating Compositional Generalization
Analyzing a Model's Command Interpretation Failure
A language model is trained on a dataset of simple commands. It successfully learns to execute individual actions like 'walk', 'run', and 'jump'. It also learns to apply the modifier 'twice' to the command 'run', correctly executing 'run twice'. However, when presented with the novel command 'jump twice', the model fails to produce the correct sequence of actions. This failure demonstrates a specific weakness in the model's ability for:
Evaluating Evidence of Generalization
Analyzing Model Performance on Novel Instructions