Google

A hard prompt, such as the instruction 'Translate the sentence into Chinese', can be seen as a discrete token sequence like $$c_1...c_5$$. When fed into a Large Language Model, these tokens are transformed into a sequence of real-valued vectors, such as $$\mathbf{h}_1...\mathbf{h}_5$$. These intermediate hidden states within the model's embedding space can be conceptually viewed as a soft prompt, illustrating the fundamental difference between human-readable text and the continuous representations used internally by the model.

Visual Representation of Hard vs. Soft Prompts

An AI researcher is examining two different methods for guiding a language model. Method 1 involves prepending the text 'Summarize the following article:' to the input. Method 2 involves inserting a sequence of trainable, continuous numerical vectors directly into the model's embedding space, which do not correspond to any actual words. Which statement correctly analyzes the representation of these two methods?

Examine the diagram below, which illustrates two different types of inputs to a language model. Identify which input (A or B) represents a soft prompt and explain your reasoning based on how it is visually represented.

**Input A:** `[Translate this sentence:]` → [Language Model]

**Input B:** `[h₁, h₂, h₃, ...]` → [Language Model]

Analysis of Prompt Representations

A language model's input can be guided in different ways. Analyze the following descriptions of how these guides are represented and match each one to the correct term.

Learn Before

Related