1Cademy - Objective Function for Context Compression into Soft Prompts

Learn Before

Continuous Prompts (Soft Prompts)

Formula

Objective Function for Context Compression into Soft Prompts

The problem of approximating a long context with a continuous representation can be formalized as an optimization task. Given a user input $\mathbf{z}$ and its full context $\mathbf{c}$ , the goal is to learn a compressed representation $\sigma$ such that the model's prediction using $\sigma$ closely matches the prediction using $\mathbf{c}$ . This objective is expressed as $\hat{\sigma} = \argmin_{\sigma} s(\hat{\mathbf{y}},\hat{\mathbf{y}}_{\sigma})$ , where $\hat{\mathbf{y}} = \argmax_{\mathbf{y}} \Pr(\mathbf{y}|\mathbf{c},\mathbf{z})$ is the prediction with the full context, $\hat{\mathbf{y}}_{\sigma} = \argmax_{\mathbf{y}_{\sigma}} \Pr(\mathbf{y}|\sigma,\mathbf{z})$ is the prediction with the compressed context, and $s(\cdot,\cdot)$ is a loss or similarity measure.