Consider a model generating the sentence 'The cat sat on the mat.' After generating the first three words ('The', 'cat', 'sat'), the model uses a function, `s(ȳ₁, ȳ₂, ȳ₃)`, to compute a summary based on these outputs. In this specific scenario, what does this summary represent, and how is it used by the model to determine the *next* word in the sequence?

Google

The formula $s(\bar{\mathbf{y}}_1 \dots \bar{\mathbf{y}}_{k-1})$ represents a function, denoted by $s$, that computes a state or context based on a sequence of the previous $k-1$ outputs. Each $\bar{\mathbf{y}}_i$ is a vector representing the $i$-th output in the sequence. This is a fundamental concept in autoregressive models where the prediction of the next element depends on the context derived from the preceding elements.

State Function from Previous Outputs

Based on the provided scenario, which model (Alpha or Beta) would be better suited for generating a sequence where each new element must be consistent with the overall context of the *entire* preceding sequence? Justify your reasoning by analyzing the information captured by each model's state function.

Analyzing State Function Impact on Sequence Generation

In a model that generates a sequence of items one by one, a function `s(ȳ₁, ..., ȳₖ₋₁)` is used to compute a summary, or 'state', from the `k-1` items that have already been generated. What is the primary purpose of this computed state?

Learn Before

Related