Describe the defining characteristic of computational tasks that involve generating text from a very long input sequence. Then, provide a novel example of such a task that is different from document summarization or code analysis.

Google

This category of long sequence modeling involves tasks where a language model produces text based on an input context that is an extensive sequence. In terms of the text generation probability notation $$\Pr(\mathbf{y}|\mathbf{x})$$, this corresponds to scenarios where the context $$\mathbf{x}$$ is a long sequence, while the generated text $$\mathbf{y}$$ is typically shorter.

Text Generation Based on Long Context

An illustrative example of text generation from a long context is the task of creating a short summary from a very long document. In this scenario, the extensive document serves as the input context, from which the model generates a concise summary.

Example of Long-Context Text Generation: Summarization

A practical application for a long-context LLM involves analyzing a vast C++ source file, potentially containing tens of thousands of lines, to generate a concise summary of the program's overall functionality.

Example of Long-Context Task: Code Functionality Outlining

A research team is evaluating different computational models for their ability to handle various language tasks. Which of the following tasks is the best example of a problem where a model must generate new text based on an input that is an exceptionally long sequence?

Consider the following business intelligence task. Based on the relationship between the information provided to a computational model and the desired output, identify the 'extensive input context' and the 'generated text', and explain your reasoning.

Learn Before

Related