Learn Before
Scaling Up via Long Sequence Adaptation
While large-scale training typically involves increasing data and computational resources, scaling up Large Language Models can also occur by adapting them to process significantly longer sequences. For example, a model might be pre-trained on extensive texts of standard length and subsequently applied to handle very long token sequences that greatly exceed the lengths encountered during its pre-training phase.
0
1
Tags
Foundations of Large Language Models
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Popular Methods for Adapting Pre-trained LLMs to Long Sequences
Strengths and Limitations of Long-Sequence Models
Pre-training and Fine-tuning Strategy for Long-Context Adaptation
Length Extrapolation in LLMs
Fine-Tuning for Architectural Adaptation in LLMs
A startup with limited computational resources and a tight deadline needs to build a system that can summarize lengthy legal documents. They have access to a powerful, general-purpose language model that was pre-trained on a massive dataset but primarily on shorter texts. Given their constraints, which of the following strategies is the most logical and efficient for them to pursue?
The primary reason for adapting existing pre-trained language models for long sequences, rather than training new models from scratch, is that pre-trained models inherently possess superior architectural designs for handling extended contexts.
Evaluating Model Development Strategies for Long-Text Analysis
Scaling Up via Long Sequence Adaptation
Fine-Tuning Pre-trained LLMs with Advanced Positional Embeddings