Learn Before
Evaluating Language Model Design Choices
Two teams are tasked with creating a language model for a mobile application that translates spoken phrases in real-time on a standard smartphone. The primary constraint is that the model must be small and fast enough to run directly on the device without a constant internet connection.
- Team A decides to build a model with a very large vocabulary and a deep, multi-layered architecture. Their goal is to create the most powerful and accurate model possible, capable of understanding a vast range of linguistic nuances.
- Team B opts for a model with a smaller, more focused vocabulary and a shallower, less complex architecture. Their goal is to prioritize computational efficiency and a small memory footprint.
Which team's design philosophy is better suited for the project's requirements? Justify your evaluation by analyzing the trade-offs each team is making.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Embedding Size in Transformer Models
Evaluating Language Model Design Choices
A research team is tasked with building a language model to analyze a large collection of specialized legal contracts. These documents contain a unique vocabulary and sentence structure not commonly found in general web text. When deciding how to approach this task, which of the following considerations is the most critical to address first to ensure the model's effectiveness?
Trade-offs in Language Model Vocabulary Design
Hidden Size in Transformer Models
Number of Attention Heads
FFN Hidden Size in Transformers
Model Depth in Transformers
Vocabulary Size in Transformers