Limitation of Perplexity for Evaluating Long-Context LLMs
While perplexity is a straightforward metric for evaluating language models, it has a significant drawback when assessing long-context capabilities. Its application tends to primarily measure a model's performance on local context, failing to adequately capture its understanding and utilization of the broader, global context within a long sequence.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.2 Generative Models - Foundations of Large Language Models
Related
Limitation of Perplexity for Evaluating Long-Context LLMs
Synthetic Tasks for Long-Context LLM Evaluation
Real-World NLP Tasks for Long-Context LLM Evaluation
A research team develops a new method to evaluate a language model's ability to process documents that are thousands of pages long. Their process involves dividing each long document into individual paragraphs, asking a specific question about the content of each paragraph in isolation, and then calculating the average accuracy across all questions. The team argues that a high average score demonstrates the model's superior long-context capabilities. Which of the following best evaluates the team's conclusion?
Evaluating a Long-Context Model Upgrade
Evaluating a New Document Summarization Model
Learn After
A research team is comparing two language models on a task that involves reading a 50-page story and then answering a question about a detail mentioned in the first chapter. Model A is specifically designed to handle very long texts, while Model B is a powerful general-purpose model. The team observes that Model B achieves a slightly lower (better) perplexity score across the entire 50-page text than Model A. However, Model A consistently answers the final question correctly, while Model B fails. What is the most likely reason for this discrepancy?
Evaluating a Model Selection Strategy
Explaining Perplexity's Limitation in Long-Context Evaluation
You are evaluating two candidate long-context LLMs...
You lead evaluation for an internal eDiscovery ass...
Your team is writing an internal evaluation checkl...
Your team is selecting an LLM for an internal "pol...
Selecting a Long-Context LLM for a Cost-Constrained Enterprise Document Assistant
Choosing Long-Context Evaluation Evidence for a High-Volume Contract Review Feature
Designing an Evaluation Plan for a Long-Context Compliance Copilot Under Latency and Cost Constraints
Reconciling Long-Context Retrieval Quality with Inference Efficiency for a Meeting-Transcript Copilot
Evaluating a Long-Context LLM for Audit-Ready Evidence Retrieval Under Throughput Constraints
Diagnosing Conflicting Long-Context Evaluation Signals for an Internal Knowledge Assistant