Limitation of Perplexity for Evaluating Long-Context LLMs

Synthetic Tasks for Long-Context LLM Evaluation

Real-World NLP Tasks for Long-Context LLM Evaluation

A research team develops a new method to evaluate a language model's ability to process documents that are thousands of pages long. Their process involves dividing each long document into individual paragraphs, asking a specific question about the content of each paragraph in isolation, and then calculating the average accuracy across all questions. The team argues that a high average score demonstrates the model's superior long-context capabilities. Which of the following best evaluates the team's conclusion?

Evaluating a Long-Context Model Upgrade

Evaluating a New Document Summarization Model