Essay

Evaluating LLM Reasoning Outputs

An engineer provides a large language model with the following math word problem: 'A grocery store had 250 apples. They sold 120 on Monday and then received a new shipment of 85 apples on Tuesday. How many apples do they have now?'

Output 1 (from a prompt asking for only the final answer): 'The store has 455 apples.'

Output 2 (from a prompt asking for a step-by-step explanation before the final answer): 'Let's break this down.

  1. Start with 250 apples.
  2. Sell 120 apples: 250 - 120 = 130 apples.
  3. Receive 85 new apples: 130 + 85 = 215 apples. Therefore, the store has 215 apples.'

Critically evaluate the two outputs. Explain the likely reason for the significant difference in correctness by analyzing the process (or lack thereof) demonstrated in each. Based on your analysis, what general conclusion can you draw about eliciting more reliable solutions from language models for multi-step problems?

0

1

Updated 2025-10-06

Contributors are:

Who are from:

Tags

Ch.5 Inference - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Evaluation in Bloom's Taxonomy

Cognitive Psychology

Psychology

Social Science

Empirical Science

Science