Google

Chain-of-thought (COT) prompting has proven to be a highly effective technique for improving performance on challenging reasoning tasks, as demonstrated by its successful application on the GSM8K mathematical reasoning benchmark.

Application of COT Prompting on GSM8K Benchmark

A large language model is given the following math word problem: 'A farmer has 15 apples. He sells 7 of them and then buys 5 more. How many apples does the farmer have now?' Which of the following outputs best exemplifies the result of applying a chain-of-thought prompting technique to solve this problem?

Based on the challenges described in the case study, what prompting technique should the team apply to both improve the model's accuracy and make its problem-solving process more transparent? Justify your choice.

Improving LLM Performance on a Reasoning Benchmark

A research team is evaluating a large language model's mathematical reasoning abilities using a benchmark composed of multi-step grade school word problems. They observe that when the model is prompted to provide only the final numerical answer, its accuracy is low. However, when they modify the prompt to instruct the model to first outline the sequence of calculations and logical steps it will take before providing the final answer, the model's accuracy on the benchmark improves dramatically. Analyze the underlying reasons for this significant performance improvement. In your response, break down the relationship between the structure of the problems in the benchmark and the process the model is guided to follow.

Learn Before

Related