Google

A simple method to score a reasoning path is to count the number of steps classified as 'correct'. This can be represented by the formula: 

$$r(\mathbf{x}, \mathbf{y}) = \sum_{k=1}^{n_s} \delta(\text{correct}, C(\mathbf{x}, \bar{\mathbf{y}}_{\le k}))$$

where:
- $$r(\mathbf{x}, \mathbf{y})$$ is the total reward for the reasoning path $$\mathbf{y}$$ given the input $$\mathbf{x}$$.
- $$n_s$$ is the total number of steps in the reasoning path.
- $$C(\mathbf{x}, \bar{\mathbf{y}}_{\le k})$$ is the classification output for step $$k$$, determined by selecting the label with the maximum probability.
- $$\delta(a, b)$$ is the Kronecker delta function, which is $${}1$$ if $$a=b$$ and $${}0$$ otherwise. In this context, it equals $${}1$$ if the classification for step $$k$$ is 'correct'.

Formula for Scoring Reasoning Paths by Counting Correct Steps

A language model generates a three-step reasoning path to solve a problem. A separate classification model evaluates each step, assigning a probability to potential labels. The final classification for a step, denoted as $C(\mathbf{x}, \bar{\mathbf{y}}_{\leq k})$, is the label with the highest probability. The overall score for the path is calculated by summing up the number of steps classified as 'correct', based on the formula: $r(\mathbf{x}, \mathbf{y}) = \sum_{k=1}^{n_s} \delta(correct, C(\mathbf{x}, \bar{\mathbf{y}}_{\leq k}))$, where $\delta$ is 1 if its two arguments are identical and 0 otherwise. Given the classifier's probability outputs below, what is the final score for the entire reasoning path?

Calculating Reasoning Path Score

A proposed method for scoring a reasoning path is given by the formula: $r(\mathbf{x}, \mathbf{y}) = \sum_{k=1}^{n_s} \delta(correct, C(\mathbf{x}, \bar{\mathbf{y}}_{\leq k}))$, where the final score is the total count of steps classified as 'correct'. Critically evaluate this scoring method. In your response, identify and explain one significant strength and one significant weakness of this approach for assessing the overall quality of a reasoning process.

Critique of a Reasoning Path Scoring Method

Consider two reasoning paths, Path A and Path B, generated to solve the same problem. Path A consists of 3 steps, all of which are classified as 'correct'. Path B consists of 5 steps, where 4 are classified as 'correct' and 1 is 'incorrect'. According to the scoring formula $r(\mathbf{x}, \mathbf{y}) = \sum_{k=1}^{n_s} \delta(correct, C(\mathbf{x}, \bar{\mathbf{y}}_{\leq k}))$, which of the following statements is true?

Learn Before

Related