Google

The critique-refine cycle is the iterative loop at the heart of sequential scaling. It consists of two main phases. First, in the 'critique' stage, a verifier evaluates the current solution and produces feedback. Second, in the 'refine' stage, the LLM is prompted with the original problem, the existing solution, and the feedback to generate a new, potentially better solution. This iterative process allows for the progressive enhancement of the solution.

Critique-Refine Cycle in Sequential Scaling

In the critique stage of an iterative refinement process, a verifier provides feedback to guide the next generation step. This feedback can take various forms, including textual critiques that identify errors or suggest improvements, numerical scores that rate the solution's quality, or a revised plan or intermediate step for the model to follow.

Feedback Mechanisms in the Critique Stage

The critique-refine cycle can be mathematically expressed as: $$\mathbf{y}_{k+1} = \mathrm{Refine}(\mathbf{x}, \mathbf{y}_k, \mathrm{Feedback}(\mathbf{y}_k))$$. In this formula, $$\mathbf{y}_{k+1}$$ represents the improved solution. It is generated by the $$\mathrm{Refine}(\cdot)$$ function, which prompts a Large Language Model with three inputs: the original problem $$\mathbf{x}$$, the previous solution $$\mathbf{y}_k$$, and the feedback on that solution, denoted as $$\mathrm{Feedback}(\mathbf{y}_k)$$. This feedback is provided by a verifier.

Formula for the Critique-Refine Cycle

The critique-refine cycle is an iterative process that can be repeated for a predetermined number of steps, denoted as K, or it can continue until the generated solution meets a specific stopping criterion.

Termination Conditions for the Critique-Refine Cycle

An AI system is tasked with generating a Python function to calculate the factorial of a number. It produces an initial version of the code. A verifier then analyzes this code and provides the following feedback: 'The function fails for an input of 0.' To continue the iterative improvement process, what is the most effective next action?

A developer designs a system to iteratively improve a summary of a long document. In each step, a verifier provides feedback on the current summary. The system then prompts the language model with the *original document* and the *verifier's feedback* to generate a new summary. Explain why this approach is likely to be less effective than the standard iterative refinement process.

Evaluating an Iterative Refinement Process

In the critique stage of sequential scaling, the verifier can provide several types of feedback to guide the refinement process. This feedback can be qualitative, such as textual critiques that pinpoint specific errors or suggest improvements. It can also be quantitative, in the form of numerical scores that reflect the overall quality of the solution. Additionally, the feedback can be directive, providing a revised plan or a new intermediate step for the next generation cycle.

Forms of Verifier Feedback in Sequential Scaling

An AI system is engaged in an iterative process to generate a recipe for a vegan chocolate cake. Below are different elements from one cycle of this process. Match each element to its corresponding role within the improvement cycle.

Learn Before

Related