Evaluating System Design for Code Generation
A software development company is designing a two-stage system to generate code snippets. The first stage generates 20 candidate snippets. The second stage uses a powerful but computationally expensive model to analyze these 20 candidates and select the single best one for the user. The team is debating two different approaches for the first stage:
- Approach A: Tune the generator to produce candidates that are highly likely to be syntactically correct and efficient, even if this results in all 20 candidates being very similar variations of the same core algorithm.
- Approach B: Tune the generator to produce a wide variety of distinct algorithms and coding styles, even if this means some candidates might contain minor errors or be less performant.
Which approach is more likely to result in a better final output after the second stage (reranking)? Justify your choice by evaluating the potential benefits and drawbacks of each approach in terms of the relationship between the initial candidates and the final selection.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A team is developing a system to generate marketing slogans. The process involves two stages: first, an initial model generates 50 potential slogans, and second, a highly accurate scoring model selects the single best slogan from that set to display. The team observes that while the final selected slogans are grammatically perfect and on-topic, they are often generic and uninspired. They also notice that the initial 50 slogans generated in each batch are usually very similar to one another. Which of the following strategies is the most sound for the team to adopt to improve the creativity of the final output?
Optimizing a Chatbot Response System
Evaluating System Design for Code Generation