Google

A scoring function problem occurs when the scoring computation fails to give the correct output a strictly higher score than the incorrect system output. In this case, the useful work is to improve how the system learns or approximates the score.

Scoring Function Problem in Inference

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

In the speech recognition example, what condition indicates a scoring function problem?

True or False: A scoring function problem means the optimization algorithm needs improvement.

A scoring function problem occurs when the correct output does not receive a _____ score than the incorrect output.

Match each score comparison outcome to its diagnostic conclusion.

Order the steps for diagnosing and responding to a scoring function problem.

Question:
Explain why the Optimization Verification test specifically requires the correct output's score to be strictly higher than the incorrect output's score, rather than merely equal or higher. Discuss what happens diagnostically if the scores are equal.

Sample answer:
The test requires Score_A(S*) to be strictly higher than Score_A(S_out) because if the scores are equal, the scoring function has not actually distinguished the correct output as preferable; a tie or lower score means the correct output was not being favored by the scoring computation. In the case Score_A(S*) ≤ Score_A(S_out), which includes equality, the scoring function has failed to give the correct output a strictly higher score than the incorrect output, so the useful work is to improve how the system learns or approximates the score.

Key points:
- The correct output must score strictly higher, not merely equal, to the incorrect output
- Score_A(S*) ≤ Score_A(S_out) includes the equality case as a failure condition
- Equal scores mean the scoring function has not favored the correct output
- This diagnosis directs effort toward improving how the score is learned or approximated

Rubric:
Full credit explains that equality does not satisfy the requirement of a strictly higher score and correctly places equality within the failure condition (Score_A(S*) ≤ Score_A(S_out)); partial credit identifies the strict inequality requirement without addressing the equality case; no credit for answers that ignore the strict inequality condition.

Explain why a strictly higher score, not just a higher score, matters for diagnosing scoring function problems.

Case context:
A speech recognition system receives audio that should be transcribed as 'I love machine learning.' The correct transcription is S*. The system instead outputs S_out, a different sentence. An engineer computes Score_A(S*) and Score_A(S_out) and finds that Score_A(S*) is less than Score_A(S_out), meaning the incorrect output received a higher score than the correct transcription.

Question:
Using the Optimization Verification test, what should the engineer conclude about the source of the error, and what should they do next?

Sample answer:
Since Score_A(S*) ≤ Score_A(S_out), the test indicates a scoring function problem: the way Score_A(.) is computed is failing to give the correct transcription a strictly higher score than the incorrect output. The engineer should not spend time on the optimization algorithm's search process; instead, the useful work is to improve how the system learns or approximates Score_A(S) for different sentences S, for example by improving the model or estimates that produce the score.

Key points:
- Score_A(S*) ≤ Score_A(S_out) signals a scoring function problem
- The scoring computation fails to rank the correct transcription above the incorrect one
- The optimization algorithm is not the source of this error
- Next step is improving how Score_A(S) is learned or approximated

Rubric:
Full credit identifies the scoring function as the fault based on the inequality and recommends improving the score-learning/approximation process; partial credit identifies the correct diagnosis but not the correct next action, or vice versa; no credit for attributing the problem to the optimization algorithm.

Diagnose a speech recognition scoring failure using the Optimization Verification test.

Question:
In one to three sentences, state what a developer should focus on improving once the Optimization Verification test confirms a scoring function problem.

Sample answer:
The developer should focus on improving how the system learns or approximates Score_A(S) for different sentences S, rather than working on the optimization algorithm.

Key points:
- Improve how Score_A(S) is learned or approximated
- Do not focus on the optimization algorithm in this case

Rubric:
Full credit for stating that effort should go toward improving how the score is learned or approximated; no credit for suggesting the optimization algorithm should be improved instead.

Learn Before

Related