Case context: You are building a route-planning system for delivery robots. For any candidate route P, the system uses a learned scoring function S(P) to estimate how good that route is, and a search algorithm to choose the route with the highest estimated score. In testing, the robot selects a route that is clearly worse than another feasible route.

Question: Using the pattern of an approximate scoring function plus an approximate maximization step, how would you analyze whether the bad route was caused by the scoring model or by the search procedure?

Sample answer: First compare the score of the best known route P_best with the score of the route chosen by the system P_chosen. If the learned scorer assigns a higher value to the worse route, so that S(P_chosen) > S(P_best), then the scoring model is the problem because it does not rank routes correctly. If the best route receives the higher score, S(P_best) > S(P_chosen), then the scoring model is consistent with the goal, and the search procedure is at fault because it failed to find the better-scoring route.

Key points:

• Treat S(P) as the approximate scoring function.
• Treat the search procedure as the approximate maximization step.
• Compare the score of the best feasible route with the score of the route actually selected.
• If the worse selected route scores higher, the scoring model is failing.
• If the better route scores higher but was not chosen, the search procedure is failing.

Rubric: The learner must say to compare the score of the best route with the score of the system-selected route. They must correctly state that S(P_chosen) > S(P_best) points to a scoring-function problem, and S(P_best) > S(P_chosen) points to a search/maximization problem.