Google

A scoring function can assign a score to each possible output for a given input, and an inference system can then search for the output that maximizes that score. A common AI design pattern is to learn an approximate scoring function and then apply an approximate maximization algorithm.

Scoring Function

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A scored inference system may compute a score for each possible output but still need an approximate search algorithm to find a high-scoring output, because exhaustive enumeration can be far too large. These approximate algorithms are not guaranteed to find the output that maximizes the score.

Approximate Search in Scored Inference

In a distance-based scoring function for knowledge graphs, the plausibility of a fact is measured by the distance between entities. This relationship is typically expressed as $$h + r \sim t$$, where $$h$$ represents the head entity, $$r$$ represents the relation, and $$t$$ represents the tail entity.

Distance-Based Scoring Function

In a similarity-based (or semantic matching) scoring function for knowledge graphs, the plausibility of a fact is measured by matching the latent semantics of entities and relations. This relationship is typically expressed using a bilinear formulation, such as $$h^T M_r \sim t^T$$, where $$h$$ and $$t$$ represent the head and tail entity vectors, and $$M_r$$ represents the relation matrix.

Semantic Matching Scoring Function

In Andrew Ng's speech recognition example, what does Score_A(S) represent?

True or False: A scoring function alone is sufficient; no separate search or maximization step is needed.

An inference system searches for the output that _____ the scoring function.

Match each scoring function component to its role in the speech recognition example.

Order the steps of the scoring function design pattern in AI system design.

Question:
Discuss why splitting an AI system into (1) learning an approximate scoring function and (2) applying an approximate maximization algorithm to search for the best output is a valuable design pattern, and how this separation helps diagnose the Optimization Verification test.

Sample answer:
Separating the system into a scoring function and a search algorithm lets engineers isolate where errors come from: if a known-good output y* scores higher than the algorithm's chosen output y under Score_x(.), the problem lies in the search/maximization algorithm; if y* scores lower or equal, the problem lies in the scoring function itself. This separation, as described in the speech recognition example, turns a single opaque system into two independently diagnosable components.

Key points:
- Scoring function estimates how good an output is for a given input
- Search/maximization algorithm finds the output that maximizes the score
- Separation allows independent diagnosis of each component
- Optimization Verification test compares scores of a correct output vs. the algorithm's output
- Determines whether errors stem from the scoring function or the search algorithm

Rubric:
Full credit for explaining that the pattern separates scoring from search, that this enables the Optimization Verification test, and that comparing scores of known-good vs. algorithm outputs isolates whether the scoring function or the search algorithm is at fault.

Explain why learning a scoring function and searching separately is a useful AI design pattern.

Case context:
A speech recognition system computes Score_A(S) = P(S|A) for an audio clip A. For a particular clip, the system outputs the sentence 'I love your city' but the correct transcription is 'I love acity.' The engineer checks Score_A('I love acity') and finds it is lower than Score_A('I love your city'), even though 'I love acity' is the correct answer.

Question:
Based on this scoring comparison, should the engineer conclude the problem is in the scoring function or in the search/maximization algorithm? Explain your reasoning.

Sample answer:
Since the correct sentence 'I love acity' received a lower score than the incorrect output 'I love your city', the scoring function itself is failing to rank the correct answer highest. This means the problem lies in the scoring function (it needs to be improved, e.g., via better training data or model), not in the search algorithm, because even a perfect search would have found and returned the higher-scoring but wrong sentence.

Key points:
- Compare Score_A of the correct sentence to Score_A of the algorithm's output
- If the correct sentence scores lower, the scoring function is at fault
- If the correct sentence scores higher or equal but wasn't chosen, the search algorithm is at fault
- This reasoning reflects the Optimization Verification test

Rubric:
Full credit for correctly identifying that a lower score for the correct sentence indicates a scoring function problem, and for explaining that this conclusion follows from the Optimization Verification test logic described in the source.

Diagnose a speech recognition error using the scoring function framework.

Question:
In one to three sentences, name and briefly describe the two components of the AI design pattern involving scoring functions.

Sample answer:
The pattern consists of (1) an approximate scoring function, Score_x(.), that estimates how good a candidate output is for a given input, and (2) an approximate maximization algorithm that searches for the output maximizing that score.

Key points:
- Approximate scoring function estimates output quality
- Approximate maximization algorithm searches for the best-scoring output

Rubric:
Full credit for naming both the scoring function and the maximization/search algorithm and briefly describing each role.

Learn Before

Related