When a primary objective function—such as the error rate in classification—is difficult to optimize directly due to non-differentiability or other mathematical complications, machine learning models instead optimize a surrogate objective. This proxy function is chosen because it is easier to compute gradients for while still aligning with the ultimate goal.

Claude

To build a formal mathematical system for learning, machine learning algorithms rely on objective functions, which are formal mathematical measures that quantify how well or poorly a model is performing its task. These functions provide the fundamental evaluation metric that optimization algorithms seek to improve.

Objective Function

Dive into Deep Learning

The cross-entropy loss function works very well for models that predict binary classes (aka the output is between 0 and 1). It is defined as -[y*log(y-hat) +(1-y)*log(1-(y-hat))]. If y=0 the left side of the function is dropped and the right side, -log(1-(y-hat)), is used. Otherwise if y=1 the right side of the function is dropped and it uses -log(y-hat). In both instances this loss function encourages probabilities that are close to the true probability. 

Cross-entropy loss

To train the parameters W and B of the logistic regression model, you need to define a cost function.

$$J(w, b) = \frac{1}{m} \sum_{i=1}^{m} L(\hat{y}^{(i)}, y^{(i)})$$

$$=-\frac{1}{m} \sum_{i=1}^{m} [y^{(i)}log(\hat{y}^{(i)}) + (1 - y^{(i)})log(1 - \hat{y}^{(i)})]$$

This loss function is Convex.

Logistic Regression Cost Function

A machine learning model is being trained for a prediction task. A key metric, the objective function, is tracked over time. The value of this function represents the magnitude of the model's error. A graph of this process shows the objective function's value consistently decreasing as the number of training iterations increases. What is the most accurate interpretation of this trend?

An engineer is training a model to predict housing prices. After running the training process for several hours, they plot the value of the model's error measurement over time. They observe that the error value remains very high and does not decrease, staying almost flat throughout the entire process. Based on this observation, analyze the effectiveness of the training process and explain what this trend indicates about the model's ability to achieve its primary goal.

Diagnosing Model Training Issues

A machine learning model is designed to predict the price of a product. For a small sample of three products, the model predicts prices of [$55, $90, $125], while the actual prices are [$50, $100, $120]. The model's performance is measured by an objective function defined as the average of the squared differences between the predicted and actual values. First, calculate the value of this objective function for the given sample. Second, explain what a lower value of this function would signify about the model's future predictions.

Calculating and Interpreting a Model's Objective Function

Surrogate Objective

A loss function is a specific type of objective function where the convention is that lower values indicate better model performance. Optimization algorithms actively seek to minimize the loss function to improve the model; any objective where higher is better can be converted into a loss function by simply flipping its sign.

Loss Function

A fundamental requirement for training modern machine learning and deep learning models is the use of differentiable objectives. Because the optimization process typically relies on gradient-based methods, such as minibatch stochastic gradient descent, the objective function (or loss function) must be mathematically differentiable with respect to the model's parameters. This differentiability allows the optimization algorithm to compute gradients, which provide the direction and magnitude of the parameter updates needed to minimize the error and improve the model's predictive performance.

Learn Before

Related