The cross-entropy loss function works very well for models that predict binary classes (aka the output is between 0 and 1). It is defined as -[y*log(y-hat) +(1-y)*log(1-(y-hat))]. If y=0 the left side of the function is dropped and the right side, -log(1-(y-hat)), is used. Otherwise if y=1 the right side of the function is dropped and it uses -log(y-hat). In both instances this loss function encourages probabilities that are close to the true probability. 

Cross-entropy loss

To train the parameters W and B of the logistic regression model, you need to define a cost function.

$$J(w, b) = \frac{1}{m} \sum_{i=1}^{m} L(\hat{y}^{(i)}, y^{(i)})$$

$$=-\frac{1}{m} \sum_{i=1}^{m} [y^{(i)}log(\hat{y}^{(i)}) + (1 - y^{(i)})log(1 - \hat{y}^{(i)})]$$

This loss function is Convex.

Logistic Regression Cost Function

A machine learning model is being trained for a prediction task. A key metric, the objective function, is tracked over time. The value of this function represents the magnitude of the model's error. A graph of this process shows the objective function's value consistently decreasing as the number of training iterations increases. What is the most accurate interpretation of this trend?

An engineer is training a model to predict housing prices. After running the training process for several hours, they plot the value of the model's error measurement over time. They observe that the error value remains very high and does not decrease, staying almost flat throughout the entire process. Based on this observation, analyze the effectiveness of the training process and explain what this trend indicates about the model's ability to achieve its primary goal.

Diagnosing Model Training Issues

A machine learning model is designed to predict the price of a product. For a small sample of three products, the model predicts prices of [$55, $90, $125], while the actual prices are [$50, $100, $120]. The model's performance is measured by an objective function defined as the average of the squared differences between the predicted and actual values. First, calculate the value of this objective function for the given sample. Second, explain what a lower value of this function would signify about the model's future predictions.

Calculating and Interpreting a Model's Objective Function

When a primary objective function—such as the error rate in classification—is difficult to optimize directly due to non-differentiability or other mathematical complications, machine learning models instead optimize a surrogate objective. This proxy function is chosen because it is easier to compute gradients for while still aligning with the ultimate goal.

Surrogate Objective

A loss function is a specific type of objective function where the convention is that lower values indicate better model performance. Optimization algorithms actively seek to minimize the loss function to improve the model; any objective where higher is better can be converted into a loss function by simply flipping its sign.

Loss Function

A fundamental requirement for training modern machine learning and deep learning models is the use of differentiable objectives. Because the optimization process typically relies on gradient-based methods, such as minibatch stochastic gradient descent, the objective function (or loss function) must be mathematically differentiable with respect to the model's parameters. This differentiability allows the optimization algorithm to compute gradients, which provide the direction and magnitude of the parameter updates needed to minimize the error and improve the model's predictive performance.

Differentiable Objectives

To build a formal mathematical system for learning, machine learning algorithms rely on objective functions, which are formal mathematical measures that quantify how well or poorly a model is performing its task. These functions provide the fundamental evaluation metric that optimization algorithms seek to improve.

University of Michigan - Ann Arbor

Claude

Google

After choosing a functional form for $\hat{f}$, based on our domain knowledge or other techniques, we need to estimate the parameters of the model based on the training data. The process of finding optimal values for the parameters of $\hat{f}$ based on the training data is called "training" or "fitting" the model.

Training or Fitting a supervised statistical learning model

A Feed forward neural network model training occurs in six stages:

- Iterate until convergence
   - Initialization
   - Forward propagation
   - Error function (Objective function)
   - Backpropagation
   - Weight update

At the end of this process, the model is ready to make predictions for unknown input data. New data can be fed to the model, a forward pass is performed, and the model generates its prediction.

Stages of Feed Forward Neural Network Learning

Gradient descent is a fundamental optimization algorithm that leverages gradients to minimize a model's loss function. Because the gradient of a function points in the direction of steepest ascent, moving the model's parameters in the opposite direction iteratively lowers the loss. Each step of such gradient-based optimization algorithms requires calculating the exact gradient of the loss with respect to the parameters.

Gradient Descent

A machine learning model is the computational machinery designed to ingest data of a specific type, apply mathematical transformations, and output predictions. In modern applications, the focus is on statistical models whose internal transformations can be estimated and refined using a dataset.

Machine Learning Model

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Reference of Foundations of Large Language Models Course

Dive into Deep Learning

We estimate the parameters $\beta_0, \beta_1, \beta_2, \dots, \beta_p$ such that we minimize the error in the following approximation:
$$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon$$

Training or Fitting a linear regression model

A model’s capacity is its ability to ﬁt a wide variety of functions.
The model $\hat{f}$ that we estimate by choosing a functional form and training, is just an approximation and almost never match the true unknown $f$. To have a closer estimate, we can choose more flexible functional forms with more parameters to estimate.
Machine learning algorithms will generally perform best when their capacity is appropriate for the true complexity of the task they need to perform and the amount of training data they are provided with.
Models with insuﬃcient capacity are unable to solve complex tasks. Models with high capacity can solve complex tasks, but when their capacity is higher than needed to solve the present task, they may overﬁt.

Supervised Statistical Model Flexibility (Capacity/Complexity)

In classification tasks, a model's output for a given input is typically a probability distribution over the set of possible labels. This distribution quantifies the model's confidence that the input belongs to each specific class.

Model Output in Classification Problems

Objective Function

When we use a feedforward neural network to accept an input $x$ from training set and produce an output $\hat{y}$, information ﬂows forward through the network. The input $x$ provides the initial information that then propagates up to the hidden units at each layer and ﬁnally produces $\hat{y}$. This is called forward propagation. 


Forward Propagation

Weights are changed to the optimal values according to the results of the backpropagation algorithm.

Since the weights are updated a small delta step at a time, several iterations are required in order for the network to learn. After each iteration, the gradient descent force updates the weights towards less and less global loss function. The amount of iterations needed to converge depends on the learning rate, the network meta-parameters, and the optimization method used.

Update Weight Iteratively Until Convergence

Initial weights are applied to all the neurons.
It is necessary to set initial weights for the first forward pass. Two basic options are to set weights to zero or to randomize them. However, this can result in a vanishing or exploding gradient, which will make it difficult to train the model. To mitigate this problem, you can use a heuristic (a formula tied to the number of neuron layers) to determine the weights. A common heuristic used for the Tanh activation is called Xavier initialization.

On top of that if the architecture being used has already been trained by someone else (such as ImageNet) the node values can be initialized to those values. This is known as transfer learning and it can be very useful to substantially increase training speed and model accuracy.

Deep Learning Weight Initialization

What is the "cache" used for in our implementation of forward propagation and backward propagation?

Which of the following statements are True? (Check all that apply).

Consider the following 1 hidden layer neural network:

Which of the following are true regarding activation outputs and vectors? (Check all that apply.)

Backpropagation is a systematic computational procedure for applying the chain rule to calculate gradients automatically. It operates by traversing a computational graph in a backwards direction—from the output loss back to the input parameters—multiplying matrices of partial derivatives at each step to determine how parameters affect the final output.

Backpropagation

A helpful website for understanding gradient descent:
https://towardsdatascience.com/gradient-descent-explained-9b953fc0d2c

Gradient Descent Reference

Gradient descent is used to properly update parameters in order to maximize the efficiency of linear regressions. Below is a step-by-step example of how to implement gradient descent.

Linear Regression and Gradient Descent

You can check your derivative computation to make sure that your implementation of back propagation is correct. You want to consider BOTH the right hand side and the left hand side derivatives. By taking a two sided derivative, you can numerically verify whether or not the function g of theta is a correct implementation of the derivative of f.

Numerical Approximation of Gradients

This is the technique used to check that our implementation is correct. There are different formulas for gradient checking and one of those is two-sided form:
          $$\frac {J(\theta + \epsilon) - J(\theta - \epsilon)} {2\epsilon}$$
Common choice for $\epsilon$ is $10^{-7}$. You shouldn't use gradient checking for the whole training data as it can be slow.

Gradient Checking

Assuming that the error function is $J(w)$ with one parameter $w$, to minimize the error, we can update the weight $w$ as follows.
$$w = w - \alpha * \frac{dJ(w)}{dw}$$
, where $\alpha$ is a learning rate, and $\frac{dJ(w)}{dw}$ is the derivative of $J(w)$ with respect to $w$.

If the error function has two or more parameters, for example, a weight $w$ and a bias $b$, we can update them one by one.
$$w = w - \alpha * \frac{\partial J(w,b)}{\partial w}$$
$$b = b - \alpha * \frac{\partial J(w,b)}{\partial b}$$
, where $\partial$ is a stylish cursive $d$, denoting the partial derivatives.

(Batch) Gradient Descent (Deep Learning Optimization Algorithm)

https://towardsdatascience.com/gradient-descent-explained-9b953fc0d2c

Gradient Descent Explained

Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may converge to another optimal point but its probability is not too much.

Consider the following "recliner chair" type of function(image below).

Obviously, this can be constructed so that there is a range in the middle where the gradient is the 0 vector, casuing the fail to find global optima. 


Why Gradient descent might fail?

DeepLearningAI. (2021, March 24). A Chat with Andrew on MLOps: From Model-centric to Data-centric AI. YouTube. https://www.youtube.com/watch?v=06-AZXmwHjo

A Chat with Andrew on MLOps: From Model-centric to Data-centric AI

Sagar, R. (2021, April 6). Big Data To Good Data: Andrew Ng Urges ML Community To Be More Data-Centric And Less Model-Centric. Analytics India Magazine. https://analyticsindiamag.com/big-data-to-good-data-andrew-ng-urges-ml-community-to-be-more-data-centric-and-less-model-centric/

Big Data to Good Data: Andrew Ng Urges ML Community To Be More Data-Centric and Less Model-Centric

Andrew Ng believes that for datasets with fewer than 10,000 data points, machine learning teams can make faster progress by focusing on improving data than improving code.

MLOps: Data-centric and Model-centric approaches

Points where the derivative of a function is 0 are known as critical points, or stationary points.

The derivative also has to be zero according to every possible directional derivative at that point in higher dimensional functions.

$$\triangledown f = 0$$

Critical Points

Optimization algorithms that use only the gradient, such as gradient descent,are called ﬁrst-order optimization algorithms.

First-order Optimization Algorithm

Optimization algorithms that also use the Hessian matrix, such as Newton’s method, are called second-order optimization algorithms

Second-order Optimization Algorithm

When the method of steepest descent is applied to a quadratic cost surface, it creates a zig-zag pattern that each line search direction is orthogonal to its previous line search direction.
Let $d_{t-1}$ be the previous search direction. At the minimum, we will find that:
$$\nabla _\theta f(\theta) \cdot d_{t-1} = 0$$
Then we know $d_t = \nabla _\theta f(\theta)$ is orthogonal to $d_{t-1}$

Method of Steepest Descent

Unlike first-order methods, second-order gradient methods use second-order derivatives. This improves optimization. 
- The most widely used second-order gradient method is the Newton method

Second-Order Gradient Methods

Gradient descent is an algorithm used to find the minimum value of a function. We will use the gradient descent algorithm to find the cost function. The idea of the algorithm is to randomly select a parameter combination at the beginning, calculate, and then find the next parameter combination that can reduce the cost function value the most, and continue to a minimum value.

https://machinelearningmastery.com/gradient-descent-for-machine-learning/

Gradient Descent Explanation

There are three variants of gradient descent, which differ in how much data we use to compute the gradient of the objective function: Batch Gradient Descent, Stochastic Gradient Descent and Mini-batch Gradient Descent. Depending on the amount of data, we make a trade-off between the accuracy of the parameter update and the time it takes to perform an update. 

Reference: https://ruder.io/optimizing-gradient-descent/

Gradient Descent Variants

Gradient descent methods use the slope of the surface. This will not necessarily point directly towards the extreme point. Local steepest direction may not be the same with the global optimum direction. 

Notes about gradient descent

Suppose you have built a neural network. You decide to initialize the weights and biases to be zero. Which of the following statements is true?

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow. 

Vanishing/exploding gradient

The training of BERT models follows a standard iterative optimization procedure used for deep neural networks. First, a large collection of training data is gathered. During each iteration, a random batch of these samples is selected, and the cumulative loss, $$\mathrm{Loss}_{\mathrm{BERT}}$$, is computed over the batch. Next, the model's parameters are updated to minimize this loss using an optimization algorithm like gradient descent or one of its variants. This cycle continues until a specific stopping condition is met, such as the convergence of the training loss.

BERT Training Process

Distributed training is an approach used when a single processor or GPU lacks the computational capacity or memory to process large amounts of training data. By distributing the workload across multiple processors, optimization algorithms like stochastic gradient descent can aggregate computations. For example, training across $$1,024$$ GPUs with a small minibatch size of $$32$$ per GPU results in an aggregate minibatch of $$32,000$$ observations, dramatically accelerating training times for massive neural networks.

Distributed Training

If all parameters of a hidden layer are initialized to a constant, such as $$\mathbf{W}^{(1)} = c$$, every hidden unit will receive the same inputs and parameters, producing identical activations during forward propagation. Consequently, during backpropagation, the gradients of the output with respect to the parameters $$\mathbf{W}^{(1)}$$ will all take the exact same value. Because gradient-based algorithms like minibatch stochastic gradient descent update the parameters using these uniform gradients, all elements of $$\mathbf{W}^{(1)}$$ will continue to have identical values after every iteration. The hidden layer will thus behave as if it has only a single unit, failing to realize the network's expressive power.

The Problem with Constant Initialization

Machine learning serves as a ubiquitous and transformative tool across various scientific and engineering disciplines. It drives significant modern advancements in fields like robotics, logistics, computational biology, particle physics, and astronomy by optimizing complex systems and accelerating research discovery.

Scientific Applications of Machine Learning

A deep learning model is a powerful type of machine learning model differentiated by its architecture, which consists of many successive transformations of data chained together from top to bottom. This depth of continuous data transformations allows these models to tackle complex tasks by automatically learning multi-level representations. By doing so, deep learning effectively replaces not only the labor-intensive process of manual feature engineering, but also the shallow models that are typically used at the end of traditional machine learning pipelines.

Deep Learning Model

Artificial General Intelligence (AGI) refers to a hypothetical, sentient artificial intelligence capable of reasoning about itself, improving its own architecture, and solving general tasks without human intervention. The textbook emphasizes that current AI systems are far from AGI; they are engineered, goal-oriented combinations of rules, heuristics, and statistical models that only give the illusion of general intelligence.

Artificial General Intelligence (AGI)

When statistical machine learning models are applied without care, they can inadvertently learn and perpetuate biases, such as racial, gender, or age bias. If these automated systems are used to drive consequential decisions, such as assessing creditworthiness or granting bail, they raise significant concerns about procedural fairness and equity.

Algorithmic Bias in Machine Learning

The application of artificial intelligence has profound societal implications, particularly concerning employment. As machine learning automates routine and menial tasks, such as harvesting operations using farm robots, it reduces costs but also displaces jobs that provide significant employment. This automation phase is considered a much more pressing, realistic concern than hypothetical scenarios of malevolent superintelligence destroying humanity.

Societal Impact of Artificial Intelligence

Machine learning is pervasive in modern life, operating widely behind the scenes to automate complex tasks. This includes financial applications like scoring creditworthiness and detecting fraudulent transactions—which form the backbone of major e-commerce payment systems—as well as internet applications where it feeds search, recommendation, personalization, and ranking algorithms.

Pervasive Machine Learning Applications

In machine learning, parametric models are algorithms that rely on a fixed, finite number of parameters to define their behavior. When these models are trained, updated, and saved, it is these specific parameters that are modified and written to disk. Because they possess a set number of parameters regardless of the dataset size, they typically contrast with nonparametric methods, whose complexity grows with the amount of available training data.

Learn Before

Related

Learn After