University of Michigan - Ann Arbor

Given X, we want $\hat{y}=P(y = 1|X)$   $0 \leq \hat{y} \leq 1$

Parameters: $w \in R^{n_{x}}, b \in R$

Output (Activation): $\hat{y}^{(i)} = a^{(i)} = \sigma (w^{T}x^{(i)} + b)$

Given $[(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), ..., (x^{(m)}, y^{(m)})]$, we want $\hat{y}^{i} \approx y^{i}$

$\sigma(z^{(i)}) = \frac{1}{1 + e^{-z^{(i)}}}$

If z is large positive, $\sigma(z^{(i)}) \approx \frac{1}{1 + 0} = 1$

If z is large negative, $\sigma(z^{(i)}) \approx \frac{1}{1 + Big Num} \approx 0$

Logistic Regression Formulation

What are the parameters of logistic regression?

 $L(\hat{y}, y) = -(ylog(\hat{y}) + (1 - y)log(1 - \hat{y}))$
- If y = 1: $L(\hat{y}, y) = -log(\hat{y}) \Rightarrow log(\hat{y}) \approx 0 \rightarrow \hat{y}  \approx 1$
- If y = 0: $L(\hat{y}, y) = -log(1 - \hat{y}) \Rightarrow log(1 - \hat{y}) \approx 0 \rightarrow \hat{y} \approx 0$

Logistic Regression Loss Function

To train the parameters W and B of the logistic regression model, you need to define a cost function.

$$J(w, b) = \frac{1}{m} \sum_{i=1}^{m} L(\hat{y}^{(i)}, y^{(i)})$$

$$=-\frac{1}{m} \sum_{i=1}^{m} [y^{(i)}log(\hat{y}^{(i)}) + (1 - y^{(i)})log(1 - \hat{y}^{(i)})]$$

This loss function is Convex.

Logistic Regression Cost Function

Neural networks are made up of nodes (each representing a Logistic regression) with layers. They are trained to map an input vector X to an output vector Y based on the training data. The hidden layers of a network contain neurons, connections, and activation functions that occur between the input and output. An activation function is applied at each layer to correctly connect the input and output. Different types of networks may use diverse types of activation functions to correctly identify the nature of how the input and output are related.
In the figure out of the 4 neurons, the Logistic regression representation of two of them are shown.

Learn Before

Related