The following webpage explains how to compute the derivative of a sigmoid function step by step 

https://kawahara.ca/how-to-compute-the-derivative-of-a-sigmoid-function-fully-worked-example/

University of Michigan - Ann Arbor

If we only have two features, $x_1$ and $x_2$, in order to minimize the loss function, we can apply gradient descent to update $w_1$, $w_2$, and $b$.
To compute the derivatives of $\mathcal L (a, y)$ with respect to $w_1$, $w_2$, and $b$, we need to compute the derivatives of $\mathcal L (a, y)$ with respect to $a$ and $z$ first.
$$\mathcal L (a, y) = -(ylog(a) + (1 - y)log(1 - a)) \Rightarrow$$
$$\frac{d\mathcal L (a, y)}{da} = -\frac{y}{a}+\frac{1-y}{1-a}$$
$$a = \sigma(z) = \frac{1}{1 + e^{-z}} \Rightarrow \frac{da}{dz} = a(1-a) \Rightarrow$$
$$\begin{aligned}
\frac{d\mathcal L (a, y)}{dz} & = \frac{d\mathcal L (a, y)}{da}\frac{da}{dz} \\
& = (-\frac{y}{a}+\frac{1-y}{1-a})*(a(1-a)) = a-y \\
\end{aligned}$$
$$\begin{aligned}
\frac{d\mathcal L (a, y)}{dw_1} & = \frac{d\mathcal L (a, y)}{dz}\frac{dz}{dw_1} = (a-y)*x_1 \\ 
\end{aligned}$$
$$\begin{aligned}
\frac{d\mathcal L (a, y)}{dw_2} & = \frac{d\mathcal L (a, y)}{dz}\frac{dz}{dw_2} = (a-y)*x_2 \\ 
\end{aligned}$$
$$\begin{aligned}
\frac{d\mathcal L (a, y)}{db} & = \frac{d\mathcal L (a, y)}{dz}\frac{dz}{db} = (a-y)*1 = (a-y)
\end{aligned}$$

Logistic regression gradient descent

How to compute the derivative of a sigmoid function

To compute logistic regression and its gradient descent on $m$ examples, we can use for loop to accumulate errors and derivatives and then average them, but it will take a long time to run on a big data set. So vectorization is a good way to get rid of explicit for loop in your code.
First, stack the $m$ examples horizontally into vectors of $X$ and $Y$, so the shape of $X$ is $(n_x, m)$, where $n_x$ is the number of features, and the shape of $Y$ is (1, m). 
Then, compute the $Z$ and $A$,
$$[z^{(1)}z^{(2)}...z^{(m)}]=Z=w^TX+b$$
$$=[(w^Tx{(1)}+b)(w^Tx{(2)}+b)...(w^Tx{(m)}+b)]$$
$$[a{(1)} a{(2)} ... a{(m)}] = A=\sigma (Z)$$
, the shape of $Z$ and $A$ should be $(1, m)$.
The derivatives of $L$ with respect to $Z$, $w$, and $b$ are
$$\frac{d\mathcal L}{dz} = A - Y$$
$$\frac{d\mathcal L}{dw} = \frac{1}{m} X {(\frac{d\mathcal L}{dz})}^T$$
$$\frac{d\mathcal L}{db} = \frac{1}{m} \sum \frac{d\mathcal L}{dz}$$

Vectorizing logistic regression on *m* examples

Input:

$$dA^{[l]}$$

   For the last layer (L): 

$$dA^{[L]} = \sum_{i=1}^{m} -\frac{y^{[1]}}{a^{[1]}} + \frac{1 - y^{[1]}}{1 - a^{[1]}}$$

Output: 

$$dA^{[l-1]},   dW^{[l]},   db^{[l]}$$

$$dZ^{[l]} = dA^{[l]} g'^{[l]}(Z^{[l]})$$

$$dW^{[l]} = \frac{1}{m} dZ^{[l]} A^{[l - 1] T}$$

$$db^{[l]} = \frac{1}{m} \sum dZ^{[l]}$$

$$dA^{[l - 1]} = W^{[l]T} dZ^{[l]}$$ 

Learn Before

Related