DL-1-week2-Basics of Neural Network Programming_week 2 quiz - neural network basics

2 Basics of Neural Network Programming

---

Date:2018.3.4

---

2.1 Binary Classification

Example:

• input:a cat image
  Three 64*64 matrices in computer corresponding to the red,green and blue.Then define a feature vector x corresponding to this image.
  The dimension of the input features x is:
  
  $$n_x = 64 \times 64 \times 64 = 12288$$
• output: 1(cat) vs 0(not cat)

2.2 Logistic Regression

Given $x$, want $\hat y = P(y=1|x)$
$x \in R^{n_x}$
Parameters: $\omega \in R^{n_x},b\in R$
Output: First consider the linear regression:$\hat y = \omega ^T x +b$, but $\hat y \in [0,1]$. Actually, In logistic regression, our output is instead going to be $\hat y$ equals the sigmoid function applied to this quantity.

$$\hat y= \sigma(\omega ^T x+b)$$
sigmoid function is:
$$\sigma (z) = \frac{1}{1+e^{-z}}$$
easily to find that :
$$0<\sigma (z) <1$$
if we let $x_0 = 1$,then $x \in R^{n_x +1}$, $\hat y = \sigma(\theta^T x)$

---

Date:2018.3.5

---

2.3 Logistic Regression cost function

Loss(error) function:
Usually we use the square loss function:

$$L(\hat y ,y)= \frac{1}{2}(\hat y - y)^2$$
but in logistic regression,the loss function is :
$$L(\hat y ,y) = -[y \log \hat y +(1-y)\log(1-\hat y)]$$
If $y=1:\quad L(\hat y,y) = -\log \hat y$. So,you want $\log \hat y$ to be as big as possible, also want $\hat y$large. But $\hat y$ is given by Sigmoid function, so $\hat y$ should be close to 1.

If $y = 0:L(\hat y ,y ) = -log(1-\hat y)$,this make $\hat y$ as small as possible.

So, if y is 0, we try to make $\hat y$ small, if y is 1, we try to make $\hat y$ large.

Coss function:
Loss function measures how well you’re doing on a single training example. However, coss function measures how well you’re doing an entire training set.

$$J(\omega,b)=\frac{1}{m}\sum \limits_{i=1}^m L(\hat y ^{(i)},y^{(i)})$$

2.4 Gradient Descent

Want to find $\omega,b$ that minimize $J(\omega,b)$
Repeat:

$$\omega :=\omega - \alpha \frac{\partial J(\omega)}{\partial \omega}$$
$$b:=b-\alpha \frac{\partial J(\omega,b)}{\partial b}$$
where $\alpha$ is the learning rate.

2.5 Derivatives

calculus and derivatives
skip

2.6 More derivatives examples

The slope of the function
skip

2.7 Computation Graph

Given a function:

$$J(a,b,c)=3(a+b\times c)$$
The step to solve it :

u=bc
v=a+u
J=3v
1
2
3

a forward path and a backward pass(back propagation)

2.8 Derivatives with a Computation Graph

chain rule

---

Date:2018.3.6

---

2.9 Logistic Regression Gradient descent

The model is:

$$z= \omega^T x +b$$ $$\hat y = a = \sigma(z)$$ $$L(a,y)=-(y\log (a) +(1-y)\log (1-a))$$

We have only two features $x_1$ and $x_2$.The step to get parameters $\omega_1,\omega_2,b$ is :

z = w1*x1+w2*x2+b
a = y^ = sigmoid(z)
L(a,y)
1
2
3

$$d\omega_1 = \frac{\partial L}{\partial a}\frac{\partial a}{\partial z}\frac{\partial z}{\partial \omega_1} = (-\frac{y}{a}+\frac{1-y}{1-a})\times a(1-a) \times x_1$$ $$=(a-y)x_1$$ $$d \omega _2 = (a-y)x_2$$ $$d b=(a-y)$$
The update steps:

w1:= w1 - \alpha dw1
w2:= w2 - \alpha dw2
b := b  - \alpha db
1
2
3

2.10 Gradient descent on m examples

The cost function:

$$J(\omega ,b )=\frac{1}{m} \sum \limits _{i=1}^{m}L(a^{(i)},y)$$ $$a^{(i)}=\hat y ^{(i)}=\sigma (z) = \sigma(\omega ^T x^{(i)}+b)$$
So the overall gradient is :
$$\frac{\partial J(\omega,b)}{\partial \omega_i}=\frac{1}{m}\sum \limits_{i=1}^m d\omega_i$$

This is only one loop for gradient descent,actually,you need more steps to use gradient descent, if the data has a lot of features,you may need 3 for loops, one for gradient descent,one for all items, one for all features, so it is too slow for the algorithm. The vectorization can accelerate the speed.

2.11 Vectorization

Vectorization is the art of getting rid of explicit for loops in your code.
Non-vectorizated:

z=0
for i in range(n_x):
    z+=w[i]*x[i]
z+=b
1
2
3
4

Vectorizated:

z = np.dot(w,x)+b
1

2.12 More vectorization examples

Whenever possible, avoid explicit for-loops.
Given

$$v = [v_1,v_2,...,v_n]^T$$ , you want to apply the exponential operation on every element of a vector.
Non-vectorization:

u = np.zeros((n,1))
for i in range(n):
    u[i]=math.exp(v[i])
1
2
3

Vectorization:

import numpy as np
u = np.exp(v)
1
2

In logistic regression derivatives:

2.13&2.14 Vectorizing Logistic Regression

$$Z = W^TX+B$$ $$A=\sigma(Z)$$ where , $Z=(z_1,z_2,z_3)^T,X=(x_1,x_2,x_3)^T$.
Let $dz = [dz^{(1)},dz^{(2)},...,dz^{(m)}],A=[a^{(1)},a^{(2)},...,a^{(m)}],Y=[y^{(1)},y^{(2)},...,y{(m)}]$
So, $$dz = A-Y$$ $$db =\frac{1}{m}dZ^T*\textbf{1}$$ $$dw = \frac{1}{m}X*dZ^T$$

---

Date:2018.3.7

---

2.15 Broadcasting in Python

import numpy as np
A = np.array([[1,2,3],[4,5,6],[7,8,9]])
cal = A.sum(axis = 0)
percentage = 100*A/cal.reshape(1,3)#reshape(),改成1行3列的数组
print(percentage)
1
2
3
4
5

2.16 A note on python/numpy vectors

Usually use a = np.random.randn(5,1),not use a = np.random.randn(5)

a = np.random.randn(5) #a.shape = (5,),rank 1 array
a = np.random.randn(5,1)#a.shape = (5,1),5 by 1 column vector
a = np.random.randn(1,5)#a.shape = (1,5),1 by 5 row vector
1
2
3

you can use assert(a.shape==(5,1)) to ensure you get a column vector otherwise a rank 1 array. Use a.reshape(5,1) if you get a rank 1 array.