优化方法（optimization method）Momentum、RMSprop、Adam、LearningRate Decay_adam learning rate decay

指数加权平均值Exponentially weighted averages

$$\begin{gathered} v_0=0 \\ {v}_1=\beta v_0+(1-\beta ){\theta }_1 \\ {v}_t=\beta v_{t-1}+(1-\beta ){\theta }_t \end{gathered}$$
$$\begin{gathered} 其中v表示第t个平均值，{\theta }_t表示第t个值，\beta \\ 是需要选定的超参数，大致可以理解为我们需要 \\ 考虑\frac{1}{1-\beta }个之前的值的影响 \end{gathered}$$
在机器学习和深度学习中,如果某个值既需要考虑以往的影响，又需要结合当前的价值，那么可以考虑使用加权平均数。
可以将$v$想象为速度，将$\theta$想象为加速度，当前的速度受到前一时间的速度和当前的加速度的影响。

上图是对温度和天气使用指数加权平均数进行预测。其中红色的${\beta }_1$选用的值更低，绿色的${\beta }_2$更高
假定$\beta =0.2$
$$\begin{gathered} v_1=0.2{\theta }_1 \\ {v}_2=0.16{\theta }_1+0.2{\theta }_2 \end{gathered}$$

可以看出上述平均值在$t$很小的时候，该平均值不能很好拟合原模型，于是引入修正方法
$$\begin{gathered} v_0=0 \\ {v}_1^{corrected}=\frac{v_1}{1-{\beta }^1} \\ ... \\ {v}_t^{corrected}=\frac{v_t}{1-{\beta }^t} \end{gathered}$$

在之后将要介绍的三种优化器都是使用上述的类似方法进行优化。

带动量的梯度下降方法（Gradient Descent with Momentum）

相比于传统的梯度下降算法，这个新的算法引入了动量，
我们有
$$\begin{gathered} par{a}_{new}=par{a}_{old}-\alpha grad \\ 其中\alpha 为学习率。 \end{gathered}$$
而动量采用了我们前面所谈到的指数加权平均数
$$\begin{gathered} 第t次迭代的更新公式为： \\ par{a}_{new}=par{a}_{old}-\alpha v_t \\ {v}_t=\beta v_{t-1}+(1-\beta )gra{d}_t \end{gathered}$$
$$\begin{gathered} 其中\alpha 为学习率，\beta 是超参数，gra{d}_t是第t才的梯度。 \\ \alpha 往往会比我们在传统梯度下降时指定的学习率更低。 \end{gathered}$$

# GRADED FUNCTION: update_parameters_with_momentum

def update_parameters_with_momentum(parameters, grads, v, beta, learning_rate):
    """
    Update parameters using Momentum
    
    Arguments:
    parameters -- python dictionary containing your parameters:
                    parameters['W' + str(l)] = Wl
                    parameters['b' + str(l)] = bl
    grads -- python dictionary containing your gradients for each parameters:
                    grads['dW' + str(l)] = dWl
                    grads['db' + str(l)] = dbl
    v -- python dictionary containing the current velocity:
                    v['dW' + str(l)] = ...
                    v['db' + str(l)] = ...
    beta -- the momentum hyperparameter, scalar
    learning_rate -- the learning rate, scalar
    
    Returns:
    parameters -- python dictionary containing your updated parameters 
    v -- python dictionary containing your updated velocities
    """

    L = len(parameters) // 2 # number of layers in the neural networks
    
    # Momentum update for each parameter
    for l in range(1, L + 1):
        
        # (approx. 4 lines)
        # compute velocities
        # v["dW" + str(l)] = ...
        # v["db" + str(l)] = ...
        # update parameters
        # parameters["W" + str(l)] = ...
        # parameters["b" + str(l)] = ...
        # YOUR CODE STARTS HERE
        v['dW'+str(l)] = beta*v['dW'+str(l)] + (1-beta)*grads['dW'+str(l)]
        v['db'+str(l)] = beta*v['db'+str(l)] +(1-beta)*grads['db'+str(l)]
        parameters['W'+str(l)] = parameters['W'+str(l)]-learning_rate*v['dW'+str(l)]
        parameters['b'+str(l)] = parameters['b'+str(l)]-learning_rate*v['db'+str(l)]
        # YOUR CODE ENDS HERE
        
    return parameters, v
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

RMSprop(Root mean sqare propagate)

相比于直接使用速度代替梯度进行计算，RMSprop采用一种类似累计均方的方法。
$$\begin{gathered} 第t次迭代： \\ {S}_0=0 \\ {S}_1=\beta S_0+(1-\beta )gra{d}_1^2 \\ ... \\ {S}_t=\beta S_{t-1}+(1-\beta )gra{d}_t^2 \\ par{a}_{new}=par{a}_{old}-\alpha \frac{grad}{\sqrt{S_t+ϵ}} \end{gathered}$$
事实上RMSprop用处相比于Momentum和Adam，范围和用途都比较少。

Adam

Adam的方法相当于上述两个方法的结合体，同时使用了v和s。
$$\begin{gathered} 第t次迭代： \\ {v}_t={\beta }_1{v}_{t-1}+(1-{\beta }_1)gra{d}_t \\ {v}_t^{corrected}=\frac{v_t}{1-{\beta }_1^t} \\ {S}_t={\beta }_2{S}_{t-1}+(1-{\beta }_2)gra{d}_t \\ {S}_t^{correctd}=\frac{S_t}{1-{\beta }_2^t} \\ par{a}_{new}=par{a}_{old}-\alpha \frac{v_t^{correctd}}{S_t^{correted}} \end{gathered}$$

# GRADED FUNCTION: update_parameters_with_adam

def update_parameters_with_adam(parameters, grads, v, s, t, learning_rate = 0.01,
                                beta1 = 0.9, beta2 = 0.999,  epsilon = 1e-8):
    """
    Update parameters using Adam
    
    Arguments:
    parameters -- python dictionary containing your parameters:
                    parameters['W' + str(l)] = Wl
                    parameters['b' + str(l)] = bl
    grads -- python dictionary containing your gradients for each parameters:
                    grads['dW' + str(l)] = dWl
                    grads['db' + str(l)] = dbl
    v -- Adam variable, moving average of the first gradient, python dictionary
    s -- Adam variable, moving average of the squared gradient, python dictionary
    t -- Adam variable, counts the number of taken steps
    learning_rate -- the learning rate, scalar.
    beta1 -- Exponential decay hyperparameter for the first moment estimates 
    beta2 -- Exponential decay hyperparameter for the second moment estimates 
    epsilon -- hyperparameter preventing division by zero in Adam updates

    Returns:
    parameters -- python dictionary containing your updated parameters 
    v -- Adam variable, moving average of the first gradient, python dictionary
    s -- Adam variable, moving average of the squared gradient, python dictionary
    """
    
    L = len(parameters) // 2                 # number of layers in the neural networks
    v_corrected = {}                         # Initializing first moment estimate, python dictionary
    s_corrected = {}                         # Initializing second moment estimate, python dictionary
    
    # Perform Adam update on all parameters
    for l in range(1, L + 1):
        # Moving average of the gradients. Inputs: "v, grads, beta1". Output: "v".
        # (approx. 2 lines)
        # v["dW" + str(l)] = ...
        # v["db" + str(l)] = ...
        # YOUR CODE STARTS HERE
        v['dW'+str(l)] = beta1*v['dW'+str(l)]+(1-beta1)*grads['dW'+str(l)]
        v['db'+str(l)] = beta1*v['db'+str(l)]+(1-beta1)*grads['db'+str(l)]
        
        # YOUR CODE ENDS HERE

        # Compute bias-corrected first moment estimate. Inputs: "v, beta1, t". Output: "v_corrected".
        # (approx. 2 lines)
        # v_corrected["dW" + str(l)] = ...
        # v_corrected["db" + str(l)] = ...
        # YOUR CODE STARTS HERE
        v_corrected['dW'+str(l)]=v['dW'+str(l)]/(1-beta1**t)
        v_corrected['db'+str(l)]=v['db'+str(l)]/(1-beta1**t)
        # YOUR CODE ENDS HERE

        # Moving average of the squared gradients. Inputs: "s, grads, beta2". Output: "s".
        #(approx. 2 lines)
        # s["dW" + str(l)] = ...
        # s["db" + str(l)] = ...
        # YOUR CODE STARTS HERE
        s['dW'+str(l)]=beta2*s['dW'+str(l)]+(1-beta2)*grads['dW'+str(l)]**2
        s['db'+str(l)]=beta2*s['db'+str(l)]+(1-beta2)*grads['db'+str(l)]**2
        
        # YOUR CODE ENDS HERE

        # Compute bias-corrected second raw moment estimate. Inputs: "s, beta2, t". Output: "s_corrected".
        # (approx. 2 lines)
        # s_corrected["dW" + str(l)] = ...
        # s_corrected["db" + str(l)] = ...
        # YOUR CODE STARTS HERE
        s_corrected['dW'+str(l)]=s['dW'+str(l)]/(1-beta2**t)
        s_corrected['db'+str(l)]=s['db'+str(l)]/(1-beta2**t)
        
        # YOUR CODE ENDS HERE

        # Update parameters. Inputs: "parameters, learning_rate, v_corrected, s_corrected, epsilon". Output: "parameters".
        # (approx. 2 lines)
        # parameters["W" + str(l)] = ...
        # parameters["b" + str(l)] = ...
        # YOUR CODE STARTS HERE
        parameters['W'+str(l)]=parameters['W'+str(l)]-learning_rate*v_corrected['dW'+str(l)]/(np.sqrt(s_corrected['dW'+str(l)])+epsilon)
        parameters['b'+str(l)]=parameters['b'+str(l)]-learning_rate*v_corrected['db'+str(l)]/(np.sqrt(s_corrected['db'+str(l)])+epsilon)
        
        # YOUR CODE ENDS HERE

    return parameters, v, s, v_corrected, s_corrected
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84

上诉的几种优化方法不仅可以加快训练速度，并且还可以一定程度上避免陷入局部最优值（可以想象为一个具有滑雪选手可以冲破一些较小的凹处、山谷，从而达到终点）

学习率下降（Learning Rate Decay）

在算法达到快要最优值得时候，我们会发现优化器会在最优值周围徘徊，因为学习率的设置，我们设置的步长往往太大了，然而初始设置步长很小又会导致速度便面，这个使用我们就可以采用一种很简单的方法使得学习率下降。
$$\begin{gathered} {\alpha }_t=f(t)\ast {\alpha }_0 \\ f是一个单调下降的函数 \end{gathered}$$
有时为了防止学习率降为0，或者经常更新导致的速度问题，$f函数$常常设置为阶段函数。

$\alpha =\frac{1}{1+decay\ast [\frac{epoches}{timeIntervel}]}$

几种优化方法的比较

源数据

• 带预测数据如上（两个对称的月牙数据）
• 模型使用三层全连接神经网络
• 经过4000次迭代
• mini_batch相同

传统梯度下降

$\uparrow$传统梯度下降算法$\uparrow$

带动量的梯度下降

Adam

GD+ LearningRate Decay

Momentum + LearningRate Decay

Adam + LearningRate Decay