全连接与卷积神经网络反向传播公式推导_全连接、卷积、循环神经网络的反向传播推导

作者：Cpp五条 | 2024-05-28 01:18:21

踩

全连接、卷积、循环神经网络的反向传播推导

全连接与卷积神经网络反向传播公式推导

全连接网络反向传播公式

BP四项基本原则：

\begin{aligned} δ_{i}^{(L)} & = ▽_{y_{i}} C o s t \cdot σ^{'} (l o g i t_{i}^{(L)}) \\ δ_{i}^{(l)} & = \sum_{j} δ_{j}^{(l + 1)} w_{j i}^{(l + 1)} σ^{'} (l o g i t_{i}^{(l)}) \\ \frac{\partial C o s t}{\partial b i a s_{i}^{(l)}} & = δ_{i}^{(l)} \\ \frac{\partial C o s t}{\partial w_{i j}^{(l)}} & = δ_{i}^{(l)} h_{j}^{(l - 1)} \end{aligned}

$\begin{aligned} \delta_i^{(L)} &= \bigtriangledown_{y_i} Cost \cdot \sigma'(logit_i^{(L)}) \\ \delta_i^{(l)} &= \sum_j \delta_j^{(l+1)} w_{ji}^{(l+1)} \sigma'(logit_i^{(l)}) \\ \frac{\partial Cost}{\partial bias_i^{(l)}} &= \delta_i^{(l)} \\ \frac{\partial Cost}{\partial w_{ij}^{(l)}} &= \delta_i^{(l)} h_j^{(l-1)} \end{aligned}$

δ_{i}^{(L)} δ_{i}^{(l)} \frac{\partial C o s t}{\partial b i a s _{i}^{(l)}} \frac{\partial C o s t}{\partial w _{i j}^{(l)}} = ▽_{y_{i}} C o s t \cdot σ^{'} (l o g i t_{i}^{(L)}) = j \sum δ_{j}^{(l + 1)} w_{j i}^{(l + 1)} σ^{'} (l o g i t_{i}^{(l)}) = δ_{i}^{(l)} = δ_{i}^{(l)} h_{j}^{(l - 1)}

其中， $(l)$ 表示第 $l$ 层，一共有L层， $i, j$ 表示当前层神经元的序号。

反向传播公式的目的主要是得到： $\frac{\partial Cost}{\partial bias_i^{(l)}}$ 和 $\frac{\partial Cost}{\partial w_{ij}^{(l)}}$ 。

在推导的过程中

\begin{aligned} \frac{\partial C o s t}{\partial b i a s_{i}^{(l)}} & = \frac{\partial C o s t}{\partial l o g i t_{i}^{(l)}} \cdot \frac{\partial l o g i t_{i}^{(l)}}{\partial b i a s_{i}^{(l)}} \\ \frac{\partial C o s t}{\partial w_{i j}^{(l)}} & = \frac{\partial C o s t}{\partial l o g i t_{i}^{(l)}} \cdot \frac{\partial l o g i t_{i}^{(l)}}{\partial w_{i j}^{(l)}} \end{aligned}

$\begin{aligned} \frac{\partial Cost}{\partial bias_i^{(l)}} &= \frac{\partial Cost}{\partial logit_i^{(l)}} \cdot \frac{\partial logit_i^{(l)}}{\partial bias_i^{(l)}} \\ \frac{\partial Cost}{\partial w_{ij}^{(l)}} &= \frac{\partial Cost}{\partial logit_i^{(l)}} \cdot \frac{\partial logit_i^{(l)}}{\partial w_{ij}^{(l)}} \end{aligned}$

\frac{\partial C o s t}{\partial b i a s _{i}^{(l)}} \frac{\partial C o s t}{\partial w _{i j}^{(l)}} = \frac{\partial C o s t}{\partial l o g i t _{i}^{(l)}} \cdot \frac{\partial l o g i t _{i}^{(l)}}{\partial b i a s _{i}^{(l)}} = \frac{\partial C o s t}{\partial l o g i t _{i}^{(l)}} \cdot \frac{\partial l o g i t _{i}^{(l)}}{\partial w _{i j}^{(l)}}

会发现都要用到 $\frac{\partial Cost}{\partial logit_i^{(l)}}$ 。

而

$logit_i^{(l)} = w_{ij}^{(l)} h_j^{(l)} + \sum_{k\ne j} w_{ik}^{(l)} h_{k}^{(l)} + bias_i^{(l)}$

所以

\begin{aligned} \frac{\partial l o g i t_{i}^{(l)}}{\partial b i a s_{i}^{(l)}} & = 1 \\ \frac{\partial l o g i t_{i}^{(l)}}{\partial w_{i j}^{(l)}} & = h_{j}^{(l)} \end{aligned}

$\begin{aligned} \frac{\partial logit_i^{(l)}}{\partial bias_i^{(l)}} &= 1 \\ \frac{\partial logit_i^{(l)}}{\partial w_{ij}^{(l)}} &= h_j^{(l)} \end{aligned}$

\frac{\partial l o g i t _{i}^{(l)}}{\partial b i a s _{i}^{(l)}} \frac{\partial l o g i t _{i}^{(l)}}{\partial w _{i j}^{(l)}} = 1 = h_{j}^{(l)}

那接下来的问题就只有求 $\frac{\partial Cost}{\partial logit_i^{(l)}}$ 了，求它可以用递推法：

为公式看起来简洁，我们把 $\frac{\partial Cost}{\partial logit_i^{(l)}}$ 记为 $\delta_i^{(l)}$ ，那么

$\delta_i^{(l)} = \frac{\partial Cost}{\partial logit_i^{(l)}} = \sum_j \frac{\partial Cost}{\partial logit_j^{(l+1)}} \cdot \frac{\partial logit_j^{(l+1)}}{\partial logit_i^{(l)}} = \sum_j \delta_j^{(l+1)} \cdot \frac{\partial logit_j^{(l+1)}}{\partial logit_i^{(l)}}$

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/Cpp五条/article/detail/634985