- 1哈工大pyltp的安装教程_哈工大ltp安装教程
- 2Spatial-Temporal Multi-Cue Network for Continuous Sign Language Recognition_distilling cross-temporal contexts for continuous
- 3Google DeepMind 大语言模型中的长形态事实性
- 4【Linux】线程同步{死锁/线程同步相关接口/由浅入深理解线程同步}
- 5使用ExLlamaV2在消费级GPU上运行Llama2 70B_valueerror: found modules on cpu/disk. using exlla
- 6tensorflow2.x RNN进行情感分析_xrnn
- 7自然语言处理相关书籍及其他资源_关于hmm的书
- 8python pandas to_sql 的用法_df.to_sql('莱芜4月份天气', engine, index=false)
- 9OpenStack集成Docker
- 10多模态连续维度情感识别
GPT和GPT2结构的区别_gpt结构图
赞
踩
GPT1结构图如下所示：
GPT2结构图如下：
注意，GPT2的最后一个LayerNorm在24个transformers或是12个transformers结构之后添加的，
这里layernormalization放在前面类似于预激活函数的设定，在另外一篇文章Identity mappings in deep
residual networks.
这里在Idenity mapping in deep residual networks之中有推导过程，这里简单的写一下
原始残差单元计算公式如下：
y
l
=
h
(
x
l
)
+
F
(
x
l
,
W
l
)
(
1
)
y_{l} = h(x_{l})+F(x_{l},W_{l})(1)
yl=h(xl)+F(xl,Wl)(1)
x
l
+
1
=
f
(
y
l
)
(
2
)
x_{l+1} = f(y_{l}) (2)
xl+1=f(yl)(2)
l为第l个单元输入的特征
其中F代表残差函数，我们假设函数h为恒等变换，则将(2)代入(1)中可以获得
x
l
+
1
=
x
l
+
∑
i
=
1
L
−
1
F
(
x
l
,
W
l
)
(
3
)
x_{l+1} = x_{l}+ \sum\limits_{i=1}^{L-1}F(x_{l},W_{l}) (3)
xl+1=xl+i=1∑L−1F(xl,Wl)(3)
通过反向传播的链式法则进行求导，可以得到结果
∂
E
∂
x
L
=
∂
E
∂
x
L
∗
∂
x
L
∂
x
l
=
∂
E
∂
x
L
(
1
+
∂
∂
x
l
∑
i
=
1
L
−
1
F
(
x
i
,
W
i
)
)
\frac{\partial E}{\partial x_{L}} = \frac{\partial E}{\partial x_{L}} * \frac{\partial x_{L}}{\partial x_{l}} = \frac{\partial E}{\partial x_{L}}(1+\frac{\partial }{\partial x_{l}}\sum \limits_{i=1}^{L-1}F(x_{i},W_{i}))
∂xL∂E=∂xL∂E∗∂xl∂xL=∂xL∂E(1+∂xl∂i=1∑L−1F(xi,Wi))
如果说这里的
h
(
x
l
)
h(x_{l})
h(xl)不为恒等函数的情况下
∂
E
∂
x
L
=
∂
E
∂
x
L
∗
∂
x
L
∂
x
l
=
∂
E
∂
x
L
(
∑
i
=
1
L
−
1
x
i
+
∂
∂
x
l
∑
i
=
1
L
−
1
F
(
x
i
,
W
i
)
)
\frac{\partial E}{\partial x_{L}} = \frac{\partial E}{\partial x_{L}} * \frac{\partial x_{L}}{\partial x_{l}} = \frac{\partial E}{\partial x_{L}}( \sum\limits_{i=1}^{L-1}x_{i}+\frac{\partial}{\partial x_{l}}\sum \limits_{i=1}^{L-1}F(x_{i},W_{i}))
∂xL∂E=∂xL∂E∗∂xl∂xL=∂xL∂E(i=1∑L−1xi+∂xl∂i=1∑L−1F(xi,Wi))
