Pytorch（二） —— 激活函数、损失函数及其梯度_pytorch 激活函数的梯度

1.激活函数

1.1 Sigmoid / Logistic

$$\begin{gathered} \delta (x)=\frac{1}{1+e^{-x}} \\ {\delta }^{\prime }(x)=\delta (1-\delta ) \end{gathered}$$

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.sigmoid(x)
plt.plot(x,y)
plt.show()
1
2
3
4
5
6

1.2 Tanh

$$\begin{gathered} tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}} \\ \frac{\partial tanh(x)}{\partial x}=1-tan{h}^2(x) \end{gathered}$$

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.tanh(x)
plt.plot(x,y)
plt.show()
1
2
3
4
5
6

1.3 ReLU

$$f(x)=max(0,x)$$

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.relu(x)
plt.plot(x,y)
plt.show()
1
2
3
4
5
6

1.4 Softmax

p i = e a i ∑ k = 1 N e a k ∂ p i ∂ a j = { p i ( 1 − p j ) i = j − p i p j i ≠ j p_i=\frac{e^{a_i}}{\sum_{k=1}^N{e^{a_k}}}\\ \frac{\partial p_i}{\partial a_j}=\left\{

$$\begin{array}{lc} p_i(1-p_j) & i=j \\ -p_ip_j&i\neq j\\ \end{array}$$ \right. pi​=∑k=1N​eak​eai​​∂aj​∂pi​​={pi​(1−pj​)−pi​pj​​i=ji​=j​

import torch.nn.functional as F
logits = torch.rand(10)
prob = F.softmax(logits,dim=0)
print(prob)
1
2
3
4

tensor([0.1024, 0.0617, 0.1133, 0.1544, 0.1184, 0.0735, 0.0590, 0.1036, 0.0861,
        0.1275])
1
2

2.损失函数

2.1 MSE

import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
mse = F.mse_loss(y,x@w)
print(mse)
1
2
3
4
5
6

tensor(238.5115)
1

2.2 CorssEntorpy

import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,10)
y = torch.randint(0,9,[100])
entropy = F.cross_entropy(x@w,y)
print(entropy)
1
2
3
4
5
6

tensor(3.6413)
1

3. 求导和反向传播

3.1 求导

• Tensor.requires_grad_()
• torch.autograd.grad()

import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
w.requires_grad_()
mse = F.mse_loss(x@w,y)
grads = torch.autograd.grad(mse,[w])
print(grads[0].shape)
1
2
3
4
5
6
7
8
9

torch.Size([64, 1])
1

3.2 反向传播

• Tensor.backward()

import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,10)
w.requires_grad_()
y = torch.randint(0,9,[100,])
entropy = F.cross_entropy(x@w,y)
entropy.backward()
w.grad.shape
1
2
3
4
5
6
7
8
9

torch.Size([64, 10])
1

by CyrusMay 2022 06 28

人生 只是 须臾的刹那
人间 只是 天地的夹缝
——————五月天（因为你 所以我）——————