CrossEntropyLoss 和NLLLoss的关系

交叉熵损失在做一件什么事?

看公式:

x是预测(不需要softmax归一化),y是label, N是batch维度的数量,交叉熵损失,干了三件事.

1. 对输入在类别维度求softmax

2. 多softmax后的数,求log

3. 对(样本数, 类别数)为shape的tensor计算NLLLoss.

其中,NLLloss做的就是log取负, 和one-hot编码点乘.相加得到最终的总损失,因为reduction默认为mean,所以除以样本数.看以下代码.

代码实现

# 
 
import torch
import torch.nn as nn
 
#
# cross entropy loss = softmax + log + nllloss
# 先softmax, 再 log,
 
# 初始化 input_ 和 target
input_ = torch.randn(3,3)
target = torch.tensor([0,2,1])
mask = torch.zeros(3,3)
mask[0,0] = 1
mask[1,2] = 1
mask[2,1] = 1
 
# 1.0 输入softmax
sft_ = nn.Softmax(dim = -1)(input_)
 
# 2.0 log
log_ = torch.log(sft_)
 
# 3.0 nllloss
loss = nn.NLLLoss()
 
print("split loss")
print(loss(log_, target))
 
# 4.0 crossentropy
print("ce loss")
loss = nn.CrossEntropyLoss()
print(loss(input_, target))
 
print("manual loss")
neg_log = 0 - log_
print(torch.sum(mask *neg_log ) / 3)
 
# ----------输出--------------
>> loss_function python crossEntropyLoss.py
>> split loss
>> tensor(1.2294)
>> ce loss
>> tensor(1.2294)
>> manual loss
>> tensor(1.2294)