Informer时序模型(代码解析)_informer模型代码

代码解析

参考资料

• 建议大家在阅读前有一定Transformer模型基础，可以先看看Transformer论文，论文下载链接
• 阅读Informer时序模型论文，重点关注作者针对Transformer模型做了哪些改进，论文下载链接
• Informer时序模型Github地址，数据没有包含在项目中，需要自行下载，这里提供下载地址 (包含代码文件和数据)

参数设定模块(main_informer)

• 值得注意的是'--model'、'--data'参数需要去掉required参数，否则运行代码可能会报'--model'、'--data'错误
• 修改完参数后运行该模块，保证代码运行不报错的情况下进行debug

参数含义

• 下面是各参数含义（注释）

# 选择模型(去掉required参数，选择informer模型)
parser.add_argument('--model', type=str, default='informer',help='model of experiment, options: [informer, informerstack, informerlight(TBD)]')

# 数据选择(去掉required参数)
parser.add_argument('--data', type=str, default='WTH', help='data')
# 数据上级目录
parser.add_argument('--root_path', type=str, default='./data/', help='root path of the data file')
# 数据名称
parser.add_argument('--data_path', type=str, default='WTH.csv', help='data file')
# 预测类型(多变量预测、单变量预测、多元预测单变量)
parser.add_argument('--features', type=str, default='M', help='forecasting task, options:[M, S, MS]; M:multivariate predict multivariate, S:univariate predict univariate, MS:multivariate predict univariate')
# 数据中要预测的标签列
parser.add_argument('--target', type=str, default='OT', help='target feature in S or MS task')
# 数据重采样(h:小时)
parser.add_argument('--freq', type=str, default='h', help='freq for time features encoding, options:[s:secondly, t:minutely, h:hourly, d:daily, b:business days, w:weekly, m:monthly], you can also use more detailed freq like 15min or 3h')
# 模型保存位置
parser.add_argument('--checkpoints', type=str, default='./checkpoints/', help='location of model checkpoints')

# 输入序列长度
parser.add_argument('--seq_len', type=int, default=96, help='input sequence length of Informer encoder')
# 先验序列长度
parser.add_argument('--label_len', type=int, default=48, help='start token length of Informer decoder')
# 预测序列长度
parser.add_argument('--pred_len', type=int, default=24, help='prediction sequence length')
# Informer decoder input: concat[start token series(label_len), zero padding series(pred_len)]

# 编码器default参数为特征列数
parser.add_argument('--enc_in', type=int, default=7, help='encoder input size')
# 解码器default参数与编码器相同
parser.add_argument('--dec_in', type=int, default=7, help='decoder input size')
parser.add_argument('--c_out', type=int, default=7, help='output size')

# 模型宽度
parser.add_argument('--d_model', type=int, default=512, help='dimension of model')
# 多头注意力机制头数
parser.add_argument('--n_heads', type=int, default=8, help='num of heads')
# 模型中encoder层数
parser.add_argument('--e_layers', type=int, default=2, help='num of encoder layers')
# 模型中decoder层数
parser.add_argument('--d_layers', type=int, default=1, help='num of decoder layers')
# 网络架构循环次数
parser.add_argument('--s_layers', type=str, default='3,2,1', help='num of stack encoder layers')
# 全连接层神经元个数
parser.add_argument('--d_ff', type=int, default=2048, help='dimension of fcn')
# 采样因子数
parser.add_argument('--factor', type=int, default=5, help='probsparse attn factor')
# 1D卷积核
parser.add_argument('--padding', type=int, default=0, help='padding type')
# 是否需要序列长度衰减
parser.add_argument('--distil', action='store_false', help='whether to use distilling in encoder, using this argument means not using distilling', default=True)
# 神经网络正则化操作
parser.add_argument('--dropout', type=float, default=0.05, help='dropout')
# attention计算方式
parser.add_argument('--attn', type=str, default='prob', help='attention used in encoder, options:[prob, full]')
# 时间特征编码方式
parser.add_argument('--embed', type=str, default='timeF', help='time features encoding, options:[timeF, fixed, learned]')
# 激活函数
parser.add_argument('--activation', type=str, default='gelu',help='activation')
# 是否输出attention
parser.add_argument('--output_attention', action='store_true', help='whether to output attention in ecoder')
# 是否需要预测
parser.add_argument('--do_predict', action='store_true', help='whether to predict unseen future data')
parser.add_argument('--mix', action='store_false', help='use mix attention in generative decoder', default=True)
# 数据读取
parser.add_argument('--cols', type=str, nargs='+', help='certain cols from the data files as the input features')
# 多核训练(windows下选择0，否则容易报错)
parser.add_argument('--num_workers', type=int, default=0, help='data loader num workers')
# 训练轮数
parser.add_argument('--itr', type=int, default=2, help='experiments times')
# 训练迭代次数
parser.add_argument('--train_epochs', type=int, default=6, help='train epochs')
# mini-batch大小
parser.add_argument('--batch_size', type=int, default=32, help='batch size of train input data')
# 早停策略
parser.add_argument('--patience', type=int, default=3, help='early stopping patience')
# 学习率
parser.add_argument('--learning_rate', type=float, default=0.0001, help='optimizer learning rate')
parser.add_argument('--des', type=str, default='test',help='exp description')
# loss计算方式
parser.add_argument('--loss', type=str, default='mse',help='loss function')
# 学习率衰减参数
parser.add_argument('--lradj', type=str, default='type1',help='adjust learning rate')
# 是否使用自动混合精度训练
parser.add_argument('--use_amp', action='store_true', help='use automatic mixed precision training', default=False)
# 是否反转输出结果
parser.add_argument('--inverse', action='store_true', help='inverse output data', default=False)

# 是否使用GPU加速训练
parser.add_argument('--use_gpu', type=bool, default=True, help='use gpu')
parser.add_argument('--gpu', type=int, default=0, help='gpu')
# GPU分布式训练
parser.add_argument('--use_multi_gpu', action='store_true', help='use multiple gpus', default=False)
# 多GPU训练
parser.add_argument('--devices', type=str, default='0,1,2,3',help='device ids of multile gpus')

# 取参数值
args = parser.parse_args()
# 获取GPU
args.use_gpu = True if torch.cuda.is_available() and args.use_gpu else False
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

数据文件参数

• 因为用的是笔记本电脑，这里只能用最小的数据集进行试验，也就是下面的WTH数据集

# 数据参数
data_parser = {
    'ETTh1':{'data':'ETTh1.csv','T':'OT','M':[7,7,7],'S':[1,1,1],'MS':[7,7,1]},
    'ETTh2':{'data':'ETTh2.csv','T':'OT','M':[7,7,7],'S':[1,1,1],'MS':[7,7,1]},
    'ETTm1':{'data':'ETTm1.csv','T':'OT','M':[7,7,7],'S':[1,1,1],'MS':[7,7,1]},
    'ETTm2':{'data':'ETTm2.csv','T':'OT','M':[7,7,7],'S':[1,1,1],'MS':[7,7,1]},
    # data：数据文件名，T：标签列，M：预测变量数(如果要预测12个特征，则为[12,12,12]),
    'WTH':{'data':'WTH.csv','T':'WetBulbCelsius','M':[12,12,12],'S':[1,1,1],'MS':[12,12,1]},
    'ECL':{'data':'ECL.csv','T':'MT_320','M':[321,321,321],'S':[1,1,1],'MS':[321,321,1]},
    'Solar':{'data':'solar_AL.csv','T':'POWER_136','M':[137,137,137],'S':[1,1,1],'MS':[137,137,1]},
}
1
2
3
4
5
6
7
8
9
10
11

• 下面是模型训练函数，这里不进行注释了

数据处理模块（data_loader）

• 从main_informer.py文件中exp.train(setting)，train方法进入exp_informer.py文件，在_get_data中找到WTH数据处理方法

data_dict = {
            'ETTh1':Dataset_ETT_hour,
            'ETTh2':Dataset_ETT_hour,
            'ETTm1':Dataset_ETT_minute,
            'ETTm2':Dataset_ETT_minute,
            'WTH':Dataset_Custom,
            'ECL':Dataset_Custom,
            'Solar':Dataset_Custom,
            'custom':Dataset_Custom,}
1
2
3
4
5
6
7
8
9

• 可以看到WTH数据处理方法为Dataset_Custom，我们进入data_loader.py文件，找到Dataset_Custom类
• __init__主要用于传各类参数，这里不过多赘述，主要对__read_data__进行说明

    def __read_data__(self):
        # 数据标准化
        self.scaler = StandardScaler()
        # 利用pandas将数据读入
        df_raw = pd.read_csv(os.path.join(self.root_path,
                                          self.data_path))
        # 如果指定了排除项
        if self.cols:
            cols=self.cols.copy()
            # 移除标签列
            cols.remove(self.target)
        else:
            # 提取数据列名；移除标签列；移除日期列
            cols = list(df_raw.columns); cols.remove(self.target); cols.remove('date')
        # 日期列+特征列+标签列(即：调整列顺序)
        df_raw = df_raw[['date']+cols+[self.target]]

        # 划分训练集
        num_train = int(len(df_raw)*0.7)
        # 划分测试集
        num_test = int(len(df_raw)*0.2)
        # 划分验证集
        num_vali = len(df_raw) - num_train - num_test
        # 计算数据起始点
        border1s = [0, num_train-self.seq_len, len(df_raw)-num_test-self.seq_len]
        border2s = [num_train, num_train+num_vali, len(df_raw)]
        border1 = border1s[self.set_type]
        border2 = border2s[self.set_type]

        # 若预测类型为M(多特征预测多特征)或MS(多特征预测单特征)
        if self.features=='M' or self.features=='MS':
            # 取除日期列的其他所有列
            cols_data = df_raw.columns[1:]
            df_data = df_raw[cols_data]
        # 若预测类型为S(单特征预测单特征)
        elif self.features=='S':
            # 取特征列
            df_data = df_raw[[self.target]]
        # 将数据进行归一化
        if self.scale:
            train_data = df_data[border1s[0]:border2s[0]]
            self.scaler.fit(train_data.values)
            data = self.scaler.transform(df_data.values)
        else:
            data = df_data.values
        # 取日期列
        df_stamp = df_raw[['date']][border1:border2]
        # 利用pandas将数据转换为日期格式
        df_stamp['date'] = pd.to_datetime(df_stamp.date)
        # 构建时间特征
        data_stamp = time_features(df_stamp, timeenc=self.timeenc, freq=self.freq)

        self.data_x = data[border1:border2]
        if self.inverse:
            self.data_y = df_data.values[border1:border2]
        else:
            # 取数据特征列
            self.data_y = data[border1:border2]
        self.data_stamp = data_stamp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

• 需要注意的是time_features函数，用来提取日期特征，比如't':['month','day','weekday','hour','minute']，表示提取月，天，周，小时，分钟。可以打开timefeatures.py
  文件进行查阅
• 同样的，对__getitem__进行说明

    def __getitem__(self, index):
        # 随机取得标签
        s_begin = index
        # 训练区间
        s_end = s_begin + self.seq_len
        # 有标签区间+无标签区间(预测时间步长)
        r_begin = s_end - self.label_len 
        r_end = r_begin + self.label_len + self.pred_len

        # 取训练数据
        seq_x = self.data_x[s_begin:s_end]
        if self.inverse:
            seq_y = np.concatenate([self.data_x[r_begin:r_begin+self.label_len], self.data_y[r_begin+self.label_len:r_end]], 0)
        else:
            # 取有标签区间+无标签区间(预测时间步长)数据
            seq_y = self.data_y[r_begin:r_end]
        # 取训练数据对应时间特征
        seq_x_mark = self.data_stamp[s_begin:s_end]
        # 取有标签区间+无标签区间(预测时间步长)对应时间特征
        seq_y_mark = self.data_stamp[r_begin:r_end]

        return seq_x, seq_y, seq_x_mark, seq_y_mark
    
    def __len__(self):
        # 返回数据长度
        return len(self.data_x) - self.seq_len- self.pred_len + 1

    def inverse_transform(self, data):
        return self.scaler.inverse_transform(data)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

Informer模型架构（model）

• 这里贴上Informer模型论文中的结构图，方便大家对照理解。
  
• K值选取原因与筛选方法
  
• 先进入exp_informer.py文件，train函数中包含有网络架构函数。

    def train(self, setting):
        # 数据加载器
        train_data, train_loader = self._get_data(flag = 'train')
        vali_data, vali_loader = self._get_data(flag = 'val')
        test_data, test_loader = self._get_data(flag = 'test')

        path = os.path.join(self.args.checkpoints, setting)
        if not os.path.exists(path):
            os.makedirs(path)

        # 记录时间
        time_now = time.time()
        # 训练steps
        train_steps = len(train_loader)
        # 早停策略
        early_stopping = EarlyStopping(patience=self.args.patience, verbose=True)

        # 优化器Adam
        model_optim = self._select_optimizer()
        # 损失函数(MSE)
        criterion =  self._select_criterion()

        # 分布式训练(windows一般不推荐)
        if self.args.use_amp:
            scaler = torch.cuda.amp.GradScaler()

        # 训练次数
        for epoch in range(self.args.train_epochs):
            iter_count = 0
            train_loss = []
            
            self.model.train()
            epoch_time = time.time()
            for i, (batch_x,batch_y,batch_x_mark,batch_y_mark) in enumerate(train_loader):
                iter_count += 1
                # 梯度归零
                model_optim.zero_grad()
                # 训练模型(网络架构)
                pred, true = self._process_one_batch(
                    train_data, batch_x, batch_y, batch_x_mark, batch_y_mark)
                # 计算损失
                loss = criterion(pred, true)
                # 加入数组
                train_loss.append(loss.item())

                # 输出信息
                if (i+1) % 100==0:
                    print("\titers: {0}, epoch: {1} | loss: {2:.7f}".format(i + 1, epoch + 1, loss.item()))
                    speed = (time.time()-time_now)/iter_count
                    left_time = speed*((self.args.train_epochs - epoch)*train_steps - i)
                    print('\tspeed: {:.4f}s/iter; left time: {:.4f}s'.format(speed, left_time))
                    iter_count = 0
                    time_now = time.time()
                
                if self.args.use_amp:
                    scaler.scale(loss).backward()
                    scaler.step(model_optim)
                    scaler.update()
                else:
                    # 反向传播
                    loss.backward()
                    # 更新梯度
                    model_optim.step()

            # 打印时间信息
            print("Epoch: {} cost time: {}".format(epoch+1, time.time()-epoch_time))
            train_loss = np.average(train_loss)
            vali_loss = self.vali(vali_data, vali_loader, criterion)
            test_loss = self.vali(test_data, test_loader, criterion)

            # 打印损失信息
            print("Epoch: {0}, Steps: {1} | Train Loss: {2:.7f} Vali Loss: {3:.7f} Test Loss: {4:.7f}".format(
                epoch + 1, train_steps, train_loss, vali_loss, test_loss))
            # 早停策略
            early_stopping(vali_loss, self.model, path)
            if early_stopping.early_stop:
                print("Early stopping")
                break

            adjust_learning_rate(model_optim, epoch+1, self.args)
        # 保存模型
        best_model_path = path+'/'+'checkpoint.pth'
        # 导入模型
        self.model.load_state_dict(torch.load(best_model_path))
        
        return self.model
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86

• 注意模型训练那一块_process_one_batch，进入该方法

        def _process_one_batch(self, dataset_object, batch_x, batch_y, batch_x_mark, batch_y_mark):
        # 将数据集放入GPU中
        batch_x = batch_x.float().to(self.device)
        batch_y = batch_y.float()

        batch_x_mark = batch_x_mark.float().to(self.device)
        batch_y_mark = batch_y_mark.float().to(self.device)

        # decoder输入
        if self.args.padding==0:
            # 创建一个全0数组，维度为batch，预测序列长度，特征数，本例中为[32,24,12]
            dec_inp = torch.zeros([batch_y.shape[0], self.args.pred_len, batch_y.shape[-1]]).float()
        elif self.args.padding==1:
            dec_inp = torch.ones([batch_y.shape[0], self.args.pred_len, batch_y.shape[-1]]).float()
        # 维度变为[32,72,12](72 = 24 + 48)，48是预测中有标签的数据量
        dec_inp = torch.cat([batch_y[:,:self.args.label_len,:], dec_inp], dim=1).float().to(self.device)
        # encoder - decoder
        if self.args.use_amp:
            with torch.cuda.amp.autocast():
                if self.args.output_attention:
                    outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0]
                else:
                    outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)
        else:
            if self.args.output_attention:
                outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0]
            else:
                # 运行到这一步，model中包含了网络架构
                # output维度[batch，预测序列长度，预测特征数]
                outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)
        if self.args.inverse:
            outputs = dataset_object.inverse_transform(outputs)
        # 如果预测类型为多特征预测单特征(取结果最后一列)
        f_dim = -1 if self.args.features=='MS' else 0

        batch_y = batch_y[:,-self.args.pred_len:,f_dim:].to(self.device)

        return outputs, batch_y
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

• 可以看到outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)，model中包含Informer的核心架构（也是最重要的部分）
• 打开model.py文件，找到Informer类，直接看forward

        def forward(self, x_enc, x_mark_enc, x_dec, x_mark_dec, 
                enc_self_mask=None, dec_self_mask=None, dec_enc_mask=None):
        # x_enc[batch,序列长度,特征列],x_mark_enc[batch,序列长度,时间特征列]
        # x_enc.shape:(32,96,12),x_mark_enc.shape:(32,96,4)
        enc_out = self.enc_embedding(x_enc, x_mark_enc)
        # enc_self_mask是数据中需要忽略的样本，本项目中为空
        enc_out, attns = self.encoder(enc_out, attn_mask=enc_self_mask)

        # 解码器embedding操作
        # x_dec维度[batch，有标签+无标签序列长度，特征列](32,72=48+24,12)
        dec_out = self.dec_embedding(x_dec, x_mark_dec)
        # 解码器decoder操作
        dec_out = self.decoder(dec_out, enc_out, x_mask=dec_self_mask, cross_mask=dec_enc_mask)
        # 利用全连接层输出结果512-->12
        dec_out = self.projection(dec_out)
        
        # dec_out = self.end_conv1(dec_out)
        # dec_out = self.end_conv2(dec_out.transpose(2,1)).transpose(1,2)
        if self.output_attention:
            return dec_out[:,-self.pred_len:,:], attns
        else:
            # 截断，只取后面24个需要预测的
            return dec_out[:,-self.pred_len:,:] # [B, L, D]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

编码器Embedding操作

• Embedding操作，在embed.py文件中

class DataEmbedding(nn.Module):
    def __init__(self, c_in, d_model, embed_type='fixed', freq='h', dropout=0.1):
        super(DataEmbedding, self).__init__()

        self.value_embedding = TokenEmbedding(c_in=c_in, d_model=d_model)
        self.position_embedding = PositionalEmbedding(d_model=d_model)
        self.temporal_embedding = TemporalEmbedding(d_model=d_model, embed_type=embed_type, freq=freq) if embed_type!='timeF' else TimeFeatureEmbedding(d_model=d_model, embed_type=embed_type, freq=freq)

        self.dropout = nn.Dropout(p=dropout)

    def forward(self, x, x_mark):
        # 12个特征列利用卷积层映射为512 + position_embedding + 4个时间特征利用全连接层映射为512
        x = self.value_embedding(x) + self.position_embedding(x) + self.temporal_embedding(x_mark)
        # 输出正则化后的embedding
        return self.dropout(x)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Encoder模块

• Encoder模块，在encoder.py文件中

class Encoder(nn.Module):
    def __init__(self, attn_layers, conv_layers=None, norm_layer=None):
        super(Encoder, self).__init__()
        self.attn_layers = nn.ModuleList(attn_layers)
        self.conv_layers = nn.ModuleList(conv_layers) if conv_layers is not None else None
        self.norm = norm_layer

    def forward(self, x, attn_mask=None):
        # x [B, L, D]
        attns = []
        if self.conv_layers is not None:
            for attn_layer, conv_layer in zip(self.attn_layers, self.conv_layers):
                # 遍历注意力架构层
                x, attn = attn_layer(x, attn_mask=attn_mask)
                # 对x做maxpool1d操作，将512-->256
                # 也就是结构中的金字塔，为了加速模型训练提出
                x = conv_layer(x)
                attns.append(attn)
            # # 遍历注意力架构层
            x, attn = self.attn_layers[-1](x, attn_mask=attn_mask)
            attns.append(attn)
        else:
            for attn_layer in self.attn_layers:
                x, attn = attn_layer(x, attn_mask=attn_mask)
                attns.append(attn)

        if self.norm is not None:
            # 执行标准化操作
            x = self.norm(x)

        return x, attns
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

• 进入EncoderLayer类，找到注意力计算架构

class EncoderLayer(nn.Module):
    def __init__(self, attention, d_model, d_ff=None, dropout=0.1, activation="relu"):
        super(EncoderLayer, self).__init__()
        d_ff = d_ff or 4*d_model
        self.attention = attention
        self.conv1 = nn.Conv1d(in_channels=d_model, out_channels=d_ff, kernel_size=1)
        self.conv2 = nn.Conv1d(in_channels=d_ff, out_channels=d_model, kernel_size=1)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)
        self.activation = F.relu if activation == "relu" else F.gelu

    def forward(self, x, attn_mask=None):
        # 传入3个x，分别用于计算Q、K、V
        new_x, attn = self.attention(
            x, x, x,
            attn_mask = attn_mask
        )
        # 残差连接
        x = x + self.dropout(new_x)

        y = x = self.norm1(x)
        y = self.dropout(self.activation(self.conv1(y.transpose(-1,1))))
        y = self.dropout(self.conv2(y).transpose(-1,1))

        return self.norm2(x+y), attn
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

• 注意代码中的new_x, attn = self.attention(x, x, x,attn_mask = attn_mask)

注意力层

• 注意力层在attn.py文件中，找到AttentionLayer类

class AttentionLayer(nn.Module):
    def __init__(self, attention, d_model, n_heads, 
                 d_keys=None, d_values=None, mix=False):
        super(AttentionLayer, self).__init__()

        d_keys = d_keys or (d_model//n_heads)
        d_values = d_values or (d_model//n_heads)

        self.inner_attention = attention
        self.query_projection = nn.Linear(d_model, d_keys * n_heads)
        self.key_projection = nn.Linear(d_model, d_keys * n_heads)
        self.value_projection = nn.Linear(d_model, d_values * n_heads)
        self.out_projection = nn.Linear(d_values * n_heads, d_model)
        self.n_heads = n_heads
        self.mix = mix

    def forward(self, queries, keys, values, attn_mask):
        # 取出batch，序列长度，特征数12(即B=32，L=96，_=12)
        B, L, _ = queries.shape
        # 同样的S=96
        _, S, _ = keys.shape
        # 多头注意力机制，这里为8
        H = self.n_heads

        # 通过全连接层将特征512-->512，映射到Q,K,V
        # 512是在进行Embedding后特征数量
        # 同时维度变为(batch，序列长度，多头注意力机制，自动计算)
        queries = self.query_projection(queries).view(B, L, H, -1)
        keys = self.key_projection(keys).view(B, S, H, -1)
        values = self.value_projection(values).view(B, S, H, -1)

        # 计算注意力
        out, attn = self.inner_attention(
            queries,
            keys,
            values,
            attn_mask
        )
        if self.mix:
            out = out.transpose(2,1).contiguous()
        # 维度batch，序列长度，自动计算值
        out = out.view(B, L, -1)
        # 连接全连接512-->512
        return self.out_projection(out), attn
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

• 注意代码中self.inner_attention，跳转到ProbAttention类
• 其中_prob_QK用于选取Q、K是非常模型核心，要认真读，贴一下公式：
  M ‾ ( q i , k ) = m a x j { q i k j T d } − 1 L k ∑ j = 1 L k q i k j T d \overline{M}_{(q_i,k)} = \mathop{max} \limits_{j} \{\frac{q_ik_j^{T}}{\sqrt{d}}\}-\frac{1}{L_{k}}\sum^{L_k}_{j=1}\frac{q_ik_j^{T}}{\sqrt{d}} M(qi​,k)​=jmax​{d ​qi​kjT​​}−Lk​1​j=1∑Lk​​d ​qi​kjT​​
• _get_initial_context计算初始V值，_update_context更新重要Q的V值

class ProbAttention(nn.Module):
    def __init__(self, mask_flag=True, factor=5, scale=None, attention_dropout=0.1, output_attention=False):
        super(ProbAttention, self).__init__()
        self.factor = factor
        self.scale = scale
        self.mask_flag = mask_flag
        self.output_attention = output_attention
        self.dropout = nn.Dropout(attention_dropout)

    def _prob_QK(self, Q, K, sample_k, n_top): # n_top: c*ln(L_q)
        # 维度[batch，头数，序列长度，自动计算值]
        B, H, L_K, E = K.shape
        _, _, L_Q, _ = Q.shape

        # 添加一个维度，相当于复制维度，当前维度为[batch，头数，序列长度，序列长度，自动计算值]
        K_expand = K.unsqueeze(-3).expand(B, H, L_Q, L_K, E)
        # 随机取样，取值范围0~96，取样维度为[序列长度，25]
        index_sample = torch.randint(L_K, (L_Q, sample_k)) # real U = U_part(factor*ln(L_k))*L_q
        # 96个Q与25个K做计算，维度为[batch，头数，Q个数，K个数，自动计算值]
        K_sample = K_expand[:, :, torch.arange(L_Q).unsqueeze(1), index_sample, :]
        # 矩阵重组，维度为[batch，头数，Q个数，K个数]
        Q_K_sample = torch.matmul(Q.unsqueeze(-2), K_sample.transpose(-2, -1)).squeeze(-2)

        # 分别取到96个Q中每一个Q跟K关系最大的值
        M = Q_K_sample.max(-1)[0] - torch.div(Q_K_sample.sum(-1), L_K)
        # 在96个Q中选出前25个
        M_top = M.topk(n_top, sorted=False)[1]

        # 取出Q特征，维度为[batch，头数，Q个数，自动计算值]
        Q_reduce = Q[torch.arange(B)[:, None, None],
                     torch.arange(H)[None, :, None],
                     M_top, :] # factor*ln(L_q)
        Q_K = torch.matmul(Q_reduce, K.transpose(-2, -1)) # factor*ln(L_q)*L_k

        return Q_K, M_top

    # 计算V值
    def _get_initial_context(self, V, L_Q):
        # 取出batch，头数，序列长度，自动计算值
        B, H, L_V, D = V.shape
        if not self.mask_flag:
            # 对25个Q以外其他Q的V值，使用平均值(让其继续平庸下去)
            V_sum = V.mean(dim=-2)
            # 先把96个V全部使用平均值代替
            contex = V_sum.unsqueeze(-2).expand(B, H, L_Q, V_sum.shape[-1]).clone()
        else: # use mask
            assert(L_Q == L_V) # requires that L_Q == L_V, i.e. for self-attention only
            contex = V.cumsum(dim=-2)
        return contex

    # 更新25个V值
    def _update_context(self, context_in, V, scores, index, L_Q, attn_mask):
        B, H, L_V, D = V.shape

        if self.mask_flag:
            attn_mask = ProbMask(B, H, L_Q, index, scores, device=V.device)
            scores.masked_fill_(attn_mask.mask, -np.inf)

        # 计算softmax值
        attn = torch.softmax(scores, dim=-1)

        # 对25个Q更新V，其他仍然为平均值
        context_in[torch.arange(B)[:, None, None],
                   torch.arange(H)[None, :, None],
                   index, :] = torch.matmul(attn, V).type_as(context_in)
        if self.output_attention:
            attns = (torch.ones([B, H, L_V, L_V])/L_V).type_as(attn).to(attn.device)
            attns[torch.arange(B)[:, None, None], torch.arange(H)[None, :, None], index, :] = attn
            return (context_in, attns)
        else:
            return (context_in, None)

    def forward(self, queries, keys, values, attn_mask):
        # 取出batch，序列长度，头数，自动计算值
        B, L_Q, H, D = queries.shape
        # 取出序列长度(相当于96个Q，96个K)
        _, L_K, _, _ = keys.shape

        # 维度转置操作，维度变为(batch，头数，序列长度，自动计算值)
        queries = queries.transpose(2,1)
        keys = keys.transpose(2,1)
        values = values.transpose(2,1)

        # 选取K的个数，模型核心，用于加速
        # factor为常数5，可以自行修改，其值越大，计算成本越高
        U_part = self.factor * np.ceil(np.log(L_K)).astype('int').item() # c*ln(L_k)
        u = self.factor * np.ceil(np.log(L_Q)).astype('int').item() # c*ln(L_q) 

        U_part = U_part if U_part<L_K else L_K
        u = u if u<L_Q else L_Q

        # Q、K选择标准
        scores_top, index = self._prob_QK(queries, keys, sample_k=U_part, n_top=u) 

        # 削弱维度对结果的影响
        scale = self.scale or 1./sqrt(D)
        if scale is not None:
            scores_top = scores_top * scale
        # 初始化V值
        context = self._get_initial_context(values, L_Q)
        # 更新25个Q的V值
        context, attn = self._update_context(context, values, scores_top, index, L_Q, attn_mask)
        
        return context.transpose(2,1).contiguous(), attn
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104

解码器Embedding操作

• 解码器的Embedding操作与编码器Embedding操作完全一致，只不过需要注意传入数组维度x_dec维度[batch，有标签+无标签序列长度，特征列](32,72=48+24,12)

Decoder模块

• 在decoder.py文件中找到Decoder类

class Decoder(nn.Module):
    def __init__(self, layers, norm_layer=None):
        super(Decoder, self).__init__()
        self.layers = nn.ModuleList(layers)
        self.norm = norm_layer

    def forward(self, x, cross, x_mask=None, cross_mask=None):
        for layer in self.layers:
            # 遍历层，需要注意的是该处计算自注意力，也就是self-attention
            # 72个Q，72个K，重复编码器中的decoder操作
            x = layer(x, cross, x_mask=x_mask, cross_mask=cross_mask)

        if self.norm is not None:
            x = self.norm(x)

        return x
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

• 代码中的layer层定义在该文件中，找到DecoderLayer类

class DecoderLayer(nn.Module):
    def __init__(self, self_attention, cross_attention, d_model, d_ff=None,
                 dropout=0.1, activation="relu"):
        super(DecoderLayer, self).__init__()
        d_ff = d_ff or 4*d_model
        self.self_attention = self_attention
        self.cross_attention = cross_attention
        self.conv1 = nn.Conv1d(in_channels=d_model, out_channels=d_ff, kernel_size=1)
        self.conv2 = nn.Conv1d(in_channels=d_ff, out_channels=d_model, kernel_size=1)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.norm3 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)
        self.activation = F.relu if activation == "relu" else F.gelu

    def forward(self, x, cross, x_mask=None, cross_mask=None):
        x = x + self.dropout(self.self_attention(
            # Decoder(序列长度为72)中的Q,K,V
            x, x, x,
            attn_mask=x_mask
        )[0])
        x = self.norm1(x)

        # cross_attention，在Encoder与Decoder间计算attention
        # 结构图中Encoder与Decoder连接线部分
        x = x + self.dropout(self.cross_attention(
            # x为Q，cross是Encoder中的K，ross是Encoder中的V
            x, cross, cross,
            attn_mask=cross_mask
        )[0])

        y = x = self.norm2(x)
        y = self.dropout(self.activation(self.conv1(y.transpose(-1,1))))
        y = self.dropout(self.conv2(y).transpose(-1,1))

        return self.norm3(x+y)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

• 这里需要注意，在Decoder板块中有两个和Encoder不一样的操作，即self-attention和corss-attention。
• self-attention是自注意力机制，比如在本例中带标签长度+预测长度为72，那么会在72个Q与72个K中进行与在Decoder中同样的筛选、更新操作
• cross-attention是交叉注意力机制，选值分别为Decoder中的Q，Encoder中的K，Encoder中的V进行与在Decoder中同样的筛选、更新操作
• 到这里model.py中的模型板块结束，回到exp_informer.py文件中的_process_one_batch，通过output变量得到预测值
• 回到exp_informer.py文件中的train函数，得到预测值与真实值，继续接下来的梯度、学习率更新，计算损失函数

结果展示

• 我用自己笔记本电脑跑的，因为没有GPU，所以耗费大概7小时（注：模型文件我放在上面的下载链接中了，包括带注释的代码文件）

train 24425
val 3485
test 6989
	iters: 100, epoch: 1 | loss: 0.4753647
	speed: 5.8926s/iter; left time: 26393.0550s
	iters: 200, epoch: 1 | loss: 0.3887450
	speed: 5.6093s/iter; left time: 24563.0934s
	iters: 300, epoch: 1 | loss: 0.3397639
	speed: 5.6881s/iter; left time: 24339.4008s
	iters: 400, epoch: 1 | loss: 0.3773919
	speed: 5.5947s/iter; left time: 23380.1260s
	iters: 500, epoch: 1 | loss: 0.3424160
	speed: 5.8912s/iter; left time: 24030.1962s
	iters: 600, epoch: 1 | loss: 0.3589063
	speed: 6.0372s/iter; left time: 24021.9204s
	iters: 700, epoch: 1 | loss: 0.3522923
	speed: 5.2896s/iter; left time: 20518.3927s
Epoch: 1 cost time: 4319.718204259872
Epoch: 1, Steps: 763 | Train Loss: 0.3825711 Vali Loss: 0.4002144 Test Loss: 0.3138740
Validation loss decreased (inf --> 0.400214).  Saving model ...
Updating learning rate to 0.0001
	iters: 100, epoch: 2 | loss: 0.3452260
	speed: 12.8896s/iter; left time: 47897.7932s
	iters: 200, epoch: 2 | loss: 0.2782844
	speed: 4.7867s/iter; left time: 17308.6180s
	iters: 300, epoch: 2 | loss: 0.2653053
	speed: 4.7938s/iter; left time: 16855.0160s
	iters: 400, epoch: 2 | loss: 0.3157508
	speed: 4.7083s/iter; left time: 16083.5403s
	iters: 500, epoch: 2 | loss: 0.3046930
	speed: 4.7699s/iter; left time: 15816.8855s
	iters: 600, epoch: 2 | loss: 0.2360453
	speed: 4.8311s/iter; left time: 15536.9307s
	iters: 700, epoch: 2 | loss: 0.2668953
	speed: 4.7713s/iter; left time: 14867.4169s
Epoch: 2 cost time: 3644.3840498924255
Epoch: 2, Steps: 763 | Train Loss: 0.2945577 Vali Loss: 0.3963071 Test Loss: 0.3274192
Validation loss decreased (0.400214 --> 0.396307).  Saving model ...
Updating learning rate to 5e-05
	iters: 100, epoch: 3 | loss: 0.2556470
	speed: 12.6569s/iter; left time: 37375.7115s
	iters: 200, epoch: 3 | loss: 0.2456252
	speed: 4.7655s/iter; left time: 13596.0810s
	iters: 300, epoch: 3 | loss: 0.2562804
	speed: 4.7336s/iter; left time: 13031.4940s
	iters: 400, epoch: 3 | loss: 0.2049552
	speed: 4.7622s/iter; left time: 12634.1883s
	iters: 500, epoch: 3 | loss: 0.2604980
	speed: 4.7524s/iter; left time: 12132.7789s
	iters: 600, epoch: 3 | loss: 0.2539216
	speed: 4.7413s/iter; left time: 11630.3915s
	iters: 700, epoch: 3 | loss: 0.2098076
	speed: 4.7394s/iter; left time: 11151.7416s
Epoch: 3 cost time: 3628.159082174301
Epoch: 3, Steps: 763 | Train Loss: 0.2486252 Vali Loss: 0.4155475 Test Loss: 0.3301197
EarlyStopping counter: 1 out of 3
Updating learning rate to 2.5e-05
	iters: 100, epoch: 4 | loss: 0.2175551
	speed: 12.6253s/iter; left time: 27649.4546s
	iters: 200, epoch: 4 | loss: 0.2459734
	speed: 4.7335s/iter; left time: 9892.9213s
	iters: 300, epoch: 4 | loss: 0.2354426
	speed: 4.7546s/iter; left time: 9461.6300s
	iters: 400, epoch: 4 | loss: 0.2267139
	speed: 4.7719s/iter; left time: 9018.9749s
	iters: 500, epoch: 4 | loss: 0.2379844
	speed: 4.8038s/iter; left time: 8598.7446s
	iters: 600, epoch: 4 | loss: 0.2434178
	speed: 4.7608s/iter; left time: 8045.7994s
	iters: 700, epoch: 4 | loss: 0.2231207
	speed: 4.7765s/iter; left time: 7594.6586s
Epoch: 4 cost time: 3649.547614812851
Epoch: 4, Steps: 763 | Train Loss: 0.2224283 Vali Loss: 0.4230270 Test Loss: 0.3334258
EarlyStopping counter: 2 out of 3
Updating learning rate to 1.25e-05
	iters: 100, epoch: 5 | loss: 0.1837259
	speed: 12.7564s/iter; left time: 18203.3974s
	iters: 200, epoch: 5 | loss: 0.1708880
	speed: 4.7804s/iter; left time: 6343.6200s
	iters: 300, epoch: 5 | loss: 0.2529005
	speed: 4.7426s/iter; left time: 5819.1675s
	iters: 400, epoch: 5 | loss: 0.2434390
	speed: 4.7388s/iter; left time: 5340.6568s
	iters: 500, epoch: 5 | loss: 0.2078404
	speed: 4.7515s/iter; left time: 4879.7921s
	iters: 600, epoch: 5 | loss: 0.2372987
	speed: 4.7986s/iter; left time: 4448.2748s
	iters: 700, epoch: 5 | loss: 0.2022571
	speed: 4.7718s/iter; left time: 3946.2739s
Epoch: 5 cost time: 3636.7107157707214
Epoch: 5, Steps: 763 | Train Loss: 0.2088229 Vali Loss: 0.4305894 Test Loss: 0.3341273
EarlyStopping counter: 3 out of 3
Early stopping
>>>>>>>testing : informer_WTH_ftM_sl96_ll48_pl24_dm512_nh8_el2_dl1_df2048_atprob_fc5_ebtimeF_dtTrue_mxTrue_test_0<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
test 6989
test shape: (218, 32, 24, 12) (218, 32, 24, 12)
test shape: (6976, 24, 12) (6976, 24, 12)
mse:0.3277873396873474, mae:0.3727897107601166
Use CPU
>>>>>>>start training : informer_WTH_ftM_sl96_ll48_pl24_dm512_nh8_el2_dl1_df2048_atprob_fc5_ebtimeF_dtTrue_mxTrue_test_1>>>>>>>>>>>>>>>>>>>>>>>>>>
train 24425
val 3485
test 6989
	iters: 100, epoch: 1 | loss: 0.4508476
	speed: 4.7396s/iter; left time: 21228.7904s
	iters: 200, epoch: 1 | loss: 0.3859568
	speed: 4.7742s/iter; left time: 20906.0895s
	iters: 300, epoch: 1 | loss: 0.3749838
	speed: 4.7690s/iter; left time: 20406.5500s
	iters: 400, epoch: 1 | loss: 0.3673764
	speed: 4.8070s/iter; left time: 20088.4627s
	iters: 500, epoch: 1 | loss: 0.3068828
	speed: 4.7643s/iter; left time: 19433.6961s
	iters: 600, epoch: 1 | loss: 0.4173551
	speed: 4.7621s/iter; left time: 18948.4516s
	iters: 700, epoch: 1 | loss: 0.2720438
	speed: 4.7609s/iter; left time: 18467.4719s
Epoch: 1 cost time: 3639.997560977936
Epoch: 1, Steps: 763 | Train Loss: 0.3788956 Vali Loss: 0.3947107 Test Loss: 0.3116618
Validation loss decreased (inf --> 0.394711).  Saving model ...
Updating learning rate to 0.0001
	iters: 100, epoch: 2 | loss: 0.3547252
	speed: 12.6113s/iter; left time: 46863.7093s
	iters: 200, epoch: 2 | loss: 0.3236437
	speed: 4.7504s/iter; left time: 17177.4475s
	iters: 300, epoch: 2 | loss: 0.2898968
	speed: 4.7720s/iter; left time: 16778.2666s
	iters: 400, epoch: 2 | loss: 0.3107039
	speed: 4.7412s/iter; left time: 16195.8892s
	iters: 500, epoch: 2 | loss: 0.2816701
	speed: 4.7244s/iter; left time: 15666.2476s
	iters: 600, epoch: 2 | loss: 0.2226012
	speed: 4.7348s/iter; left time: 15227.0618s
	iters: 700, epoch: 2 | loss: 0.2239729
	speed: 4.8806s/iter; left time: 15208.0025s
Epoch: 2 cost time: 3635.6160113811493
Epoch: 2, Steps: 763 | Train Loss: 0.2962583 Vali Loss: 0.4018708 Test Loss: 0.3213752
EarlyStopping counter: 1 out of 3
Updating learning rate to 5e-05
	iters: 100, epoch: 3 | loss: 0.2407307
	speed: 12.5584s/iter; left time: 37084.8281s
	iters: 200, epoch: 3 | loss: 0.2294409
	speed: 5.1105s/iter; left time: 14580.3263s
	iters: 300, epoch: 3 | loss: 0.3180184
	speed: 5.9484s/iter; left time: 16376.0364s
	iters: 400, epoch: 3 | loss: 0.2101320
	speed: 5.7987s/iter; left time: 15384.0189s
	iters: 500, epoch: 3 | loss: 0.2701742
	speed: 5.5463s/iter; left time: 14159.6749s
	iters: 600, epoch: 3 | loss: 0.2391748
	speed: 4.8338s/iter; left time: 11857.4335s
	iters: 700, epoch: 3 | loss: 0.2280931
	speed: 4.7718s/iter; left time: 11228.1147s
Epoch: 3 cost time: 3975.2745430469513
Epoch: 3, Steps: 763 | Train Loss: 0.2494072 Vali Loss: 0.4189631 Test Loss: 0.3308771
EarlyStopping counter: 2 out of 3
Updating learning rate to 2.5e-05
	iters: 100, epoch: 4 | loss: 0.2260314
	speed: 12.7037s/iter; left time: 27821.0994s
	iters: 200, epoch: 4 | loss: 0.2191769
	speed: 4.7906s/iter; left time: 10012.3575s
	iters: 300, epoch: 4 | loss: 0.2044496
	speed: 4.7498s/iter; left time: 9452.0362s
	iters: 400, epoch: 4 | loss: 0.2167130
	speed: 4.7545s/iter; left time: 8985.9758s
	iters: 500, epoch: 4 | loss: 0.2340788
	speed: 4.7329s/iter; left time: 8471.8863s
	iters: 600, epoch: 4 | loss: 0.2137127
	speed: 4.7037s/iter; left time: 7949.1748s
	iters: 700, epoch: 4 | loss: 0.1899967
	speed: 4.7049s/iter; left time: 7480.8388s
Epoch: 4 cost time: 3624.2080821990967
Epoch: 4, Steps: 763 | Train Loss: 0.2222918 Vali Loss: 0.4390603 Test Loss: 0.3350959
EarlyStopping counter: 3 out of 3
Early stopping
>>>>>>>testing : informer_WTH_ftM_sl96_ll48_pl24_dm512_nh8_el2_dl1_df2048_atprob_fc5_ebtimeF_dtTrue_mxTrue_test_1<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
test 6989
test shape: (218, 32, 24, 12) (218, 32, 24, 12)
test shape: (6976, 24, 12) (6976, 24, 12)
mse:0.3116863965988159, mae:0.36840054392814636
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180

• 跑完以后项目文件中会生成两个文件夹，checkpoints文件夹中存放模型文件，后缀名为.pht；results文件夹中有3个文件，pred.npy为预测值，true.npy为真实值
• 作者在GitHub上留下了关于预测的具体方法，这里因为篇幅原因就不继续写了，可以看后续Informer时序模型(自定义项目)