devbox
我家小花儿
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Transformer for Time Series Prediction完整代码_=时间序列预测 transformer源代码

作者：我家小花儿 | 2024-04-23 15:07:12

赞

踩

=时间序列预测 transformer源代码

时序预测小白，看到很多Transfirmer在时间序列预测上应用的代码，一头雾水，很烦，所以想自己记录一下，以求真正把Transformer怎么用在时间序列预测上讲清楚，捋明白。

我直接用代码来讲，用一个公开数据集(ETTh1)来做例子。

首先导包

  1. import sys
  2. import argparse
  3. import os
  4. import torch
  5. import torch.nn as nn
  6. import torch.nn.functional as F
  7. import math
  8. import time
  9. import numpy as np
  10. import pandas as pd
  11. from torch.utils.data import Dataset, DataLoader
  12. from typing import List
  13. from pandas.tseries import offsets
  14. from pandas.tseries.frequencies import to_offset
  15. import random
  16. import matplotlib.pyplot as plt

这些包的作用后面都会有。

然后设置随机种子

  1. sys.argv = ['run.py']
  2. fix_seed = 2024
  3. random.seed(fix_seed)
  4. torch.manual_seed(fix_seed)
  5. np.random.seed(fix_seed)

然后传入参数

  1. parser = argparse.ArgumentParser(description='Transformer')
  2. parser.add_argument('--model', type=str, required=False, default='Transformer', help='model of experiment')
  3. parser.add_argument('--data', type=str, required=False, default='ETTh1', help='dataset')
  4. parser.add_argument('--root_path', type=str, default='./data/', help='root path of the data file')
  5. parser.add_argument('--data_path', type=str, default='ETTh1.csv', help='data file')    
  6. parser.add_argument('--features', type=str, default='MS', help='forecasting task, options:[M, S, MS]; M:multivariate predict multivariate, S:univariate predict univariate, MS:multivariate predict univariate')
  7. parser.add_argument('--target', type=str, default='OT', help='target feature in S or MS task')
  8. parser.add_argument('--freq', type=str, default='h', help='freq for time features encoding, options:[s:secondly, t:minutely, h:hourly, d:daily, b:business days, w:weekly, m:monthly]')
  9. parser.add_argument('--seq_len', type=int, default=96, help='input sequence length of Transformer encoder')
  10. parser.add_argument('--label_len', type=int, default=48, help='start token length of Transformer decoder')
  11. parser.add_argument('--pred_len', type=int, default=24, help='prediction sequence length')
  12. parser.add_argument('--enc_in', type=int, default=7, help='encoder input size')
  13. parser.add_argument('--dec_in', type=int, default=7, help='decoder input size')
  14. parser.add_argument('--c_out', type=int, default=7, help='output size')
  15. parser.add_argument('--d_model', type=int, default=512, help='dimension of model')
  16. parser.add_argument('--n_heads', type=int, default=8, help='num of heads')
  17. parser.add_argument('--e_layers', type=int, default=6, help='num of encoder layers')
  18. parser.add_argument('--d_layers', type=int, default=2, help='num of decoder layers')
  19. parser.add_argument('--d_ff', type=int, default=2048, help='dimension of fcn')
  20. parser.add_argument('--padding', type=int, default=0, help='padding type')
  21. parser.add_argument('--dropout', type=float, default=0.05, help='dropout')
  22. parser.add_argument('--activation', type=str, default='gelu',help='activation')
  23. parser.add_argument('--output_attention', action='store_true', help='whether to output attention in ecoder')
  24. parser.add_argument('--num_workers', type=int, default=0, help='data loader num workers')
  25. parser.add_argument('--train_epochs', type=int, default=6, help='number of train epochs')
  26. parser.add_argument('--batch_size', type=int, default=32, help='batch size of train input data')
  27. parser.add_argument('--patience', type=int, default=3, help='early stopping patience')
  28. parser.add_argument('--learning_rate', type=float, default=0.0001, help='optimizer learning rate')
  29. parser.add_argument('--loss', type=str, default='mse',help='loss function')
  30. parser.add_argument('--lradj', type=str, default='type1', help='adjust learning rate')
  31. parser.add_argument('--inverse', action='store_true', help='inverse output data', default=False)
  32. parser.add_argument('--use_gpu', type=bool, default=True, help='use gpu')
  33. parser.add_argument('--gpu', type=int, default=0, help='gpu')
  34. parser.add_argument('--is_rolling_predict', type=bool, default=True, help='rolling predict')
  35. parser.add_argument('--rolling_data_path', type=str, default='ETTh1-Test.csv', help='data file')
  36. parser.add_argument('--do_predict', action='store_true', default=True, help='whether to predict unseen future data')
  37. args = parser.parse_args()
  38. args.use_gpu = True if torch.cuda.is_available() and args.use_gpu else False
  39. data_parser = {
  40.     'ETTh1':{'data':'ETTh1.csv','T':'OT','M':[7,7,7],'S':[1,1,1],'MS':[7,7,1]},
  41.     'ETTh2':{'data':'ETTh2.csv','T':'OT','M':[7,7,7],'S':[1,1,1],'MS':[7,7,1]},
  42.     'ETTm1':{'data':'ETTm1.csv','T':'OT','M':[7,7,7],'S':[1,1,1],'MS':[7,7,1]},
  43.     'ETTm2':{'data':'ETTm2.csv','T':'OT','M':[7,7,7],'S':[1,1,1],'MS':[7,7,1]},
  44.     'WTH':{'data':'WTH.csv','T':'WetBulbCelsius','M':[12,12,12],'S':[1,1,1],'MS':[12,12,1]},
  45.     'ECL':{'data':'ECL.csv','T':'MT_320','M':[321,321,321],'S':[1,1,1],'MS':[321,321,1]},
  46.     'Solar':{'data':'solar_AL.csv','T':'POWER_136','M':[137,137,137],'S':[1,1,1],'MS':[137,137,1]},
  47. }
  48. if args.data in data_parser.keys():
  49.     data_info = data_parser[args.data]
  50.     args.data_path = data_info['data']
  51.     args.target = data_info['T']  # 'OT'
  52.     args.enc_in, args.dec_in, args.c_out = data_info[args.features]  # 7, 7, 1
  53.  
  54. print('Args in experiment:')
  55. print(args)

可以看出我们设置的一些默认参数

然后下面就开始写类和函数了，共有12个类和3个函数。

首先是TokenEmbedding

  1. class TokenEmbedding(nn.Module):
  2.     def __init__(self, c_in, d_model):
  3.         super(TokenEmbedding, self).__init__()
  4.         self.tokenConv = nn.Conv1d(in_channels=c_in, out_channels=d_model, kernel_size=3, padding=1, padding_mode='circular')
  5.         # weight initialization
  6.         for m in self.modules():
  7.             if isinstance(m, nn.Conv1d):
  8.                 nn.init.kaiming_normal_(m.weight, mode='fan_in', nonlinearity='leaky_relu')
  9.  
  10.     
  11.     def forward(self, x):
  12.         # x is a tensor with Size([32, 96, 7])
  13.         x = self.tokenConv(x.permute(0, 2, 1)).transpose(1,2)
  14.         # After Conv1d, x is a torch with Size([32, 96, 512])
  15.         return x

然后是

  1. class PositionalEmbedding(nn.Module):
  2.     def __init__(self, d_model, max_len=5000):
  3.         super(PositionalEmbedding, self).__init__()
  4.         # Compute the positional encodings once in log space.
  5.         pe = torch.zeros(max_len, d_model).float()  # pe is a tensor with Size([5000, 512]) with all 0.
  6.         pe.require_grad = False
  7.         position = torch.arange(0, max_len).float().unsqueeze(1)  # position is a tensor with Size([5000, 1])
  8.         div_term = (torch.arange(0, d_model, 2).float() * -(math.log(10000.0) / d_model)).exp()  # div_term is a tensor with Size([256])
  9.         # position * div_term is a tensor with Size([5000, 256])
  10.         pe[:, 0::2] = torch.sin(position * div_term)  # 把sin结果赋给奇数列
  11.         pe[:, 1::2] = torch.cos(position * div_term)  # 把cos结果赋给偶数列
  12.         pe = pe.unsqueeze(0)  # pe is a tensor with Size([1, 5000, 512])
  13.         self.register_buffer('pe', pe)
  14.  
  15.     
  16.     def forward(self, x):
  17.         return self.pe[:, :x.size(1)]  # return is a tensor with Size([1, 96, 512])

然后是时间特征的嵌入

  1. class TimeFeatureEmbedding(nn.Module):
  2.     def __init__(self, d_model, freq='h'):
  3.         super(TimeFeatureEmbedding, self).__init__()
  4.         freq_map = {'h':4, 't':5, 's':6, 'm':1, 'a':1, 'w':2, 'd':3, 'b':3}
  5.         d_inp = freq_map[freq]  # 4
  6.         self.embed = nn.Linear(d_inp, d_model)  # 4, 512
  7.     
  8.  
  9.     def forward(self, x):
  10.         return self.embed(x)

数据嵌入

  1. class DataEmbedding(nn.Module):
  2.     def __init__(self, c_in, d_model, freq, dropout):
  3.         super(DataEmbedding, self).__init__()
  4.         self.value_embedding = TokenEmbedding(c_in=c_in, d_model=d_model)
  5.         self.position_embedding = PositionalEmbedding(d_model=d_model)
  6.         self.time_feature_embedding = TimeFeatureEmbedding(d_model=d_model, freq=freq)
  7.         self.dropout = nn.Dropout(p=dropout)
  8.  
  9.  
  10.     def forward(self, x, x_mark):
  11.         # x is a tensor with Size([32, 96, 7])
  12.         # x_mark is a tensor with Size([32, 96, 4])
  13.         x = self.value_embedding(x) + self.position_embedding(x) + self.time_feature_embedding(x_mark)
  14.         return self.dropout(x)  # return is a tensor with Size([32, 96, 512])

掩码的实现

  1. class TriangularCausalMask():
  2.     def __init__(self, B, L, device="cpu"):
  3.         mask_shape = [B, 1, L, L]
  4.         with torch.no_grad():
  5.             self._mask = torch.triu(torch.ones(mask_shape, dtype=torch.bool), diagonal=1).to(device)
  6.  
  7.     @property
  8.     def mask(self):
  9.         return self._mask

注意力机制

  1. class FullAttention(nn.Module):
  2.     def __init__(self, mask_flag=True, factor=5, scale=None, attention_dropout=0.1, output_attention=False):
  3.         super(FullAttention, self).__init__()
  4.         self.scale = scale  # None
  5.         self.mask_flag = mask_flag  # False
  6.         self.output_attention = output_attention  # False
  7.         self.dropout = nn.Dropout(attention_dropout)  # attention_dropout = 0.05
  8.         
  9.     
  10.     def forward(self, queries, keys, values, attn_mask):
  11.         # shape of queries is [b, l, h, e]
  12.         # shape of keys is [b, s, h, e]
  13.         # shape of values is [b, s, h, d]
  14.         B, L, H, E = queries.shape  # 32, 96, 8, 64
  15.         _, S, _, D = values.shape  # 96  64
  16.         scale = self.scale or 1. / math.sqrt(E)  # 0.125
  17.         scores = torch.einsum("blhe,bshe->bhls", queries, keys)  # scores is a tensor with Size([32, 8, 96, 96])
  18.         
  19.         # self.mask_flag = False in encoder but self.mask_flag = True in decoder
  20.         if self.mask_flag:
  21.             if attn_mask is None:
  22.                 attn_mask = TriangularCausalMask(B, L, device=queries.device)
  23.             scores.masked_fill_(attn_mask.mask, -np.inf)
  24.         
  25.         A = self.dropout(torch.softmax(scale * scores, dim=-1))  # A is a tensor with Size([32, 8, 96, 96])
  26.         V = torch.einsum("bhls,bshd->blhd", A, values)  # V is a tensor with Size([32, 96, 8, 64])
  27.         if self.output_attention:
  28.             return (V.contiguous(), A)
  29.         else:  # 走这条路
  30.             return (V.contiguous(), None)

注意力层

  1. class AttentionLayer(nn.Module):
  2.     def __init__(self, attention, d_model, n_heads, d_keys=None, d_values=None):
  3.         super(AttentionLayer, self).__init__()
  4.         d_keys = d_keys or (d_model // n_heads)  # 512 / 8 = 64
  5.         d_values = d_values or (d_model // n_heads)  # 512 / 8 = 64
  6.         self.inner_attention = attention  # FullAttention(...)
  7.         self.query_projection = nn.Linear(d_model, d_keys * n_heads)  # 512 512
  8.         self.key_projection = nn.Linear(d_model, d_keys * n_heads)  # 512 512
  9.         self.value_projection = nn.Linear(d_model, d_values * n_heads)  # 512 512
  10.         self.out_projection = nn.Linear(d_values * n_heads, d_model)  # 512 512  # 线性拼接8个头
  11.         self.n_heads = n_heads  # 8
  12.  
  13.     
  14.     def forward(self, queries, keys, values, attn_mask):
  15.         B, L, _ = queries.shape  # B = 32, L = 96
  16.         _, S, _ = keys.shape  # S = 96
  17.         H = self.n_heads  # H = 8
  18.         queries = self.query_projection(queries).view(B, L, H, -1)  # queries is a tensor with Size([32, 96, 8, 64]) 
  19.         keys = self.key_projection(keys).view(B, S, H, -1)  # keys is a tensor with Size([32, 96, 8, 64])
  20.         values = self.value_projection(values).view(B, S, H, -1)  # values is a tensor with Size([32, 96, 8, 64])
  21.         out, attn = self.inner_attention(queries, keys, values, attn_mask)  # out is a tensor with Size([32, 96, 8, 64])
  22.         out = out.view(B, L, -1)  # out is a tensor with Size([32, 96, 512])
  23.         return self.out_projection(out), attn

编码器层

  1. class EncoderLayer(nn.Module):
  2.     def __init__(self, attention, d_model, d_ff=None, dropout=0.1, activation="relu"):
  3.         super(EncoderLayer, self).__init__()
  4.         d_ff = d_ff or 4 * d_model  # 512 * 4 = 2048
  5.         self.attention = attention
  6.         self.conv1 = nn.Conv1d(in_channels=d_model, out_channels=d_ff, kernel_size=1)
  7.         self.conv2 = nn.Conv1d(in_channels=d_ff, out_channels=d_model, kernel_size=1)
  8.         self.norm1 = nn.LayerNorm(d_model)
  9.         self.norm2 = nn.LayerNorm(d_model)
  10.         self.dropout = nn.Dropout(dropout)
  11.         self.activation = F.relu if activation == "relu" else F.gelu
  12.  
  13.     
  14.     def forward(self, x, attn_mask=None):
  15.         # x [B, L, D]  32, 96, 512
  16.         new_x, attn = self.attention(x, x, x, attn_mask = attn_mask)  # new_x is a tensor with Size([32, 96, 512])
  17.         x = x + self.dropout(new_x)  # connection  # x is a tensor with Size([32, 96, 512])
  18.         y = x = self.norm1(x)  # x and y are a tensor with Size([32, 96, 512])
  19.         y = self.dropout(self.activation(self.conv1(y.transpose(-1,1))))
  20.         y = self.dropout(self.conv2(y).transpose(-1,1))  # y is a tensor with Size([32, 96, 512])
  21.         return self.norm2(x+y), attn

编码器

  1. class Encoder(nn.Module):
  2.     def __init__(self, attn_layers, conv_layers=None, norm_layer=None):
  3.         super(Encoder, self).__init__()
  4.         self.attn_layers = nn.ModuleList(attn_layers)
  5.         self.norm = norm_layer
  6.  
  7.     
  8.     def forward(self, x, attn_mask=None):
  9.         # x [B, L, D] [32, 96, 512]
  10.         attns = []
  11.         for attn_layer in self.attn_layers:
  12.             x, attn = attn_layer(x, attn_mask=attn_mask)
  13.             attns.append(attn)
  14.  
  15.         if self.norm is not None:
  16.             x = self.norm(x)
  17.         return x, attns

解码器层

  1. class DecoderLayer(nn.Module):
  2.     def __init__(self, self_attention, cross_attention, d_model, d_ff=None, dropout=0.1, activation="relu"):
  3.         super(DecoderLayer, self).__init__()
  4.         d_ff = d_ff or 4 * d_model
  5.         self.self_attention = self_attention
  6.         self.cross_attention = cross_attention
  7.         self.conv1 = nn.Conv1d(in_channels=d_model, out_channels=d_ff, kernel_size=1)
  8.         self.conv2 = nn.Conv1d(in_channels=d_ff, out_channels=d_model, kernel_size=1)
  9.         self.norm1 = nn.LayerNorm(d_model)
  10.         self.norm2 = nn.LayerNorm(d_model)
  11.         self.norm3 = nn.LayerNorm(d_model)
  12.         self.dropout = nn.Dropout(dropout)
  13.         self.activation = F.relu if activation == "relu" else F.gelu
  14.  
  15.     def forward(self, x, cross, x_mask=None, cross_mask=None):
  16.         x = x + self.dropout(self.self_attention(x, x, x, attn_mask=x_mask)[0])
  17.         x = self.norm1(x)
  18.         x = x + self.dropout(self.cross_attention(x, cross, cross, attn_mask=cross_mask)[0])
  19.         y = x = self.norm2(x)
  20.         y = self.dropout(self.activation(self.conv1(y.transpose(-1,1))))
  21.         y = self.dropout(self.conv2(y).transpose(-1,1))
  22.         return self.norm3(x+y)

解码器

  1. class Decoder(nn.Module):
  2.     def __init__(self, layers, norm_layer=None):
  3.         super(Decoder, self).__init__()
  4.         self.layers = nn.ModuleList(layers)
  5.         self.norm = norm_layer
  6.  
  7.     
  8.     def forward(self, x, cross, x_mask=None, cross_mask=None):
  9.         # x is the output of decoder_embedding
  10.         # cross is the output of encoder
  11.         for layer in self.layers:
  12.             x = layer(x, cross, x_mask=x_mask, cross_mask=cross_mask)
  13.         if self.norm is not None:
  14.             x = self.norm(x)
  15.         return x

最重要的Transformer类

  1. class Transformer(nn.Module):
  2.     def __init__(self, enc_in, dec_in, c_out, seq_len, label_len, pred_len, factor, d_model, n_heads, e_layers, d_layers, d_ff, 
  3.                 dropout, freq, activation, output_attention, device):
  4.         super(Transformer, self).__init__()
  5.         self.pred_len = pred_len  # 24
  6.         self.output_attention = output_attention  # False
  7.         # Embedding
  8.         self.enc_embedding = DataEmbedding(enc_in, d_model, freq, dropout)
  9.         self.dec_embedding = DataEmbedding(dec_in, d_model, freq, dropout)
  10.  
  11.         # Encoder
  12.         self.encoder = Encoder(
  13.             [
  14.                 EncoderLayer(
  15.                     AttentionLayer(FullAttention(False, factor, attention_dropout=dropout, output_attention=output_attention), d_model, n_heads),
  16.                     d_model,
  17.                     d_ff,
  18.                     dropout=dropout,
  19.                     activation=activation
  20.                 ) for l in range(e_layers)
  21.             ],
  22.             norm_layer=torch.nn.LayerNorm(d_model)
  23.         )
  24.         
  25.         # Decoder
  26.         self.decoder = Decoder(
  27.             [
  28.                 DecoderLayer(
  29.                     AttentionLayer(FullAttention(True, factor, attention_dropout=dropout, output_attention=True), d_model, n_heads),
  30.                     AttentionLayer(FullAttention(False, factor, attention_dropout=dropout, output_attention=False), d_model, n_heads),
  31.                     d_model,
  32.                     d_ff,
  33.                     dropout=dropout,
  34.                     activation=activation,
  35.                 )
  36.                 for l in range(d_layers)
  37.             ],
  38.             norm_layer=nn.LayerNorm(d_model)
  39.         )
  40.         self.projection = nn.Linear(d_model, c_out, bias=True)
  41.     
  42.     
  43.     def forward(self, x_enc, x_mark_enc, x_dec, x_mark_dec, enc_self_mask=None, dec_self_mask=None, dec_enc_mask=None):
  44.         enc_embedding_out = self.enc_embedding(x_enc, x_mark_enc)  # enc_out is a tensor with Size([32, 96, 512])
  45.         enc_out, attns = self.encoder(enc_embedding_out, attn_mask=enc_self_mask)  # enc_out is a tensor with Size([32, 96, 512])
  46.         dec_embedding_out = self.dec_embedding(x_dec, x_mark_dec)
  47.         dec_out = self.decoder(dec_embedding_out, enc_out, x_mask=dec_self_mask)  # dec_out is a tensor with Size([32, 72, 512])
  48.         model_output = self.projection(dec_out)  # enc_out is a tensor with Size([32, 72, 1])
  49.         return model_output[:, -self.pred_len:, :] # [B, L, D]  [32, 24, 1]

调整学习率的方法

  1. def adjust_learning_rate(optimizer, epoch, args):
  2.     if args.lradj=='type1':  # 走这条路
  3.         lr_adjust = {epoch: args.learning_rate * (0.5 ** ((epoch-1) // 1))}
  4.     elif args.lradj=='type2':
  5.         lr_adjust = {
  6.             2: 5e-5, 4: 1e-5, 6: 5e-6, 8: 1e-6, 
  7.             10: 5e-7, 15: 1e-7, 20: 5e-8
  8.         }
  9.     
  10.     if epoch in lr_adjust.keys():
  11.         lr = lr_adjust[epoch]
  12.         for param_group in optimizer.param_groups:
  13.             param_group['lr'] = lr
  14.         print('Updating learning rate to {}'.format(lr))  # 这里输出

早停类

  1. class EarlyStopping:
  2.     def __init__(self, patience, verbose, delta=0):
  3.         self.patience = patience  # 3
  4.         self.verbose = verbose  # True
  5.         self.counter = 0
  6.         self.best_score = None
  7.         self.early_stop = False
  8.         self.val_loss_min = np.Inf
  9.         self.delta = delta
  10.  
  11.     def __call__(self, val_loss, model, path):
  12.         score = -val_loss
  13.         if self.best_score is None:
  14.             self.best_score = score
  15.             self.save_checkpoint(val_loss, model, path)
  16.         elif score < self.best_score + self.delta:
  17.             self.counter += 1
  18.             print(f'EarlyStopping counter: {self.counter} out of {self.patience}')
  19.             if self.counter >= self.patience:
  20.                 self.early_stop = True
  21.         else:
  22.             self.best_score = score
  23.             self.save_checkpoint(val_loss, model, path)
  24.             self.counter = 0
  25.  
  26.     def save_checkpoint(self, val_loss, model, path):
  27.         if self.verbose:
  28.             print(f'Validation loss decreased ({self.val_loss_min:.6f} --> {val_loss:.6f}).  Saving model ...')  # 这里输出
  29.         torch.save(model.state_dict(), path + '/' + 'checkpoint.pth')
  30.         self.val_loss_min = val_loss

数据预处理

  1. class StandardScaler():
  2.     def __init__(self):
  3.         self.mean = 0.
  4.         self.std = 1.
  5.     
  6.     def fit(self, data):
  7.         self.mean = data.mean(0)
  8.         self.std = data.std(0)
  9.  
  10.     def transform(self, data):
  11.         mean = torch.from_numpy(self.mean).type_as(data).to(data.device) if torch.is_tensor(data) else self.mean
  12.         std = torch.from_numpy(self.std).type_as(data).to(data.device) if torch.is_tensor(data) else self.std
  13.         return (data - mean) / std
  14.  
  15.     def inverse_transform(self, data):
  16.         mean = torch.from_numpy(self.mean).type_as(data).to(data.device) if torch.is_tensor(data) else self.mean
  17.         std = torch.from_numpy(self.std).type_as(data).to(data.device) if torch.is_tensor(data) else self.std
  18.         if data.shape[-1] != mean.shape[-1]:
  19.             mean = mean[-1:]
  20.             std = std[-1:]
  21.         return (data * std) + mean

时间特征的提取

  1. class TimeFeature:
  2.     def __init__(self):
  3.         pass
  4.  
  5.     def __call__(self, index: pd.DatetimeIndex) -> np.ndarray:
  6.         pass
  7.  
  8.     def __repr__(self):
  9.         return self.__class__.__name__ + "()"
  10.  
  11. class SecondOfMinute(TimeFeature):
  12.     """Minute of hour encoded as value between [-0.5, 0.5]"""
  13.     def __call__(self, index: pd.DatetimeIndex) -> np.ndarray:
  14.         return index.second / 59.0 - 0.5
  15.  
  16. class MinuteOfHour(TimeFeature):
  17.     """Minute of hour encoded as value between [-0.5, 0.5]"""
  18.     def __call__(self, index: pd.DatetimeIndex) -> np.ndarray:
  19.         return index.minute / 59.0 - 0.5
  20.  
  21. class HourOfDay(TimeFeature):
  22.     """Hour of day encoded as value between [-0.5, 0.5]"""
  23.     def __call__(self, index: pd.DatetimeIndex) -> np.ndarray:
  24.         return index.hour / 23.0 - 0.5
  25.  
  26. class DayOfWeek(TimeFeature):
  27.     """Hour of day encoded as value between [-0.5, 0.5]"""
  28.     def __call__(self, index: pd.DatetimeIndex) -> np.ndarray:
  29.         return index.dayofweek / 6.0 - 0.5
  30.  
  31. class DayOfMonth(TimeFeature):
  32.     """Day of month encoded as value between [-0.5, 0.5]"""
  33.     def __call__(self, index: pd.DatetimeIndex) -> np.ndarray:
  34.         return (index.day - 1) / 30.0 - 0.5
  35.  
  36. class DayOfYear(TimeFeature):
  37.     """Day of year encoded as value between [-0.5, 0.5]"""
  38.     def __call__(self, index: pd.DatetimeIndex) -> np.ndarray:
  39.         return (index.dayofyear - 1) / 365.0 - 0.5
  40.  
  41. class MonthOfYear(TimeFeature):
  42.     """Month of year encoded as value between [-0.5, 0.5]"""
  43.     def __call__(self, index: pd.DatetimeIndex) -> np.ndarray:
  44.         return (index.month - 1) / 11.0 - 0.5
  45.  
  46. class WeekOfYear(TimeFeature):
  47.     """Week of year encoded as value between [-0.5, 0.5]"""
  48.     def __call__(self, index: pd.DatetimeIndex) -> np.ndarray:
  49.         return (index.week - 1) / 52.0 - 0.5
  50.  
  51. def time_features_from_frequency_str(freq_str: str) -> List[TimeFeature]:
  52.     features_by_offsets = {
  53.         offsets.YearEnd: [],
  54.         offsets.QuarterEnd: [MonthOfYear],
  55.         offsets.MonthEnd: [MonthOfYear],
  56.         offsets.Week: [DayOfMonth, WeekOfYear],
  57.         offsets.Day: [DayOfWeek, DayOfMonth, DayOfYear],
  58.         offsets.BusinessDay: [DayOfWeek, DayOfMonth, DayOfYear],
  59.         offsets.Hour: [HourOfDay, DayOfWeek, DayOfMonth, DayOfYear],  # 找到这一项
  60.         offsets.Minute: [
  61.             MinuteOfHour,
  62.             HourOfDay,
  63.             DayOfWeek,
  64.             DayOfMonth,
  65.             DayOfYear,
  66.         ],
  67.         offsets.Second: [
  68.             SecondOfMinute,
  69.             MinuteOfHour,
  70.             HourOfDay,
  71.             DayOfWeek,
  72.             DayOfMonth,
  73.             DayOfYear,
  74.         ],
  75.     }
  76.  
  77.     offset = to_offset(freq_str)  # <Hour>
  78.     for offset_type, feature_classes in features_by_offsets.items():
  79.         if isinstance(offset, offset_type):
  80.             return [cls() for cls in feature_classes]  # return [HourOfDay(), DayOfWeek(), DayOfMonth(), DayOfYear()]
  81.  
  82.     supported_freq_msg = f"""
  83.     Unsupported frequency {freq_str}
  84.     The following frequencies are supported:
  85.         Y   - yearly
  86.             alias: A
  87.         M   - monthly
  88.         W   - weekly
  89.         D   - daily
  90.         B   - business days
  91.         H   - hourly
  92.         T   - minutely
  93.             alias: min
  94.         S   - secondly
  95.     """
  96.     raise RuntimeError(supported_freq_msg)
  97.  
  98. def time_features(dates, timeenc, freq):
  99.     """
  100.     > `time_features` takes in a `dates` dataframe with a 'dates' column and extracts the date down to `freq` where freq can be any of the following if `timeenc` is 0: 
  101.     > * m - [month]
  102.     > * w - [month]
  103.     > * d - [month, day, weekday]
  104.     > * b - [month, day, weekday]
  105.     > * h - [month, day, weekday, hour]
  106.     > * t - [month, day, weekday, hour, *minute]
  107.     > 
  108.     > If `timeenc` is 1, a similar, but different list of `freq` values are supported (all encoded between [-0.5 and 0.5]): 
  109.     > * Q - [month]
  110.     > * M - [month]
  111.     > * W - [Day of month, week of year]
  112.     > * D - [Day of week, day of month, day of year]
  113.     > * B - [Day of week, day of month, day of year]
  114.     > * H - [Hour of day, day of week, day of month, day of year]
  115.     > * T - [Minute of hour*, hour of day, day of week, day of month, day of year]
  116.     > * S - [Second of minute, minute of hour, hour of day, day of week, day of month, day of year]
  118.     *minute returns a number from 0-3 corresponding to the 15 minute period it falls into.
  119.     """
  120.     if timeenc==0:
  121.         dates['month'] = dates.date.apply(lambda row:row.month,1)
  122.         dates['day'] = dates.date.apply(lambda row:row.day,1)
  123.         dates['weekday'] = dates.date.apply(lambda row:row.weekday(),1)
  124.         dates['hour'] = dates.date.apply(lambda row:row.hour,1)
  125.         dates['minute'] = dates.date.apply(lambda row:row.minute,1)
  126.         dates['minute'] = dates.minute.map(lambda x:x//15)
  127.         freq_map = {
  128.             'y':[],'m':['month'],'w':['month'],'d':['month','day','weekday'],
  129.             'b':['month','day','weekday'],'h':['month','day','weekday','hour'],
  130.             't':['month','day','weekday','hour','minute'],
  131.         }
  132.         return dates[freq_map[freq.lower()]].values
  133.     if timeenc==1:  # 走这条路
  134.         list1 = []
  135.         dates = pd.to_datetime(dates.date.values)
  136.         for feat in time_features_from_frequency_str(freq):
  137.             # print(feat(dates))
  138.             list1.append(feat(dates))
  139.         
  140.         return np.vstack(list1).transpose(1, 0)

数据集类

  1. class MyDataset(Dataset):
  2.     def __init__(self, root_path, flag, size, features, data_path, target, inverse, timeenc, freq):
  3.         # size [seq_len, label_len, pred_len]
  4.         self.seq_len = size[0]  # 96
  5.         self.label_len = size[1]  #  48
  6.         self.pred_len = size[2]  # 24
  7.         assert flag in ['train', 'test', 'val']
  8.         type_map = {'train':0, 'val':1, 'test':2}
  9.         self.set_type = type_map[flag]  # 0
  10.         self.features = features  # 'MS'
  11.         self.target = target  # 'OT'
  12.         self.inverse = inverse  # False
  13.         self.timeenc = timeenc  # 1
  14.         self.freq = freq  # 'h'
  15.         self.root_path = root_path  # './data/'
  16.         self.data_path = data_path  # 'ETTh1.csv'
  17.         self.__read_data__()
  18.  
  19.     
  20.     def __read_data__(self):
  21.         self.scaler = StandardScaler()
  22.         df_raw = pd.read_csv(os.path.join(self.root_path, self.data_path))  # a dataframe of 17420 * 8
  23.         border1s = [0, int(len(df_raw) * 0.6) - self.seq_len, int(len(df_raw) * 0.8) - self.seq_len]  # border1s = [0, 10356, 13840]
  24.         border2s = [int(len(df_raw) * 0.6), int(len(df_raw) * 0.8), int(len(df_raw) * 1)]  # border2s = [10452, 13936, 17420]
  25.         border1 = border1s[self.set_type]  # 0
  26.         border2 = border2s[self.set_type]  # 10452
  27.         
  28.         if self.features=='M' or self.features=='MS':
  29.             df_data = df_raw[df_raw.columns[1:]]  # a dataframe of 17420 * 7 (without 'date' column)
  30.         elif self.features=='S':
  31.             df_data = df_raw[[self.target]]
  32.  
  33.         train_data = df_data[border1s[0]:border2s[0]]  # a dataframe of 10452 * 7
  34.         self.scaler.fit(train_data.values)
  35.         data = self.scaler.transform(df_data.values)  # a 2D-array of shape (17420, 7)            
  36.         df_stamp = df_raw[['date']][border1:border2]  # a dataframe of 10452 * 1 (only 'date' column)
  37.         df_stamp['date'] = pd.to_datetime(df_stamp.date)  # convert to datatimens
  38.         data_stamp = time_features(df_stamp, timeenc=self.timeenc, freq=self.freq)  # a 2D-array of shape (10452, 4)
  39.         self.data_x = data[border1:border2]  # a 2D-array of shape (10452, 7)
  40.         
  41.         if self.inverse:
  42.             self.data_y = df_data.values[border1:border2]
  43.         else:  # 走这条路
  44.             self.data_y = data[border1:border2]  # a 2D-array of shape (10452, 7)
  45.         
  46.         self.data_stamp = data_stamp  # a 2D-array of shape (10452, 4)
  47.     
  48.     
  49.     def __getitem__(self, index):
  50.         s_begin = index
  51.         s_end = s_begin + self.seq_len
  52.         r_begin = s_end - self.label_len
  53.         r_end = r_begin + self.label_len + self.pred_len
  54.         seq_x = self.data_x[s_begin:s_end]  # a 2D-array of shape (96, 7)
  55.         if self.inverse:
  56.             seq_y = np.concatenate([self.data_x[r_begin:r_begin+self.label_len], self.data_y[r_begin+self.label_len:r_end]], 0)
  57.         else:  # 走这条路
  58.             seq_y = self.data_y[r_begin:r_end]  # a 2D-array of shape (72, 7)
  59.         
  60.         seq_x_mark = self.data_stamp[s_begin:s_end]  # a 2D-array of shape (96, 4)
  61.         seq_y_mark = self.data_stamp[r_begin:r_end]  # a 2D-array of shape (72, 4)
  62.         return seq_x, seq_y, seq_x_mark, seq_y_mark
  63.     
  64.     
  65.     def __len__(self):
  66.         return len(self.data_x) - self.seq_len - self.pred_len + 1  # 10452 - 96 - 24 + 1 = 10333
  67.  
  68.     
  69.     def inverse_transform(self, data):
  70.         return self.scaler.inverse_transform(data)

评价指标

  1. def RSE(pred, true):
  2.     return np.sqrt(np.sum((true-pred)**2)) / np.sqrt(np.sum((true-true.mean())**2))
  3.  
  4. def CORR(pred, true):
  5.     u = ((true-true.mean(0))*(pred-pred.mean(0))).sum(0) 
  6.     d = np.sqrt(((true-true.mean(0))**2*(pred-pred.mean(0))**2).sum(0))
  7.     return (u/d).mean(-1)
  8.  
  9. def MAE(pred, true):
  10.     return np.mean(np.abs(pred-true))
  11.  
  12. def MSE(pred, true):
  13.     return np.mean((pred-true)**2)
  14.  
  15. def RMSE(pred, true):
  16.     return np.sqrt(MSE(pred, true))
  17.  
  18. def MAPE(pred, true):
  19.     return np.mean(np.abs((pred - true) / true))
  20.  
  21. def MSPE(pred, true):
  22.     return np.mean(np.square((pred - true) / true))
  23.  
  24. def metric(pred, true):
  25.     mae = MAE(pred, true)
  26.     mse = MSE(pred, true)
  27.     rmse = RMSE(pred, true)
  28.     mape = MAPE(pred, true)
  29.     mspe = MSPE(pred, true)
  30.     return mae,mse,rmse,mape,mspe

Exp类

  1. class Exp_Transformer:
  2.     def __init__(self, args):
  3.         self.args = args
  4.         self.device = self._acquire_device()
  5.         self.model = self._build_model().to(self.device)
  6.     
  7.     def _acquire_device(self):
  8.         os.environ["CUDA_VISIBLE_DEVICES"] = str(self.args.gpu)  # 指定使用显卡cuda:{0}
  9.         device = torch.device('cuda:{}'.format(self.args.gpu))
  10.         print('Use GPU: cuda:{}'.format(self.args.gpu))  # 这里输出
  11.         return device
  12.     
  13.     def _build_model(self):
  14.         model = Transformer(
  15.             self.args.enc_in,  # 7
  16.             self.args.dec_in,  # 7
  17.             self.args.c_out,  # 1
  18.             self.args.seq_len,  # 96
  19.             self.args.label_len,  # 48
  20.             self.args.pred_len,  # 24
  21.             self.args.factor,  # 5
  22.             self.args.d_model,  # 512
  23.             self.args.n_heads, # 8
  24.             self.args.e_layers, # 2
  25.             self.args.d_layers, # 1
  26.             self.args.d_ff,  # 2048
  27.             self.args.dropout,  # 0.05
  28.             self.args.freq,  # 'h'
  29.             self.args.activation,  # 'gelu'
  30.             self.args.output_attention,  # False
  31.             self.device  # torch.device('cuda:{0}')
  32.         ).float()
  33.         return model
  34.     
  35.     
  36.     def _get_data(self, flag):
  37.         Data = MyDataset
  38.         if flag == 'test':
  39.             shuffle_flag = False
  40.             drop_last = True
  41.             batch_size = self.args.batch_size
  42.             freq = self.args.freq
  43.         elif flag=='pred':
  44.             shuffle_flag = False
  45.             drop_last = False
  46.             batch_size = 1
  47.             freq = self.args.freq
  48.             Data = Dataset_Pred
  49.         else:  # flag == 'train' or flag == 'val'
  50.             shuffle_flag = True
  51.             drop_last = True
  52.             batch_size = self.args.batch_size  # 32
  53.         
  54.         data_set = Data(
  55.             root_path=self.args.root_path,  # './data/'
  56.             flag=flag,  # 'train'
  57.             size=[self.args.seq_len, self.args.label_len, self.args.pred_len],  # [96, 48, 24]
  58.             features=self.args.features,  # 'MS'
  59.             data_path=self.args.data_path,  # 'ETTh1.csv'
  60.             target=self.args.target,  # 'OT'
  61.             inverse=self.args.inverse,  # False
  62.             timeenc=1,
  63.             freq='h',
  64.         )  # 训练时data_set是MyDataset类的一个对象
  65.         print(flag, len(data_set))  # 这里输出
  66.         data_loader = DataLoader(data_set, batch_size=batch_size, shuffle=shuffle_flag, num_workers=args.num_workers, drop_last=drop_last)
  67.         return data_set, data_loader
  68.     
  69.     
  70.     def train(self, setting):
  71.         train_data, train_loader = self._get_data(flag = 'train')
  72.         vali_data, vali_loader = self._get_data(flag = 'val')
  73.         test_data, test_loader = self._get_data(flag = 'test')
  74.         path = os.path.join(self.args.checkpoints, setting)
  75.         if not os.path.exists(path):
  76.             os.makedirs(path)
  77.         time_now = time.time()
  78.         train_steps = len(train_loader)  # 266 = 8521 // 32
  79.         early_stopping = EarlyStopping(patience=self.args.patience, verbose=True)
  80.         model_optim = self._select_optimizer()
  81.         criterion =  self._select_criterion()
  82.         hist = np.zeros(self.args.train_epochs)
  83.  
  84.         for epoch in range(self.args.train_epochs):  # 循环6次
  85.             iter_count = 0
  86.             train_loss = []
  87.             self.model.train()
  88.             epoch_time = time.time()
  89.             for i, (batch_x,batch_y,batch_x_mark,batch_y_mark) in enumerate(train_loader):
  90.                 # batch_x is a tensor with Size([32, 96, 7])
  91.                 # batch_y is a tensor with Size([32, 72, 7])
  92.                 # batch_x_mark is a tensor with Size([32, 96, 4])
  93.                 # batch_y_mark is a tensor with Size([32, 72, 4])
  94.                 iter_count += 1
  95.                 model_optim.zero_grad()
  96.                 pred, true = self._process_one_batch(train_data, batch_x, batch_y, batch_x_mark, batch_y_mark)
  97.                 loss = criterion(pred, true)
  98.                 train_loss.append(loss.item())
  99.                 if (i+1) % 100==0:
  100.                     print("\titers: {0}, epoch: {1} | loss: {2:.7f}".format(i + 1, epoch + 1, loss.item()))  # 这里输出
  101.                     speed = (time.time() - time_now) / iter_count
  102.                     left_time = speed * ((self.args.train_epochs - epoch) * train_steps - i)
  103.                     print('\tspeed: {:.4f}s/iter; left time: {:.4f}s'.format(speed, left_time))  # 这里输出
  104.                     iter_count = 0
  105.                     time_now = time.time()
  106.                 
  107.                 loss.backward()
  108.                 model_optim.step()
  109.  
  110.             print("Epoch: {} cost time: {}".format(epoch+1, time.time()-epoch_time))  # 这里输出
  111.             train_loss = np.average(train_loss)
  112.             hist[epoch] = train_loss
  113.             vali_loss = self.vali(vali_data, vali_loader, criterion)
  114.             test_loss = self.vali(test_data, test_loader, criterion)
  115.  
  116.             print("Epoch: {0}, Steps: {1} | Train Loss: {2:.7f} Vali Loss: {3:.7f} Test Loss: {4:.7f}".format(
  117.                 epoch + 1, train_steps, train_loss, vali_loss, test_loss))  # 这里输出
  118.             early_stopping(vali_loss, self.model, path)
  119.             if early_stopping.early_stop:
  120.                 print("Early stopping")  # 这里输出
  121.                 break
  122.  
  123.             adjust_learning_rate(model_optim, epoch+1, self.args)
  124.         
  125.         
  126.         plt.plot(hist, label="Training loss")
  127.         plt.legend()
  128.         plt.xticks(range(1, len(hist) + 1))  # len(hist) + 1 是因为范围不包括结束值
  129.         plt.show()
  130.         best_model_path = path+'/'+'checkpoint.pth'
  131.         self.model.load_state_dict(torch.load(best_model_path))
  132.         return self.model
  133.     
  134.     
  135.     def vali(self, vali_data, vali_loader, criterion):
  136.         self.model.eval()
  137.         total_loss = []
  138.         for i, (batch_x,batch_y,batch_x_mark,batch_y_mark) in enumerate(vali_loader):
  139.             pred, true = self._process_one_batch(vali_data, batch_x, batch_y, batch_x_mark, batch_y_mark)
  140.             loss = criterion(pred.detach().cpu(), true.detach().cpu())
  141.             total_loss.append(loss)
  142.         total_loss = np.average(total_loss)
  143.         self.model.train()
  144.         return total_loss
  145.  
  146.     def test(self, setting):
  147.         test_data, test_loader = self._get_data(flag='test')
  148.         self.model.eval()
  149.         
  150.         preds = []
  151.         trues = []
  152.         
  153.         for i, (batch_x,batch_y,batch_x_mark,batch_y_mark) in enumerate(test_loader):
  154.             pred, true = self._process_one_batch(test_data, batch_x, batch_y, batch_x_mark, batch_y_mark)
  155.             preds.append(pred.detach().cpu().numpy())
  156.             trues.append(true.detach().cpu().numpy())
  157.  
  158.         preds = np.array(preds)
  159.         trues = np.array(trues)
  160.         print('test shape:', preds.shape, trues.shape)
  161.         preds = preds.reshape(-1, preds.shape[-2], preds.shape[-1])
  162.         trues = trues.reshape(-1, trues.shape[-2], trues.shape[-1])
  163.         print('test shape:', preds.shape, trues.shape)
  164.         prediction = []
  165.         ground_truth = []
  166.         for i in range(preds.shape[0] - 1):
  167.             prediction.append(preds[i][0].item())
  168.             ground_truth.append(trues[i][0].item())
  169.         for j in range(preds.shape[1] - 1):
  170.             prediction.append(preds[-1][j+1].item())
  171.             ground_truth.append(trues[-1][j+1].item())
  172.  
  173.         plt.plot(prediction, label='Prediction')  
  174.         plt.plot(ground_truth, label='Ground Truth')  
  175.         # 添加图例  
  176.         plt.legend()
  177.         # 设置x轴和y轴的标签  
  178.         plt.xlabel('Index')  
  179.         plt.ylabel('Value')
  180.         # 显示图表  
  181.         plt.show()
  182.         
  183.         # result save
  184.         folder_path = './results/' + setting +'/'
  185.         if not os.path.exists(folder_path):
  186.             os.makedirs(folder_path)
  187.  
  188.         mae, mse, rmse, mape, mspe = metric(preds, trues)
  189.         print('mse:{}, mae:{}'.format(mse, mae))
  190.         np.save(folder_path+'metrics.npy', np.array([mae, mse, rmse, mape, mspe]))
  191.         np.save(folder_path+'pred.npy', preds)
  192.         np.save(folder_path+'true.npy', trues)
  193.         return
  194.     
  195.     
  196.     def _select_optimizer(self):
  197.         model_optim = torch.optim.Adam(self.model.parameters(), lr=self.args.learning_rate)
  198.         return model_optim
  199.     
  200.     
  201.     def _select_criterion(self):
  202.         criterion =  nn.MSELoss()
  203.         return criterion
  204.     
  205.     
  206.     def _process_one_batch(self, dataset_object, batch_x, batch_y, batch_x_mark, batch_y_mark):
  207.         batch_x = batch_x.float().to(self.device)
  208.         batch_y = batch_y.float()  # 这个不能放GPU上
  209.         batch_x_mark = batch_x_mark.float().to(self.device)
  210.         batch_y_mark = batch_y_mark.float().to(self.device)
  211.  
  212.         # decoder input
  213.         if self.args.padding==0:  # 走这条路
  214.             dec_inp = torch.zeros([batch_y.shape[0], self.args.pred_len, batch_y.shape[-1]]).float()  # dec_inp is a tensor with Size([32, 24, 7])
  215.         elif self.args.padding==1:
  216.             dec_inp = torch.ones([batch_y.shape[0], self.args.pred_len, batch_y.shape[-1]]).float()
  217.         dec_inp = torch.cat([batch_y[:, :self.args.label_len, :], dec_inp], dim=1).float().to(self.device)  # dec_inp is a tensor with Size([32, 72, 7])
  218.         # encoder - decoder
  219.         if self.args.output_attention:
  220.             outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)[0]
  221.         else:  # 走这条路
  222.             outputs = self.model(batch_x, batch_x_mark, dec_inp, batch_y_mark)  # outputs is a tensor with Size([32, 24, 1])
  223.         if self.args.inverse:
  224.             outputs = dataset_object.inverse_transform(outputs)
  225.         f_dim = -1 if self.args.features=='MS' else 0
  226.         batch_y = batch_y[:, -self.args.pred_len:, f_dim:].to(self.device)  # batch_y is a tensor with Size([32, 24, 1])
  227.         return outputs, batch_y

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/我家小花儿/article/detail/474521
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号