devbox
我家自动化
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

22- Pytorch实现天气分类 (Pytorch系列) (项目二十二)_四种天气数据集

作者：我家自动化 | 2024-05-17 04:21:10

赞

踩

四种天气数据集

项目要点

  • 4种天气数据的分类:   cloudy,  rain,  shine,  sunrise.
  • all_img_path = glob.glob(r'G:\01-project\08-深度学习\day 56 迁移学习\dataset/*.jpg')     # 指定文件夹   # import glob
  • 获取随机数列: index = np.random.permutation(len(all_img_path))
  • 建立数组和索引的关联: idx_to_species = dict((i, c) for i, c in enumerate(species))
  • transform = transforms.Compose([transforms.Resize((96, 96)), transforms.ToTensor()])    # 转换为tensor  # 定义transform
  • 数据由numpy转换为tensor: torch.from_numpy(np.array(label)).long()
  • 判断图片的通道数: if np.array(img).shape[-1] == 3
  • 打开文件夹图片: img = Image.open(all_img_path[0])
  • 数据转换为ndarray: np.asarray(img).shape
  • train_d1 = torch.utils.data.DataLoader(train_ds, batch_size = 16, shuffle = True, collate_fn = MyDataset.collate_fn, drop_last = True)      # 定义dataloader  # 最后一批数据直接不用

定义模型:

  • 添加卷积层: self.conv1 = nn.Conv2d(3, 32, 3)
  • 添加激活层: x = self.pool(F.relu(self.conv1(x)))
  • 添加BN层: self.bn1 = nn.BatchNorm2d(32)     # x = self.bn1(x)
  • 添加Flatten层: x = nn.Flatten()(x)     # 用来将输入“压平”，即把多维的输入一维化，# 常用在从 卷积层到全连接层的过渡。
  • 添加卷积层: self.fc1 = nn.Linear(64 * 10 * 10, 1024)   
  • 添加激活层: x = F.relu(self.fc1(x))
  • 添加dropout: self.dropout = nn.Dropout()    # 防止过拟合
  • 添加输出层: self.fc3 = nn.Linear(256, 4)   
    • x = self.fc3(x)
  • 定义程序运行位置: device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
  • 定义优化器: optimizer = optim.Adam(model.parameters(), lr=0.001)
  • 定义loss: loss_fn = nn.CrossEntropyLoss()
  • 定义梯度下降:
  1. for x, y in train_loader:
  2.         x, y = x.to(device), y.to(device)
  3.         y_pred = model(x)
  4.         loss = loss_fn(y_pred, y)
  5.         optimizer.zero_grad()
  6.         loss.backward()
  7.         optimizer.step()
  8.         
  9.         with torch.no_grad():
  10.             y_pred = torch.argmax(y_pred, dim=1)
  11.             correct += (y_pred == y).sum().item()
  12.             total += y.size(0)
  13.             running_loss += loss.item()

一 自定义数据集分类

 4种天气数据的分类: cloudy,  rain,  shine,  sunrise.

1.1 导包

  1. import torch
  2. import numpy as np
  3. from torchvision import transforms
  4. import glob
  5. from PIL import Image
  6. import torch.nn.functional as F
  7. import torch.optim as optim
  8. import matplotlib.pyplot as plt

1.2 数据导入

  • 指定文件夹
all_img_path = glob.glob(r'G:\01-project\08-深度学习\day 56 迁移学习\dataset/*.jpg')
  • 打乱顺序 
  1. # permutation 排列组合
  2. # 借助ndarray的索引取值的方法, 打乱数据
  3. index = np.random.permutation(len(all_img_path))
  4. index   # array([ 175, 1027,  530, ...,    4,  831,   65])
  1. species = ['cloudy', 'rain', 'shine', 'sunrise']
  2. # 建立类别和索引之间的映射关系
  3. idx_to_species = dict((i, c) for i, c in enumerate(species))
  4. # {0: 'cloudy', 1: 'rain', 2: 'shine', 3: 'sunrise'}
  5. # 生成所有图片的label
  6. all_labels = []
  7.  
  8. for img in all_img_path:
  9.     for i,c in enumerate(species):
  10.         if c in img:
  11.             all_labels.append(i)
  •  数据格式转换
  1. all_labels = np.array(all_labels, dtype=np.int64)[index]
  2. all_labels   # array([0, 3, 2, ..., 0, 3, 0], dtype=int64)
  1. all_img_path = np.array(all_img_path)[index]
  2. all_img_path

1.3 数据拆分

  1. # 手动划分一下训练数据和测试数据
  2. split = int(len(all_img_path) * 0.8)  # int 只取整数部分
  3.  
  4. train_imgs = all_img_path[:split]
  5. train_labels = all_labels[:split]
  6.  
  7. test_imgs = all_img_path[split:]
  8. test_labels = all_labels[split:]
  • 定义 transform
  1. # 定义transform
  2. transform = transforms.Compose([transforms.Resize((96, 96)),
  3.                                 transforms.ToTensor()])  # 转换为tensor

1.4 数据处理

  1. class MyDataset(torch.utils.data.Dataset):
  2.     def __init__(self, img_paths, labels, transform):  # 接受初始化数据
  3.         self.imgs = img_paths
  4.         self.labels = labels
  5.         self.transforms = transform
  6.         
  7.     def __getitem__(self, index):   # 取上面的数据
  8.         # 根据index获取item
  9.         img_path = self.imgs[index]
  10.         label = self.labels[index]
  11.         
  12.         # 通过PIL的Image读取图片
  13.         img = Image.open(img_path)
  14.         if np.array(img).shape[-1] == 3:
  15.             data = self.transforms(img)
  16.         
  17.             return data, torch.from_numpy(np.array(label)).long()
  18.         else:
  19.             # 否则为有问题的图片
  20.             print(img_path)
  21.             # print(np.array(img).shape)
  22.             # print(np.array(img))
  23.             return self.__getitem__(index+1)
  24.     
  25.     def __len__(self):  # 调用数据时, 返回长度
  26.         return len(self.imgs)  # 返回个数
  27.     
  28.     # 重写collate_fn
  29.     @staticmethod
  30.     def collate_fn(batch):
  31.         # batch是列表, 长度是batch_size
  32.         # 列表的每个元素是一个元组(x, y)
  33.         # [(x1, y1), (x2, y2).......]
  34.         # collate_fn 的作用, 把所有的x,y分别放到一起, x在一起, y在一起.
  35.         # 把batch中返回值为空的部分过滤掉
  36.         batch = [sample for sample in batch if sample is not None]
  37.         # 简单方法, 直接调用默认的collate方法
  38.         # from torch.utils.data.dataloader import default_collate
  39.         # return default_collate(batch)
  40.         
  41.         # 方式二
  42.         imgs, labels = zip(*batch)
  43.         return torch.stack(imgs, 0), torch.stack(labels, 0)
  44.  
  45. dataset = MyDataset(all_img_path, all_labels, transform)
  46. len(dataset)   # 1122
  1. train_ds = MyDataset(train_imgs, train_labels, transform)
  2. test_ds = MyDataset(test_imgs, test_labels, transform)
  3. # dataloader
  4. train_d1 = torch.utils.data.DataLoader(train_ds, batch_size = 16,
  5.                                        shuffle = True,
  6.                                        collate_fn=MyDataset.collate_fn,
  7.                                        drop_last = True)  # 最后一批数据直接不用
  8. test_d1 = torch.utils.data.DataLoader(test_ds, batch_size = 16 * 2,
  9.                                       collate_fn=MyDataset.collate_fn, drop_last = True)
  10. for x, y in train_d1:
  11.     print(x.shape,y.shape)

  1. imgs, labels = next(iter(train_d1))
  2. imgs.shape     # torch.Size([16, 3, 96, 96])
  3. labels     # tensor([3, 3, 2, 3, 2, 3, 3, 0, 1, 1, 0, 3, 2, 0, 1, 1])

1.5 定义模型

  1. # 定义模型  # 添加BN层
  2. import torch.nn as nn
  3. class Net(nn.Module):
  4.     def __init__(self):
  5.         super().__init__()
  6.         self.conv1 = nn.Conv2d(3, 32, 3) # 卷积  # (96, 96, 3)  --> (32, 94, 94)
  7.         # 主要做标准化处理
  8.         self.bn1 = nn.BatchNorm2d(32)
  9.         self.pool = nn.MaxPool2d(2, 2) # 池化   # (32, 47, 47)
  10.         self.conv2 = nn.Conv2d(32, 32, 3)    # (32, 45, 45) --> pooling --> (32, 22, 22)
  11.         self.bn2 = nn.BatchNorm2d(32)
  12.         self.conv3 = nn.Conv2d(32, 64, 3)   # (64, 22, 22)  --> pooling --> (64, 10, 10)
  13.         self.bn3 = nn.BatchNorm2d(64)
  14.         self.dropout = nn.Dropout()  # 防止过拟合    # 
  15.         
  16.         # batch, channel, height, width, 64
  17.         # 全连接层
  18.         self.fc1 = nn.Linear(64 * 10 * 10, 1024)
  19.         self.bn_fc1 = nn.BatchNorm1d(1024)
  20.         self.fc2 = nn.Linear(1024, 256)
  21.         self.bn_fc2 = nn.BatchNorm1d(256)
  22.         self.fc3 = nn.Linear(256, 4)
  23.         
  24.     def forward(self, x):
  25.         x = self.pool(F.relu(self.conv1(x)))
  26.         x = self.bn1(x)
  27.         x = self.pool(F.relu(self.conv2(x)))
  28.         x = self.bn2(x)
  29.         x = self.pool(F.relu(self.conv3(x)))
  30.         x = self.bn3(x)
  31.         
  32.         # x.view(-1, 64 * 10 * 10)
  33.         # flatten , Flatten层用来将输入“压平”，即把多维的输入一维化，
  34.         # 常用在从卷积层到全连接层的过渡。
  35.         x = nn.Flatten()(x)
  36.         x = F.relu(self.fc1(x))
  37.         x = self.bn_fc1(x)
  38.         x = self.dropout(x)
  39.         x = F.relu(self.fc2(x))
  40.         x = self.bn_fc2(x)
  41.         x = self.dropout(x)
  42.         x = self.fc3(x)
  43.         return x
  1. device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
  2. device   # device(type='cpu')
  3.  
  4. # 生成模型
  5. model = Net()
  6. # 把模型拷贝到GPU
  7. if torch.cuda.is_available():
  8.     model.to(device)

1.6 定义训练

  1. optimizer = optim.Adam(model.parameters(), lr=0.001)
  2. loss_fn = nn.CrossEntropyLoss()
  3.  
  4. # 定义训练过程
  5. def fit(epoch, model, train_loader, test_loader):
  6.     correct = 0
  7.     total = 0
  8.     running_loss = 0
  9.     
  10.     for x, y in train_loader:
  11.         x, y = x.to(device), y.to(device)
  12.         y_pred = model(x)
  13.         loss = loss_fn(y_pred, y)
  14.         optimizer.zero_grad()
  15.         loss.backward()
  16.         optimizer.step()
  17.         
  18.         with torch.no_grad():
  19.             y_pred = torch.argmax(y_pred, dim=1)
  20.             correct += (y_pred == y).sum().item()
  21.             total += y.size(0)
  22.             running_loss += loss.item()
  23.             
  24.     epoch_loss = running_loss / len(train_loader.dataset)
  25.     epoch_acc = correct / total
  26.     
  27.     # 测试过程
  28.     test_correct = 0
  29.     test_total = 0
  30.     test_running_loss = 0
  31.     with torch.no_grad():
  32.         for x, y in test_loader:
  33.             x, y = x.to(device), y.to(device)
  34.             y_pred = model(x)
  35.             loss = loss_fn(y_pred, y)
  36.             y_pred = torch.argmax(y_pred, dim=1)
  37.             test_correct += (y_pred == y).sum().item()
  38.             test_total += y.size(0)
  39.             test_running_loss += loss.item()
  40.     test_epoch_loss = test_running_loss / len(test_loader.dataset)
  41.     test_epoch_acc = test_correct /test_total
  42.     
  43.     print('epoch', epoch,
  44.           'loss', round(epoch_loss, 3),
  45.           'accuracy', round(epoch_acc, 3),
  46.           'test_loss', round(test_epoch_loss, 3),
  47.           'test_accuracy', round(test_epoch_acc, 3))
  48.     return epoch_loss, epoch_acc, test_epoch_loss, test_epoch_acc
  • 指定训练
  1. # 指定训练次数
  2. epochs = 10
  3. train_loss = []
  4. train_acc = []
  5. test_loss = []
  6. test_acc = []
  7.  
  8. for epoch in range(epochs):
  9.     epoch_loss, epoch_acc, test_epoch_loss, test_epoch_acc = fit(epoch, model,
  10.                                                                  train_d1, test_d1)
  11.     train_loss.append(epoch_loss)
  12.     train_acc.append(epoch_acc)
  13.     
  14.     test_loss.append(epoch_loss)
  15.     test_acc.append(epoch_acc)

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/我家自动化/article/detail/581948
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号