菜鸟追梦旅行

这个屌丝很懒，什么也没留下！

热门标签

PyTorch 1.7 Video 初体验（Video Datasets，Video IO，Video Classification Models，Video Transform）_runtimeerror: not compiled with video_reader suppo

作者：菜鸟追梦旅行 | 2024-03-05 19:01:51

踩

runtimeerror: not compiled with video_reader support, to enable video_reader

Environment 环境

Win 10
Anaconda Navigator
PyCharm
cuda 10.1
torch 1.7.1
torchvision 0.8.2
Python 3.8

Reference 参考链接

Anaconda Navigator 版本的升级：https://www.cnblogs.com/developerchen/p/8879516.html

打开 Anaconda Prompt，输入以下命令：

conda install -c continuumcrew anaconda-navigator=1.5.1

conda update --all

torch 1.7.1 的安装：https://pytorch.org/get-started/locally/

打开 Anaconda Prompt，切换到相应环境，输入以下命令：

pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Pytorch 1.7.1 官方文档：https://pytorch.org/docs/stable/index.html

Video Datasets 视频数据集 & 加载

UCF101：https://pytorch.org/docs/stable/torchvision/datasets.html#ucf101
HMDB51：https://pytorch.org/docs/stable/torchvision/datasets.html#hmdb51
Kinetics400：https://pytorch.org/docs/stable/torchvision/datasets.html#kinetics-400
......

加载 UCF101 数据集


import torchvision.datasets as datasets
 
data = datasets.UCF101(
    root='path/UCF-101',
    annotation_path='path/UCF101TrainTestSplits-RecognitionTask/ucfTrainTestlist',
    frames_per_clip=16,
    num_workers=0
)
 
print(data)

返回值：

video (Tensor[T, H, W, C]): the `T` video frames
audio(Tensor[K, L]): the audio frames, where `K` is the number of channels and `L` is the number of points
label (int): class of the video clip

注意：

win 10 系统下运行该代码一定要加上 num_workers=0，不然会报出如下错误

还需要安装 PyAV 这个库，安装命令：pip install av

在导入 UCF101 数据时，由于 windows 路径用的是“\”，所以在加载数据集时会报出如下错误：

原因 & 解决方案：https://stackoverflow.com/questions/61522539/i-cant-import-the-ucf-101-dataset-torchvision-list-index-out-of-range-error

原因：trainlist01/02/03.txt 和 testlist01/02/03.txt 中的 video path 长这样：ApplyEyeMakeup/v_ApplyEyeMakeup_g01_c01.avi 和 windows 系统路径要求的斜杠（ \ ）不一样

我用的是其中的第一种解决方案：把 trainlist01/02/03.txt 和 testlist01/02/03.txt 中的 / 全部替换为 \

加载 HMDB51 数据集

参数：

root (string) – Root directory of the HMDB51 Dataset.

annotation_path (str) – Path to the folder containing the split files.

frames_per_clip (int) – Number of frames in a clip.

step_between_clips (int) – Number of frames between each clip.

fold (int, optional) – Which fold to use. Should be between 1 and 3.

train (bool, optional) – If True, creates a dataset from the train split, otherwise from the test split.

transform (callable, optional) – A function/transform that takes in a TxHxWxC video and returns a transformed version.

返回值：

video (Tensor[T, H, W, C]): the `T` video frames
audio(Tensor[K, L]): the audio frames, where `K` is the number of channels and `L` is the number of points
label (int): class of the video clip

加载 Kinetics 400 数据集

参数：

root (string) – Root directory of the Kinetics-400 Dataset.

frames_per_clip (int) – number of frames in a clip

step_between_clips (int) – number of frames between each clip

transform (callable, optional) – A function/transform that takes in a TxHxWxC video and returns a transformed version.

返回值：

video (Tensor[T, H, W, C]): the `T` video frames
audio(Tensor[K, L]): the audio frames, where `K` is the number of channels and `L` is the number of points
label (int): class of the video clip

Video I/O 视频 I/O 操作

官方文档：

torchvision.io.read_video()

源码：https://pytorch.org/docs/stable/_modules/torchvision/io/video.html#read_video

Parameters

filename (str) – path to the video file

start_pts (int if pts_unit = 'pts', optional) – float / Fraction if pts_unit = ‘sec’, optional the start presentation time of the video

end_pts (int if pts_unit = 'pts', optional) – float / Fraction if pts_unit = ‘sec’, optional the end presentation time

pts_unit (str, optional) – unit in which start_pts and end_pts values will be interpreted, either ‘pts’ or ‘sec’. Defaults to ‘pts’.

Returns

vframes (Tensor[T, H, W, C]) – the T video frames

aframes (Tensor[K, L]) – the audio frames, where K is the number of channels and L is the number of points

info (Dict) – metadata for the video and audio. Can contain the fields video_fps (float) and audio_fps (int)

补充知识：什么是时间戳？什么是 pts？

https://blog.csdn.net/tanningzhong/article/details/105564589

时间戳单位

前面我们提到采样率，感觉到采样率是个很大的单位，一般标准的音频AAC采样率达到了44kHz,视频采样率也规定在90000Hz.所以我们衡量时间的单位不能再是秒，毫秒这种真实的时间单位，我们的单位应该转换为采样率，也就是一个采样的时间为音视频的时间单位，这就是时间戳的真实值。当我们要播放和控制时，我们再将时间戳根据采样率转换为真实的时间即可。

一句话，时间戳不是真实的时间是采样次数。比如时间戳是160，我们不能认为是160秒或者160毫秒，应该是160个采样。要换算真实时间，我们必须知道采样率，比如8000，那么说明1秒被划分成8000分之一，如果你要明确160个采样占用的时间，则160*(1/8000)即可，即20毫秒。

时间戳增量

就是一帧图像和另外一帧图像之间的时间戳差值，或者一帧音频和一帧音频的时间戳差值。同理时间戳增量也是采样个数的差值不是真实时间差值，还是要根据采样率才能换算成真实时间。

所以对于视频和音频的时间戳计算要一定明确帧率是多少，采样率是多少。

比如视频而言，帧率25，那么对于90000的采样率来说，一帧占用的采样数就是90000/25也就是3600，说明每帧图像的时间戳增量应该是3600，换算成实际时间就是3600*（1/90000）=0.04秒=40毫秒，这也和1/25=0.04秒=40毫秒一致。

对于AAC音频，一帧1024个采样，采样频率是44kHz,所以一帧的播放时间应该是1024*（1/44100）=0.0232秒=23.22毫秒。

用个 Example 更直观的理解这两个概念：


import torchvision.io as io
 
 
vframes, aframes, info = io.read_video(
    filename='path/v_ApplyEyeMakeup_g01_c01.avi',
    pts_unit='pts',
    end_pts=3
)
 
print(vframes.shape)
print(info)
 
 
# output:
# torch.Size([3, 240, 320, 3])
# {'video_fps': 25.0, 'audio_fps': 44100}
 
 
# --------------------------------------------------------------------
 
 
import torchvision.io as io
 
 
vframes, aframes, info = io.read_video(
    filename='path/v_ApplyEyeMakeup_g01_c01.avi',
    pts_unit='sec',
    end_pts=3
)
 
print(vframes.shape)
print(info)
 
 
# output:
# torch.Size([75, 240, 320, 3])
# {'video_fps': 25.0, 'audio_fps': 44100}

torchvision.io.read_video_timestamps()

源码：https://pytorch.org/docs/stable/_modules/torchvision/io/video.html#read_video_timestamps

Parameters

filename (str) – path to the video file

pts_unit (str, optional) – unit in which timestamp values will be returned either ‘pts’ or ‘sec’. Defaults to ‘pts’.

Returns

pts (List[int] if pts_unit = ‘pts’) – List[Fraction] if pts_unit = ‘sec’ presentation timestamps for each one of the frames in the video.

video_fps (float, optional) – the frame rate for the video

Example：


import torchvision.io as io
 
v_pts, v_fps = io.read_video_timestamps(
    filename='path/v_ApplyEyeMakeup_g01_c01.avi',
    pts_unit='pts'
)
 
print(v_pts)
print(v_fps)
 
 
# output
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164]
# 25.0
 
 
 
# ---------------------------------------------------------------------------
 
 
 
 
import torchvision.io as io
 
v_pts, v_fps = io.read_video_timestamps(
    filename='path/v_ApplyEyeMakeup_g01_c01.avi',
    pts_unit='sec'
)
 
print(v_pts)
print(v_fps)
 
 
# output
# [Fraction(1, 25), Fraction(2, 25), Fraction(3, 25), Fraction(4, 25), Fraction(1, 5), Fraction(6, 25), Fraction(7, 25), Fraction(8, 25), Fraction(9, 25), Fraction(2, 5), Fraction(11, 25), Fraction(12, 25), Fraction(13, 25), Fraction(14, 25), Fraction(3, 5), Fraction(16, 25), Fraction(17, 25), Fraction(18, 25), Fraction(19, 25), Fraction(4, 5), Fraction(21, 25), Fraction(22, 25), Fraction(23, 25), Fraction(24, 25), Fraction(1, 1), Fraction(26, 25), Fraction(27, 25), Fraction(28, 25), Fraction(29, 25), Fraction(6, 5), Fraction(31, 25), Fraction(32, 25), Fraction(33, 25), Fraction(34, 25), Fraction(7, 5), Fraction(36, 25), Fraction(37, 25), Fraction(38, 25), Fraction(39, 25), Fraction(8, 5), Fraction(41, 25), Fraction(42, 25), Fraction(43, 25), Fraction(44, 25), Fraction(9, 5), Fraction(46, 25), Fraction(47, 25), Fraction(48, 25), Fraction(49, 25), Fraction(2, 1), Fraction(51, 25), Fraction(52, 25), Fraction(53, 25), Fraction(54, 25), Fraction(11, 5), Fraction(56, 25), Fraction(57, 25), Fraction(58, 25), Fraction(59, 25), Fraction(12, 5), Fraction(61, 25), Fraction(62, 25), Fraction(63, 25), Fraction(64, 25), Fraction(13, 5), Fraction(66, 25), Fraction(67, 25), Fraction(68, 25), Fraction(69, 25), Fraction(14, 5), Fraction(71, 25), Fraction(72, 25), Fraction(73, 25), Fraction(74, 25), Fraction(3, 1), Fraction(76, 25), Fraction(77, 25), Fraction(78, 25), Fraction(79, 25), Fraction(16, 5), Fraction(81, 25), Fraction(82, 25), Fraction(83, 25), Fraction(84, 25), Fraction(17, 5), Fraction(86, 25), Fraction(87, 25), Fraction(88, 25), Fraction(89, 25), Fraction(18, 5), Fraction(91, 25), Fraction(92, 25), Fraction(93, 25), Fraction(94, 25), Fraction(19, 5), Fraction(96, 25), Fraction(97, 25), Fraction(98, 25), Fraction(99, 25), Fraction(4, 1), Fraction(101, 25), Fraction(102, 25), Fraction(103, 25), Fraction(104, 25), Fraction(21, 5), Fraction(106, 25), Fraction(107, 25), Fraction(108, 25), Fraction(109, 25), Fraction(22, 5), Fraction(111, 25), Fraction(112, 25), Fraction(113, 25), Fraction(114, 25), Fraction(23, 5), Fraction(116, 25), Fraction(117, 25), Fraction(118, 25), Fraction(119, 25), Fraction(24, 5), Fraction(121, 25), Fraction(122, 25), Fraction(123, 25), Fraction(124, 25), Fraction(5, 1), Fraction(126, 25), Fraction(127, 25), Fraction(128, 25), Fraction(129, 25), Fraction(26, 5), Fraction(131, 25), Fraction(132, 25), Fraction(133, 25), Fraction(134, 25), Fraction(27, 5), Fraction(136, 25), Fraction(137, 25), Fraction(138, 25), Fraction(139, 25), Fraction(28, 5), Fraction(141, 25), Fraction(142, 25), Fraction(143, 25), Fraction(144, 25), Fraction(29, 5), Fraction(146, 25), Fraction(147, 25), Fraction(148, 25), Fraction(149, 25), Fraction(6, 1), Fraction(151, 25), Fraction(152, 25), Fraction(153, 25), Fraction(154, 25), Fraction(31, 5), Fraction(156, 25), Fraction(157, 25), Fraction(158, 25), Fraction(159, 25), Fraction(32, 5), Fraction(161, 25), Fraction(162, 25), Fraction(163, 25), Fraction(164, 25)]
# 25.0

torchvision.io.write_video()

源码：https://pytorch.org/docs/stable/_modules/torchvision/io/video.html#write_video

Parameters

filename (str) – path where the video will be saved

video_array (Tensor[T, H, W, C]) – tensor containing the individual frames, as a uint8 tensor in [T, H, W, C] format

fps (Number) – frames per second

class torchvision.io.VideoReader(path, stream='video')

官方文档：https://pytorch.org/docs/stable/torchvision/io.html#fine-grained-video-api

Fine-grained video-reading API. Supports frame-by-frame reading of various streams from a single video container.

Parameters

path (string) – Path to the video file in supported format

stream (string, optional) – descriptor of the required stream, followed by the stream id, in the format {stream_type}:{stream_id}. Defaults to "video:0". Currently available options include ['video', 'audio']

注意：我使用的时候报出了如下错误。原因是 VideoReader 还在测试【Beta】中，网上有人说安装 ffmpeg 后就可以了，但是我试了不管是在系统还是在 conda 下安装都没有用，还是等正式推出之后再说吧。。。

参考：

"RuntimeError: Not compiled with video_reader support" raises when I use the new fine-grained VideoReader API. https://github.com/pytorch/vision/issues/2934#issuecomment-718834813
官方解释报错链接（还在测试【Beta】中）：https://github.com/pytorch/vision/releases/tag/v0.8.0
ffmpeg 的 conda 下载与安装：conda install ffmpeg
ffmpeg 的 windows 下载与安装：https://www.zhihu.com/question/288655694/answer/1605692761

常用函数：

__next__() ：Decodes and returns the next frame of the current stream

Returns：

a dictionary with fields data and pts containing decoded frame and corresponding timestamp

get_metadata()：Returns video metadata

Returns：

dictionary containing duration and frame rate for every stream

seek(time_s: float)：Seek within current stream.

Parameters

time_s (float) – seek time in seconds

Video Transform 视频变换操作

官方源码：

https://github.com/pytorch/vision/blob/master/torchvision/transforms/_functional_video.py（比下一个链接更底层一点）
https://github.com/pytorch/vision/blob/master/torchvision/transforms/_transforms_video.py（上一个链接包装了一下）

我暂时没有找到官方文档，不过从源码里的注释里也能明白。

第二个链接里官方给出的 video 相关的 Transform 函数如下：

RandomCropVideo
RandomResizedCropVideo
CenterCropVideo
NormalizeVideo
ToTensorVideo
RandomHorizontalFlipVideo

ToTensorVideo()

Convert tensor data type from uint8 to float, divide value by 255.0 and permute the dimensions of clip tensor.

和图片的 ToTensor() 操作类似，但要注意维度的顺序！

Args:
clip (torch.tensor, dtype=torch.uint8): Size is (T, H, W, C)

Return:
clip (torch.tensor, dtype=torch.float): Size is (C, T, H, W)

NormalizeVideo()

Normalize the video clip by mean subtraction and division by standard deviation.

和图片的 Normalize() 函数是一致的。不过图片通常使用 ImageNet 的 mean 和 std，视频用的是 Kinetics-400 的 mean = [0.43216, 0.394666, 0.37645] and std = [0.22803, 0.22145, 0.216989]（来源：https://pytorch.org/docs/stable/torchvision/models.html#video-classification）。

Args:
mean (3-tuple): pixel RGB mean
std (3-tuple): pixel RGB standard deviation
inplace (boolean): whether do in-place normalization

RandomHorizontalFlipVideo()

Flip the video clip along the horizonal direction with a given probability.

没有 Video Vertically Flip 也能理解吧

Args:
p (float): probability of the clip being flipped. Default value is 0.5

CenterCropVideo()

Args:
clip (torch.tensor): Video clip to be cropped. Size is (C, T, H, W)

crop_size: int / tuple

Returns:
torch.tensor: central cropping of video clip. Size is (C, T, crop_size, crop_size)

RandomCropVideo()

Args:
clip (torch.tensor): Video clip to be cropped. Size is (C, T, H, W)

size: int / tuple

Returns:
torch.tensor: randomly cropped/resized video clip.

RandomResizedCropVideo()

Args:
clip (torch.tensor): Video clip to be cropped. Size is (C, T, H, W)

  scale:【Default】(0.08, 1.0)

  ratio:【Default】(3.0 / 4.0, 4.0 / 3.0)

  interpolation_mode:【Default】"bilinear"

Returns:
torch.tensor: randomly cropped/resized video clip.

Example


import torchvision.transforms as transform
import torchvision.transforms._transforms_video as v_transform
import torchvision.io as io
 
 
vframes, aframes, info = io.read_video(
    filename='path/v_ApplyEyeMakeup_g01_c01.avi',
    pts_unit='pts',
)
 
trans = transform.Compose([
    v_transform.ToTensorVideo(),
    v_transform.RandomHorizontalFlipVideo(),
    v_transform.RandomResizedCropVideo(112),
])
 
print(vframes.shape)
print(trans(vframes))
print(trans(vframes).shape)
 
 
 
 
# output:
# 原来的 video clip tensor's shape：torch.Size([164, 240, 320, 3])
# Transform 后的 video clip tensor's shape：torch.Size([3, 164, 112, 112])

Video Classification Model 视频动作分类模型

官方文档：https://pytorch.org/docs/stable/torchvision/models.html#video-classification

源码：https://pytorch.org/docs/stable/_modules/torchvision/models/video/resnet.html

模型：

ResNet 3D 18
ResNet MC 18
ResNet (2+1)D

这些模型我没太详细接触过，文档里已经非常贴心的给出了相应的论文：https://arxiv.org/abs/1711.11248。

Parameters

pretrained (bool) – If True, returns a model pre-trained on Kinetics-400

progress (bool) – If True, displays a progress bar of the download to stderr

Returns

Network

Example


import torchvision.models.video as v_model
 
model = v_model.r3d_18(pretrained=True)
 
print(model)

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/菜鸟追梦旅行/article/detail/193177

PyTorch 1.7 Video 初体验（Video Datasets，Video IO，Video Classification Models，Video Transform）_runtimeerror: not compiled with video_reader suppo

目录

Environment 环境

Reference 参考链接

Video Datasets 视频数据集 & 加载

加载 UCF101 数据集

加载 HMDB51 数据集

加载 Kinetics 400 数据集

Video I/O 视频 I/O 操作

torchvision.io.read_video()

torchvision.io.read_video_timestamps()

torchvision.io.write_video()

class torchvision.io.VideoReader(path, stream='video')

Video Transform 视频变换操作

ToTensorVideo()

NormalizeVideo()

RandomHorizontalFlipVideo()

CenterCropVideo()

RandomCropVideo()

RandomResizedCropVideo()

Example

Video Classification Model 视频动作分类模型

Example