- Each video is analyzed frame by frame,each frame in a video will go through a CNN
- AlexNet is used as the baseline model
- input image size is 170 x 170 x 3

Single-frame

- First convolutional layer is fed  information from a fixed number of frames.
- Filter size of the first convolutional layer is 11 x 11 x 3 x T pixels

Early Fusion

- two separate single frame work network
- penultimate fully connected layer learns the spatio-temporal features 

Late Fusion

- slow learning of temporal information by the network

Slow Fusion

Videos are sequences of images, where each image is revealed as time progresses and due to persistence of vision we perceive them as moving images.  Videos are represented  using 4 dimensional tensors( one channel dimension.one temporal dimension( time ) and two spatial dimension).There are different neural network models that can be used to learn the spatio-temporal features.

- Single-frame
- Early Fusion
- Late Fusion
- Slow Fusion

San Jose State University

"Large-scale Video Classification with Convolutional Neural Networks" is one of the first papers that have mentioned CNN for video classification. Two main questions addressed by this paper are 

 - "what temporal connectivity pattern in a CNN architecture is best at taking advantage of local motion information present in the video?"

- "How does the additional motion information influence the predictions
of a CNN and how much does it improve performance overall? "

Convolutional Neural Networks for Video classification

Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014). Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 1725-1732).

Large-scale Video Classification with Convolutional Neural Networks

Time Information Fusion in CNNs

In order to reduce runtime  and increase the classification accuracy , low resolution images can be used.

- two streams for handling two spatial resolutions
- context stream receives downsampled frames with reduced spatial resolution.
- fovea stream receives the high resolution center part of the frame,
 

Multiresolution CNNs

- As for the CNN Models (single frame/ early fusion/ late fusion/ slow fusion), they learn well compared to the baseline model. The single-frame model already displays strong performance.
- Compared to the wide gap relative to the feature-based baseline, the variation among different CNN architectures are comparatively insignificant.
- Multiresolution CNN architectures are 2-4 times faster due to reduced dimensions from input layer.


Quantitative Findings of the Sports-1M video classification experiments using CNNs and feature histogram baseline models

Motion-aware network should in theory benefit from motion information. In fact, the slow fusion networks are more likely to underperform when there is camera motion present, implying they struggle to learn complete invariance across all possible angles and speeds of camera translation and zoom.

Difference between single frame network and slow fusion (motion-aware) networks

The models process about inputs from 5 video clips per second for full frame networks and up to 20 clips per second for multi-resolution networks. The networks have each seen approximately 500 million examples throughout a 1-month training period.

Training CNN Models for Sports-1M Video Classification

- Sports - 1M dataset consists of 1 million YouTube videos of varying length belonging to 487 sports categories.
- UCF-101 Activity recognition dataset which contains 13,320 videos consisting of 101 classes mainly classified into 5 groups.
    - Human object interaction
    - Body motion
    - Human - human interaction
    - Playing instruments
    - Sports

Datasets used for Experimentation

Transferring the low level features learned on the dataset Sports-1M to train the dataset UCF-101.

Learn Before

Related

Learn After