- slow learning of temporal information by the network

San Jose State University

Videos are sequences of images, where each image is revealed as time progresses and due to persistence of vision we perceive them as moving images.  Videos are represented  using 4 dimensional tensors( one channel dimension.one temporal dimension( time ) and two spatial dimension).There are different neural network models that can be used to learn the spatio-temporal features.

- Single-frame
- Early Fusion
- Late Fusion
- Slow Fusion

Time Information Fusion in CNNs

Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014). Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 1725-1732).

Large-scale Video Classification with Convolutional Neural Networks

- Each video is analyzed frame by frame,each frame in a video will go through a CNN
- AlexNet is used as the baseline model
- input image size is 170 x 170 x 3

Single-frame

- First convolutional layer is fed  information from a fixed number of frames.
- Filter size of the first convolutional layer is 11 x 11 x 3 x T pixels

Early Fusion

- two separate single frame work network
- penultimate fully connected layer learns the spatio-temporal features 

Learn Before

Related