"Large-scale Video Classification with Convolutional Neural Networks" is one of the first papers that have mentioned CNN for video classification. Two main questions addressed by this paper are

• "what temporal connectivity pattern in a CNN architecture is best at taking advantage of local motion information present in the video?"

• "How does the additional motion information influence the predictions of a CNN and how much does it improve performance overall? "