The children of this node are results of the paper linked to by reference "CONVOLUTIONAL RECURRENT NEURAL NETWORKS FOR MUSIC CLASSIFICATION." It compares common CNN structures to a CRNN structure in the context of music classification.

CRNNs for Music Classification

A Convolutional Recurrent Neural Network (CRNN) is a hybrid architecture that combines a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). The goal of a CRNN is to leverage CNNs for local feature extraction and RNNs for temporal summarization of the extracted features. This combination is particularly effective for classifying inputs that are visually complex and possess temporal characteristics.

Nova Southeastern University

Google

Convolutional networks, also known as convolutional neural networks, or CNNs, are a specialized kind of neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1-D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D grid of pixels. 

The network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers.

The CNN gets its power from the following concepts:
- Sparse connectivity
- Parameter sharing

Convolutional Neural Network (CNN)

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

Choi, K., Fazekas, G., Sandler, M., & Cho, K. (2017, March). Convolutional recurrent neural networks for music classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2392-2396). IEEE. https://doi.org/10.1109/icassp.2017.7952585

Convolutional Recurrent Neural Networks for Music Classification (Choi et al., 2017)

A helpful website for understanding how CNN works:
https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53

CNN Reference

CNNs are currently being applied to several applications, including:

 - Computer vision—medical image analysis, image recognition, face detection and video classification
 - Natural Language Processing (NLP)—semantic parsing, sentence modeling and search query retrieval
 - Drug discovery—the discovery of chemical features and prediction of medicinal benefits

Applications of Convolutional Neural Networks

CNNs and RNNs are not mutually exclusive, as both can perform classification of image and text inputs, creating an opportunity to combine the two network types for increased effectiveness. This is especially true if the input to be classified is visually complex with added temporal characteristics that a CNN alone would be unable to process.

Typically, when these two network types are combined, sometimes referred to as a CRNN, inputs are first processed by CNN layers whose outputs are then fed to RNN layers. CNN Long Short-Term Memory (LSTM) architectures are particularly promising, as they facilitate analysis of inputs over longer periods than could be achieved with lower-level RNN architecture types.

Currently, these hybrid architectures are being explored for use in applications like video scene labeling, emotion detection or gesture recognition, video identification or gait recognition, and DNA sequence prediction.

Hybrid of Convolutional and Recurrent Neural Network

- Python: conv_forward
- TensorFlow: tf.nn.conv2d
- Keras: Conv2D
- Pytorch: torch.nn.Conv2d
- Caffe: message ConvolutionParameter
- Theano: theano.tensor.nnet.conv.conv2d

Methods to Calculate Convolution in Python

CNNs are made up of three-layer types—convolutional, pooling and fully-connected (FC).

In the convolutional layers, the input matrix is passed through a set of filters that output a feature map. This output is then sent to a pooling layer, which reduces the size of the feature map. This helps reduce the processing time by condensing the map to its most essential information.

The convolutional and pooling processes are repeated several times, with the number of repeats depending on the network, after which the condensed feature map outputs are sent to a series of FC layers. These FC layers then flatten the maps together and compare the probabilities of each feature occurring in conjunction with the others, until the best classification is determined.

Convolutional Neural Networks Architecture

A 3D CNN is similar to a 2D CNN except that it differs in (non-exhaustive listing):
1- 3D Convolution Layers

2- 3D MaxPool Layers

3D Convolutional Neural Network

Visualizing and Understanding Convolutional Networks Paper

- Convolutional neural network can be used for obtaining high- dimensional images rather than being used in class prediction for logistic regression  or value prediction for regression problem

- Output of the CNN model can be a tensor S(i,j,k) which represent the pixel(j,k) of ith class, such a collaborative  effort can be used to precisely form a perfect image as the output.

Strategies to get output dimension same as the input are 

- Pooling is completely avoided or can use pooling with unit stride.

- Pixel-wise labelling involves initial guess of image pixels and refining this guess by interacting with neighboring pixels by parameter sharing.

Structured Output from CNN

Questions about the ReLU.

The Neocognitron, introduced by Fukushima in 1982, represents one of the earliest architectures and foundational references in the development of Convolutional Neural Networks (CNNs).

Neocognitron

In object detection, a recognition system should not be overly concerned with the precise location of an object within an image. For instance, similar to finding the character Waldo in chaotic scenes, a detector can sweep an image to assign likelihood scores to different patches, indicating where the target might be. Convolutional Neural Networks (CNNs) systematize this spatial invariance, exploiting it to learn useful representations with fewer parameters.

Spatial Invariance in Object Detection

The locality principle asserts that the earliest layers of a computer vision network should focus exclusively on local regions, without regard for the contents of the image in distant areas. Mathematically, this constrains the network's parameters such that outside a localized range ($$|a| > \Delta$$ or $$|b| > \Delta$$), the convolution weights are set to zero ($$[\mathbf{V}]_{a, b} = 0$$). The hidden representation is computed as:

$$[\mathbf{H}]_{i, j} = u + \sum_{a = -\Delta}^{\Delta} \sum_{b = -\Delta}^{\Delta} [\mathbf{V}]_{a, b} [\mathbf{X}]_{i+a, j+b}$$

By restricting the network to glean relevant information only from a bounded window around each pixel, this principle drastically reduces the number of learnable parameters (e.g., from $$4 \times 10^6$$ down to $$4 \Delta^2$$).

Locality Principle in Computer Vision

A Fully Convolutional Network (FCN) is designed for dense prediction tasks, such as semantic segmentation. The model architecture begins with a standard Convolutional Neural Network (CNN) to extract image features. Next, it employs a $$1 	imes 1$$ convolutional layer to transform the number of channels to match the number of target classes. Finally, it uses a transposed convolutional layer to scale the height and width of the feature maps back to the dimensions of the original input image. The resulting output has the same spatial dimensions as the input, with each output channel representing the predicted classes for the pixel at the corresponding spatial position.

Fully Convolutional Network (FCN) Architecture

Convolutional Recurrent Neural Network (CRNN)

RNNs are currently being applied to several applications, including:

 - Temporal analysis—time-series anomaly detection and time-series prediction
 - Computer vision—Image description, video tagging and video analysis
 - NLP—Sentiment analysis, speech recognition, language modeling, machine translation and text generation

Applications of RNN

The simplest form of RNN (“vanilla” RNN) is similar to a regular neural network, only it contains a loop that allows the model to carry forward results from previous neuron layers.

RNN Basic Structure

RNNs come in many variants, including:
 - Bidirectional RNNs
 - Memory Units (LSTMs) RNNs
 - Fully recurrent 
 - Elman/Jordan networks, or Simple Recurrent Networks (SRN)
 - Recursive neural network
 - Hopfield
 - Echo State Network (ESN)
 - Stacked RNNs
 - Hierarchical
 - Neural Turing machines (NTMs)
 - Differentiable Neural Computer (DNC) 
 - Recurrent Multi-Layer Perceptron (RMLP)
 - Independent RNN (IndRNN)
 - Neural history compressor 
 - Second order RNNs 
 - Gated recurrent unit (GRU)
 - Continuous-Time Recurrent Neural Network (CTRNN)
 - Multiple Timescales Recurrent Neural Network (MTRNN) 
 - Neural Network Pushdown Automata (NNPDA)

RNN Extensions and Types

Given the predictions for each time step $\hat{y}^{(t)}$, together with the ground truth labels $y^{(t)}$, we can calculate the loss on step t
$$J^{(t)} = \mathcal{L} (\hat{y}^{(t)}, y^{(t)})$$
To get the overall loss, we only need to average these:
$$J = \frac{1}{T} \sum_{t=1}^T J^{(t)} $$

Loss Function for RNN

Both RNNs and HMMs make use of dynamic state. This means that these models depend both on the input system and current state. Additionally, RNNs have distributed state acquiring them with larger memory and computational capacity(which isn't true about HMMs). 

RNNs(Recurrent Neural Networks) vs HMMs (Hidden Markov Models)

Unlike feedforward neural networks:

1. Make use of sequence of inputs by using their memory
2. Cost function is applied at each time step unlike feedforward neural networks were cost is applied only to the final output. 

RNNs vs Feedforward Neural Networks

Why is an RNN (Recurrent Neural Network) used for machine translation, say translating English to French? (Check all that apply.)

In an RNN model, predictions are only based on knowledge/features/inputs from previous nodes. However, information from the rest of the sentence might be useful in the previous step as well.




Example: In the following examples, name entity recognition of Teddy can be different based on information that come after 'Teddy".




He said, "Teddy Roosevelt was a great president"
He said, "Teddy bear are on sale!"

RNN Problem

If we consider Tx the length of input and Ty the length of output:

1- Many-to-many ( Tx=Ty)

2- Many-to_many (Tx!=Ty)

3- Many-to-one

4-One_to_Many


Different types of RNN (in terms of input/output)

Gradients propagated many times tend to vanish or explode. Recurrent networks involve the composition of the same function many times which can result in nonlinear behavior. The figure shows the function after each time the transition function has been composed.



Long Term Dependencies Problem

"Conditioning" in the context of sequence to sequence learning in RNNs is the process of computing the probability of obtaining the output sequence conditioned on the input sequence, or p(y|x). The network is used to model this conditional probability mapping

We can extend a model to represent a distribution P(y|x) by using 3 common methods:

* As an extra input at each time step, or
* As the initial state h(0), or
* Both


Modeling Sequences Conditioned on Context with RNNs

As mentioned in the introduction to long-term dependencies, there are 3 ways to address long-term dependencies, and 1 of them is "Leakage units and other multi-timescale strategies", which will be introduced in this unit

This idea has led to numerous approaches, I will introduce three approaches here.

1 - Adding Skip Connections through Time

Adding direct connections from variables in the distant past to current variables is one way to get coarse time scales.
The gradient may vanish or exponentially about the time step. The algorithm introduces circular connections with d-delay to alleviate this problem. The rate of exponential decrease of the derivative is now related to τ/d correlation rather than τ. Since there is both a delay and a single-step connection, the gradient may still explode exponentially into t. This allows the learning algorithm to capture longer dependencies.

2- Leaky Units and a Spectrum of Diﬀerent Time Scales

The idea is similar to the sliding average model and shadow variables in TensorFlow.
The sliding average actually contains information over a long period of time.
In the long-term dependence problem, we can use the sliding average instead of the real parameters, making more long-term information to be considered when the parameters are iterated.

3- Removing connection

Unlike skiping connections, which add edges, the unit can self-select favorable dependencies. Whereas a removing connection directly removes the connection of length 1, forcing the connection to be longer.

Leaky Units and Other Strategies for Multiple Time Scales

For all hidden state vectors in a recurrent neural network, a pooling function can be applied to compute their element-wise mean: $$\mathbf{h}_{\text{mean}} = \frac{1}{n} \sum_{i=1}^{n} \mathbf{h}_{i}$$ Alternatively, an element-wise maximum can also be used as the pooling function.

Pooling Layer in RNN

One reason for the inability of RNNs to carry forward critical information is that the hidden layers are being asked to perform two tasks simultaneously: 
- Provide information useful for the current decision
- Updating and carrying forward information required for future decisions

Inability of RNNs to Carry Forward Critical Information

Stacked RNNs consist of multiple networks where the output of one layer serves as the input to a subsequent layer, as shown in Fig. 9.10.

Stacked RNNs generally outperform single-layer networks. One reason for this
success seems to be that the network induces representations at differing levels of abstraction across layers. The optimal number of stacked RNNs is specific to each application and to each training set. However, as the number of stacks is increased the training costs rise quickly.

Stacked RNNs

A bidirectional RNN combines two independent bidirectional RNN
independent RNNs, one where the input is processed from the start to the end, and the other from the end to the start. We then concatenate the two representations computed by the networks into a single vector that captures both the left and right contexts of an input at each point in time. 

Fig. 9.11 illustrates such a bidirectional network that concatenates the outputs of the forward and backward pass. Other simple ways to combine the forward and backward contexts include element-wise addition or multiplication. The output at each step in time thus captures information to the left and to the right of the current input. In sequence labeling applications, these concatenated outputs can serve as the basis for a local labeling decision.

Bidirectional RNNs

In recurrent neural networks, the length of the input sequence introduces a unique dimension of depth. To influence the final output, inputs from the first time step must propagate through a chain of $$T$$ layers corresponding to each time step. During backpropagation, this creates a chain of matrix-products of length $$\mathcal{O}(T)$$, which frequently leads to numerical instability in the form of vanishing or exploding gradients.

Sequence Depth in Recurrent Neural Networks

A vanilla Recurrent Neural Network (RNN) module can be implemented concisely by wrapping high-level deep learning APIs, such as nn.RNN in PyTorch or tf.keras.layers.SimpleRNN in TensorFlow. The custom module initializes the built-in recurrent layer and defines a forward pass that accepts an input sequence along with an optional initial hidden state. The built-in layer efficiently processes the sequence to compute and return both the output sequence and the final updated hidden state.

Concise RNN Module Implementation

As of current versions, the JAX-based Flax linen API does not provide a built-in RNNCell for the concise implementation of a vanilla Recurrent Neural Network (RNN). While developers can easily access advanced recurrent variants natively—such as Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs)—a standard vanilla RNN must be implemented from scratch using lower-level tensor operations.

Learn Before

Related

Learn After