








def conv_forward(A_prev, W, b, hparameters):
    """
    Implements the forward propagation for a convolution function
    Arguments:
    A_prev -- output activations of the previous layer, numpy array of shape (m, n_H_prev, n_W_prev, n_C_prev)
    W -- Weights, numpy array of shape (f, f, n_C_prev, n_C)
    b -- Biases, numpy array of shape (1, 1, 1, n_C)
    hparameters -- python dictionary containing "stride" and "pad"
    Returns:
    Z -- conv output, numpy array of shape (m, n_H, n_W, n_C)
    cache -- cache of values needed for the conv_backward() function
    """

    ### START CODE HERE ###
    # Retrieve dimensions from A_prev's shape (≈1 line)
    (m, n_H_prev, n_W_prev, n_C_prev) = A_prev.shape

    # Retrieve dimensions from W's shape (≈1 line)
    (f, f, n_C_prev, n_C) = W.shape

    # Retrieve information from "hparameters" (≈2 lines)
    stride = hparameters['stride']
    pad = hparameters['pad']

    # Compute the dimensions of the CONV output volume using the formula given above. Hint: use int() to floor. (≈2 lines)
    n_H = int((n_H_prev - f + (2 * pad)) / stride + 1)
    n_W = int((n_W_prev - f + (2 * pad)) / stride + 1)

    # Initialize the output volume Z with zeros. (≈1 line)
    Z = np.zeros((m, n_H, n_W, n_C))

    # Create A_prev_pad by padding A_prev
    A_prev_pad = zero_pad(A_prev, pad)

    for i in range(m):  # loop over the batch of training examples
        a_prev_pad = A_prev[i]  # Select ith training example's padded activation
        for h in range(n_H):  # loop over vertical axis of the output volume
            for w in range(n_W):  # loop over horizontal axis of the output volume
                for c in range(n_C):  # loop over channels (= #filters) of the output volume

                    # Find the corners of the current "slice" (≈4 lines)
                    vert_start = stride * h
                    vert_end = stride * h + f
                    horiz_start = stride * w
                    horiz_end = stride * w + f

                    # Use the corners to define the (3D) slice of a_prev_pad (See Hint above the cell). (≈1 line)
                    a_slice_prev = A_prev_pad[i, vert_start:vert_end, horiz_start:horiz_end, :]

                    # Convolve the (3D) slice with the correct filter W and bias b, to get back one output neuron. (≈1 line)
                    Z[i, h, w, c] = conv_single_step(a_slice_prev, W[:, :, :, c], b[:, :, :, c])

    ### END CODE HERE ###

    # Making sure your output shape is correct
    assert (Z.shape == (m, n_H, n_W, n_C))

    # Save information in "cache" for the backprop
    cache = (A_prev, W, b, hparameters)

    return Z, cache

Python: conv_forward

Computes a 2-D convolution given input and 4-D filters tensors.
tf.nn.conv2d(
    input, filters, strides, padding, data_format='NHWC', dilations=None,
    name=None
)
1.input:	A Tensor. Must be one of the following types: half, bfloat16, float32, float64. A Tensor of rank at least 4. The dimension order is interpreted according to the value of data_format; with the all-but-inner-3 dimensions acting as batch dimensions. See below for details.

2.filters:	A Tensor. Must have the same type as input. A 4-D tensor of shape [filter_height, filter_width, in_channels, out_channels]

3. strides	An int or list of ints that has length 1, 2 or 4. The stride of the sliding window for each dimension of input. If a single value is given it is replicated in the H and W dimension. By default the N and C dimensions are set to 1. The dimension order is determined by the value of data_format, see below for details.

4. padding	Either the string "SAME" or "VALID" indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. When explicit padding is used and data_format is "NHWC", this should be in the form [[0, 0], [pad_top,pad_bottom], [pad_left, pad_right], [0, 0]]. When explicit padding used and data_format is "NCHW", this should be in the form [[0, 0], [0, 0],[pad_top, pad_bottom], [pad_left, pad_right]].

5. data_format	An optional string from: "NHWC", "NCHW". Defaults to "NHWC". Specify the data format of the input and output data. With the default format "NHWC", the data is stored in the order of: batch_shape + [height, width, channels]. Alternatively, the format could be "NCHW", the data storage order of: batch_shape + [channels, height, width].

6. dilations	An int or list of ints that has length 1, 2 or 4, defaults to 1. The dilation factor for each dimension ofinput. If a single value is given it is replicated in the H and W dimension. By default the N and C dimensions are set to 1. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of data_format, see above for details. Dilations in the batch and depth dimensions if a 4-d tensor must be 1.

7. name	A name for the operation (optional).

TensorFlow: tf.nn.conv2d


This layer creates a convolution kernel that is convolved with the layer input to produce a tensor of outputs. If use_bias is True, a bias vector is created and added to the outputs. Finally, if activation is not None, it is applied to the outputs as well.

When using this layer as the first layer in a model, provide the keyword argument input_shape (tuple of integers, does not include the sample axis), e.g. input_shape=(128, 128, 3) for 128x128 RGB pictures in data_format="channels_last".


Keras: Conv2D class

CLASStorch.nn.Conv2d(in_channels: int, out_channels: int, kernel_size: Union[T, Tuple[T, T]], stride: Union[T, Tuple[T, T]] = 1, padding: Union[T, Tuple[T, T]] = 0, dilation: Union[T, Tuple[T, T]] = 1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros')





Pytorch: CONV2D

1. Layer type Convolution

2.Input   n * c_i * h_i * w_i

3.Output n * c_o * h_o * w_o, where h_o = (h_i + 2 * pad_h - kernel_h) / stride_h + 1 and w_o likewise.

4. The Convolution layer convolves the input image with a set of learnable filters, each producing one feature map in the output image.


Caffe： Convolution Layer Sample

https://buildmedia.readthedocs.org/media/pdf/theano/latest/theano.pdf

Theano Documentation Release 0.10.0beta1

- Python: conv_forward
- TensorFlow: tf.nn.conv2d
- Keras: Conv2D
- Pytorch: torch.nn.Conv2d
- Caffe: message ConvolutionParameter
- Theano: theano.tensor.nnet.conv.conv2d

University of Michigan - Ann Arbor

Convolutional networks, also known as convolutional neural networks, or CNNs, are a specialized kind of neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1-D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D grid of pixels. 

The network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers.

The CNN gets its power from the following concepts:
- Sparse connectivity
- Parameter sharing

Convolutional Neural Network (CNN)

A helpful website for understanding how CNN works:
https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53

CNN Reference

CNNs are currently being applied to several applications, including:

 - Computer vision—medical image analysis, image recognition, face detection and video classification
 - Natural Language Processing (NLP)—semantic parsing, sentence modeling and search query retrieval
 - Drug discovery—the discovery of chemical features and prediction of medicinal benefits

Applications of Convolutional Neural Networks

CNNs and RNNs are not mutually exclusive, as both can perform classification of image and text inputs, creating an opportunity to combine the two network types for increased effectiveness. This is especially true if the input to be classified is visually complex with added temporal characteristics that a CNN alone would be unable to process.

Typically, when these two network types are combined, sometimes referred to as a CRNN, inputs are first processed by CNN layers whose outputs are then fed to RNN layers. CNN Long Short-Term Memory (LSTM) architectures are particularly promising, as they facilitate analysis of inputs over longer periods than could be achieved with lower-level RNN architecture types.

Currently, these hybrid architectures are being explored for use in applications like video scene labeling, emotion detection or gesture recognition, video identification or gait recognition, and DNA sequence prediction.

Hybrid of Convolutional and Recurrent Neural Network

Methods to Calculate Convolution in Python

CNNs are made up of three-layer types—convolutional, pooling and fully-connected (FC).

In the convolutional layers, the input matrix is passed through a set of filters that output a feature map. This output is then sent to a pooling layer, which reduces the size of the feature map. This helps reduce the processing time by condensing the map to its most essential information.

The convolutional and pooling processes are repeated several times, with the number of repeats depending on the network, after which the condensed feature map outputs are sent to a series of FC layers. These FC layers then flatten the maps together and compare the probabilities of each feature occurring in conjunction with the others, until the best classification is determined.

Convolutional Neural Networks Architecture

A 3D CNN is similar to a 2D CNN except that it differs in (non-exhaustive listing):
1- 3D Convolution Layers

2- 3D MaxPool Layers

3D Convolutional Neural Network

Visualizing and Understanding Convolutional Networks Paper

- Convolutional neural network can be used for obtaining high- dimensional images rather than being used in class prediction for logistic regression  or value prediction for regression problem

- Output of the CNN model can be a tensor S(i,j,k) which represent the pixel(j,k) of ith class, such a collaborative  effort can be used to precisely form a perfect image as the output.

Strategies to get output dimension same as the input are 

- Pooling is completely avoided or can use pooling with unit stride.

- Pixel-wise labelling involves initial guess of image pixels and refining this guess by interacting with neighboring pixels by parameter sharing.

Structured Output from CNN

A Convolutional Recurrent Neural Network (CRNN) is a combination of a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). 
-CNN are specialized networks for processing data that use a specialized, linear mathematical operation, also known as a convolution. 
-RNN is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time adds additional layers of comprehension on top of the previous inputs. 
-The goal of a CRNN is to take advantage of CNNs for local feature extraction and RNNs for temporal summarization of the extracted features.


Convolutional Recurrent Neural Network (CRNN) 

Questions about the ReLU.

The Neocognitron, introduced by Fukushima in 1982, represents one of the earliest architectures and foundational references in the development of Convolutional Neural Networks (CNNs).

Neocognitron

In object detection, a recognition system should not be overly concerned with the precise location of an object within an image. For instance, similar to finding the character Waldo in chaotic scenes, a detector can sweep an image to assign likelihood scores to different patches, indicating where the target might be. Convolutional Neural Networks (CNNs) systematize this spatial invariance, exploiting it to learn useful representations with fewer parameters.

Spatial Invariance in Object Detection

The locality principle asserts that the earliest layers of a computer vision network should focus exclusively on local regions, without regard for the contents of the image in distant areas. Mathematically, this constrains the network's parameters such that outside a localized range ($$|a| > \Delta$$ or $$|b| > \Delta$$), the convolution weights are set to zero ($$[\mathbf{V}]_{a, b} = 0$$). The hidden representation is computed as:

$$[\mathbf{H}]_{i, j} = u + \sum_{a = -\Delta}^{\Delta} \sum_{b = -\Delta}^{\Delta} [\mathbf{V}]_{a, b} [\mathbf{X}]_{i+a, j+b}$$

By restricting the network to glean relevant information only from a bounded window around each pixel, this principle drastically reduces the number of learnable parameters (e.g., from $$4 \times 10^6$$ down to $$4 \Delta^2$$).

Locality Principle in Computer Vision

A Fully Convolutional Network (FCN) is designed for dense prediction tasks, such as semantic segmentation. The model architecture begins with a standard Convolutional Neural Network (CNN) to extract image features. Next, it employs a $$1 	imes 1$$ convolutional layer to transform the number of channels to match the number of target classes. Finally, it uses a transposed convolutional layer to scale the height and width of the feature maps back to the dimensions of the original input image. The resulting output has the same spatial dimensions as the input, with each output channel representing the predicted classes for the pixel at the corresponding spatial position.

Learn Before

Related

Learn After