In relation-based knowledge transfer, feature map relationships between two neural network layers can be modeled using a Gram matrix. This matrix summarizes relations by calculating the inner products between pairs of feature maps from the two layers. Singular Value Decomposition (SVD) is then applied to the resulting Gram matrix to extract essential information from these feature map correlations.

University of California, Berkeley

Google


Relation-based knowledge investigates the relationships between different layers or data samples, rather than outputs of specific layers. 

Relation-based knowledge 


In the context of convolutional neural networks, a feature map is the output produced by a convolutional layer. It acts as a learned representation, capturing specific spatial features—such as patterns across height and width dimensions—that are then passed forward as input to subsequent layers in the network.

Feature Map

Singular value decomposition (SVD) is an unsupervised dimensionality reduction technique used for feature extraction.  The singular value decomposition of an m x n matrix M is a factorization of the form U$$\sum$$M*.  U is an m x m unitary matrix. $$\sum$$ is a m x n diagonal matrix. V is a v x v unitary matrix.
Image link: [www.wikiwand.com/en/Singular_value_decomposition](https://www.wikiwand.com/en/Singular_value_decomposition)
 

Singular value decomposition (SVD)


Individual knowledge distillation can be implemented by directly distilling the teacher’s individual soft targets into the student. This transferred knowledge contains feature information and mutual relations of data samples. 

Knowledge Distillation Methods 


The distillation loss for relation-based knowledge transfer, based on the relations of feature maps, is calculated as:

$$L_{RelD}(f_t, f_s) = L_{R^1}(\psi_t(\acute{f_t}, \check{f_t}), \psi_s(\acute{f_s}, \check{f_s}))$$

Where:
- $$f_t$$ and $$f_s$$ are feature maps of the teacher and student models, respectively.
- $$\acute{f_t}$$ and $$\check{f_t}$$ are pairs of feature maps chosen from the teacher.
- $$\acute{f_s}$$ and $$\check{f_s}$$ are pairs of feature maps chosen from the student.
- $$\psi_t(\cdot)$$ and $$\psi_s(\cdot)$$ are similarity functions for pairs of feature maps from the models.
- $$L_{R^1}(\cdot)$$ is the correlation function between the teacher and student feature maps.

Distillation Loss for Relation-Based Knowledge

Gram Matrix for Feature Map Relationships

For any given element $$x$$ within a layer of a convolutional neural network, its receptive field consists of all the elements from all preceding layers that can potentially influence the calculation of $$x$$ during forward propagation. Because a single output element aggregates information from a window of inputs, its receptive field may encompass an area significantly larger than its own dimensions.

Receptive Field in Convolutional Neural Networks

Just as input images are represented as third-order tensors, hidden representations in convolutional neural networks are also formulated as third-order tensors, denoted as $$\mathsf{H}$$. Instead of a single value per spatial location, there is an entire vector of hidden representations for each location. These representations can be conceptualized as multiple two-dimensional grids stacked together, commonly referred to as channels or feature maps, where lower layers might recognize basic features like edges and deeper layers capture complex textures.

Hidden Representation as a 3rd-Order Tensor

To effectively detect objects of varying sizes, an object detection system can generate anchor boxes at multiple scales by utilizing feature maps of different spatial resolutions. Intuitively, smaller objects are more likely to appear on an image than larger ones; for example, a $$1 	imes 1$$ object can appear in more possible locations on a given image than a $$2 	imes 2$$ object. Therefore, a larger feature map provides more individual spatial units, allowing the system to uniformly sample a greater number of regions. This dense sampling is highly suitable for generating smaller anchor boxes to detect small objects. Conversely, by reducing the spatial dimensions of the feature map, the system samples fewer regions, which is ideal for placing larger anchor boxes intended to capture larger objects.

Multiscale Anchor Box Generation

According to the description of SVD algorithm, the columns of U are known as Left-singular Vectors.

Left-singular Vectors

According to the description of SVD algorithm, the columns of V are known as Right-singular Vectors.

Right-singular Vectors

Kalman, D. (1996). A singularly valuable decomposition: The SVD of a matrix. The College Mathematics Journal, 27(1), 2–23.

Kalman (1996)

In singular value decomposition (SVD), the elements along the diagonal of the matrix $$\Sigma$$ are known as the singular values.

Learn Before

Related