Based on the provided scenario, identify the correct shape for the single-stream 'key' weight matrix and explain the fundamental reason for the engineer's error.

Google

In a multi-head attention mechanism, the key weight matrix for an individual attention head, which can be denoted as $k_h^k$, has a specific shape defined as $d \times \frac{d_h}{M}$. In this formula, $d$ is the dimension of the input representation, $d_h$ is the total dimension of the key projection across all heads, and $M$ is the number of attention heads.

Shape of Key Weight Matrix per Head

In a neural network component that uses parallel processing 'channels' to analyze input, an input representation with a dimension of 512 is transformed. This transformation is split across 8 parallel channels. For the 'key' transformation, the total dimension across all 8 channels is also 512. What is the shape of the learnable weight matrix used for the 'key' transformation within a single one of these channels?

Debugging a Dimensionality Mismatch

An attention mechanism is configured with 12 parallel processing heads. It receives input data where each element has a dimension of 768. The total dimension for the 'key' projection, when combined across all 12 heads, is also 768. What are the dimensions of the specific weight matrix used to compute the 'key' for a single head? (Format your answer as 'rows x columns', e.g., '100 x 50').

Learn Before

Related