Formula

Shape of a Concatenated Weight Sub-Matrix (Whk\mathbf{W}_h^k)

Each sub-matrix Whk\mathbf{W}_h^k within a larger weight matrix Wh\mathbf{W}_h—which is constructed by concatenating MM sub-matrices—has dimensions of d×dhMd \times \frac{d_h}{M}. In this specification, dd denotes the number of rows, dhd_h is the total number of columns for the complete matrix Wh\mathbf{W}_h, and MM is the total count of sub-matrices. This guarantees that joining the MM sub-matrices horizontally results in the combined matrix having exactly dhd_h columns.

0

1

Updated 2026-04-21

Contributors are:

Who are from:

Tags

Ch.2 Generative Models - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences