The GeGLU (GELU-based Gated Linear Unit) activation function is utilized in the architecture of modern Large Language Models. For instance, the Gemma family of models incorporates GeGLU.

Google

GeGLU is a variant of the Gated Linear Unit (GLU) family of activation functions. It is created by specifying the internal non-linear activation function, $\sigma(\cdot)$, to be the Gaussian Error Linear Unit (GELU). This choice distinguishes GeGLU from other GLU variants.

GeGLU (GELU-based Gated Linear Unit)

Reference of Foundations of Large Language Models Course

The GeGLU (GELU-based Gated Linear Unit) activation function is defined by the following formula: 

$\sigma_{\text{geglu}}(\mathbf{h}) = \sigma_{\text{gelu}}(\mathbf{hW}_1 + \mathbf{b}_1) \odot (\mathbf{W}_2 + \mathbf{b}_2)$ 

In this equation, $\mathbf{h}$ represents the input, while $\mathbf{W}_1, \mathbf{W}_2, \mathbf{b}_1$, and $\mathbf{b}_2$ are learnable model parameters (weights and biases). The function $\sigma_{\text{gelu}}$ is the Gaussian Error Linear Unit (GELU) activation, and $\odot$ signifies the element-wise product.

Learn Before

Related