The Gaussian Error Linear Unit (GELU) activation function is defined by the following formula, which is applied element-wise to an input vector $\mathbf{h}$:

$\sigma_{\text{gelu}}(\mathbf{h}) = \mathbf{h} \text{Pr}(h \le \mathbf{h})$

Here, $h$ is a random variable following the standard normal distribution, $\mathcal{N}(0, 1)$. The term $\text{Pr}(h \le \mathbf{h})$ is an informal notation representing the cumulative distribution function (CDF) of the standard normal distribution, commonly denoted by $\Phi(\mathbf{h})$. When applied to the input vector $\mathbf{h}$, this term results in a new vector where each entry is the percentile (CDF value) corresponding to the respective entry in $\mathbf{h}$. Therefore, the formula can be simplified to:

$\sigma_{\text{gelu}}(\mathbf{h}) = \mathbf{h} \Phi(\mathbf{h})$