GELU (Gaussian Error Linear Unit) Formula

Applications of GELU in Large Language Models

An activation function is defined by its behavior of weighting an input value by that value's corresponding cumulative probability from a standard normal distribution (mean=0, variance=1). Given two inputs, x = -3 and y = 3, which statement best describes their respective outputs, f(x) and f(y)?

Hendrycks and Gimpel [2016] on GELU

An activation function is designed to scale its input value by the probability that a randomly drawn value from a standard normal distribution (mean=0, variance=1) is less than or equal to that input. How does this function's output for a small negative input (e.g., -0.1) compare to the output of a function that simply sets all negative inputs to zero?

Activation Function Selection for a Language Model

Diagnosing Training Instability When Changing Normalization and FFN Activations

Choosing an FFN Activation and Normalization Pair Under Deployment Constraints

Explaining a Distribution Shift Caused by Swapping LayerNorm for RMSNorm and GELU for SwiGLU

Root-Cause Analysis of FFN Output Drift After Swapping Normalization and Activation

Selecting a Normalization + FFN Activation Change After Quantization Regressions

Interpreting Activation/Normalization Interactions from FFN Telemetry

You are reviewing a teammate’s proposed Transforme...

In a transformer feed-forward block, your team is ...

You’re debugging a transformer FFN refactor where ...

You’re reviewing a PR that changes a transformer b...