The tangent propagation algorithm is a regularization technique for neural networks that introduces an extra penalty to make the classifier's outputs locally invariant to known factors of variation. These factors correspond to movement along the manifold where examples of the same class concentrate. Local invariance is achieved by requiring the gradient $\nabla_x f(x)$ to be orthogonal to known manifold tangent vectors $v^{(i)}$ at $x$.

Equivalently, the directional derivative of $f$ at $x$ in the directions $v^{(i)}$ is kept small by adding a regularization penalty: $\Omega(f) = \sum_{i} ((\nabla_x f(x))^\top v^{(i)})^2$

The tangent vectors are derived a priori from knowledge of transformations that should not alter the output. While related to data augmentation (its non-infinitesimal counterpart) and the tangent distance algorithm, tangent propagation only regularizes the model against infinitesimal perturbations, which can be challenging for networks using rectified linear units.