The **kernel trick** in Support Vector Machines (SVMs) allows the model to learn complex decision boundaries without explicitly transforming original data points into a new, high-dimensional feature space. Instead, a kernelized SVM computes these boundaries using only similarity calculations (such as dot products) between pairs of points, leaving the high-dimensional transformed feature representation implicit.

University of Michigan - Ann Arbor

Google

The kernel function in an SVM model calculates the similarity in the new feature space of two points in the original feature space.

The SVM Gaussian kernel function is more complicated than the linear kernel function model, and it is easy to overfit.

SVM Kernel Function

A kernel is a function that computes the dot product of two vectors in a new, often higher-dimensional feature space without explicitly transforming the vectors into that space. This is known as the "kernel trick." A kernel function, commonly denoted by $$K(\vec{x}_1, \vec{x}_2)$$, acts as an implicit feature mapping and provides a measure of similarity or distance between inputs. In the dual formulation of Support Vector Machines (SVMs), replacing the standard dot product with $$K(\vec{x}^i, \vec{x}^j)$$ allows the model to efficiently learn non-linear decision boundaries in the original feature space. While closely related to the general notion of an inner product, a kernel need not strictly be an inner product.

Kernel Functions

- Radial Basis Function (RBF) Kernel
- Polynomial Kernel

Popular SVM Kernel Functions

SVM Kernel Trick

What does SVM do to solve the linear inseparable problem:

What does SVM do to solve the linear inseparable problem?

Basic types of kernel functions include:

- **Polynomial Kernel**: $$K(x, y) = (x \cdot y + 1)^p$$
- **Radial Basis Function (RBF) Kernel**: $$K(x, y) = \exp\left(-\frac{||x - y||^2}{2\sigma^2}\right)$$
- **Sigmoid Kernel**: $$K(x, y) = \tanh(C(x \cdot y) + \delta)$$

Basic Kernel Types

The **quadratic kernel** provides a clear example of an implicit feature mapping. Instead of explicitly computing feature mappings $$f(\vec{x})$$ and $$f(\vec{z})$$ and then calculating their dot product $$f(\vec{x}) \cdot f(\vec{z})$$, the kernel efficiently computes this inner product in one step.

To see how the kernel and an explicit feature mapping relate, consider input vectors in $$\mathbb{R}^2$$. The explicit mapping to a higher-dimensional space, $$f([x_1, x_2]^T) = [x_1^2, x_2^2, \sqrt{2}x_1x_2, \sqrt{2}x_1, \sqrt{2}x_2, 1]^T$$, mathematically corresponds directly to the quadratic kernel function:

$$K(\vec{x},\vec{z}) = (\vec{x} \cdot \vec{z} + 1)^2$$

The accompanying proof visually demonstrates that evaluating $$K(\vec{x},\vec{z})$$ achieves the same exact result as the dot product of the transformed features, allowing the model to avoid the computationally expensive mapping step.

Learn Before

Related