Linear decision boundaries need not be unique, and in fact, there are infinitely many linear decision boundaries that could classify a linearly separable dataset.

Consider the data and classifiers below. We note that the blue and green classifier both have 0 training error, though the blue classifier probably does not **generalize** to test data very well. 

The assumption and motivation for using SVMs is that by finding the separating hyperplane that both *minimizes* training error, and *maximizes* the distance from the plane to the points, we will find a more **generalizable** model. 


Non Uniqueness of Linear Decision Boundary

Even though soft-margin SVM allows us to fit a classifier when a few outliers lay outside of their linearly separable classes, it doesn’t mean that a linear decision boundary should **always** be used. For example consider the below scatterplot. A soft-margin SVM might converge to a find a solution by drawing a diameter to the circle, however this is not at all sufficient, since we will have 50% accuracy. Your classifier is effectively a coin toss. 

We would instead want to fit a boundary that looks like the inner circle in the below image. In comes non-linear Kernel methods to save the day!

img src - [www.mathworks.com/help/stats/support-vector-machines-for-binary-classification.html](https://www.mathworks.com/help/stats/support-vector-machines-for-binary-classification.html) 

Addressing Explicit Non-Linearity


The motivation for using SVM’s is to generate a separating hyperplane for a binary dataset, which is **linearly separable**. 

In topological parlance, we are assuming that the positive and negative classes are contained in **disconnected** sets. 

Visually in $$\mathbb{R}^2$$, this means that we can draw a line between all of the data points, such that all of the positive points are on one side of the line, or hyperplane, and all of the negative points are on the other. 

Interestingly, a dataset might not be linearly separable in $$\mathbb{R}^k$$, but might be linearly separable in some $$d > k$$ dimensional space. More on this when talking about kernels.


University of Michigan - Ann Arbor

- Hard Margin SVM (Solving the maximum margin separator problem)
- Soft Margin SVM (Allow minimally misclassified points)

Types of SVM

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Linear Decision Boundaries, and the Assumption of Hard Margin


The hard margin classifier will not even find a solution for data that is not linearly separable (in its original feature space). In that case we prefer to use a *support vector classifier*, also known as a *soft-margin* classifier. From the below figure, we see that our data is not linearly separable, due to some noisy data points. The soft margin version of SVM will allow us to build a classifier that has <100% training accuracy but will perform well with most of the observations. 


Learn Before

Related

Learn After